Confirm the season-aware start probability on GW4 (pre-registered), and the returning-injury case #33
Labels
No labels
architecture
cleanup
dependencies
enhancement
needs-info
needs-triage
performance
priority: high
priority: medium
ready-for-agent
ready-for-human
reliability
security
testing
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
solvreven/FPL#33
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
4af4cc9rebuilt start and play probability after the old estimator lost to "did he start last round?" by a third at 26/27 GW3 (Brier 0.1471 vs 0.0964; starters predicted 0.74, started 0.89; non-starters predicted 0.22, started 0.08). The replacement — this season's rounds at h=1.5 with last season as a prior worth half a round, plus availability-weighted club normalisation — was chosen on GW1-3, so GW1-3 cannot validate it.To do once GW4 is ingested:
python scripts/backtest_starts_boundary.py --gw 4, run once. Bar (pre-registered indocs/testing-notes.md): new below old by >= 0.01 AND new below LAST or within 0.01 of it. A miss reverts to the old estimator pending re-measurement; it is not re-tuned on GW4.Known weakness, measured only on GW2's 50 flagged players: a player whose zeros this season came from an injury reads as having lost his place until he starts again. Dropping a flagged player's zeros was scored and made GW2 worse (0.1197 -> 0.1256), so it was not adopted. The per-deadline snapshots hold the flag history that could separate the two; nothing reads them for this yet.
GW4 confirmation, run once on 2026-09-15: passed.
python scripts/backtest_starts_boundary.py --gw 4, at4da18ca, after the chores ingested GW4. n = 420. GW4 has no snapshot for its own deadline, so availability is 1.0 for everyone (the script says so).The estimator stays. Calibration, by starts in the last three rounds:
The over-prediction at the top that GW3 showed (0.94 against 0.89) is still there. Recorded in
docs/testing-notes.md(621e46b).Leaving this open for the second half of the issue: the returning-injury case is still unmeasured beyond GW2's 50 flagged players.
The returning-injury audit, run once on the rounds so far (
scripts/audit_returning_injuries.pyfrom #43, ataaca006, 2026-10-16). It reads the stored forecasts, takes the injury flags at each deadline, and uses "started last round" as the control (LAST, 0.85/0.15).The three measured on GW6 were all injured in GW5 and last played in GW4:
The Brier score is 0.380 for ours against 0.722 for LAST. All three started, and ours priced two of them well above LAST and one below it. Three players cannot show whether the season-aware start chance handles a return from injury, so this case stays open. The audit will be re-run as rounds come in.
🤖 Generated with Claude Code
Triage Notes
Category: enhancement — the remaining scope is model validation/research, not a newly confirmed implementation defect.
State: needs-triage — keep open for evaluation as more returning-injury evidence becomes available.
Established:
scripts/audit_returning_injuries.py, with coverage intests/test_audit_returning_injuries.py. It reads stored forecasts and each deadline's flags, includes both returning starters and players who remain benched, and compares with LAST (0.85/0.15). Creating another audit would duplicate existing work.2474892: the existing read-only audit for GW2, GW5 and GW6 returns UNMEASURED for all three because this checkout has nodata/pitsnapshot archive. Its exit status is 1, explicitly signalling missing evidence. This does not contradict the results obtained on the snapshot-bearing checkout.backend/web/priors.py), the audit, its tests,docs/testing-notes.md, and the start-probability wiki page. No.out-of-scope/rejection records are present in this checkout.Remaining work / next review trigger:
The completed GW4 work and implemented audit need no new implementation ticket. The remaining evaluation stays here.
Agent Brief
Category: enhancement
Summary: Prepare and verify prospective returning-injury evidence collection, then produce a bounded audit report from whatever genuine evidence is available.
Current behavior and decision:
The maintainer confirmed on 2026-10-18 that no local snapshots exist. The earlier proposal to run a historical audit locally therefore has an unavailable input. The documented Windows backup is not accessible on this machine; its continued existence elsewhere is unknown. Do not search indefinitely for it or assume it can be restored.
The pre-registered GW4 confirmation already passed. The returning-injury audit and the immutable snapshot collector already exist. The latest issue discussion reports three measured GW6 cases and nine unmeasured histories on another checkout; these are previously reported evidence, not locally reproduced results. This checkout's outcome file currently lists GW1–4.
Desired behavior:
An agent can complete a finite evidence-preparation phase now, without waiting for a future match or claiming historical flags have been recovered. Deliver a reproducible collection/audit protocol, verify the existing path, capture genuine current inputs when the required local forecast inputs are available, and report exactly what remains unmeasured. Reuse the current collector, point-in-time selector and audit; this is not a request for another estimator or another scheduler.
Key interfaces and invariants:
snapshot_indexandchoose_snapshotare authoritative: forecasts must be built and captured before the target deadline, project that round, and satisfy the previous-round evidence rule. A capture made now cannot stand for a past deadline.audit(gws, pit_dir=..., rounds_path=...)and its CLI accept explicit target rounds and input locations. Use stored forecasts and each deadline's flags; LAST remains 0.85/0.15. The cohort is available (a) at the target deadline after consecutive zero-minute rounds injury-flagged (i) at their own deadlines. Include starters and non-starters alike.Acceptance criteria:
Out of scope:
Changing the estimator, dropping injury zeros, rerunning or retuning the completed GW4 confirmation, reconstructing historical flags from live data, treating the earlier three-player cohort as validation, adding a parallel scheduler, installing a persistent service or paid data source, or silently extending the audit into an open-ended study. If a concrete collection defect is discovered, report its reproduction and propose a separately scoped fix.
Handoff:
Keep #33 open after this phase. If model work is warranted later, propose a separate change with a pre-registered decision rule and genuinely unseen confirmation data. If future rounds are the only remaining dependency, record the exact next review trigger and move out of ready-for-agent rather than repeatedly assigning the same empty audit.
This brief supersedes the prior needs-triage recommendation for the next executable phase.
ready-for-agentrefers to evidence preparation and the bounded diagnostic above, not to an approved model change.Evidence-preparation handoff
Submitted PR #44 for the bounded agent brief. The PR preserves the model-validation question here.
Verified: 141 focused tests passed, including valid isolated snapshot fixtures and missing/wrong-deadline/late-evidence refusals. Runbook examples were syntax-checked; official target discovery and early-window/missing-archive refusal paths were executed. Independent review found no substantive issues.
Available evidence: the explicit GW2/3/5/6 audit returned UNMEASURED throughout (exit 1), because the maintainer-confirmed local snapshot archive is absent. Local outcomes stop at GW4, and the forecast cache is absent too. Earlier GW4 confirmation and GW6 cohort figures remain attributed to the original comments; neither was reproduced or retuned.
Next operational steps: official data inspected at 2026-10-17 23:06 UTC names GW8, deadline 2026-10-23 17:30 UTC. GW7 was only 5/10 fixtures finished/provisional, so no valid GW8 decision snapshot could be made. Prepare the persistent checkout, ingest completed rounds, wait for and ingest GW7, then rebuild and capture inside the existing pre-deadline window (14:30–17:30 UTC on 23 October). Validate through the production selector and preserve the archive to an explicit backup destination. No configured FPL collection path was found on the inspected host surfaces; the runbook names the existing manual/supervised route and its uptime limits. No service or backup was installed.
Disposition: preparation is submitted for review, not model validation. Keep #33 open and return it to needs-triage after this handoff so another agent is not assigned the identical empty audit. Next evaluation trigger: genuine prospective snapshots plus completed outcomes and the required consecutive injury-flag history. GW8 alone cannot reconstruct GW7; a GW8 injury followed by a GW9 return could become measurable after GW9 only if the full history passes the guards. Any model change still needs a separate pre-registered rule and unseen confirmation data.