Confirm the season-aware start probability on GW4 (pre-registered), and the returning-injury case #33

Open
opened 2026-09-13 14:46:01 +02:00 by solvreven · 5 comments
Owner

4af4cc9 rebuilt start and play probability after the old estimator lost to "did he start last round?" by a third at 26/27 GW3 (Brier 0.1471 vs 0.0964; starters predicted 0.74, started 0.89; non-starters predicted 0.22, started 0.08). The replacement — this season's rounds at h=1.5 with last season as a prior worth half a round, plus availability-weighted club normalisation — was chosen on GW1-3, so GW1-3 cannot validate it.

To do once GW4 is ingested: python scripts/backtest_starts_boundary.py --gw 4, run once. Bar (pre-registered in docs/testing-notes.md): new below old by >= 0.01 AND new below LAST or within 0.01 of it. A miss reverts to the old estimator pending re-measurement; it is not re-tuned on GW4.

Known weakness, measured only on GW2's 50 flagged players: a player whose zeros this season came from an injury reads as having lost his place until he starts again. Dropping a flagged player's zeros was scored and made GW2 worse (0.1197 -> 0.1256), so it was not adopted. The per-deadline snapshots hold the flag history that could separate the two; nothing reads them for this yet.

`4af4cc9` rebuilt start and play probability after the old estimator lost to "did he start last round?" by a third at 26/27 GW3 (Brier 0.1471 vs 0.0964; starters predicted 0.74, started 0.89; non-starters predicted 0.22, started 0.08). The replacement — this season's rounds at h=1.5 with last season as a prior worth half a round, plus availability-weighted club normalisation — was **chosen on GW1-3**, so GW1-3 cannot validate it. **To do once GW4 is ingested:** `python scripts/backtest_starts_boundary.py --gw 4`, run once. Bar (pre-registered in `docs/testing-notes.md`): new below old by >= 0.01 AND new below LAST or within 0.01 of it. A miss reverts to the old estimator pending re-measurement; it is not re-tuned on GW4. Known weakness, measured only on GW2's 50 flagged players: a player whose zeros this season came from an injury reads as having lost his place until he starts again. Dropping a flagged player's zeros was scored and made GW2 worse (0.1197 -> 0.1256), so it was not adopted. The per-deadline snapshots hold the flag history that could separate the two; nothing reads them for this yet.
Author
Owner

GW4 confirmation, run once on 2026-09-15: passed.

python scripts/backtest_starts_boundary.py --gw 4, at 4da18ca, after the chores ingested GW4. n = 420. GW4 has no snapshot for its own deadline, so availability is 1.0 for everyone (the script says so).

estimator Brier
old 0.1577
new 0.1134
LAST (started last round) 0.1075
SEASON 0.1211
  • Bar part 1, new below old by at least 0.01: it is below by 0.0443. Passed.
  • Bar part 2, new below LAST or within 0.01 of it: it is 0.0059 above LAST. Passed.

The estimator stays. Calibration, by starts in the last three rounds:

starts n predicted observed
3 of 3 163 0.95 0.88
2 of 3 58 0.65 0.64
1 of 3 53 0.35 0.40
0 of 3 143 0.08 0.04

The over-prediction at the top that GW3 showed (0.94 against 0.89) is still there. Recorded in docs/testing-notes.md (621e46b).

Leaving this open for the second half of the issue: the returning-injury case is still unmeasured beyond GW2's 50 flagged players.

**GW4 confirmation, run once on 2026-09-15: passed.** `python scripts/backtest_starts_boundary.py --gw 4`, at `4da18ca`, after the chores ingested GW4. n = 420. GW4 has no snapshot for its own deadline, so availability is 1.0 for everyone (the script says so). | estimator | Brier | |---|---| | old | 0.1577 | | **new** | **0.1134** | | LAST (started last round) | 0.1075 | | SEASON | 0.1211 | - Bar part 1, new below old by at least 0.01: it is below by 0.0443. Passed. - Bar part 2, new below LAST or within 0.01 of it: it is 0.0059 above LAST. Passed. The estimator stays. Calibration, by starts in the last three rounds: | starts | n | predicted | observed | |---|---|---|---| | 3 of 3 | 163 | 0.95 | 0.88 | | 2 of 3 | 58 | 0.65 | 0.64 | | 1 of 3 | 53 | 0.35 | 0.40 | | 0 of 3 | 143 | 0.08 | 0.04 | The over-prediction at the top that GW3 showed (0.94 against 0.89) is still there. Recorded in `docs/testing-notes.md` (`621e46b`). Leaving this open for the second half of the issue: the returning-injury case is still unmeasured beyond GW2's 50 flagged players.
Author
Owner

The returning-injury audit, run once on the rounds so far (scripts/audit_returning_injuries.py from #43, at aaca006, 2026-10-16). It reads the stored forecasts, takes the injury flags at each deadline, and uses "started last round" as the control (LAST, 0.85/0.15).

round verdict why n
GW2 UNMEASURED no fully evidenced cohort: 14 players have no GW1 row or FPL status 0
GW5 UNMEASURED no snapshot can stand for the GW4 decision (the pit rule refuses a later clock) 0
GW6 PARTIAL 9 flag histories are unmeasured 3

The three measured on GW6 were all injured in GW5 and last played in GW4:

player ours LAST started
Tete 0.070 0.15 yes
Mendy 0.585 0.15 yes
N. Gonzalez 0.678 0.15 yes

The Brier score is 0.380 for ours against 0.722 for LAST. All three started, and ours priced two of them well above LAST and one below it. Three players cannot show whether the season-aware start chance handles a return from injury, so this case stays open. The audit will be re-run as rounds come in.

🤖 Generated with Claude Code

**The returning-injury audit, run once on the rounds so far** (`scripts/audit_returning_injuries.py` from #43, at `aaca006`, 2026-10-16). It reads the stored forecasts, takes the injury flags at each deadline, and uses "started last round" as the control (LAST, 0.85/0.15). | round | verdict | why | n | |---|---|---|---| | GW2 | UNMEASURED | no fully evidenced cohort: 14 players have no GW1 row or FPL status | 0 | | GW5 | UNMEASURED | no snapshot can stand for the GW4 decision (the pit rule refuses a later clock) | 0 | | GW6 | PARTIAL | 9 flag histories are unmeasured | 3 | The three measured on GW6 were all injured in GW5 and last played in GW4: | player | ours | LAST | started | |---|---|---|---| | Tete | 0.070 | 0.15 | yes | | Mendy | 0.585 | 0.15 | yes | | N. Gonzalez | 0.678 | 0.15 | yes | The Brier score is 0.380 for ours against 0.722 for LAST. All three started, and ours priced two of them well above LAST and one below it. Three players cannot show whether the season-aware start chance handles a return from injury, so this case stays open. The audit will be re-run as rounds come in. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Collaborator

This was generated by AI during triage.

Triage Notes

Category: enhancement — the remaining scope is model validation/research, not a newly confirmed implementation defect.
State: needs-triage — keep open for evaluation as more returning-injury evidence becomes available.

Established:

  • The pre-registered GW4 confirmation is complete and passed, as recorded in the 2026-09-15 comment. Do not repeat it or tune against GW4.
  • The returning-injury audit is already implemented in scripts/audit_returning_injuries.py, with coverage in tests/test_audit_returning_injuries.py. It reads stored forecasts and each deadline's flags, includes both returning starters and players who remain benched, and compares with LAST (0.85/0.15). Creating another audit would duplicate existing work.
  • The latest evidence in this issue (2026-10-17 comment) is GW6 PARTIAL: three measured players, nine unmeasured flag histories. The reported Brier scores are 0.380 for ours versus 0.722 for LAST. These are prior reported results, not a fresh validation, and are insufficient to settle the model question.
  • Local verification at 2474892: the existing read-only audit for GW2, GW5 and GW6 returns UNMEASURED for all three because this checkout has no data/pit snapshot archive. Its exit status is 1, explicitly signalling missing evidence. This does not contradict the results obtained on the snapshot-bearing checkout.
  • Redundancy review covered the season-aware estimator (backend/web/priors.py), the audit, its tests, docs/testing-notes.md, and the start-probability wiki page. No .out-of-scope/ rejection records are present in this checkout.

Remaining work / next review trigger:

  1. On the checkout holding the original snapshots, run the existing audit for explicitly named, newly completed gameweeks once outcomes and their deadline snapshots are available. Preserve the report's code/forecast provenance, cohort rows, exclusions and missing histories.
  2. Review the growing cohort before deciding whether a model change is warranted. The audit is exploratory; no returning-injury model-adoption or closure threshold is specified in this issue. Any proposed change needs a pre-registered decision rule and genuinely unseen confirmation data.
  3. Keep missing historical flags unmeasured; today's flags cannot reconstruct them. Do not revive the rejected zero-dropping heuristic or close this issue merely because the measurement tooling exists.

The completed GW4 work and implemented audit need no new implementation ticket. The remaining evaluation stays here.

> *This was generated by AI during triage.* ## Triage Notes **Category:** enhancement — the remaining scope is model validation/research, not a newly confirmed implementation defect. **State:** needs-triage — keep open for evaluation as more returning-injury evidence becomes available. **Established:** - The pre-registered GW4 confirmation is complete and passed, as recorded in the 2026-09-15 comment. Do not repeat it or tune against GW4. - The returning-injury audit is already implemented in `scripts/audit_returning_injuries.py`, with coverage in `tests/test_audit_returning_injuries.py`. It reads stored forecasts and each deadline's flags, includes both returning starters and players who remain benched, and compares with LAST (0.85/0.15). Creating another audit would duplicate existing work. - The latest evidence in this issue (2026-10-17 comment) is GW6 PARTIAL: three measured players, nine unmeasured flag histories. The reported Brier scores are 0.380 for ours versus 0.722 for LAST. These are prior reported results, not a fresh validation, and are insufficient to settle the model question. - Local verification at `2474892`: the existing read-only audit for GW2, GW5 and GW6 returns UNMEASURED for all three because this checkout has no `data/pit` snapshot archive. Its exit status is 1, explicitly signalling missing evidence. This does not contradict the results obtained on the snapshot-bearing checkout. - Redundancy review covered the season-aware estimator (`backend/web/priors.py`), the audit, its tests, `docs/testing-notes.md`, and the start-probability wiki page. No `.out-of-scope/` rejection records are present in this checkout. **Remaining work / next review trigger:** 1. On the checkout holding the original snapshots, run the existing audit for explicitly named, newly completed gameweeks once outcomes and their deadline snapshots are available. Preserve the report's code/forecast provenance, cohort rows, exclusions and missing histories. 2. Review the growing cohort before deciding whether a model change is warranted. The audit is exploratory; no returning-injury model-adoption or closure threshold is specified in this issue. Any proposed change needs a pre-registered decision rule and genuinely unseen confirmation data. 3. Keep missing historical flags unmeasured; today's flags cannot reconstruct them. Do not revive the rejected zero-dropping heuristic or close this issue merely because the measurement tooling exists. The completed GW4 work and implemented audit need no new implementation ticket. The remaining evaluation stays here.
Collaborator

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Prepare and verify prospective returning-injury evidence collection, then produce a bounded audit report from whatever genuine evidence is available.

Current behavior and decision:
The maintainer confirmed on 2026-10-18 that no local snapshots exist. The earlier proposal to run a historical audit locally therefore has an unavailable input. The documented Windows backup is not accessible on this machine; its continued existence elsewhere is unknown. Do not search indefinitely for it or assume it can be restored.

The pre-registered GW4 confirmation already passed. The returning-injury audit and the immutable snapshot collector already exist. The latest issue discussion reports three measured GW6 cases and nine unmeasured histories on another checkout; these are previously reported evidence, not locally reproduced results. This checkout's outcome file currently lists GW1–4.

Desired behavior:
An agent can complete a finite evidence-preparation phase now, without waiting for a future match or claiming historical flags have been recovered. Deliver a reproducible collection/audit protocol, verify the existing path, capture genuine current inputs when the required local forecast inputs are available, and report exactly what remains unmeasured. Reuse the current collector, point-in-time selector and audit; this is not a request for another estimator or another scheduler.

Key interfaces and invariants:

  • The immutable point-in-time collector captures official bootstrap/fixtures plus the local forecast and current-season inputs, recording absent artifacts explicitly. A successful command alone does not prove a usable snapshot.
  • snapshot_index and choose_snapshot are authoritative: forecasts must be built and captured before the target deadline, project that round, and satisfy the previous-round evidence rule. A capture made now cannot stand for a past deadline.
  • audit(gws, pit_dir=..., rounds_path=...) and its CLI accept explicit target rounds and input locations. Use stored forecasts and each deadline's flags; LAST remains 0.85/0.15. The cohort is available (a) at the target deadline after consecutive zero-minute rounds injury-flagged (i) at their own deadlines. Include starters and non-starters alike.
  • The existing pre-deadline chore performs collection only while the application runs. Describe that operating dependency accurately. A manual smoke run does not establish unattended collection.

Acceptance criteria:

  • Produce a durable report and runnable command sequence identifying the checkout/code SHA, interpreter, input paths, artifact destinations and known historical coverage gaps. Record the maintainer's no-local-snapshots statement; preserve the earlier issue results as attributed historical evidence.
  • Inspect the existing collection and audit entry points and run their relevant existing tests. Demonstrate with isolated test inputs that valid frozen evidence is accepted and missing, wrong-deadline or post-deadline evidence is refused. Synthetic fixtures must never be stored or reported as historical observations.
  • Determine the next future deadline from official event data at execution time and record the named round and UTC deadline before collection. Check the local forecast and previous-round inputs. If prerequisites are valid, make one genuine current capture and validate its contents with the production selector. If they are unavailable or stale, report the specific prerequisite and repair command; do not manufacture a cache, relax the selector or label an incomplete capture usable.
  • Define the prospective collection sequence: freeze forecast and status evidence before every relevant deadline, ingest outcomes only after the round finishes, then run the audit for an explicitly recorded set of completed rounds. Explain that an injury return can require several consecutive deadline snapshots; the first new snapshot does not reconstruct an earlier absence history.
  • For any genuine, eligible completed-round evidence available during the task, save the original machine-readable audit output and a human-readable summary: per-round status, cohort rows, forecast provenance, Brier versus LAST, calibration, exclusions and missing-history reasons. If none exists, report UNMEASURED and name the evidence needed next. Do not wait indefinitely for future outcomes.
  • Record how the existing collection path is invoked on this host, whether unattended execution is actually configured, where the archive persists, and the exact remaining operational steps if it is not. Do not claim a backup or schedule exists without verifying it.
  • Post the artifact references and findings here. Distinguish completion of evidence preparation from resolution of the model question. An explicit insufficient-evidence result completes this bounded phase; it does not validate the estimator or justify closing #33.

Out of scope:
Changing the estimator, dropping injury zeros, rerunning or retuning the completed GW4 confirmation, reconstructing historical flags from live data, treating the earlier three-player cohort as validation, adding a parallel scheduler, installing a persistent service or paid data source, or silently extending the audit into an open-ended study. If a concrete collection defect is discovered, report its reproduction and propose a separately scoped fix.

Handoff:
Keep #33 open after this phase. If model work is warranted later, propose a separate change with a pre-registered decision rule and genuinely unseen confirmation data. If future rounds are the only remaining dependency, record the exact next review trigger and move out of ready-for-agent rather than repeatedly assigning the same empty audit.

This brief supersedes the prior needs-triage recommendation for the next executable phase. ready-for-agent refers to evidence preparation and the bounded diagnostic above, not to an approved model change.

> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Prepare and verify prospective returning-injury evidence collection, then produce a bounded audit report from whatever genuine evidence is available. **Current behavior and decision:** The maintainer confirmed on 2026-10-18 that no local snapshots exist. The earlier proposal to run a historical audit locally therefore has an unavailable input. The documented Windows backup is not accessible on this machine; its continued existence elsewhere is unknown. Do not search indefinitely for it or assume it can be restored. The pre-registered GW4 confirmation already passed. The returning-injury audit and the immutable snapshot collector already exist. The latest issue discussion reports three measured GW6 cases and nine unmeasured histories on another checkout; these are previously reported evidence, not locally reproduced results. This checkout's outcome file currently lists GW1–4. **Desired behavior:** An agent can complete a finite evidence-preparation phase now, without waiting for a future match or claiming historical flags have been recovered. Deliver a reproducible collection/audit protocol, verify the existing path, capture genuine current inputs when the required local forecast inputs are available, and report exactly what remains unmeasured. Reuse the current collector, point-in-time selector and audit; this is not a request for another estimator or another scheduler. **Key interfaces and invariants:** - The immutable point-in-time collector captures official bootstrap/fixtures plus the local forecast and current-season inputs, recording absent artifacts explicitly. A successful command alone does not prove a usable snapshot. - `snapshot_index` and `choose_snapshot` are authoritative: forecasts must be built and captured before the target deadline, project that round, and satisfy the previous-round evidence rule. A capture made now cannot stand for a past deadline. - `audit(gws, pit_dir=..., rounds_path=...)` and its CLI accept explicit target rounds and input locations. Use stored forecasts and each deadline's flags; LAST remains 0.85/0.15. The cohort is available (`a`) at the target deadline after consecutive zero-minute rounds injury-flagged (`i`) at their own deadlines. Include starters and non-starters alike. - The existing pre-deadline chore performs collection only while the application runs. Describe that operating dependency accurately. A manual smoke run does not establish unattended collection. **Acceptance criteria:** - [ ] Produce a durable report and runnable command sequence identifying the checkout/code SHA, interpreter, input paths, artifact destinations and known historical coverage gaps. Record the maintainer's no-local-snapshots statement; preserve the earlier issue results as attributed historical evidence. - [ ] Inspect the existing collection and audit entry points and run their relevant existing tests. Demonstrate with isolated test inputs that valid frozen evidence is accepted and missing, wrong-deadline or post-deadline evidence is refused. Synthetic fixtures must never be stored or reported as historical observations. - [ ] Determine the next future deadline from official event data at execution time and record the named round and UTC deadline before collection. Check the local forecast and previous-round inputs. If prerequisites are valid, make one genuine current capture and validate its contents with the production selector. If they are unavailable or stale, report the specific prerequisite and repair command; do not manufacture a cache, relax the selector or label an incomplete capture usable. - [ ] Define the prospective collection sequence: freeze forecast and status evidence before every relevant deadline, ingest outcomes only after the round finishes, then run the audit for an explicitly recorded set of completed rounds. Explain that an injury return can require several consecutive deadline snapshots; the first new snapshot does not reconstruct an earlier absence history. - [ ] For any genuine, eligible completed-round evidence available during the task, save the original machine-readable audit output and a human-readable summary: per-round status, cohort rows, forecast provenance, Brier versus LAST, calibration, exclusions and missing-history reasons. If none exists, report UNMEASURED and name the evidence needed next. Do not wait indefinitely for future outcomes. - [ ] Record how the existing collection path is invoked on this host, whether unattended execution is actually configured, where the archive persists, and the exact remaining operational steps if it is not. Do not claim a backup or schedule exists without verifying it. - [ ] Post the artifact references and findings here. Distinguish completion of evidence preparation from resolution of the model question. An explicit insufficient-evidence result completes this bounded phase; it does not validate the estimator or justify closing #33. **Out of scope:** Changing the estimator, dropping injury zeros, rerunning or retuning the completed GW4 confirmation, reconstructing historical flags from live data, treating the earlier three-player cohort as validation, adding a parallel scheduler, installing a persistent service or paid data source, or silently extending the audit into an open-ended study. If a concrete collection defect is discovered, report its reproduction and propose a separately scoped fix. **Handoff:** Keep #33 open after this phase. If model work is warranted later, propose a separate change with a pre-registered decision rule and genuinely unseen confirmation data. If future rounds are the only remaining dependency, record the exact next review trigger and move out of ready-for-agent rather than repeatedly assigning the same empty audit. This brief supersedes the prior needs-triage recommendation for the next executable phase. `ready-for-agent` refers to evidence preparation and the bounded diagnostic above, not to an approved model change.
Collaborator

This was generated by AI during triage.

Evidence-preparation handoff

Submitted PR #44 for the bounded agent brief. The PR preserves the model-validation question here.

Verified: 141 focused tests passed, including valid isolated snapshot fixtures and missing/wrong-deadline/late-evidence refusals. Runbook examples were syntax-checked; official target discovery and early-window/missing-archive refusal paths were executed. Independent review found no substantive issues.

Available evidence: the explicit GW2/3/5/6 audit returned UNMEASURED throughout (exit 1), because the maintainer-confirmed local snapshot archive is absent. Local outcomes stop at GW4, and the forecast cache is absent too. Earlier GW4 confirmation and GW6 cohort figures remain attributed to the original comments; neither was reproduced or retuned.

Next operational steps: official data inspected at 2026-10-17 23:06 UTC names GW8, deadline 2026-10-23 17:30 UTC. GW7 was only 5/10 fixtures finished/provisional, so no valid GW8 decision snapshot could be made. Prepare the persistent checkout, ingest completed rounds, wait for and ingest GW7, then rebuild and capture inside the existing pre-deadline window (14:30–17:30 UTC on 23 October). Validate through the production selector and preserve the archive to an explicit backup destination. No configured FPL collection path was found on the inspected host surfaces; the runbook names the existing manual/supervised route and its uptime limits. No service or backup was installed.

Disposition: preparation is submitted for review, not model validation. Keep #33 open and return it to needs-triage after this handoff so another agent is not assigned the identical empty audit. Next evaluation trigger: genuine prospective snapshots plus completed outcomes and the required consecutive injury-flag history. GW8 alone cannot reconstruct GW7; a GW8 injury followed by a GW9 return could become measurable after GW9 only if the full history passes the guards. Any model change still needs a separate pre-registered rule and unseen confirmation data.

> *This was generated by AI during triage.* ## Evidence-preparation handoff Submitted **[PR #44](https://git.kristofferopsahl.com/solvreven/FPL/pulls/44)** for the bounded agent brief. The PR preserves the model-validation question here. - [Readiness report and acceptance evidence](https://git.kristofferopsahl.com/solvreven/FPL/src/commit/151976e9323773e6356c2435acde6e0693e20b13/docs/benchmarks/2026-10-18-returning-injury/report.md) - [Prospective collection and audit runbook](https://git.kristofferopsahl.com/solvreven/FPL/src/commit/151976e9323773e6356c2435acde6e0693e20b13/docs/returning-injury-evidence.md) - [Original scoped audit output](https://git.kristofferopsahl.com/solvreven/FPL/src/commit/151976e9323773e6356c2435acde6e0693e20b13/docs/benchmarks/2026-10-18-returning-injury/audit.json) **Verified:** 141 focused tests passed, including valid isolated snapshot fixtures and missing/wrong-deadline/late-evidence refusals. Runbook examples were syntax-checked; official target discovery and early-window/missing-archive refusal paths were executed. Independent review found no substantive issues. **Available evidence:** the explicit GW2/3/5/6 audit returned UNMEASURED throughout (exit 1), because the maintainer-confirmed local snapshot archive is absent. Local outcomes stop at GW4, and the forecast cache is absent too. Earlier GW4 confirmation and GW6 cohort figures remain attributed to the original comments; neither was reproduced or retuned. **Next operational steps:** official data inspected at 2026-10-17 23:06 UTC names GW8, deadline 2026-10-23 17:30 UTC. GW7 was only 5/10 fixtures finished/provisional, so no valid GW8 decision snapshot could be made. Prepare the persistent checkout, ingest completed rounds, wait for and ingest GW7, then rebuild and capture inside the existing pre-deadline window (14:30–17:30 UTC on 23 October). Validate through the production selector and preserve the archive to an explicit backup destination. No configured FPL collection path was found on the inspected host surfaces; the runbook names the existing manual/supervised route and its uptime limits. No service or backup was installed. **Disposition:** preparation is submitted for review, not model validation. Keep #33 open and return it to needs-triage after this handoff so another agent is not assigned the identical empty audit. Next evaluation trigger: genuine prospective snapshots plus completed outcomes and the required consecutive injury-flag history. GW8 alone cannot reconstruct GW7; a GW8 injury followed by a GW9 return could become measurable after GW9 only if the full history passes the guards. Any model change still needs a separate pre-registered rule and unseen confirmation data.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
solvreven/FPL#33
No description provided.