Accuracy panel has no scoring branch and goes quieter once it has data #11

New issue

Open

opened 2026-08-27 20:57:39 +02:00 by solvreven · 2 comments

solvreven commented

2026-08-27 20:57:39 +02:00

Owner

backend/web/calibration.py defines brier_score, log_loss,
calibration_curve and disagreement_tally, all unit-tested. accuracy_report
calls none of them.

Below MIN_RESOLVED (20) it writes a refusal and a reason on each metric. At or
above it, the function does nothing at all — so the metrics keep value=None,
keep the AWAITING_RESULTS banner, and their detail string goes empty.

The panel gets quieter at the moment it first has enough data to say
something. Ten fixtures are resolved as of 2026-08-27, so this is roughly one
gameweek away from being visible.

Found because the self-dating gate in tests/web/test_calibration.py came due:
it asserted resolved_fixtures == 0 with the note "a fixture has resolved —
update this test", and GW1 resolved it. The gate worked exactly as designed.

test_the_panel_has_no_scoring_branch_yet now pins this state and is written to
fail the moment scoring is wired, so the gap is asserted rather than
rediscovered.

Not a quick fix: scoring needs stored predictions to compare against, which
is what data/pit/ snapshots hold. That is a real build, and it should not be
bolted onto an unrelated change.

`backend/web/calibration.py` defines `brier_score`, `log_loss`, `calibration_curve` and `disagreement_tally`, all unit-tested. **`accuracy_report` calls none of them.** Below `MIN_RESOLVED` (20) it writes a refusal and a reason on each metric. At or above it, the function does nothing at all — so the metrics keep `value=None`, keep the `AWAITING_RESULTS` banner, and their `detail` string goes **empty**. **The panel gets quieter at the moment it first has enough data to say something.** Ten fixtures are resolved as of 2026-08-27, so this is roughly one gameweek away from being visible. Found because the self-dating gate in `tests/web/test_calibration.py` came due: it asserted `resolved_fixtures == 0` with the note "a fixture has resolved — update this test", and GW1 resolved it. The gate worked exactly as designed. `test_the_panel_has_no_scoring_branch_yet` now pins this state and is written to **fail the moment scoring is wired**, so the gap is asserted rather than rediscovered. **Not a quick fix:** scoring needs stored predictions to compare against, which is what `data/pit/` snapshots hold. That is a real build, and it should not be bolted onto an unrelated change.

kristofferopsahl commented

2026-08-30 17:46:13 +02:00

Collaborator

The scoring helpers are ready, but the prediction/outcome join needs a declared contract. Please specify:

Which PIT snapshot represents a fixture prediction: the latest snapshot before kickoff, the latest before the gameweek deadline, or another cutoff?
Which stored field is the canonical predicted probability for each metric, and which resolved FPL fields define the observed outcomes?
How should postponed/double-gameweek fixtures and players absent from a snapshot be treated?
Should the report score every player-fixture prediction or aggregate first to fixture/team level?
Is MIN_RESOLVED = 20 counted in fixtures, player-fixtures, or another observation unit?

These choices materially change the Brier/log-loss sample and cannot be recovered safely from the current placeholder branch.

The scoring helpers are ready, but the prediction/outcome join needs a declared contract. Please specify: 1. Which PIT snapshot represents a fixture prediction: the latest snapshot before kickoff, the latest before the gameweek deadline, or another cutoff? 2. Which stored field is the canonical predicted probability for each metric, and which resolved FPL fields define the observed outcomes? 3. How should postponed/double-gameweek fixtures and players absent from a snapshot be treated? 4. Should the report score every player-fixture prediction or aggregate first to fixture/team level? 5. Is `MIN_RESOLVED = 20` counted in fixtures, player-fixtures, or another observation unit? These choices materially change the Brier/log-loss sample and cannot be recovered safely from the current placeholder branch.

solvreven commented

2026-08-30 19:42:01 +02:00

Author

Owner

Answering all five. These are the right questions — the join is where a scoring harness silently grades the wrong thing.

1. The prediction is the latest PIT snapshot before the DEADLINE, not before kickoff.

The deadline is when the squad locks and the decision is made; that is the moment the prediction was acted on. A snapshot taken between the deadline and kickoff has seen team news the decision could not use, and grading against it flatters the model. This is the vaastav ep_this trap the snapshots exist to prevent — their historical file carries a field scraped after the gameweek, so any backtest touching it sees the future.

data/pit/<stamp>/manifest.json already carries captured_at and next_gw, so the selection is: newest snapshot where next_gw == N and captured_at <= deadline(N). There are 14 snapshots; GW1 has seven and GW2 six, so a "latest before deadline" rule is not hypothetical — it picks a different one from "any snapshot for that gameweek".

Record WHICH snapshot each score came from in the report. A number that cannot name its input is the thing this repo keeps finding.

2. Canonical fields.

Match outcome (RPS, the existing backtest_rps metric): predicted = the three-way from the snapshot's solved market/model lambdas; observed = team_h_score/team_a_score from the resolved fixture. Note the snapshot stores our fit AND the market — score ours, and report the market alongside as the benchmark, never blend them.
Clean sheet: predicted cs_h/cs_a; observed = opponent score == 0 and the fixture finished.
Minutes / P(60+): predicted start_prob; observed from event/{gw}/live/ minutes. This already has a harness (backtest_minutes.py, Brier 0.0894 against a 0.1926 base rate) — the panel should read the same definition rather than inventing a second one.
Goalscorer: p_score_sim, against realised goals. scripts/score_goalscorer.py already does this and the chore runner calls it; the panel should consume its output, not re-derive.

3. Postponed and doubles.

Postponed: excluded, and counted in the report as excluded. Silently dropping them makes the denominator move for a reason the reader cannot see.
Double gameweeks: two fixtures, scored independently. A player's gameweek total is not a prediction we made; the fixture-level ones are.
Player absent from the snapshot: excluded and counted, never imputed. A player who did not exist in the cache has no prediction, and treating that as a prediction of zero is precisely the absence-as-value failure.

4. Aggregate to FIXTURE level for the match metrics; keep player-fixture for minutes and goalscorer. They are different claims: "Arsenal 2-1" is one prediction, "Saka starts" is 22 per fixture. Mixing them into one Brier gives the player metrics ~20x the weight of the match ones and the number stops meaning anything.

5. MIN_RESOLVED = 20 in FIXTURES, for the match metrics. At 10 fixtures a gameweek that is two rounds, which is the right order for "enough to say anything". For player-level metrics the floor should be separate and much higher — backtest_minutes runs on 23,884 rows for a reason.

One thing to fix while you are in there. accuracy_report currently writes a refusal on each metric below the floor and does nothing at or above it, so the metrics keep value=None, keep the AWAITING_RESULTS banner, and their detail goes empty — the panel gets quieter the moment it has enough data to speak. tests/web/test_calibration.py::test_the_panel_has_no_scoring_branch_yet pins that gap and is written to fail the moment scoring is wired, so it will tell you when this issue is done.

Answering all five. These are the right questions — the join is where a scoring harness silently grades the wrong thing. **1. The prediction is the latest PIT snapshot before the DEADLINE, not before kickoff.** The deadline is when the squad locks and the decision is made; that is the moment the prediction was acted on. A snapshot taken between the deadline and kickoff has seen team news the decision could not use, and grading against it flatters the model. This is the vaastav `ep_this` trap the snapshots exist to prevent — their historical file carries a field scraped *after* the gameweek, so any backtest touching it sees the future. `data/pit/<stamp>/manifest.json` already carries `captured_at` and `next_gw`, so the selection is: newest snapshot where `next_gw == N` and `captured_at <= deadline(N)`. There are 14 snapshots; GW1 has seven and GW2 six, so a "latest before deadline" rule is not hypothetical — it picks a different one from "any snapshot for that gameweek". Record WHICH snapshot each score came from in the report. A number that cannot name its input is the thing this repo keeps finding. **2. Canonical fields.** - *Match outcome (RPS, the existing `backtest_rps` metric):* predicted = the three-way from the snapshot's solved market/model lambdas; observed = `team_h_score`/`team_a_score` from the resolved fixture. Note the snapshot stores our fit AND the market — score **ours**, and report the market alongside as the benchmark, never blend them. - *Clean sheet:* predicted `cs_h`/`cs_a`; observed = opponent score == 0 **and** the fixture finished. - *Minutes / P(60+):* predicted `start_prob`; observed from `event/{gw}/live/` minutes. This already has a harness (`backtest_minutes.py`, Brier 0.0894 against a 0.1926 base rate) — the panel should read the same definition rather than inventing a second one. - *Goalscorer:* `p_score_sim`, against realised goals. `scripts/score_goalscorer.py` already does this and the chore runner calls it; the panel should consume its output, not re-derive. **3. Postponed and doubles.** - *Postponed:* excluded, and **counted in the report as excluded**. Silently dropping them makes the denominator move for a reason the reader cannot see. - *Double gameweeks:* two fixtures, scored independently. A player's gameweek total is not a prediction we made; the fixture-level ones are. - *Player absent from the snapshot:* excluded and counted, never imputed. A player who did not exist in the cache has no prediction, and treating that as a prediction of zero is precisely the absence-as-value failure. **4. Aggregate to FIXTURE level for the match metrics; keep player-fixture for minutes and goalscorer.** They are different claims: "Arsenal 2-1" is one prediction, "Saka starts" is 22 per fixture. Mixing them into one Brier gives the player metrics ~20x the weight of the match ones and the number stops meaning anything. **5. `MIN_RESOLVED = 20` in FIXTURES**, for the match metrics. At 10 fixtures a gameweek that is two rounds, which is the right order for "enough to say anything". For player-level metrics the floor should be separate and much higher — `backtest_minutes` runs on 23,884 rows for a reason. **One thing to fix while you are in there.** `accuracy_report` currently writes a refusal on each metric below the floor and **does nothing at or above it**, so the metrics keep `value=None`, keep the AWAITING_RESULTS banner, and their `detail` goes empty — the panel gets *quieter* the moment it has enough data to speak. `tests/web/test_calibration.py::test_the_panel_has_no_scoring_branch_yet` pins that gap and is written to fail the moment scoring is wired, so it will tell you when this issue is done.

solvreven referenced this issue

2026-08-30 19:53:57 +02:00

The season stops being hardcoded, the chores run themselves, and Norsk Tipping drives the market #19