solvreven/FPL

Watch

Fork 0

Accuracy panel has no scoring branch and goes quieter once it has data #11

New issue

Closed

opened 2026-08-27 20:57:39 +02:00 by solvreven · 5 comments

solvreven commented

2026-08-27 20:57:39 +02:00

Owner

backend/web/calibration.py defines brier_score, log_loss,
calibration_curve and disagreement_tally, all unit-tested. accuracy_report
calls none of them.

Below MIN_RESOLVED (20) it writes a refusal and a reason on each metric. At or
above it, the function does nothing at all — so the metrics keep value=None,
keep the AWAITING_RESULTS banner, and their detail string goes empty.

The panel gets quieter at the moment it first has enough data to say
something. Ten fixtures are resolved as of 2026-08-27, so this is roughly one
gameweek away from being visible.

Found because the self-dating gate in tests/web/test_calibration.py came due:
it asserted resolved_fixtures == 0 with the note "a fixture has resolved —
update this test", and GW1 resolved it. The gate worked exactly as designed.

test_the_panel_has_no_scoring_branch_yet now pins this state and is written to
fail the moment scoring is wired, so the gap is asserted rather than
rediscovered.

Not a quick fix: scoring needs stored predictions to compare against, which
is what data/pit/ snapshots hold. That is a real build, and it should not be
bolted onto an unrelated change.

`backend/web/calibration.py` defines `brier_score`, `log_loss`, `calibration_curve` and `disagreement_tally`, all unit-tested. **`accuracy_report` calls none of them.** Below `MIN_RESOLVED` (20) it writes a refusal and a reason on each metric. At or above it, the function does nothing at all — so the metrics keep `value=None`, keep the `AWAITING_RESULTS` banner, and their `detail` string goes **empty**. **The panel gets quieter at the moment it first has enough data to say something.** Ten fixtures are resolved as of 2026-08-27, so this is roughly one gameweek away from being visible. Found because the self-dating gate in `tests/web/test_calibration.py` came due: it asserted `resolved_fixtures == 0` with the note "a fixture has resolved — update this test", and GW1 resolved it. The gate worked exactly as designed. `test_the_panel_has_no_scoring_branch_yet` now pins this state and is written to **fail the moment scoring is wired**, so the gap is asserted rather than rediscovered. **Not a quick fix:** scoring needs stored predictions to compare against, which is what `data/pit/` snapshots hold. That is a real build, and it should not be bolted onto an unrelated change.

kristofferopsahl commented

2026-08-30 17:46:13 +02:00

Collaborator

The scoring helpers are ready, but the prediction/outcome join needs a declared contract. Please specify:

Which PIT snapshot represents a fixture prediction: the latest snapshot before kickoff, the latest before the gameweek deadline, or another cutoff?
Which stored field is the canonical predicted probability for each metric, and which resolved FPL fields define the observed outcomes?
How should postponed/double-gameweek fixtures and players absent from a snapshot be treated?
Should the report score every player-fixture prediction or aggregate first to fixture/team level?
Is MIN_RESOLVED = 20 counted in fixtures, player-fixtures, or another observation unit?

These choices materially change the Brier/log-loss sample and cannot be recovered safely from the current placeholder branch.

The scoring helpers are ready, but the prediction/outcome join needs a declared contract. Please specify: 1. Which PIT snapshot represents a fixture prediction: the latest snapshot before kickoff, the latest before the gameweek deadline, or another cutoff? 2. Which stored field is the canonical predicted probability for each metric, and which resolved FPL fields define the observed outcomes? 3. How should postponed/double-gameweek fixtures and players absent from a snapshot be treated? 4. Should the report score every player-fixture prediction or aggregate first to fixture/team level? 5. Is `MIN_RESOLVED = 20` counted in fixtures, player-fixtures, or another observation unit? These choices materially change the Brier/log-loss sample and cannot be recovered safely from the current placeholder branch.

solvreven commented

2026-08-30 19:42:01 +02:00

Author

Owner

Answering all five. These are the right questions — the join is where a scoring harness silently grades the wrong thing.

1. The prediction is the latest PIT snapshot before the DEADLINE, not before kickoff.

The deadline is when the squad locks and the decision is made; that is the moment the prediction was acted on. A snapshot taken between the deadline and kickoff has seen team news the decision could not use, and grading against it flatters the model. This is the vaastav ep_this trap the snapshots exist to prevent — their historical file carries a field scraped after the gameweek, so any backtest touching it sees the future.

data/pit/<stamp>/manifest.json already carries captured_at and next_gw, so the selection is: newest snapshot where next_gw == N and captured_at <= deadline(N). There are 14 snapshots; GW1 has seven and GW2 six, so a "latest before deadline" rule is not hypothetical — it picks a different one from "any snapshot for that gameweek".

Record WHICH snapshot each score came from in the report. A number that cannot name its input is the thing this repo keeps finding.

2. Canonical fields.

Match outcome (RPS, the existing backtest_rps metric): predicted = the three-way from the snapshot's solved market/model lambdas; observed = team_h_score/team_a_score from the resolved fixture. Note the snapshot stores our fit AND the market — score ours, and report the market alongside as the benchmark, never blend them.
Clean sheet: predicted cs_h/cs_a; observed = opponent score == 0 and the fixture finished.
Minutes / P(60+): predicted start_prob; observed from event/{gw}/live/ minutes. This already has a harness (backtest_minutes.py, Brier 0.0894 against a 0.1926 base rate) — the panel should read the same definition rather than inventing a second one.
Goalscorer: p_score_sim, against realised goals. scripts/score_goalscorer.py already does this and the chore runner calls it; the panel should consume its output, not re-derive.

3. Postponed and doubles.

Postponed: excluded, and counted in the report as excluded. Silently dropping them makes the denominator move for a reason the reader cannot see.
Double gameweeks: two fixtures, scored independently. A player's gameweek total is not a prediction we made; the fixture-level ones are.
Player absent from the snapshot: excluded and counted, never imputed. A player who did not exist in the cache has no prediction, and treating that as a prediction of zero is precisely the absence-as-value failure.

4. Aggregate to FIXTURE level for the match metrics; keep player-fixture for minutes and goalscorer. They are different claims: "Arsenal 2-1" is one prediction, "Saka starts" is 22 per fixture. Mixing them into one Brier gives the player metrics ~20x the weight of the match ones and the number stops meaning anything.

5. MIN_RESOLVED = 20 in FIXTURES, for the match metrics. At 10 fixtures a gameweek that is two rounds, which is the right order for "enough to say anything". For player-level metrics the floor should be separate and much higher — backtest_minutes runs on 23,884 rows for a reason.

One thing to fix while you are in there. accuracy_report currently writes a refusal on each metric below the floor and does nothing at or above it, so the metrics keep value=None, keep the AWAITING_RESULTS banner, and their detail goes empty — the panel gets quieter the moment it has enough data to speak. tests/web/test_calibration.py::test_the_panel_has_no_scoring_branch_yet pins that gap and is written to fail the moment scoring is wired, so it will tell you when this issue is done.

Answering all five. These are the right questions — the join is where a scoring harness silently grades the wrong thing. **1. The prediction is the latest PIT snapshot before the DEADLINE, not before kickoff.** The deadline is when the squad locks and the decision is made; that is the moment the prediction was acted on. A snapshot taken between the deadline and kickoff has seen team news the decision could not use, and grading against it flatters the model. This is the vaastav `ep_this` trap the snapshots exist to prevent — their historical file carries a field scraped *after* the gameweek, so any backtest touching it sees the future. `data/pit/<stamp>/manifest.json` already carries `captured_at` and `next_gw`, so the selection is: newest snapshot where `next_gw == N` and `captured_at <= deadline(N)`. There are 14 snapshots; GW1 has seven and GW2 six, so a "latest before deadline" rule is not hypothetical — it picks a different one from "any snapshot for that gameweek". Record WHICH snapshot each score came from in the report. A number that cannot name its input is the thing this repo keeps finding. **2. Canonical fields.** - *Match outcome (RPS, the existing `backtest_rps` metric):* predicted = the three-way from the snapshot's solved market/model lambdas; observed = `team_h_score`/`team_a_score` from the resolved fixture. Note the snapshot stores our fit AND the market — score **ours**, and report the market alongside as the benchmark, never blend them. - *Clean sheet:* predicted `cs_h`/`cs_a`; observed = opponent score == 0 **and** the fixture finished. - *Minutes / P(60+):* predicted `start_prob`; observed from `event/{gw}/live/` minutes. This already has a harness (`backtest_minutes.py`, Brier 0.0894 against a 0.1926 base rate) — the panel should read the same definition rather than inventing a second one. - *Goalscorer:* `p_score_sim`, against realised goals. `scripts/score_goalscorer.py` already does this and the chore runner calls it; the panel should consume its output, not re-derive. **3. Postponed and doubles.** - *Postponed:* excluded, and **counted in the report as excluded**. Silently dropping them makes the denominator move for a reason the reader cannot see. - *Double gameweeks:* two fixtures, scored independently. A player's gameweek total is not a prediction we made; the fixture-level ones are. - *Player absent from the snapshot:* excluded and counted, never imputed. A player who did not exist in the cache has no prediction, and treating that as a prediction of zero is precisely the absence-as-value failure. **4. Aggregate to FIXTURE level for the match metrics; keep player-fixture for minutes and goalscorer.** They are different claims: "Arsenal 2-1" is one prediction, "Saka starts" is 22 per fixture. Mixing them into one Brier gives the player metrics ~20x the weight of the match ones and the number stops meaning anything. **5. `MIN_RESOLVED = 20` in FIXTURES**, for the match metrics. At 10 fixtures a gameweek that is two rounds, which is the right order for "enough to say anything". For player-level metrics the floor should be separate and much higher — `backtest_minutes` runs on 23,884 rows for a reason. **One thing to fix while you are in there.** `accuracy_report` currently writes a refusal on each metric below the floor and **does nothing at or above it**, so the metrics keep `value=None`, keep the AWAITING_RESULTS banner, and their `detail` goes empty — the panel gets *quieter* the moment it has enough data to speak. `tests/web/test_calibration.py::test_the_panel_has_no_scoring_branch_yet` pins that gap and is written to fail the moment scoring is wired, so it will tell you when this issue is done.

solvreven referenced this issue

2026-08-30 19:53:57 +02:00

The season stops being hardcoded, the chores run themselves, and Norsk Tipping drives the market #19

solvreven referenced this issue

2026-09-20 16:03:13 +02:00

Small findings from 2026-09-13, each verified on the live page or in code #34

solvreven referenced this issue

2026-09-26 01:08:02 +02:00

Squad rail becomes a league-wide captaincy board, star-rated #31

solvreven commented

2026-09-26 01:08:02 +02:00

Author

Owner

Partial progress in bf7e50a (pushed):

It no longer goes quieter once it has data. With results in and no scorer, the metrics are NOT_SCORED with the count ("50 results are in; nothing scores them against the snapshots yet"). Before, they kept the AWAITING_RESULTS dataclass default with an empty detail, so the panel said "forecasts not yet scored" with 50 results in.
The goalscorer metric is scored. It reads the rounds scripts/score_goalscorer.py banks in data/scores, through score_history.verdict. A round whose market was priced after its deadline therefore stays out here, as it does in the verdict. On screen, for GW5: 0.2439 log loss on 237 appearances, beside the corrected market (0.2508, Norsk Tipping with the margin removed) and the base rate (0.2792) on the same round. The tile says how many more valid rounds are needed before one source is preferred. It is bound to its metric by name, not by list position. Tests read an empty data/scores unless they pass their own, so no result depends on whether a round has been scored on the machine running them.
Not done: match outcome (RPS) and clean sheets (Brier) still have no scorer. The prediction/outcome join from your five questions and the answers of 30 Aug isn't built. accuracy_report still computes no score itself, and test_the_panel_has_no_scoring_branch_yet still pins that.

Leaving this open for those two metrics.

🤖 Generated with Claude Code

Partial progress in `bf7e50a` (pushed): - **It no longer goes quieter once it has data.** With results in and no scorer, the metrics are `NOT_SCORED` with the count ("50 results are in; nothing scores them against the snapshots yet"). Before, they kept the `AWAITING_RESULTS` dataclass default with an empty detail, so the panel said "forecasts not yet scored" with 50 results in. - **The goalscorer metric is scored.** It reads the rounds `scripts/score_goalscorer.py` banks in `data/scores`, through `score_history.verdict`. A round whose market was priced after its deadline therefore stays out here, as it does in the verdict. On screen, for GW5: 0.2439 log loss on 237 appearances, beside the corrected market (0.2508, Norsk Tipping with the margin removed) and the base rate (0.2792) on the same round. The tile says how many more valid rounds are needed before one source is preferred. It is bound to its metric by name, not by list position. Tests read an empty `data/scores` unless they pass their own, so no result depends on whether a round has been scored on the machine running them. - **Not done:** match outcome (RPS) and clean sheets (Brier) still have no scorer. The prediction/outcome join from your five questions and the answers of 30 Aug isn't built. `accuracy_report` still computes no score itself, and `test_the_panel_has_no_scoring_branch_yet` still pins that. Leaving this open for those two metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

solvreven referenced this issue from a commit

2026-09-26 15:58:09 +02:00

docs: design for match-outcome and clean-sheet scoring by refit (#11)

solvreven referenced this issue from a commit

2026-09-26 15:58:09 +02:00

docs: implementation plan for match-outcome and clean-sheet scoring (#11)

solvreven commented

2026-09-26 15:58:43 +02:00

Author

Owner

Both remaining metrics are wired and pushed. Commits 1e3a4cb..ff6e6d1; the design is in docs/superpowers/specs/2026-09-26-match-scoring-design.md.

The contract from 30 Aug assumed something the snapshots don't hold. Answer 2 said the snapshot stores "our fit AND the market". It stores neither per fixture:

The cache holds player-level projections only. p_cs is a player's chance of clean-sheet points, minutes included.
For a snapshot's own round, the projections ran on the market's goal rates. So our fit's lambdas were computed in engine.project_fixture and thrown away.

Offered a choice between storing forecasts from GW6 on and refitting, the user chose to refit:

Our side: scripts/score_matches.py refits today's model through pipeline.strengths_deps on each pit.choose_snapshot snapshot's frozen results and team list. The path is always named, because None would read the live file.
Market and control: it scores beside the market that snapshot froze (converted with rho = 0, the model its lambdas were solved under) and 25/26's base rates.
Banks: it writes data/scores/matches_gw{N}.json with the scorer's code digest.

A scheduler job (match_scores) rescores whenever a played round lacks a bank from the code on disk, and after_gameweek.py runs it as step 4/4. The panel says "refit" on every tile and no longer promises "never recomputed". The rest of your five answers stand:

the snapshot rule (now pit.choose_snapshot);
postponed and moved fixtures excluded and named;
fixture level;
a floor of 20 fixtures.

A fresh-context review found no Critical issues and five Important ones, all fixed in 60ab678:

A bank that parses but has the wrong shape stopped every chore. Every field a reader touches is now checked.
"Market" pooled the book consensus (GW1–2) and Norsk Tipping (GW5) under one word. The tile now names the book per round.
Exclusions were counted and never shown.
Four refusal causes were shown as one.
Two connections had no test.

The market-age rule (44159af). GW1's consensus was priced 22 days before its deadline, with the transfer window open. A market is now the benchmark only if it was priced in the week before the deadline; it is kept for the record either way. This was decided after the first run, and it narrowed our lead.

Results (lower is better; GW1, GW2, GW5; GW3 and GW4 have no snapshot that can stand for them):

	ours	market (GW2 consensus + GW5 Norsk Tipping, 20 fixtures)	base rate
RPS	0.2097 on 30; 0.2070 on those 20	0.2161	0.2327
Clean-sheet Brier	0.1989 on 30; 0.2046 on those 20	0.2133	0.2018

All paired bootstrap intervals span zero, so the reading is level with the market, as for the goalscorer.

Still not wired (they were out of the spec's scope): calibration_curve and disagreement_tally, and the minutes metric from your answer 2. As far as the two metrics this issue names go, #11 is done. Closing it is the user's call.

🤖 Generated with Claude Code

Both remaining metrics are wired and pushed. Commits `1e3a4cb`..`ff6e6d1`; the design is in `docs/superpowers/specs/2026-09-26-match-scoring-design.md`. **The contract from 30 Aug assumed something the snapshots don't hold.** Answer 2 said the snapshot stores "our fit AND the market". It stores neither per fixture: - The cache holds player-level projections only. `p_cs` is a player's chance of clean-sheet points, minutes included. - For a snapshot's own round, the projections ran on the market's goal rates. So our fit's lambdas were computed in `engine.project_fixture` and thrown away. Offered a choice between storing forecasts from GW6 on and refitting, the user chose to **refit**: - **Our side:** `scripts/score_matches.py` refits today's model through `pipeline.strengths_deps` on each `pit.choose_snapshot` snapshot's frozen results and team list. The path is always named, because `None` would read the live file. - **Market and control:** it scores beside the market that snapshot froze (converted with rho = 0, the model its lambdas were solved under) and 25/26's base rates. - **Banks:** it writes `data/scores/matches_gw{N}.json` with the scorer's code digest. A scheduler job (`match_scores`) rescores whenever a played round lacks a bank from the code on disk, and `after_gameweek.py` runs it as step 4/4. The panel says "refit" on every tile and no longer promises "never recomputed". The rest of your five answers stand: - the snapshot rule (now `pit.choose_snapshot`); - postponed and moved fixtures excluded and named; - fixture level; - a floor of 20 fixtures. **A fresh-context review** found no Critical issues and five Important ones, all fixed in `60ab678`: - A bank that parses but has the wrong shape stopped every chore. Every field a reader touches is now checked. - "Market" pooled the book consensus (GW1–2) and Norsk Tipping (GW5) under one word. The tile now names the book per round. - Exclusions were counted and never shown. - Four refusal causes were shown as one. - Two connections had no test. **The market-age rule (`44159af`).** GW1's consensus was priced 22 days before its deadline, with the transfer window open. A market is now the benchmark only if it was priced in the week before the deadline; it is kept for the record either way. This was decided after the first run, and it narrowed our lead. **Results** (lower is better; GW1, GW2, GW5; GW3 and GW4 have no snapshot that can stand for them): | | ours | market (GW2 consensus + GW5 Norsk Tipping, 20 fixtures) | base rate | |---|---|---|---| | RPS | 0.2097 on 30; 0.2070 on those 20 | 0.2161 | 0.2327 | | Clean-sheet Brier | 0.1989 on 30; 0.2046 on those 20 | 0.2133 | 0.2018 | All paired bootstrap intervals span zero, so the reading is **level with the market**, as for the goalscorer. **Still not wired** (they were out of the spec's scope): `calibration_curve` and `disagreement_tally`, and the minutes metric from your answer 2. As far as the two metrics this issue names go, #11 is done. Closing it is the user's call. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

solvreven referenced this issue

2026-09-26 15:58:43 +02:00

Squad rail becomes a league-wide captaincy board, star-rated #31

solvreven commented

2026-09-26 17:13:35 +02:00

Author

Owner

The rest of the panel is done, in e5c1f77 (pushed). It now scores five things, not three, and every comparison carries a verdict.

Expected points, the app's central number, were scored nowhere. scripts/score_players.py reads the forecast each snapshot stored before its deadline and scores it against what each player scored. FPL's own ep_next from the same frozen bootstrap is scored beside it, and used only when that bootstrap was for the round. GW1/2/5, 1,267 player-rounds: MAE 1.737 v FPL 1.758, level (95% interval −0.078 to +0.038). Ranking is ahead of FPL in all three rounds (rank correlation 0.53/0.55/0.68 v 0.40/0.41/0.63). The 80% range held 85%.
The chance of starting (start_prob × availability), against "started last round". GW2 and GW5, 1,269 players: Brier 0.0968 v 0.1151, ahead (−0.032 to −0.005). GW1 has no round before it and says so.
Verdicts. Every comparison reads ahead, behind or level from a seeded paired bootstrap (calibration.paired_verdict). A difference is called only when the 95% interval excludes zero. The match tiles now read "level with the market".
The two empty sections are filled, and each names what it counts. The disagreement tally covers clean sheets on the rounds the market was the benchmark for: "8 to 11 over 19 disagreements (clean sheets, GW2, 5)". The calibration chart shows the chance of starting: "GW2, 5 · 1,269 players". Both are computed in the scorers' summaries, so test_the_panel_computes_no_score_itself still holds.

The scheduler keeps both new banks current, and after_gameweek.py runs the player scorer as step 5/5. The commit message records sixteen mutation cuts, all red.

Still open, deliberately: accuracy of the fit 2–5 rounds ahead (the window beyond the imminent round) is queued, not built. GW3 and GW4 stay out of every total, because no snapshot stands for them.

🤖 Generated with Claude Code

The rest of the panel is done, in `e5c1f77` (pushed). It now scores five things, not three, and every comparison carries a verdict. - **Expected points**, the app's central number, were scored nowhere. `scripts/score_players.py` reads the forecast each snapshot stored before its deadline and scores it against what each player scored. FPL's own `ep_next` from the same frozen bootstrap is scored beside it, and used only when that bootstrap was for the round. GW1/2/5, 1,267 player-rounds: MAE **1.737 v FPL 1.758, level** (95% interval −0.078 to +0.038). Ranking is ahead of FPL in all three rounds (rank correlation 0.53/0.55/0.68 v 0.40/0.41/0.63). The 80% range held 85%. - **The chance of starting** (`start_prob` × availability), against "started last round". GW2 and GW5, 1,269 players: Brier **0.0968 v 0.1151, ahead** (−0.032 to −0.005). GW1 has no round before it and says so. - **Verdicts.** Every comparison reads ahead, behind or level from a seeded paired bootstrap (`calibration.paired_verdict`). A difference is called only when the 95% interval excludes zero. The match tiles now read "level with the market". - **The two empty sections are filled**, and each names what it counts. The disagreement tally covers clean sheets on the rounds the market was the benchmark for: "8 to 11 over 19 disagreements (clean sheets, GW2, 5)". The calibration chart shows the chance of starting: "GW2, 5 · 1,269 players". Both are computed in the scorers' summaries, so `test_the_panel_computes_no_score_itself` still holds. The scheduler keeps both new banks current, and `after_gameweek.py` runs the player scorer as step 5/5. The commit message records sixteen mutation cuts, all red. **Still open, deliberately:** accuracy of the fit 2–5 rounds ahead (the window beyond the imminent round) is queued, not built. GW3 and GW4 stay out of every total, because no snapshot stands for them. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

solvreven referenced this issue

2026-09-26 17:13:35 +02:00

Squad rail becomes a league-wide captaincy board, star-rated #31

kristofferopsahl referenced this issue from a pull request that will close it,

2026-10-15 21:41:27 +02:00

Split application domains and cache player delivery #43

kristofferopsahl closed this issue

2026-10-15 22:01:06 +02:00