Understat: current-season npxG, a live squad list, and a chore that fetches it #30
No reviewers
Labels
No labels
architecture
cleanup
dependencies
performance
priority: high
priority: medium
reliability
security
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
solvreven/FPL!30
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/understat-current-npxg"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stacked on #29, which is stacked on #28. Merge in that order and each reduces to its own commits.
Answers the question "are we using Understat, and can it update itself after gameweeks?" — the answer to the first was yes, but narrowly, and the investigation turned up a defect bigger than the one it set out to fix.
The defect: the join resolved names against a five-week-old squad list
scripts/fetch_understat.pyread FPL elements fromdata/backfill/bootstrap.json— a 25 July snapshot with 558 elements against 651 live. The 93 missing were all summer signings, and an unresolved player is skipped with a barecontinue.So the join was silently blindest on exactly the cohort the file exists to serve — its own docstring says "players whose PL rates are position anchors", which is to say new signings with no PL history.
Goretzka is in that gap. CLAUDE.md already records him as one of the five 2026-08-28 transfers a free source had right while our own snapshot did not. Same staleness, one artefact over, sitting there since July.
Measured on the tracked model input:
foreign_rates.json(2025/26)understat_2026_27.json(new)The gains are the right names — Di Gregorio, Maitland-Niles, Norton-Cuffy, Fernández-Pardo, all cross-league summer signings.
One surviving player's rates moved when zero should have. Chased rather than shrugged at: element 316 Emersonn, same
understat_id14082, minutes 1517→1607, games 27→28. Understat revised its own 2025/26 Toulouse data after the July pull. Not a wrong-identity match — and worth knowing that a "completed" season still moves.Why Understat rather than FotMob, for xG
Checked both rather than trusting the 2026-08-02 audit note. Current-season xG/xA is already ingested free from the FPL API (
expected_goals,expected_assistsper gameweek, wired atpipeline.pyand feeding the rate build), so scraping either site for that would duplicate what we have.What Understat uniquely adds is npxG, which FPL does not publish and which
split_open_playalready consumes.FotMob's
api/data/leagues?id=47does return 200 with 742KB — the "no public API" note is out of date — butstats.playersis a 43-row leaderboard, not squad-wide, with no npxG and no shotmap, and the deep-stats endpoint 404s. It buys ratings and a transfers feed, not xG depth. That lands separately, inpre_deadline_steps, where squad freshness actually matters.Season choice: two seasons of one source, never blended
Each row carries the
seasonit was measured in, so which one won is visible in the row rather than inferred from which file it came out of. The current season only displaces last season once it clears 450 minutes.Verified a true no-op today: 320 rows in, 320 out, byte-identical, because the current season's maximum is 270 minutes. A no-op by construction, not by luck — and the switchover therefore cannot fire on real data until ~GW6, so it is forced with a synthetic fixture.
No season is named in the code. Files are discovered by glob and ordered by the
seasonfield inside them. CLAUDE.md counts fifteen sites of the hardcoded-gameweek family, every guard for it too narrow by exactly the axis the next instance used — so the guard here is not a grep for a filename.test_the_filename_never_decides_the_seasonputs the data inunderstat_something_else_entirely.jsonand asserts the row still wins on its own field.The chore
understat_watch, besident_watch, keyed on coverage — "does the banked file reach the round that has been played" — answered from the file, so a hand-run fetch counts.Not folded into
after_gameweek.py, measured rather than assumed: Understat had GW2 by 48h after that round's last kickoff, whileafter_gameweekfires the moment the round ends. Chaining it there would fetch before Understat published and leave us permanently one round behind.The fetch is always followed by a rebuild — these files are read at cache-build time and nowhere else, so a fetch alone refreshes something no projection sees.
A bug that was live for about a minute, and the sharpest thing here. Coverage counted all five leagues. Ligue 1 had played 3 rounds against the PL's 2, so it returned 3, never fell below the denominator, and the watch would have been permanently satisfied without firing once. Counting one population while describing another — the "20 fixtures" coverage line in a different file.
Verification
BOOTSTRAP_URLin the source and stayed GREEN when the constant was renamed, over code that wouldNameError. Now drivesload_elementsand reads back the URL actually requested.🤖 Generated with Claude Code
https://claude.ai/code/session_014adnmRztkdAPwinWXnNQ7N
Removes the six-book consensus from the served surfaces and makes NT the market for match odds, anytime goalscorer and anytime assist. The consensus files stay on disk and score_goalscorer.py still scores both every round, so this is a code change, not a lost comparison. NT does not open its markets all at once -- measured 2026-09-01/02, goals open ~4 days out and assists ~2, per fixture -- so a single pre-deadline pull could never carry both. nt_watch in chores.py polls on COVERAGE ("does the newest banked run carry both markets for every fixture of the next round", answered from the manifests on disk) rather than on a clock or a completion marker, so a pull someone ran by hand counts. The denominator is the round's fixture count from FPL, never the run's. Four producer-without-a-consumer defects surfaced while wiring it, three created by the NT-only switch itself: disagreements and fixtures_without_market were computed and discarded by main.matches; gw{n}_market_nt.json was written by nt_resolve with nothing rebuilding the cache to read it; and n_fixtures counted match_odds while the table was drawn from a different subset. _unpriced_but_ours no longer returns one flat list. "No book priced X -- the omission is the bookmakers' saying he is unlikely to feature" was printed over five players in two fixtures NT had not opened at all, which is a verdict reported where the book had said nothing. It returns {"declined": [...], "unopened": [...]} keyed on the FPL team id, because the two need opposite responses. pre_deadline_steps is now pull -> resolve -> rebuild -> snapshot, so the frozen cache is the one built from the frozen prices. snapshot_pit.odds_for takes both market files; the raw NT run is zipped rather than pointed at, since prune_nt_runs can delete the directory. Retires _books_expect_him: the coverage-share gate cannot be computed from one book. Reported rather than patched around -- raising our own start_prob floor would let the weakest number in the system decide membership. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014adnmRztkdAPwinWXnNQ7NAdds `understat_watch` to chores.py, beside `nt_watch`. It asks "does the banked current-season file reach the round that has been played" and answers it FROM THE FILE, so a fetch someone ran by hand counts and the ledger is never the evidence -- a completion marker is not evidence of what was completed. NOT folded into after_gameweek.py, and the reason is measured rather than assumed. Understat had GW2 by 48h after that round's last kickoff, while after_gameweek fires the moment the round ends. Chaining it there would fetch before Understat published and leave us permanently one round behind: self-healing in principle, never actually caught up. One observation bounds the lag at <=48h and cannot pin it tighter, which is the argument for keying on coverage rather than on a clock -- the design does not need the number. THE FETCH IS ALWAYS FOLLOWED BY A REBUILD. foreign_rates.json and the current-season file are read at CACHE BUILD time and nowhere else, so a fetch with no rebuild behind it refreshes a file no projection ever sees. That is the shape nt_resolve sat in for a week, writing gw{n}_market_nt.json that only a rebuild reads. No season is written down: the season id and the output filename are both DERIVED from the label FPL reports (understat_season_arg, understat_out_path). A BUG THAT WAS LIVE FOR ABOUT A MINUTE, and the sharpest thing here. Coverage counted all five leagues. Ligue 1 had played 3 rounds while the PL had played 2, so it returned 3, never fell below the denominator, and the watch would have been PERMANENTLY SATISFIED without ever firing once. The denominator is the FPL gameweek, so the numerator has to be EPL rows -- counting one population while describing another is the "20 fixtures" coverage line in a different file. Pinned by test_coverage_counts_only_the_league_the_denominator_belongs_to. 8 tests here, 21 across the three Understat files. Every cut RED, including two on the WIRING rather than the mechanism: removing the planner hook goes red while every helper test stays green, and making the watch fire unconditionally goes red too. A mechanism nothing calls is a comment. Verified at commit time: the 21 targeted tests pass and all mutation cuts go RED. The full suite was still running; its result is reported separately rather than asserted here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014adnmRztkdAPwinWXnNQ7NView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.