No scheduler: every recurring data ingest is a manual chore #13
Labels
No labels
architecture
cleanup
dependencies
performance
priority: high
priority: medium
reliability
security
testing
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
solvreven/FPL#13
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Nothing in the repo runs on a schedule — no CI workflow, no cron, no task.
Every recurring ingest depends on someone remembering:
scripts/fetch_current_season.pyscripts/fetch_market.py --gw Nscripts/fetch_props.py --gw Nscripts/snapshot_pit.pyscripts/build_cache.pyIf the ingest is not run the model silently stops learning;
archive_only_warningnow states that positively in the build warnings, butonly once someone looks.
Cost is not the blocker — measured 2026-08-27. The Odds API bulk endpoint
bills 1 credit per market for the entire slate, so a full match-odds refresh
is 2 credits:
Daily match odds ~60/month, weekly props ~40/month — about 100 against a 500
budget. The earlier assumption of per-event pricing (~20-30 per refresh) was
wrong and is what made this look unaffordable.
Suggested fix. A scheduled job chaining ingest -> cache rebuild -> snapshot,
with the credit budget stated and the balance logged per run (the
events()call is free and returns the quota headers).
The scheduler needs operational choices that the repository cannot infer. Please confirm:
Once those are fixed, the workflow can encode the cadence and budget mechanically.
A scheduler now exists on branch
fix/gw2-window-and-provenance(a89fb53, 2026-08-30 14:58 Oslo) — pushed just now, which is why it was invisible when the questions above were written. Answering them against what is actually built, so the work is not done twice.It is not a CI/cron design, so most of the questions above do not arise:
backend/web/chores.py), started in the FastAPI lifespan beside the existingautorefresh.Refresher. No runner, no workflow file.events.after_gameweek.py, the NT pull,snapshot_pit.py), so there is one code path; the rebuilt cache reaches the app through the existing mtime hot-reload. The ledgerdata/chores.jsonis gitignored deliberately: committing it would hand a fresh clone a completion marker for work that never happened there.Verified end-to-end on live data, not just in tests: on first start it detected GW1 as the newest finished round, ran ingest → rebuild → score unprompted, and the server hot-reloaded the cache it built (
built_at 2026-08-30T12:40:12Z). Ledger now readsafter_gameweek:1 OK.Guards, each a way an automated chore is worse than none: refuses a round still being played; runs each round once; records outcome not attempt; stops after 3 attempts rather than polling FPL; never overlaps itself; never snapshots after a deadline; and publishes its state on
/api/status, rendered in the header — a scheduler that works silently cannot be told from one that died. 37 tests, mutation-checked (12 guards cut one at a time; the two that stayed green were a double-guarded property and one test that could not fail, which was rewritten).The one question that is genuinely still open is architectural, and it is the reader's to make: in-process means it only runs while the server does. A deadline passing with the app closed is recorded
MISSEDand shown in the warning colour, because a point-in-time snapshot cannot be taken retrospectively. A Forgejo Actions version would not have that gap, at the cost of secrets, write-back policy and a second code path. I would not build both.One correction to the issue title: it says every recurring ingest is manual. Bootstrap, fixtures and the squad sync stopped being manual in
9d9bd4d(5-minute atomic refresh, age published).Taking the decision I left open above: keep it in-process. Do not build the Forgejo Actions version.
The trade is one gap against three costs.
The gap: an in-process scheduler only runs while the server does. A deadline passing with the app closed leaves no point-in-time snapshot, and that one cannot be repaired afterwards.
What that gap actually costs, measured rather than assumed. On first start it marked GW1 and GW2's pre-deadline windows
OK, notMISSED— becausesnapshot_exists_forreadsdata/pit/and found seven snapshots taken before GW1's deadline and six before GW2's, every one run by hand. The chore's job is that a snapshot EXISTS before the deadline, not that this code took it. So the realistic failure is "the app was closed for the three hours before a Friday deadline AND nobody ran the script" — and when it happens it is stated in the header in the warning colour, naming the round, rather than being silent.What CI would cost:
data/cache/projections.jsonis ~2 MB and is machine state, not source. Committing it back makes every rebuild a commit; not committing it means the runner's work never reaches the host that serves the app — which is the only place it is wanted.And the deciding point: the app is the only consumer. A cache built on a runner has to reach the machine serving the site, and the only thing that machine is guaranteed to be doing when it serves the site is running the server. Putting the scheduler there removes the transport problem entirely.
If the gap ever bites — i.e. a
MISSEDactually appears — the cheap fix is not CI. It is a Windows Task Scheduler entry on the same host runningscripts/after_gameweek.py, which is the same script, on the same machine, with no secrets and no write-back. That keeps one code path and closes the gap for the cost of one scheduled task.I would revisit this only if the app moves off the local host, at which point the transport problem changes and so does the answer.
Closed after PR #19 merged as
6e888a5. The implementation described in the latest issue comment is now on main.