Make cache, squad, and override persistence atomic and concurrency-safe #3

New issue

Closed

opened 2026-08-23 22:37:27 +02:00 by kristofferopsahl · 1 comment

kristofferopsahl commented

2026-08-23 22:37:27 +02:00

Collaborator

Finding

Persistent JSON state is written directly to the destination path:

backend/web/pipeline.py:774-780 (save_cache)
backend/web/main.py:1131-1136 (squad)
backend/web/overrides.py:87-103 (read-modify-write overrides)

FastAPI runs synchronous endpoints in a worker thread pool, so overlapping refresh, firm-up, override, and squad requests can interleave. Direct Path.write_text() also exposes readers to partially written JSON. get_state() explicitly anticipates a half-written cache, but retrying later does not prevent lost writes or a process crash leaving a truncated file. Override updates have a classic read-modify-write lost-update race.

Suggested direction

Serialize mutations with an application-level lock.
Write to a temporary file in the same directory, flush/fsync as appropriate, then atomically replace the destination.
Ensure in-memory state is published only after durable persistence succeeds, or define rollback behaviour.
Add concurrent-request and interrupted-write tests for cache, squad, and overrides.

Acceptance criteria

Readers see either the complete old document or the complete new document, never partial JSON.
Concurrent mutations cannot silently overwrite each other.
A failed write leaves the previous usable state intact.

## Finding Persistent JSON state is written directly to the destination path: - `backend/web/pipeline.py:774-780` (`save_cache`) - `backend/web/main.py:1131-1136` (squad) - `backend/web/overrides.py:87-103` (read-modify-write overrides) FastAPI runs synchronous endpoints in a worker thread pool, so overlapping refresh, firm-up, override, and squad requests can interleave. Direct `Path.write_text()` also exposes readers to partially written JSON. `get_state()` explicitly anticipates a half-written cache, but retrying later does not prevent lost writes or a process crash leaving a truncated file. Override updates have a classic read-modify-write lost-update race. ## Suggested direction - Serialize mutations with an application-level lock. - Write to a temporary file in the same directory, flush/fsync as appropriate, then atomically replace the destination. - Ensure in-memory state is published only after durable persistence succeeds, or define rollback behaviour. - Add concurrent-request and interrupted-write tests for cache, squad, and overrides. ## Acceptance criteria - Readers see either the complete old document or the complete new document, never partial JSON. - Concurrent mutations cannot silently overwrite each other. - A failed write leaves the previous usable state intact.

kristofferopsahl added the

reliability

priority: high

labels

2026-08-23 22:37:27 +02:00