{"page":"/incidents","as_of":"2026-11-03","scope":["Entries come from this repository, its git history and stored capture receipts.","History starts on 3 November 2026, when collection began. A short list reflects a short history, not a clean record.","Times and recovery durations appear where stored clocks establish them."],"incidents":[{"id":"2026-11-03-player-sitemap-503","title":"players.xml sitemap returned 503 on the shared database host","date":"2026-11-03","status":"resolved","started_at":null,"ended_at":null,"recovery_seconds":null,"recovery_note":"The fix was committed at 2026-11-03 20:30:04 UTC.","datasets":["Public pages"],"impact":"The player sitemap answered 503, and page visits retried a heavy aggregate every two minutes. API responses were not part of the reported failure.","cause":"The player index aggregate ran on the visitor path and hit its 6 second statement timeout on the shared database host.","remediation":"A scheduled job now builds the index and stores one row in public_snapshots, and visitors read that row. Scheduled refreshes were then bounded by a deadline.","evidence":[{"kind":"git commit","reference":"e5d8e398","shows":"States the 6 second timeout, the 503 and the retry behavior, and moves the index off the visitor path."},{"kind":"git commit","reference":"2b01d421","shows":"Bounds the scheduled refresh."},{"kind":"git commit","reference":"77c4a064","shows":"Adds the sitemap index."}]},{"id":"2026-11-03-dependency-packaging","title":"Dependency packaging error before later source jobs","date":"2026-11-03","status":"resolved","started_at":null,"ended_at":null,"recovery_seconds":null,"recovery_note":"Found and fixed before later successful source jobs.","datasets":["Collection workers"],"impact":"Failed runs were not counted as successful evidence.","cause":"A dependency packaging error.","remediation":"Fixed, followed by successful scheduled source jobs.","evidence":[{"kind":"repository document","reference":"docs/final-execution-followup.md","shows":"States the error and that later source jobs succeeded."}]},{"id":"2026-11-03-native-reconciliation-backoff","title":"Native sportsbook recovery slowed by accumulated retry backoff","date":"2026-11-03","status":"resolved","started_at":"2026-11-03T21:10:24Z","ended_at":null,"recovery_seconds":null,"recovery_note":"All three subscribed native scopes were unavailable at 21:13:18 UTC. After a controlled restart with the fix, all three were verified available; new baseline receipts arrived at 21:15:37–44 UTC.","datasets":["Native sportsbook subscribed scopes"],"impact":"All three subscribed native scopes were marked stale and required resynchronization. Their authenticated snapshot and live delivery readiness were unavailable during the observed outage.","cause":"Planned periodic reconciliation boundaries accumulated a lifetime retry counter, increasing subsequent recovery waits toward ten minutes despite healthy sessions between boundaries.","remediation":"Planned reconciliation now waits 30 seconds and resets that failure streak. Genuine rapid failures retain exponential backoff. The corrected worker was restarted and all three subscribed scopes were verified available.","evidence":[{"kind":"repository document","reference":"docs/native-reconciliation-incident-2026-11-03.md","shows":"Read-only production observations, receipt clocks, journal evidence and verified recovery."},{"kind":"git commit","reference":"4400e700","shows":"Corrects planned reconciliation backoff and adds regression tests."}]},{"id":"2026-11-03-database-cutover","title":"Planned database cutover to an owned PostgreSQL server","date":"2026-11-03","status":"resolved","started_at":"2026-11-03T19:15:28Z","ended_at":null,"recovery_seconds":null,"recovery_note":"The old database was fenced read-only at 19:15:28 UTC. The commit recording that all three collection jobs had completed on the new database is dated 19:23:07 UTC, so collection resumed within at most 7 minutes 39 seconds of the fence. Capture gaps are on /reliability.","datasets":["Pick6","DFS","Sportsbook (ESPN-attributed mainlines)"],"impact":"Collection writes stopped during the move. On the old copy the last Pick6 capture is 19:11:42 UTC, the last DFS capture 19:02:06 UTC and the last sportsbook capture 19:15:06 UTC. All 47 tables, 533,222 rows, matched by count and digest after the restore.","cause":"Planned. The repository records a move to owned storage at lower cost, with storage at about 653 MB against an 800 MiB admission guard at 18:28 UTC.","remediation":"Workers and timers were restarted against the new database, and API checks with a temporary key returned 200 and a revoked key returned 401. The old copy stays read-only for rollback.","evidence":[{"kind":"repository document","reference":"docs/shared-database-result.md","shows":"Fence time, table and row counts, digest match, restart and verification."},{"kind":"git commit","reference":"397ce5d5","shows":"Dated 19:23:07 UTC, records the cutover as done."},{"kind":"repository document","reference":"docs/final-execution-followup.md","shows":"Storage measured at about 653 MB against the 800 MiB guard at 18:28 UTC."},{"kind":"stored capture receipts","reference":"pick6_captures, dfs_captures, batches on the pre-cutover copy","shows":"Last capture clock per dataset before the fence."}]},{"id":"2026-11-03-pick6-target-failures","title":"Pick6 category failures failed the whole batch","date":"2026-11-03","status":"resolved","started_at":"2026-11-03T16:45:35Z","ended_at":null,"recovery_seconds":null,"recovery_note":"The repository records a successful batch at 16:40:29 UTC and failures for targets attempted at 16:45:35, 16:50:39 and 16:55:43 UTC. pick6_batch_runs, added by this fix, starts at 17:05:49 UTC with no failed run on the pre-cutover copy.","datasets":["Pick6"],"impact":"One failing category ended its batch, so later due categories waited. Captures already stored stayed valid. A source cooldown was also in effect, and it was not cleared or bypassed.","cause":"The collector awaited each target with no per-target isolation, so one category's decode or ingest failure stopped the batch.","remediation":"Failures are now isolated per target, back off from 15 minutes up to 6 hours, and are stored with an enumerated error code. Discovery-only responses no longer count as complete scopes.","evidence":[{"kind":"repository document","reference":"docs/pick6-reliability-result.md","shows":"The measured baseline, the attempt clocks and the change."},{"kind":"stored capture receipts","reference":"pick6_batch_runs on the pre-cutover copy","shows":"First run clock and run statuses."}]},{"id":"2026-11-03-collection-gap","title":"Collection gap of about nine to eleven hours on all three source jobs","date":"2026-11-03","status":"resolved","started_at":"2026-11-03T05:36:36Z","ended_at":"2026-11-03T16:40:37Z","recovery_seconds":39841,"recovery_note":"Measured between stored capture receipts. Pick6: no capture from 05:36:36 to 16:28:53 UTC, 10 h 52 min. DFS: 07:37:45 to 16:40:19 UTC, 9 h 2 min. Sportsbook: 07:36:51 to 16:40:37 UTC, 9 h 3 min. The range above spans all three.","datasets":["Pick6","DFS","Sportsbook (ESPN-attributed mainlines)","Game context (ESPN)"],"impact":"No new observations were stored for those hours, and these changes were not recorded and cannot be recovered. Existing rows stayed readable and aged into stale. Nothing was backfilled.","cause":"The stored receipts show that captures stopped and do not record why.","remediation":"Collection resumed at 16:28 to 16:40 UTC. The same day the workers gained per-target failure isolation and durable run records, which the next entry describes.","evidence":[{"kind":"stored capture receipts","reference":"pick6_captures, dfs_captures, batches, context_captures on the pre-cutover copy","shows":"The longest gap between consecutive capture clocks per dataset."}]},{"id":"2026-11-03-native-source-denied","title":"Native sportsbook endpoints denied automated requests","date":"2026-11-03","status":"open","started_at":"2026-11-03T00:39:31Z","ended_at":null,"recovery_seconds":null,"recovery_note":"Not recovered as a general feed. Narrow subscribed native scopes launched later the same day, and broader probes were denied and stopped, so broad native coverage does not exist.","datasets":["Sportsbook (native scopes)"],"impact":"Requests for broad native markets returned HTTP 403, so no native markets were imported from them. ESPN-attributed mainlines stayed separately labeled, and broad native player props remain unavailable.","cause":"The source denied the requests. Collection stopped after each denial, with no retry from another route.","remediation":"Scope-specific native collection with its own admission budget and cooldowns. See /data/coverage-report/completeness for the scopes that exist now.","evidence":[{"kind":"repository document","reference":"docs/source-native-result.md","shows":"HTTP 403 at 00:39:31 UTC and the decision to stop."},{"kind":"repository document","reference":"DK-PUBLIC-RESULT.md","shows":"HTTP 403 receipt at 04:42:20 UTC and zero markets imported."},{"kind":"git commit","reference":"8cdf2714","shows":"Launches the narrow native scopes."}]}]}