Incident history

What this list covers

• Entries come from this repository, its git history and stored capture receipts.
• History starts on 3 November 2026, when collection began. A short list reflects a short history, not a clean record.
• Times and recovery durations appear where stored clocks establish them.

Summary

players.xml sitemap returned 503 on the shared database host

2026-11-03. Status: resolved.

• Affected: Public pages.

Impact

The player sitemap answered 503, and page visits retried a heavy aggregate every two minutes. API responses were not part of the reported failure.

Cause

The player index aggregate ran on the visitor path and hit its 6 second statement timeout on the shared database host.

Recovery

The fix was committed at 2026-11-03 20:30:04 UTC.

What changed

A scheduled job now builds the index and stores one row in public_snapshots, and visitors read that row. Scheduled refreshes were then bounded by a deadline.

Evidence

• git commit, e5d8e398: States the 6 second timeout, the 503 and the retry behavior, and moves the index off the visitor path.
• git commit, 2b01d421: Bounds the scheduled refresh.
• git commit, 77c4a064: Adds the sitemap index.

Dependency packaging error before later source jobs

2026-11-03. Status: resolved.

• Affected: Collection workers.

Impact

Failed runs were not counted as successful evidence.

Cause

A dependency packaging error.

Recovery

Found and fixed before later successful source jobs.

What changed

Fixed, followed by successful scheduled source jobs.

Evidence

• repository document, docs/final-execution-followup.md: States the error and that later source jobs succeeded.

Native sportsbook recovery slowed by accumulated retry backoff

2026-11-03. Status: resolved.

• Started: Nov 3, 2026, 21:10 UTC.
• Affected: Native sportsbook subscribed scopes.

Impact

All three subscribed native scopes were marked stale and required resynchronization. Their authenticated snapshot and live delivery readiness were unavailable during the observed outage.

Cause

Planned periodic reconciliation boundaries accumulated a lifetime retry counter, increasing subsequent recovery waits toward ten minutes despite healthy sessions between boundaries.

Recovery

All three subscribed native scopes were unavailable at 21:13:18 UTC. After a controlled restart with the fix, all three were verified available; new baseline receipts arrived at 21:15:37–44 UTC.

What changed

Planned reconciliation now waits 30 seconds and resets that failure streak. Genuine rapid failures retain exponential backoff. The corrected worker was restarted and all three subscribed scopes were verified available.

Evidence

• repository document, docs/native-reconciliation-incident-2026-11-03.md: Read-only production observations, receipt clocks, journal evidence and verified recovery.
• git commit, 4400e700: Corrects planned reconciliation backoff and adds regression tests.

Planned database cutover to an owned PostgreSQL server

2026-11-03. Status: resolved.

• Started: Nov 3, 2026, 19:15 UTC.
• Affected: Pick6, DFS, Sportsbook (ESPN-attributed mainlines).

Impact

Collection writes stopped during the move. On the old copy the last Pick6 capture is 19:11:42 UTC, the last DFS capture 19:02:06 UTC and the last sportsbook capture 19:15:06 UTC. All 47 tables, 533,222 rows, matched by count and digest after the restore.

Cause

Planned. The repository records a move to owned storage at lower cost, with storage at about 653 MB against an 800 MiB admission guard at 18:28 UTC.

Recovery

The old database was fenced read-only at 19:15:28 UTC. The commit recording that all three collection jobs had completed on the new database is dated 19:23:07 UTC, so collection resumed within at most 7 minutes 39 seconds of the fence. Capture gaps are on /reliability.

What changed

Workers and timers were restarted against the new database, and API checks with a temporary key returned 200 and a revoked key returned 401. The old copy stays read-only for rollback.

Evidence

• repository document, docs/shared-database-result.md: Fence time, table and row counts, digest match, restart and verification.
• git commit, 397ce5d5: Dated 19:23:07 UTC, records the cutover as done.
• repository document, docs/final-execution-followup.md: Storage measured at about 653 MB against the 800 MiB guard at 18:28 UTC.
• stored capture receipts, pick6_captures, dfs_captures, batches on the pre-cutover copy: Last capture clock per dataset before the fence.

Pick6 category failures failed the whole batch

2026-11-03. Status: resolved.

• Started: Nov 3, 2026, 16:45 UTC.
• Affected: Pick6.

Impact

One failing category ended its batch, so later due categories waited. Captures already stored stayed valid. A source cooldown was also in effect, and it was not cleared or bypassed.

Cause

The collector awaited each target with no per-target isolation, so one category's decode or ingest failure stopped the batch.

Recovery

The repository records a successful batch at 16:40:29 UTC and failures for targets attempted at 16:45:35, 16:50:39 and 16:55:43 UTC. pick6_batch_runs, added by this fix, starts at 17:05:49 UTC with no failed run on the pre-cutover copy.

What changed

Failures are now isolated per target, back off from 15 minutes up to 6 hours, and are stored with an enumerated error code. Discovery-only responses no longer count as complete scopes.

Evidence

• repository document, docs/pick6-reliability-result.md: The measured baseline, the attempt clocks and the change.
• stored capture receipts, pick6_batch_runs on the pre-cutover copy: First run clock and run statuses.

Collection gap of about nine to eleven hours on all three source jobs

2026-11-03. Status: resolved.

• Started: Nov 3, 2026, 05:36 UTC.
• Ended: Nov 3, 2026, 16:40 UTC.
• Affected: Pick6, DFS, Sportsbook (ESPN-attributed mainlines), Game context (ESPN).

Impact

No new observations were stored for those hours, and these changes were not recorded and cannot be recovered. Existing rows stayed readable and aged into stale. Nothing was backfilled.

Cause

The stored receipts show that captures stopped and do not record why.

Recovery

Measured between stored capture receipts. Pick6: no capture from 05:36:36 to 16:28:53 UTC, 10 h 52 min. DFS: 07:37:45 to 16:40:19 UTC, 9 h 2 min. Sportsbook: 07:36:51 to 16:40:37 UTC, 9 h 3 min. The range above spans all three.

What changed

Collection resumed at 16:28 to 16:40 UTC. The same day the workers gained per-target failure isolation and durable run records, which the next entry describes.

Evidence

• stored capture receipts, pick6_captures, dfs_captures, batches, context_captures on the pre-cutover copy: The longest gap between consecutive capture clocks per dataset.

Native sportsbook endpoints denied automated requests

2026-11-03. Status: open.

• Started: Nov 3, 2026, 00:39 UTC.
• Affected: Sportsbook (native scopes).

Impact

Requests for broad native markets returned HTTP 403, so no native markets were imported from them. ESPN-attributed mainlines stayed separately labeled, and broad native player props remain unavailable.

Cause

The source denied the requests. Collection stopped after each denial, with no retry from another route.

Recovery

Not recovered as a general feed. Narrow subscribed native scopes launched later the same day, and broader probes were denied and stopped, so broad native coverage does not exist.

What changed

Scope-specific native collection with its own admission budget and cooldowns. See /data/coverage-report/completeness for the scopes that exist now.

Evidence

• repository document, docs/source-native-result.md: HTTP 403 at 00:39:31 UTC and the decision to stop.
• repository document, DK-PUBLIC-RESULT.md: HTTP 403 receipt at 04:42:20 UTC and zero markets imported.
• git commit, 8cdf2714: Launches the narrow native scopes.