Rebuilding a live-sports data platform

The problem

The client’s live-sports data pipeline had failed on every major event weekend for over a year. Match results took minutes to appear in the app. Standings lagged by hours. Every weekend engineering was firefighting the same shape of incident: queue backpressure, dropped events, then a cascade of stale data across product surfaces.

The team had rewritten the ingest layer twice. Both rewrites shipped, and both failed under peak load within a month.

What I did

I ran a two-week focused discovery before touching any code.

• Mapped every event source, every consumer, and every backing store — 14 systems in the flow.
• Instrumented the pipeline end-to-end and captured a full weekend of production traces.
• Named the actual failure mode: a fan-out step that scaled fine in unit tests but blocked the entire pipeline whenever any single downstream consumer slowed down.

The discovery ended with a written plan, not a proposal. Three phases, with a stop-work checkpoint at the end of each.

The build:

• Replaced the fan-out step with a durable per-consumer log on Postgres, keyed by consumer offset.
• Reworked the ingest layer as a single-writer state machine so I could reason about the invariants.
• Added a small library of end-to-end tests that reproduce every historical incident.

Outcome

By week ten:

• Ingest p99 latency: 12s → 1s
• Event-weekend uptime: 71% → 99.9% (measured across three consecutive event weekends after cutover)
• On-call incidents: -85%

Just as important: the team stopped firefighting weekends and started shipping features again. The first new feature — historical replays — shipped six weeks after the pipeline stabilized. It had been backlogged for nine months.

The stack

Go, Postgres (logical replication and advisory locks), a small durable-log helper library the team now owns, and boring observability with Prometheus and Grafana. No queues, no orchestrators, no service mesh.