Incident postmortem

On the morning of the outage the ingest tier stopped acknowledging writes.

Timeline

The first alert fired and the on-call engineer acknowledged it within minutes.

Remediation

We added a backpressure signal between the two tiers and re-ran the load test.