← All posts
May 20, 2026
Lessons from recovering 14,000 authentication tokens in production
How Python automation, Athena queries, and encrypted restoration workflows brought 14k auth tokens back from DynamoDB during a production incident.
incident-responseawspythondynamodb
The Incident
A production incident at Nielsen Media required us to recover 14,000 authentication tokens that had been inadvertently affected by a deployment. The data lived in DynamoDB and needed to come back fast.
The Approach
- Python automation to process logs and assemble the recovery payload.
- Athena queries to identify exactly which rows needed restoration.
- EMS-encrypted restoration workflows to put the data back safely.
What I Learned
- Pre-built recovery scripts pay off. We had Athena queries and Python scripts ready from a prior exercise, which cut the recovery time dramatically.
- Encryption-in-transit matters. EMS-encrypted restoration prevented the tokens from ever sitting in plaintext at rest.
- Communicate constantly. The platform team, application owners, and on-call rotation need a single shared status during incidents like this.
Key Takeaway
Incident recovery isn’t the time to discover you don’t have the tooling. Practice on a regular cadence and keep the scripts in the same repo as the production code.