Production incident: what to prepare before you call
During an incident, time is rarely lost in the fix: it is lost reconstructing the facts. Preparing four pieces of information before the first call changes the rhythm of the whole intervention.
RGAIX team · · 6 min read
Describe the observable impact, not the assumed cause
Start with what you can see: the affected services, the users concerned, the start time and the journeys that still work. A page that no longer responds and a transaction that fails do not call for the same investigation, and the second is often more serious than the first.
Explicitly separate observed facts from hypotheses. “The database is saturated” is a hypothesis; “orders have been failing since 2:20 pm with a 500 error” is a fact. A hypothesis stated as a fact sends the investigation in one direction, and it then takes time to come back.
Retrace the latest changes
List the deployments, infrastructure changes, configuration edits and data operations close to the start of the incident. Keep their references: version number, release id, ticket, author.
A correlation in time helps the investigation without being, on its own, proof of a cause. Many incidents happen after a release without being caused by it: a change in volume, an expired certificate or a scheduled job can coincide.
Organise the conversation before it organises you
Identify two people: a contact able to authorise the actions that change the system, and a technical contact who knows the application. Without the first, the intervention stops at every decision; without the second, every question costs a round trip.
Agree on a single channel and a rhythm for situation updates. Write down the actions already taken: that is what avoids duplicates and, above all, contradictory changes launched in parallel by two people unaware of each other.
Share what is useful, never the secrets
Prepare the error messages and the relevant log extracts, removing the personal data that plays no part in the diagnosis. Technical access is organised through a suitable channel, with a defined scope and duration.
- No password, API key or token in a form or an email.
- Timestamped log extracts, with the time zone.
- The exact error codes rather than a paraphrase of them.
- A capture of the failing journey, from the user’s side.
What changes after recovery
A correct HTTP response does not prove the service is restored: the affected business journeys must be checked with your own people. Does an order actually go through end to end, confirmation email included?
The report then separates three things: the immediate fix, the cause — identified or still uncertain — and the prevention work. Blurring the three produces a reassuring and useless document.
A well-documented incident beats a quickly forgotten one: it is the only moment when an organisation learns something about its own system.
Related pages
IT troubleshooting & emergencies
Your production is down.
Discover the service ↗READ NEXTCTO, squad or expert reinforcement: which model should you choose?
Three ways of working together, three ways of splitting responsibility. How to pick the one that matches what you are actually missing.
Read the article ↗READ NEXTRTO and RPO: asking the right business questions before the architecture
Two numbers decide your continuity architecture. They are defined with the business, not in an infrastructure spreadsheet.
Read the article ↗Let’s talk about what is blocking you.
An emergency, a profile to find or a project to structure: let’s start with your need.