When a cloud application becomes slow, unavailable, error-prone, or suspiciously inconsistent, this runbook gives developers and IT teams a repeatable way to contain impact, collect evidence, recover safely, communicate clearly, and learn from the incident.
Overview
A production incident is easier to manage when the team separates urgent action from later analysis. The first objective is to protect users and limit the blast radius. The second is to restore a reliable service. Only after those goals are underway should the team pursue a complete explanation.
Use this cloud incident response runbook for outages, elevated error rates, severe latency, failed deployments, dependency failures, authentication problems, and possible data integrity issues. Adapt the steps to your architecture, cloud provider, observability stack, and service-level objectives. The names of dashboards and commands will vary, but the operating sequence remains useful across many app development platforms and cloud deployment models.
Severity matrix
Define severity by user impact and business risk rather than by the component that failed.
- Critical: The application is broadly unavailable, core transactions are failing, data may be exposed or corrupted, or there is no safe workaround. Assign an incident lead immediately and begin frequent stakeholder updates.
- High: A major feature, region, customer group, or integration is materially degraded, but service remains partially available. Page the appropriate on-call owners and establish a recovery plan.
- Moderate: The issue affects a limited workflow or small segment, has a workaround, or is trending toward greater impact. Track it with a named owner and a time-bound investigation.
- Low: The issue has limited operational impact and no immediate risk of escalation. Record it for normal engineering work, while monitoring for change.
If the severity is uncertain, start with the higher level and reduce it when evidence supports doing so. Under-classifying an incident can delay coordination; over-classifying it can be corrected without losing the initial response window.
The first response
- Acknowledge the alert or report and record the time, reporter, affected service, and initial symptoms.
- Open an incident channel or shared workspace. Assign an incident lead, communications owner, technical investigators, and a scribe when staffing allows.
- Check whether users are currently affected. Test the smallest representative request from an approved location or synthetic check.
- Freeze non-essential production changes. Avoid unrelated deployments, migrations, configuration edits, and manual data manipulation until the situation is understood.
- Publish a short internal update: what is known, what is unknown, what action is underway, and when the next update will arrive.
What to track
Track symptoms, scope, recent changes, and recovery signals. A useful incident record should allow someone who was not on call to reconstruct the event without relying on memory.
Service health
- Availability by endpoint, region, tenant, or customer segment.
- Request volume, success rate, and status-code distribution, especially server errors, timeouts, and authorization failures.
- Latency percentiles rather than only averages, separated by route and dependency where possible.
- Container, virtual machine, function, or worker saturation: CPU, memory, concurrency, queue depth, connection pools, and throttling.
- Health-check and synthetic-monitor results from more than one location.
Application and infrastructure evidence
Preserve logs before changing retention settings or restarting components that may hold useful context. Capture a time-bounded sample using a consistent timezone and include correlation or request identifiers. Record representative error messages, stack traces, deployment versions, configuration revisions, and affected routes.
Use traces to follow a slow or failed request across the frontend, API, background worker, database, and external services. Compare healthy and unhealthy traces instead of examining only the slowest example. For serverless app hosting, include invocation duration, cold-start indicators where available, concurrency, retries, and platform throttling signals.
Change and dependency history
Build a timeline of changes immediately before the first symptom. Include application releases, feature flags, infrastructure changes, secret or certificate updates, schema changes, autoscaling events, dependency releases, and provider notices. “No deployment occurred” is useful evidence, but it does not rule out a configuration, dependency, traffic, or infrastructure change.
Check databases, queues, caches, object storage, identity providers, payment systems, messaging services, and APIs outside your direct control. Compare their health and latency with the application timeline. If authentication is involved, review token validation, clock differences, key rotation, callback URLs, and provider status without exposing credentials or tokens in the incident channel. For background on selecting an authentication layer, see this authentication comparison.
Data integrity and security signals
Treat unexpected data changes as a separate risk from ordinary downtime. Check duplicate records, missing writes, delayed jobs, inconsistent read replicas, failed transactions, and unusual administrative activity. Preserve access logs and relevant audit records. Do not copy sensitive production data into public debugging tools or paste secrets, tokens, customer records, or private URLs into chat.
If compromise, unauthorized access, or regulated data exposure is plausible, follow the organization’s security escalation process immediately. Containment may require actions that differ from a normal availability incident, such as preserving evidence, restricting access, or involving legal and privacy teams.
Cadence and checkpoints
A predictable cadence reduces confusion during a fast-moving incident. The incident lead should adjust the interval to the severity, but every update should state the same four things: current impact, confirmed facts, actions in progress, and the next checkpoint.
Decision tree for triage
- Is the issue reproducible? If no, compare monitoring, logs, customer reports, and affected segments. If yes, capture a minimal failing request and its correlation ID.
- Is the scope global or isolated? Compare regions, tenants, routes, versions, and network paths. Isolation can point toward routing, configuration, permissions, or a localized dependency.
- Did a recent change precede the symptom? If yes, pause the rollout and assess rollback or flag disablement. If no, examine traffic, capacity, dependencies, certificates, scheduled jobs, and provider events.
- Can the risk be reduced safely? Consider removing a faulty feature flag, shifting traffic, increasing capacity within tested limits, disabling a non-critical integration, or placing a system in read-only mode.
- Is recovery verified? Do not close the incident because one dashboard looks normal. Confirm user journeys, error rates, latency, queues, database behavior, and data correctness.
Rollback and recovery workflow
Prefer the smallest reversible action that addresses the likely failure. A rollback is appropriate when the previous version is known to be compatible and the change is a credible cause. A feature-flag change or traffic shift may be safer when reverting code would repeat a database or infrastructure transition. For schema-related incidents, avoid destructive reverse migrations without reviewing dependencies and backups. The database migration rollback playbook provides a useful companion checklist.
After mitigation, watch the system for a defined observation period. Confirm that retries are not creating duplicate work, queues are draining at a safe rate, caches are repopulating, and resource usage is stabilizing. Document every manual action, including who performed it and how it can be reversed.
Communication template
Use plain language and avoid unconfirmed root-cause claims:
Status: Investigating / Mitigating / Monitoring / Resolved
Impact: [who or what is affected, including known regions or workflows]
Start time: [time and timezone]
Known facts: [confirmed symptoms and measurements]
Action: [current mitigation or investigation]
Next update: [time or condition]
Customer-facing updates should explain the effect and current status without exposing internal architecture or speculative details. Use the same facts internally and externally, but tailor the level of technical detail.
How to interpret changes
Incident metrics become useful when compared with a known baseline. A rise in latency with stable traffic may indicate a dependency, database, or resource problem. A rise in latency and traffic together may indicate saturation, inefficient queries, or insufficient capacity. A sharp error-rate increase immediately after a release makes the release a strong lead, not automatic proof.
Separate correlation from causation by changing one variable at a time where practical. Compare affected and unaffected routes, versions, regions, and tenants. Look for the first measurable deviation, not merely the most dramatic graph. A service can appear recovered while asynchronous work remains stalled, error budgets are still being consumed, or data repair is incomplete.
For recurring incidents, track these fields after each event:
- Detection time, acknowledgment time, mitigation time, and full recovery time.
- Primary symptom, confirmed cause, contributing factors, and affected components.
- Alert that fired, alert that should have fired, and signals that were missing or misleading.
- Rollback, failover, feature-flag, capacity, or manual recovery action used.
- Customer impact, data impact, follow-up owner, and due date.
Do not turn the postmortem into a search for individual blame. Examine system design, defaults, permissions, testing, deployment controls, observability, documentation, and decision-making conditions. The goal is to make the next response faster and the same failure less likely.
Postmortem template
- Summary: What happened and what users experienced.
- Timeline: Detection, escalation, key observations, mitigations, recovery, and closure.
- Impact: Duration, scope, failed workflows, data effects, and customer communications.
- Technical explanation: Trigger, contributing conditions, and why existing controls did not prevent or detect it earlier.
- What went well: Signals, automation, documentation, or coordination that helped.
- What did not go well: Gaps in alerts, ownership, testing, capacity, access, or recovery procedures.
- Actions: Specific changes with an owner, priority, acceptance condition, and target date.
When to revisit
Review this production incident runbook at least quarterly and after any major change to the application architecture, cloud provider, deployment pipeline, database, identity system, observability tools, or on-call structure. Revisit it immediately after a significant incident, a failed recovery exercise, or a handoff that exposes unclear ownership.
During each review, test the alert links, dashboards, escalation paths, backup and restore assumptions, rollback instructions, status-page access, and permissions needed to act. Remove obsolete commands and replace provider-specific steps when the platform changes. Validate that logs, traces, and metrics still contain the fields the team expects; an observability migration can silently invalidate an old query.
Run a short incident drill on a recurring schedule. Choose one scenario—deployment regression, database saturation, authentication failure, dependency outage, or suspected data issue—and ask the team to identify the incident lead, evidence sources, safe mitigation, communication channel, and recovery test. Record friction rather than judging performance.
Before the next on-call rotation, keep a compact version of this checklist easy to find:
- Acknowledge, classify, and assign roles.
- Confirm user impact and freeze risky changes.
- Capture metrics, logs, traces, versions, and dependency status.
- Choose the smallest reversible mitigation.
- Communicate at a predictable cadence.
- Verify recovery across user journeys, data, queues, and dependencies.
- Schedule a blameless postmortem and track actions to completion.
Pair this runbook with your production readiness checklist before launch and with the modern app stack architecture guide when reviewing boundaries between frontend, backend, database, and hosting layers. A runbook is valuable only when it reflects the system that actually runs in production.
