Skip to content
Post Mortem

Playbook ยท Decide in 15 minutes

A change broke something. Should we roll back?

Compare the cost of the bad change with the cost of undoing it.

HIGHTake a breath. Do these in order.

How you might know

  • Customers report problems right after a release.
  • Data looks missing or wrong.
  • An AI agent or teammate made a change to the live system.

Do this now

  1. Confirm the change is the likely cause

    Check what changed and when the problems started. If they line up, it is likely the cause.

  2. Check whether rollback is safe for data

    Changes to the database may not be reversible. Ask: if we undo this, do we lose anything customers added since?

  3. Take a backup or snapshot first

    Copy the current state so you do not lose evidence or data.

  4. Roll back the smallest thing that restores safety

    Revert the specific release or feature, not the whole system, if you can.

  5. Verify it worked

    Test the main flows and check customer reports. Confirm monitoring is normal.

  6. Tell people what happened

    Customers and your team should know what changed and whether any data was affected.

Do not do this

  • Do not roll back blindly if data changed in ways that cannot be undone.
  • Do not skip verification.
  • Do not let an agent fix its own mistake on live data without a person checking.

Who to tell

  • Incident lead
  • Anyone who relied on the new behavior
  • Customers if data was affected

What to save before you clean up

  • Deployment and migration logs
  • A copy of the data before the rollback

In the next 24 hours

  • Write the timeline while it is fresh.
  • Decide how to re-release safely.
  • Add a rollback plan to every risky release.

Official help for the platforms involved

Links open each provider's own documentation, so you do not have to search for it.

What to write in the post mortem afterwards

  • Why the change shipped without a rollback plan
  • How quickly rollback was decided
  • How to test or gate risky changes

Finished this one?

Add it to What I learned and earn 30 Trust. You can open your list from the badge at the bottom of the page.