A team redeploys the previous application after a failed release. The earlier code starts successfully, then cannot read records written in the new format. The deployment rolled back. The user journey did not recover.
Identify what the release changes
In a fictional order-processing application, a release changes a status field, a configuration value and the format of an exported file. Restoring the prior container does not necessarily restore any of those. An external system may already have consumed the new file.
For each change, identify what can be reversed, what must remain compatible and what needs a separate recovery action. Avoid describing an external action as reversible merely because the code that initiated it can be replaced. A sent notification or accepted order has its own consequences.
Keep a usable recovery version
Google’s release-engineering chapter describes controlled, repeatable releases and the mechanisms around getting software into service. For this example, that discipline means retaining the previous application artifact together with the configuration references and compatibility assumptions needed to run it.
Record where the artifact is retained and who can restore it. Check that the recovery operator can reach the required environment through an approved access path. A command copied into a document is still unproven until its prerequisites have been exercised.
Source: Google SRE · Release Engineering.
Test recovery with new data
A useful test environment contains synthetic records from before and after the proposed change. Deploy the candidate, exercise the new behavior, then restore the earlier application using the planned procedure. Read and update both sets of records through the user interface or API.
If the old version cannot operate safely, choose a different compatibility strategy or a forward recovery plan. Make that decision before release. Recovery can require a corrective deployment; it should not depend on discovering the incompatibility during an incident.
- Does the earlier version understand newly written data?
- Which configuration must accompany it?
- Have queued jobs or downstream systems observed the new behavior?
- What user action establishes that recovery worked?
Assign responsibility for recovery
The team needs a decision rule and an owner for stopping a release. Record the observed failure, the chosen recovery path and the result. If restoring old code could make the situation worse, the operator needs that warning as part of the plan.
The useful outcome of rehearsal is shared understanding: what can go back, what cannot and how the team proves the service is usable again. A retained version becomes a recovery capability only when those pieces connect.