Runbook automation: automate the steps, keep the judgement

Runbook automation is the practice of turning documented manual procedures into scripts or workflows that a system can execute — either fully, or with a human approving each stage. It is the natural next step once a runbook has been followed enough times to be trusted, and the natural disaster if applied before that point.

The sequencing matters more than the tooling. A procedure that is inconsistent when a person runs it will be consistently wrong when a machine does, faster and at greater scale. Automate what is already reliable; document and stabilise what is not.

Which procedures are worth runbook automation

• Frequent — run often enough that the effort pays back. A quarterly procedure rarely justifies it.
• Deterministic — the same inputs always produce the same correct action, with no judgement in the middle.
• Low-blast-radius, at least initially. Automate the diagnosis and the reversible actions before the destructive ones.
• Currently reliable when done manually, with a runbook people actually follow.
• Boring. The procedures people dread and postpone are the ones where automation removes real risk, because the manual version is where mistakes cluster.

Assisted automation before full automation

The intermediate step is undervalued: the system gathers the diagnostics, presents them, proposes the action, and waits for a human to approve. That captures most of the time saving — the tedious data-gathering — while keeping a person in the decision. It also builds the evidence you need to automate fully later, because you accumulate a record of how often the proposed action was the right one. Teams that jump straight to unattended execution usually lack that evidence and are relying on optimism.

Automation needs its own runbook: what it does, how to tell whether it ran, how to stop it, and what to do when it fails halfway. An automated procedure that fails in the middle leaves the system in a state no manual runbook describes, and that is the scenario where teams discover the automation was the only thing that understood the process.

Keep the audit trail

Automated actions taken during an incident need to be visible afterwards: what ran, when, with what parameters, triggered by whom or by what, and what it changed. Without that, the incident timeline has gaps exactly where the system acted, and the postmortem cannot distinguish what the automation did from what the humans did. Emit the same record you would expect a person to leave.

Automation decays too

Scripts encode assumptions about hostnames, credentials, API versions and directory layouts, and every one of those changes eventually. Unlike a stale document, stale automation fails loudly at the worst moment — or, worse, succeeds against the wrong target. Exercise automated procedures on a schedule in a non-production environment, and treat a failed exercise with the same seriousness as a failed backup restore.

Keeping the catalogue

Ettex Records holds the catalogue of automated procedures with owner, last exercise date and blast radius, Ettex Sheets tracks the candidate list with frequency and time saved so automation effort goes where it pays, and the underlying documents are the ones described in incident response plan and in the runbook itself.

Plainly: this is records and spreadsheets, not an automation platform. Execution belongs in purpose-built tooling; what this covers is deciding what to automate and keeping track of what you have.