Design for the day it breaks

Assume the failure

An automation that has run for two years is not a solved problem — it is an unexploded one. Power fails, disks fill, passwords rotate, a vendor changes an API, a file arrives empty. All of these will happen. The only variable you control is what the system does in response.

Three behaviours, best to worst

Fail and recover. The system notices the problem, waits, tries again, and succeeds. Best where the failure is genuinely transient — a network blip, a busy server. Bound your retries: three attempts with increasing gaps, then give up loudly. Unbounded retries are not resilience, they are a machine hammering a broken thing forever.

Fail and stop. The system notices the problem, refuses to continue, and says so. This is the right default for anything that writes, deletes, sends or charges. A stopped automation is an inconvenience. An automation that continues past a condition it did not understand is an incident.

Fail and continue. The system hits a problem, ignores it, and carries on. This is how you get a backup that copies nothing for three weeks and reports success every morning. Never build this on purpose — and it is worth knowing that you rarely do. It is what you get by default when nobody decided.

The condition you must not gate on

Here is a mistake I have made more than once, and it has a shape worth memorising: never gate an automation on a latch that a human has to remember to clear.

A flag gets set — “paused: on limited internet”, “a show is currently on air”, “maintenance in progress”. The condition ends. Nobody clears the flag, because clearing it was never anybody's job. Weeks later you discover a job that has been quietly refusing to run, reporting no error, because it is still obeying an instruction that stopped being true.

Gate on live state instead: is the process actually running right now? is the disk actually mounted right now? is the connection actually metered right now? Live state clears itself, because it was never a claim about the world — it was a reading of it.

Prove the failure path

The recovery path you have never tested does not work. That is not pessimism; it is the observed rate. Unplug the drive. Change the password. Feed it an empty file. Do it on a Tuesday afternoon while you are calm, because the alternative is discovering it at 3am while you are not.