Silence is the dangerous failure
The thing that actually bites
When people imagine automation going wrong, they picture something dramatic: a crash, a red screen, a deleted file. In practice, across every system I run, the dominant failure mode is not a crash.
It is silence. The job simply stops happening, and nothing anywhere says so.
What silence looks like
I audited my own infrastructure and found ten scheduled jobs that should have been running and were not. Among them: a backup safety net whose next run time had become infinity — a scheduling quirk meant it could never fire again, ever. A content pipeline that had last run six weeks earlier. An index that fed a daily selection, twenty days stale, so the selection had been quietly drawing from an old list for three weeks.
Not one of these had produced an error. Not one had sent an alert. Every dashboard was green, because a job that never runs cannot fail.
Why silence is so hard to see
Monitoring is almost always built to watch runs: did the last run succeed? If a job never runs, there is no last run to evaluate, so the monitoring has nothing to complain about. The absence of bad news reads identically to good news.
The habit that catches it
When something is not working, ask the questions in this order:
- Is it scheduled at all? Not “is it installed”, not “is it enabled” — does it have a real next-run time in the future? An enabled job with no valid schedule is the commonest form of this bug.
- Did it run? When last, exactly?
- Did it do anything? A job can run, exit successfully, and have transferred nothing, written nothing, sent nothing.
- Only then: did it run correctly?
Nearly everyone starts at question four. Most real faults are at question one.
Build the heartbeat
The fix is small and it is the highest-value thing in this course. Have the automation record a timestamp somewhere every time it genuinely completes work — a file, a row, a line. Then have something separate watch that timestamp and complain when it gets old.
The watcher must be separate. A job cannot report its own absence; the moment it is not running is exactly the moment it cannot tell you anything. And be precise about what “completes work” means: never let an error write the done-marker. I once found hundreds of records permanently flagged as processed whose entire content was an error message — the failure had been recorded as a result, so nothing would ever retry them. A failure is not a result.