Six lessons from automation that runs while you sleep

Most of what I run at home runs without me. Jobs fire on timers, do work, publish results, and tell me only if something interesting happened. That is the point of automation and it is also the danger of it: a job nobody watches can be broken for two weeks before anyone notices.

These are lessons that cost me something. None are clever. Most are the kind of thing that reads as obvious and is not obvious until it has happened to you.

1. A monotonic timer can stop firing and say nothing

The first one is specific enough to be worth naming exactly, because it is a trap in a very widely used tool.

Systemd timers come in two broad shapes. Calendar timers fire at wall-clock times: every hour on the hour, every Tuesday at five. Monotonic timers fire relative to an event: fifteen minutes after boot, an hour after the last run.

Monotonic looks simpler and for many jobs it is. But a monotonic timer can end up in a state where it has elapsed and has no next trigger scheduled, and when that happens it simply stops. No error, no failed unit, no alert. The job just never runs again.

I lost sixteen days of data collection to exactly this before noticing the numbers had gone flat. The fix is to use calendar timers with catch-up enabled for anything whose job is to happen regularly, and to reserve monotonic timers for things genuinely tied to an event.

The generalisation: prefer scheduling primitives that fail loudly. A job that runs and errors is visible. A job that silently stops existing is not.

2. Your success message will lie to you eventually

I had a fetch step that printed how many new items it had found. It printed the same number every run, forever, because it was counting the wrong thing — total candidates after de-duplication rather than genuinely new ones.

The consequence was not the wrong number. The consequence was that “quiet” and “broken” became indistinguishable. When someone eventually asked why nothing had been published for three days, the logs said the pipeline had found plenty of work each time. It had not. I could not tell the difference between a slow news week and a dead feed, and so I could not tell there was nothing wrong — which, it turned out, there mostly wasn’t, but three other real bugs were hiding behind that one bad counter.

Make the summary line report the thing you would actually use to decide whether to investigate. If it cannot distinguish “nothing happened” from “nothing worked”, it is worse than no summary at all.

3. Never let a raw exception reach a log

A library that raises on a failed HTTP request will usually put the request URL in the exception message. If that URL contains an API token — and for a lot of messaging and notification APIs it does, in the path — then an unhandled error prints your credential.

On an interactive run you see it and shrug. On a scheduled run it goes to the system journal, where it sits, in plain text, in a file that gets rotated into backups.

Every outbound call now goes through a small wrapper that catches, discards the original exception, and re-raises with only the method name and the service’s own error description. It took twenty minutes to write and it closed a hole I would not have found by looking.

4. A deterministic check that is too broad fails with total confidence

I had a validation step whose job was to reject any text claiming a vulnerability was being exploited when the source did not support that claim. It was implemented as a substring search for the word “exploit”.

Which meant it rejected “no known exploitation”. And “not currently exploited”. Both of which are true, and both of which are precisely what a careful piece of writing says.

It sat there quietly discarding correct output, with nothing anywhere indicating that the check was the thing that was wrong. The failure looked like a generation problem, so I spent time on the generator.

The lesson is not “handle negation”, though it is now negation-aware. The lesson is that a deterministic guard that is too broad is worse than a fuzzy one, because it fails with complete confidence and produces no reasoning you can inspect. If you write a rule-based check, probe it in both directions: things it should catch, and things it must not.

5. Approval gates must not block

An early version of a review step posted a message and then waited, in-process, for someone to approve it. Six-hour timeout, which felt generous.

It fails on the most common real case in the world: the job runs at half past midnight, everyone is asleep, and by seven in the morning the process has already given up — having sat idle for six hours holding a finished piece of work.

Approval is now durable state on disk. The job posts the request, writes a record, and exits. A separate small job runs periodically, matches approvals to pending records, and acts on them. A response at any hour works, including after a reboot, because nothing is waiting in memory.

The related trap: a double-click on an approve button delivers two events. If both arrive in the same batch, naive handling processes both and does the work twice. Claim the token before doing any work, and re-check that the pending record still exists.

6. Write down how it works somewhere that is not the machine

The expensive one.

I had a box that held the only copy of a system’s configuration and the credentials it used. It died. The code existed in a repository; the thing that made the code run did not.

Everything now lives in version control — the scripts, the unit files, the configuration, the documentation of why each odd decision is the way it is. Secrets stay out and go somewhere they can be reissued. The test I apply is: if this machine vanished tonight, is rebuilding it a boring afternoon or a research project?

The related discipline is to write down the reasons, not just the settings. Half the notes I keep are of the form “this looks wrong and must stay this way, here is what happens if you ‘fix’ it”. Those notes have saved me from myself more than once, because six months later I am a different person with less context and an appetite for tidying up.

The thread running through all of them

Every one of these is a failure that was invisible rather than loud. The timer that stopped, the counter that lied, the check that rejected correct output, the gate that gave up while everyone slept — none of them raised an error. Some of them looked like success.

If you take one thing: when you build something that runs unattended, spend your effort on making failure visible, not on making it unlikely. You will not prevent it. You can absolutely arrange to find out.