Auditing a homelab after it breaks: what the documentation got wrong

Something in my homelab fell over. I got it back, and then I did the thing I would have told anyone else to do first: I went through the machines properly and compared what was actually running to what my notes said was running.

Six things were wrong. What struck me afterwards was not that there were six — it was that all six were wrong in the same direction. Every one of them was a case of the documentation faithfully recording what I had intended, and never being corrected when reality went somewhere else.

That pattern is more useful than any individual finding, so this is mostly about the pattern. The findings are here because specifics are what make a pattern believable.

What actually broke was boring

The most plausible cause of the outage was that the hypervisor’s root filesystem was completely full. 114 MB free on a 39 GB volume. Nothing interesting — but a full root on a hypervisor breaks the cluster filesystem, blocks guests from starting, and stops logging, which is why the symptoms looked far more dramatic than the cause.

Roughly 30 of the 37 GB in use was cache, stale archives, or an application that had no business living on the host’s root volume at all. Two package-manager cache directories alone accounted for about 11 GB, and clearing them lost me nothing that could not be re-downloaded. The volume went from 100% full to about 70% in under a minute.

So the failure that cost me an outage and a genuine fright was a housekeeping job I had never scheduled. I find that more instructive than a dying disk would have been, because a dying disk is bad luck and this was just absence of attention.

The docs described the plan, not the machine

The clearest example: every document I had described a particular GPU workload as running in its own virtual machine, with the card passed through to it. It does not. It runs directly on the hypervisor, with the card attached to the host.

Here is the interesting part. The endpoint was documented correctly — the address and port that other things connect to were right everywhere, and so was the code that used them. Only the prose describing what sat on the other end was wrong.

That is not a coincidence, and once you notice it you see it everywhere. Anything that gets executed stays true, because when it drifts something breaks and you fix it. Prose has no such feedback loop. It was correct on the day it was written, it was never executed again, and nothing in the system had any way of noticing it had quietly become false.

It mattered for reasons beyond tidiness. Because I believed that workload sat in a VM, I believed it was inside the VM backup scheme. It was not. And a failure of that service was never an isolated service failure — it was a host failure that took everything else down with it, which is roughly what happened.

Finished projects leave their instructions lying around

Several documents described an in-progress migration away from an older server, with that older server repeatedly marked “sacred — do not modify”. The migration had finished months earlier. The server no longer existed at all.

Nothing about those files announced that they were history. They read exactly like current operating instructions, in the confident imperative voice you use when writing instructions, and they sat in the same directories as the documents that were still true.

Stale documentation is worse than none, for the same reason a wrong map is worse than no map: it stops you looking. Anything superseded now gets moved and banner-marked. And every document carries the date it describes rather than just the date it was edited — “as found, 30 July” is a claim I can go and check, where “how it works” is a claim that expires without telling anyone.

“I have backups” was a belief, not a fact

There was no backup job. Not a failing one — the configuration file that would define scheduled backups did not exist at all.

There were two archives sitting on the host, which is presumably why I had never worried about it. Both were four months old. Neither was the machine that mattered: one was an unrelated guest, and the other turned out to be a leftover from a previous occupant of the same VM ID, from a build I had replaced long ago.

So everything the automation depended on — the code, the accumulated assets, the database, the authentication tokens that would each need re-issuing by hand — existed in exactly one place, on one virtual machine, on one physical machine that had just demonstrated it could fall over.

The fix was ordinary: nightly snapshots of every guest to network storage, seven days retained. What I could not have known without measuring is that it costs about 47 GB and eleven and a half minutes a night, in snapshot mode, with nothing taken offline. That number is the whole point. Before I measured it, “set up proper backups” was an unbounded task I could keep deferring. Once it was eleven minutes at three in the morning, there was nothing left to defer.

Measure the medium before you trust the tool

While moving a large archive to the network storage, the tool I reached for by habit managed 8-11 MB/s and projected somewhere between thirteen and twenty hours. A plain sequential write to the same share measured just under 80 MB/s, and the backup job itself achieved about 100.

I never established exactly why, and for this job I did not need to. The useful conclusion was that my habitual tool was running eight times slower than the medium was capable of, and that a single sequential stream was what this hardware was actually good at. Ten minutes of measurement replaced a night of waiting.

A related discovery in the same archive: a live 32 GB swapfile inside a directory I was about to copy wholesale. Copying a swapfile in use is both pointless and unsafe, and it had been inflating my estimate of how much real data was there by about 7%.

Auditing the secrets is how I exposed the secrets

Five older files contained live credentials in plain text. Passwords, an API token, the lot. The newer notes were clean and had an explicit rule against it which had been followed — the older ones predated the rule, and nobody had gone back.

Two things came out of that worth passing on.

The first is that two of those files held different versions of what should have been the same token. At least one was a mistyped copy that had presumably never worked, and nobody had ever noticed, because nothing used that copy. A secret duplicated across five files is not one secret in five places. It is five secrets that diverge in silence.

The second is more uncomfortable. In the course of reviewing those files during the audit, I read them into an AI chat session. Which means the audit that found the exposed credentials also exposed them further, and both had to be treated as compromised and rotated. The lesson is not “do not use the tools”. It is that reading a file counts as handling its contents, and I had not thought of an audit as an activity with a blast radius.

What I actually changed

Backups exist and have been verified. The rebuild instructions now live somewhere other than the machine they describe, with a copy off-site, because documentation kept only on the server is a recovery plan that depends on the server. Secrets are out of the documents. Anything superseded is marked as history rather than left looking current.

But the change that will matter most is the smallest. I now record when a document was last checked against reality, not when it was last edited. Those are different dates, and only one of them tells you anything.

The audit was uncomfortable in the way audits are — there is a particular feeling that arrives with discovering your only backups are four months old and of the wrong machine. It also took an afternoon. Given the alternative was finding all of this out during the next failure instead of after this one, it was the cheapest afternoon I have spent on any of it.