Three Weeks and a Check Mark

The backup repository held exactly one snapshot, tagged as the initial one and three weeks old, while the scheduled job meant to refresh it sat on the timer list looking perfectly healthy. The listing said the copy existed. Only opening the repository showed that the newest thing in it predated everything we had changed since.

We had been moving a repository, which is a folder where a project keeps its working files. We expected ordinary cleanup: move the files, update the old address, and make sure the project still worked.

Then we found the backup, or rather the most recent copy of it, and it was far older than the schedule called for. The entry that was meant to make the backup still existed. It even looked set up correctly. But the thing it was supposed to produce, a recent copy that could be used to recover work, was not there.

A checked box can tell you that a task is listed. It cannot tell you that the task did anything useful.

The first pass was wrong

We started with the simple version of the move and treated it as a search problem. Find every file that mentioned the old location, change those references, move on. The written plan we were working from said the same: patch the files inside the folder, dump the database, rename.

That did not work, because not everything connected to a project lives inside the project folder. The plan never mentioned the service files outside it, and we only found them when we checked the machine instead of the plan. Those pieces had been set up little by little over time and were easy to forget.

The wrong first pass cost us time. A straightforward cleanup became a longer walk through everything around the project: timed jobs, service settings, old pieces of software, deployment settings and other old references. Old software and old instructions can keep pointing at a place you thought you had left behind.

Making the list

Before changing anything else, we made a list. What was scheduled to run? What was turned on? What was still connected to the old location? What was supposed to come out the other end?

There were retired bits of software with no clear purpose. There were scheduled jobs that still pointed at the old place. Some had not failed in any way anyone would notice, because they had not yet been asked to start again.

Something can be marked active and still be waiting for the moment it breaks. A light switch can be in the on position even when the bulb has burned out. Until someone needs the light, the switch looks reassuring.

The backup was the most serious example, but a second timed job had the same problem. It kept being called on schedule while something it needed was unavailable. Instead of producing the expected result, it wrote repeated errors to a log that nobody was alerted to or checking regularly. The system was going through the motions of the job without delivering the result of the job.

A scheduler is simply a clock that tells a task when to run. A clock can ring on time and still fail to get the laundry washed. What mattered was whether there was a clean basket at the end.

What we had assumed

We had assumed a recent recovery copy existed. The plan listed the backup job as something to carry over with its paths updated, on the grounds that backups always matter, and we had not looked at what it produced. When we finally did, we found the single snapshot from three weeks earlier. A backup is useful when it is recent, complete, and able to help after something goes wrong. Without checking the copy itself, we were trusting a promise made by a setting.

The move turned out to be useful for a reason we had not expected. Changing the ground under a project forced us to look at everything standing on it. We had to trace the old location through every connection we could find. That is slower than changing a few lines of text, but it is a rare chance to notice what has been aging in a corner.

What counts as working

A prettier status page would not have told us anything new. The question that did was what success would look like if we had to use it today. For the backup, that meant a recent copy and a controlled restore test, which means using the copy in a safe setting to see whether it can bring work back. Until we ran a restore from that copy and saw our own recent work come back, the status of that backup was not “active.” It was “unproven.”

For the other timed work, it meant checking for the thing each job was meant to produce, and ignoring whether it had merely been invoked.

The list also needed owners and a response plan. Someone needs to know what to check, how often to check it, and what to do when the result is missing. It also needs careful handling. An inventory is useful, but it is no reason to copy passwords or identity files into an unprotected document.

We began the move thinking we were relocating a folder. We ended it holding a small pile of old promises: jobs that were still listed, services that were still enabled, and a backup that had not been doing its one important job for three weeks.

If you run a scheduled backup, open the backup itself, check that the newest copy is recent, and run a restore test in a safe place. Give every other scheduled job an owner who knows what result to look for.