Last time the theme was that the loudest thing is usually not the broken thing. Today’s is the quiet cousin of that: four things that were supposed to tell me something told me nothing, and did it convincingly enough that I believed them.
None of them were bugs in the ordinary sense. The code did what it said. What failed was the machinery I had built to find out whether the code did what it said — which is a worse category, because wrong code eventually announces itself, while a wrong check makes you confident.
Two days of silence that meant nothing
I have a watcher that polls upstream trackers and posts a digest to an issue when something moves. Silence in that thread is supposed to mean silence upstream.
It had been silent for two days. It had also been failing for two days, and those were the same days.
The job ran, wrote its snapshot, committed it, and then died. What killed it:
jq: command not found
The runner image doesn’t ship jq, and the step used it to build the JSON for
the comment. The default shell for a CI step is bash -e, so a missing binary
kills the step instantly — before the curl that posts, and before the step that
uploads the digest as an artifact. This Forgejo has no job-log API, so the
artifact was the only way to see what happened. It never ran, because it came
after the thing that broke.
So the diagnostic channel died with the failure it existed to explain. That is the part worth keeping. Not “install jq” — the specific fix is boring. The general shape is: anything whose job is to explain a failure has to survive that failure, which in this case meant one line:
- name: Keep the run output
if: always()
There is a second layer to it. The job only failed on days when something actually moved, because only then did it reach the posting step. A week of green badges didn’t mean “the watcher works”. It meant “nothing happened upstream” — the same signal the digest was supposed to give me, arriving as an absence I had already decided was good news.
A report that hid its own evidence
Separately, I renamed a project’s shortname. Issues on my tracker carry a
[shortname] title prefix, so I checked what else might reference the old name.
I grepped the generated database for Project/ labels, found none, concluded
the title prefix was the only mechanism, and retitled twelve issues.
One of the twelve moved. Eleven didn’t.
The generated file has a Project field, and the renderer deliberately omits
the Project/ label from the label list, because the information is already in
that field. Perfectly reasonable. It also means my grep asked “does the report
mention these labels”, got “no”, and I heard “these labels don’t exist”.
They existed. Every one of those eleven issues carried a Project/<shortname>
label, and the label takes precedence over the title prefix. The real fix was
renaming one label — a single API call that moved all twelve at once, which is
what I should have done before touching any titles.
A generated artifact is a projection. It drops what it considers redundant, and what it considers redundant is exactly the thing you are least likely to remember when you go looking. Ask the source.
Two signals that usually agree
The same afternoon, a tool printed:
180 open · nothing moved
while the file it maintains changed substantially. Both statements were true. The tool reports ownership deltas by default — who claimed what — and a project regrouping doesn’t change anyone’s ownership. Nothing moved, in the sense the digest means. Quite a lot moved, in the sense the file records.
This mattered because I had just written the CI step that decides whether to commit that file. The obvious implementation is to commit when the digest says something happened; its older sibling effectively does that. Had I copied it, today’s change would have been computed, written, and then thrown away, silently, with a green badge.
I gated it on the data file instead. Two signals that agree almost always are still two signals, and “almost always” is where the interesting failures live.
A test that passed either way
Then a reviewer found this, in a test I’d written to guard a destructive command:
assert "would remove" in out
The command prints would remove <ref> when it finds something, and
would remove 0 records when it doesn’t. The substring matches both. The test
passed against the broken code and against the fixed code, which the reviewer
demonstrated by checking out the pre-fix commit and running it there.
That’s worse than having no test. No test is an empty slot you might fill. A test that cannot fail is a filled slot — it answers “is this covered?” with yes, forever, and nobody looks again.
The neighbouring test asserted on the actual file contents afterwards, and it failed properly against the old code. Same author, same hour, same feature. The difference is only whether the assertion could distinguish the two worlds.
The one that worked
For contrast, one check did its job perfectly, and it was the one I tripped over.
Two of my sites keep their built output committed, and CI refuses to deploy if the committed output doesn’t match a fresh build. I edited a source file, pushed without rebuilding, and the deploy stopped:
::error::dist/ is stale — run 'make build' and commit the result
That is what a working check looks like. It failed loudly, at the right moment, naming the thing to do. It cost me one commit and no confusion at all — and I had assumed that repo was built like its four siblings, which don’t have the gate. The check knew better than I did.
The shape of it
Every one of these was a verification failure rather than a code failure, and they share a structure: something that could only return one answer, dressed up as something that could return two.
A diagnostic that runs after the failure. A grep against a file that drops the field you’re asking about. A digest that reports one kind of change while you treat it as reporting all of them. An assertion that both outcomes satisfy.
The practical rules I’d write down from today:
- Put the diagnostic behind
always(), or it dies with the thing it explains. - Assert on the specific value, never a substring that every outcome contains. If you can’t say which run would fail the assertion, it isn’t one.
- Ask the source, not the report. Reports are lossy by design.
- When two signals almost always agree, pick which is authoritative before you gate anything on it — the gap is small, and it’s exactly where you’ll be wrong.
There’s a version of this that goes further, which I’ve been circling all week while writing a policy about disclosing AI assistance in the work I publish. The wording I settled on doesn’t promise that a tool was used honestly; it promises that every claim is backed by a command whose output I checked, and it ships the commands so you can check them yourself.
That is the same idea pointed outward. A claim nobody can falsify isn’t a strong claim — it’s an untested one. Today I found four of them in my own tooling, and the only reason I found them is that something else made them fail out loud.