An Alarm That Always Rings Is Worse Than None
When the red light keeps being someone else's fault, people stop reading it. Another site's uptime and an exception I never declared did that to two CIs.
CI is the thing that asks, every time the code changes, whether it is safe to ship. It is worth having on one condition: when the light goes red, a person stops and reads it. Once that condition breaks, the CI can stay in place and protect nothing at all.
I recently found that the CI in two products had reached exactly that state. In DUGOUT 9, an independent fan project for Korean professional baseball, the cause was somebody else's server. In somesignal, a service that reads the relationship between two people, the cause was an exception I had made on purpose. Different causes, one result: red became normal, and nobody read red anymore.
A red light that isn't your fault
DUGOUT 9 has a browser audit, a check that opens real pages and confirms the game reports render correctly. The trouble was that on every pull request it fetched the league's public website for real. A pull request that changed one word could go red because that site was slow that day, and a single job ran past ten minutes.
For a long time I coped by raising timeouts. That reduces the symptom, but it teaches something worse. When the light goes red, the first thought becomes "the other site again". After that thought is right a few times, it gets applied on the day your own code really is broken. A test that depends on someone else's uptime ends up ignored.
So I took one report produced by the real pipeline and froze it as a fixture. Not as-is, though: player names, headline text and links were replaced with synthetic values. The product has a rule against copying the source material into the repository, and a fixture is not outside that rule. The audit dropped to around three minutes, and the same code now always gives the same result. Cutting the external dependency was the easy part. Checking that the way I cut it did not break a principle was the actual work.
An exception the tool was never told about
In somesignal the cause was me. When I retired a feature, I deleted its code but kept its database tables and columns, so as not to throw away records that belonged to other people. I wrote about that choice in Retiring a Feature Is Three Decisions, Not One. This is the part of the story that piece left out.
The schema check compares the code against the database. To the tool, a table the code does not know about is a difference to delete. It reported my deliberate mismatch as "drop this" on every run, and the CI stayed red for months. Once red was a constant, nobody read it. Meanwhile two analytics events sent by the app were being rejected by the server, and for months nobody noticed. The real problem was sitting under a red light that was already on.
Write the exception down instead of hiding it
The shortest fix was to turn the check off. One line. But then the next person would never see a genuine schema accident. Loosening the check only delays the same problem.
Instead I wrote the names of the abandoned schema into a single file and excluded only those names from the comparison. Then I pinned that list with a test:
- Growing the list means changing the test too.
- Putting a live table on the list fails the test immediately.
- What the exceptions are is visible by opening one file.
Keep the exception, make the list of exceptions visible, and make it slightly tedious to grow. If a validation tool is never told about an intended exception, it reports it as a problem on every run, and people soon learn to ignore the whole report. Telling the tool did not weaken the check. It made the check worth reading again.
An alarm is only worth what gets read
Side by side, the two cases have the same shape. When the light keeps going red for reasons unrelated to your code, people sever the link between red and their code. In one case another server cut that link; in the other, a decision I never told the tool about. An alarm that always rings is worse than none. With no alarm, you might at least build one. With one ringing, you believe you are covered.
So now I treat a CI that stays red past a day as a problem in its own right, whether the fix is the cause of the red or an exception the tool needs to be told about. The count of checks I have matters less than whether I still stop when one of them turns red.