367 tests passed on a blank screen, so we built an agent that doesn't trust tests
Our build pipeline had five agents and a perfect record of green checkmarks. Then a test suite passed on a modal nobody could see. Vex is the sixth agent, and its only job is to disbelieve the other five.
A while ago I shipped a blog modal on this site. Click a card, a dialog opens with the full post. I had written a test suite for it: 367 assertions covering the title, the body text, the close button, the keyboard handling, every tag. All 367 passed.
The modal was completely blank.
Two bugs, stacked. A rule that hid the card when the modal opened was also hiding the copy of the card inside the modal, because the copy carried the same class. And a global margin reset was beating the browser's own rule for centring a dialog, so the thing sat 270 pixels off to one side. Either one alone would have been obvious. Together they produced an empty box in the corner of the screen.
The tests passed because text inside a hidden element is still readable to a test. Every assertion asked "is this text present?" and the honest answer was yes. Present, and invisible. The suite was measuring the wrong property with total confidence.
That was the week I stopped treating a passing test as evidence.
The gap nobody owns
Our build pipeline runs as a relay. Storm takes a half-formed idea and argues it into a scored plan. Rune turns the plan into a spec precise enough to execute. Forge builds the thing. Echo writes down what happened so the next project starts from what we learned.
It is a good chain, and it had one structural hole: the only thing checking Forge's work was Forge. The agent that wrote the code also wrote the tests for the code, then reported that the tests passed. That is not a lie, and it is not even carelessness. It is just the oldest problem in software, which is that the person who wrote something is the worst possible person to find what is missing from it. They test what they thought about. The bug lives in what they did not.
So we added a sixth member who arrives after Forge and before Echo, and whose entire remit is adversarial. Vex does not build anything. It does not improve anything. It tries to break what was just built, and it reports what it found.
Four passes, and a rule about findings
Vex runs the same four passes every time, in the same order:
- Functional. Does it do the thing. Not does it render, not does it compile. Does the user get the outcome.
- Quality. Is it sound, or is it held together by something that will come apart on the second edit.
- Security. A pass against the OWASP Top 10, which matters more than it sounds for internal tools, because internal tools are where people stop checking.
- Spec match. Does it do what Rune actually wrote down, or does it do an adjacent thing that feels close enough.
One rule governs the output: every finding names a file and a line. No "consider improving error handling". No "this could be more robust". If Vex cannot point at the exact place, it does not get to raise it. That rule exists because vague findings are free to generate and expensive to action, and an agent that produces a hundred of them per run is worse than no agent at all.
It cuts the other way too. When Vex clears something, it says so explicitly. "Checked, clear" is a finding. Silence is not.
What it has actually caught
Two from the last fortnight, both from work on our own tools.
The first: a display component sized itself from a hand-written list of names, while a separate configuration file also declared sizes. Two sources of truth for one number. Anything not named in the list quietly fell through to a default. It rendered. It looked fine. It was simply the wrong size, and the configuration file that appeared to control it had no effect whatsoever. Nothing failed, so nothing would have surfaced it.
The second is the one I think about more. We had added a check to confirm that an animation stops for people who have asked their system to reduce motion, which matters because the animation in question is flashing light. The check passed. Then we broke the code on purpose to confirm the check would catch it, which is a habit we picked up the hard way.
It still passed.
The check was looking for a class name somewhere in a block of styles. The name was still there, in a different rule a few lines down that did something else entirely. The test was satisfied by a coincidence. We had written a test about accessibility that would have let a seizure risk through, and we only found out because we tried to break it.
Why it is summoned, not always on
The obvious move would be to run Vex on everything automatically. We deliberately did not.
An adversary that runs constantly becomes background noise within about a week. You start skimming its output. Then you start closing it. A gate you have learned to ignore is worse than no gate, because it still costs time and now it also provides false comfort.
So Vex is called. Someone decides a piece of work is ready to be gated, and summons it. That decision is the point. It means every Vex run has a person behind it who intends to act on what comes back.
What changed in practice
Less than you might expect, and in a direction I did not predict. We do not ship noticeably slower. What changed is that the phrase "the tests pass" stopped being the end of a conversation.
The habit that came out of it is now standard on our builds: when you write a check, break the code on purpose and confirm that check goes red. A test you have never seen fail is a test you have no evidence about. Most of the bad tests we have found were not wrong about the code. They were asking a question whose answer was yes no matter what the code did.
That is the whole argument for a sixth agent. Five agents building and checking their own work will produce green checkmarks. Whether the thing works is a different question, and somebody has to be in the room whose job is to ask it.
Vex is that somebody. It is not pleasant to work with. That is the feature.