Step26

What vibe coding gets wrong, and how we caught it

28 October 2026 · 4 min read

vibe-codingclaudefailuresprocesstesting

A naval hit seen from above, with large fireballs over dark water and a messy foam patch.

Plenty of writing about vibe coding are about what it can do. This one is the other list, kept in the same spirit as our honesty write-up: what went wrong on Brick & Mortar, in roughly the order we found it, and what caught each one. None of these ended the project. Several would have, had nothing been checking.

1. "Done" that was not

Agents report success fluently. Early on, our plan and handoff files filled with entries marked finished that nobody had proved. When the tracker role audited the queue on 14 October, 380 handoffs were open. After checking each against a commit or a dated decision, 197 were closed as done, 113 as superseded by a later decision, and 70 stayed open. Several commit messages in the history carry the word UNTESTED on purpose, so a reader cannot mistake a draft for a result.

What caught it: a rule that "done" needs a proof file, and a role whose only job is to find work marked done without one.

2. The line-ending commit

Our source files are mostly CRLF, some are LF. On 6 October a tool wrote two logs and a script out with the other ending. Nothing in the content changed, but the commit that put it right shows 8,280 lines changed in one file, 720 in another and 3,182 in a third: 6,091 insertions and 6,091 deletions, all whitespace. Two smaller "restore the line endings" commits followed on 6 and 7 October. The history is still readable, but a real change hidden in that noise would not be.

What caught it: the diff stat looked absurd. Now a patch script has to preserve the ending and the rulebook says so.

3. Overwritten work

Two kinds. A script written for one job reused the filename of another role's draft and replaced it; the handoff note records "no harm" and the two files were separated. And on 2 October a commit was needed to restore takes that had to be put back into the master cinematic timeline. Git is the undo button for an agent. Commit often, commit per finished row, and never let a role write over another's files.

What caught it: ownership lists, and commits small enough to restore from.

4. Stale bases

On 10 October a missile-trail commit was built on an old copy of two files and dropped some test hooks. It compiled. Nothing failed until a test needed them, and a later commit restored them.

What caught it: a test that used the hooks. Tests catch what compilers do not.

5. Notes that go out of date

The plan for Big Battle Mode says the suite is 66 of 66. The build state a few hours later says 71 of 71, because the new tests were added. Both were true when written. Our own notes also grew 148 per cent in two and a half days, so we made a role whose job is to make them smaller (see The documents ate the budget). A release audit found 28 hits of an old unit name still sitting in hint strings; the ids were right, the text was not.

What caught it: an audit script that greps for what must not ship, and a rule that notes name their date.

6. Determinism bugs

A lockstep game needs every PC to compute the same thing. When Claude audited the whole simulation, it found sixteen places where two PCs could disagree: civilians walking on the clock's random numbers, shells starting from a smoothed picture of a barrel, maths that rounds differently by CPU. It also found, much later, that our trailer set pieces replayed exactly at normal speed and not at five times speed, so we shoot every take at 1x.

What caught it: a 64-bit fingerprint of the whole game, compared tick by tick between two runs.

7. Confidence about things it cannot see

Claude cannot see the game unless you show it. It will say a model "matches the reference" or the sea "looks right". On day two every unit icon showed a turret sunk into its hull; the game itself was fine.

What caught it: a person looking, and later review boards with overlays and numbers.

Patterns across the list

Every failure was cheap to find and expensive to ignore. A diff stat, a test, a grep, a person looking.
Almost none came from code that was hard. They came from process: state, ownership, notes, endings.
We did not fix them by being careful. We fixed them by adding a check, then writing the check into the rulebook.

Try it yourself

Keep a file called FAILURES. When something goes wrong, write one line: what, how you found it, and which check would have found it sooner. After two weeks, build the three checks that come up most. That list is more useful than any prompt template.

Next: The serious part - determinism, tests, bots and friends (29 October 2026)

Start at Step 1 All 27 steps