Step27

The serious part - determinism, tests, bots and friends

29 October 2026 · 4 min read

vibe-codingtestingdeterminismplaytestshipping

A top-down view of a green island with a white-capped mountain, a river and a lagoon, with small bases and ships offshore, from an automated AI-versus-AI test match.

There is a version of vibe coding that is all vibes: type a wish, accept what comes, ship it. We wanted a real-time strategy game that two people can play online, that holds up for an hour and that is worth a stranger's time. That needs the opposite of vibes at the edges. Claude writes most of the code. Everything that decides whether the code is allowed to stay is measurement.

This step is the safety net, in four layers.

Layer 1: the fingerprint

In a lockstep game only the players' orders cross the internet, so every PC must compute exactly the same result. We prove that with a 64-bit hash of the whole simulation, taken on every tick of a test run and compared between runs (the long version is in Two PCs, one simulation).

On 24 October, after the Big Battle work, the ordinary skirmish fingerprint matched the baseline of 22 October on every tick sampled, for example 6aabd3efa96bc394 at tick 0 and 1965899024e3d021 at tick 8014. That is the line that lets us say a big new mode did not break the old ones. Without it, we would be hoping.

Layer 2: the suite

test.ps1 runs the automation tests headless. It has 71 tests and all of them pass as of the Big Battle commit. They cover rules, definitions, rigs, win and lose conditions, a stuck-unit check and the new mode's settings, planner and battle code. A change that breaks one fails loudly, whoever or whatever made it. Claude is allowed to be wrong, because the suite is allowed to say so.

Layer 3: bots as testers

A person cannot play 50 Tower Defense levels a hundred times. Three bots of different skill can. They taught us that the wave generator we wrote on day one was a first guess, and we tuned until the middle bot won 36 of 50 levels, with the rest listed by the wave they die on (the bots step). A bot also shows what it cannot tell you: it does not know if the game is fun. A packaged exe, built for friends, beat three levels by bot before it was zipped. The log's last line is the one we like: not verified: a played session; real two-PC Steam multiplayer.

The same idea runs missions: every campaign mission has a scripted win route, a lose route and a checkpoint on Easy, and a script plays them.

Layer 4: friends on real PCs

Bots play perfectly legally on the machine they were built on. People have other machines. We have shipped ten friend builds, and the first real online match between two PCs on 25 September stayed in sync for 4,080 checks.

The most useful report so far came on 24 October. A friend running an i7-12700K and an RTX 3070 (all three of the test PCs in that round were the same) sent a screenshot of the first Tower Defense level: the water was black, with white specks. They also had a texture-loading problem that survived a restart.

That is a bug no test in our suite could find, because it depends on the graphics card, the driver or an effect that behaves differently on another machine. It went into the queue as an engine bug with the screenshot attached. When this step was written it was still listed as under investigation, and we do not know yet whether it is the card, the driver or our own water. We will say when we do.

The same round brought notes that the camera needed rotation, that buildings should rotate when placed, that WASD should move the camera, and that on the second campaign mission clicking the water under the bridge to place something also destroys the bridge by accident. None of those would come from a bot. They came from a person trying to play.

The Tower Defense interface over brick apartment blocks in falling snow — A Tower Defense level in snow. One level of fifty, and the one where a friend's test PC showed black water.

Where the human sits

The owner plays every build and gives the verdict on models, balance, story and look. Friends play the build. Scripts and bots check what can be checked. Claude writes, runs the checks and reads their output. If you took the person out, the game would still compile and still pass all 71 tests. It would not be any good.

What is not proved

We keep this list because it is part of being serious:

A real two-PC match through Steam from the packaged build.
A long soak: eight AIs on the biggest map for an hour of simulated time.
Medium, Hard and Extreme balance in the campaign.
Live performance at 600 or more units in the Big Battle mode.
Anyone outside the project having fun, apart from a handful of friends.

Try it yourself

Ship your safety net before your features. Make one run reproducible, hash its output, and compare two runs. Then give it to someone with a different computer, and ask for a screenshot.

Next: coming soon. Subscribe by RSS to catch it.

Start at Step 1 All 27 steps