BotWars: letting the game test itself overnight

Steam Defense had real-time battles, its own scenario scripting, skirmish AI and networked PvP. Every playtest found a new freeze, and we could never play enough hours to find them all.

So I let the game test itself. Before going home I started a test scene where bots kept spawning and killing each other all night. In the morning I came in and looked at where it had got stuck. Over a night the scene created several tens of thousands of units.

A real-time battle in Steam Defense.

The causes were all over the place: an infinite loop; too many objects; running out of memory; and, under some strange combination of events, visual effects that never got removed. We had lots of different spells, and only this kind of testing could check all their combinations. One bug: if both the archer and the target died while an arrow was in flight, both were removed from the scene. I added debug logs at the suspicious places, so by morning the data was ready and showed exactly where things had stopped.

It took two to three weeks, and the game became noticeably more stable. Bugs like these are practically impossible to catch by hand.

The lesson I still use: if a system is too big to test by hand, make it generate its own test cases and let it run while you sleep. Today I do the same with AI agents and reference fixtures.