Testing!

A challenge that lies ahead for me is testing the game I’m building. Unlike previous games I’ve worked on I’m working on my own with a minimal budget, so I have no QA department at my disposal with an army of young, keen, cunning testers doing their worst to try to break the game for months on end. There are “for hire” QA teams, but that is far beyond my budget. Therefore I need to do something else.

Beta Testing

Part of the solution is to give close friends a beta copy of the game and let them play it, noting their feedback which they provide in return for a free copy of the final game. I expect that this type of testing will be most useful for gameplay feedback – finding out whether the game is actually fun, whether players get lost in my levels and so on. Of course, obvious crash bugs and other functional things will be found, but I wouldn’t expect any beta testers to play through daily builds of the game, smoke-testing all game assets, getting full coverage on all possible hardware or spending hours trying to find an easy way to reproduce intermittent bugs. For this I need something else.

Automated Testing

It seems that my only realistic option is automated testing. Bear in mind, I’m not talking about Test Driven Development, Unit Testing, and other source code based testing methodologies – though of course they do contribute towards the overall testing and QA of the game. I’m talking about automated testing of builds of the whole game.

In my experience so far, there has been little automated testing of the games I’ve worked on. Perhaps there would be a hastily written script which loads every level, every character, dynamic object, and so on, checking for crashes before QA get the build, but I’ve never seen anything very much more sophisticated.

I think the main reason for this lack of automated testing is that in the large companies I’ve worked for, there’s always a QA department in the building that can do the testing manually – from a producer’s point of view, adding automated testing is extra upfront work that takes time away from making the game. However, I think this sort of testing is worth the effort, even for these large companies. After all, even the most diligent human tester can miss things – on many an occasion I’ve received bug reports that turn out to have been unnoticed in the game for months and thought “How could they have missed that?”. The simple answer is usually that it wasn’t on the test plan and was probably found by a lucky accident. This is a problem that also afflicts automated testing – it can only test what you tell it to test, far more so than human testers. I’m not sure there is a good answer to this short of a Culture Mind.

What I’d Like To Have

What I would like is a framework to perform the following tests on every build of the game and report results for every test, adding failures into a decent bug database. It would also be good to allow easy promotion of an intended fail (e.g. a change in a graphical effect) to become the new baseline for a pass. This is not an exhaustive list. I’m sure I could come up with more things to test – suggestions are welcome in the comments!

  1. Smoke-testing. Loading every level, character, model, texture that the game uses to check they all load without crashing.
  2. Render and animate every level, character, model in the game, comparing results with screenshots from previous builds.
  3. Run a recorded playthrough of the entire game. I was thinking something like the old playthrough that went on behind the main menu in Quake. Of course this wouldn’t suit all game types, though perhaps could be done in a limited fashion for those. This needs to test for crashes, dumping a callstack, game inputs and internal state, and other appropriate data for a programmer to look at.
  4. To find performance hotspots (ideally from the playthrough data) and take PIX/GPA/<insert performance tool> grabs. Comparisons with previous builds are essential here as we need to be able to find when and where performance problems occur.
  5. To soak test every level for a given number of hours.
  6. To test all UI screens, and check that options actually give the correct results from rendering/audio/difficulty/etc (as applicable).
  7. To test rendering at all available resolutions (if appropriate).
  8. The ability to run any subset of these tests on demand (rather than overnight).
  9. The ability to add new tests quickly and easily.

I don’t envisage anything being able to do all of this straight out of the box, but a framework I could gradually add to would be ideal – even better if I can keep as much of its code separate from the game code as possible to avoid further bugs.

If you, dear reader, know of such a framework that can be set up to do this sort of thing I’d be eternally grateful if you could let me know in the comments. I’ve heard of companies having this sort of thing in their tool-sets, but the solutions tend to be proprietary and/or expensive. It’s most probably infeasible to write something myself that would do all of the above without taking longer to implement than the game itself.