You are coding away and suddenly realize that there is a vital piece of new data you need to propagate through the battlefield of your game (or application) for your change.  Your heart sinks… you are going to have to take drastic action.  It will be tricky to safely create and navigate that vital piece of data through the lines.

A complex web of method signatures fill the field like barbed wire between your data and its goal – painful for both friend and foe.  Tracing back from your goal, you come across comments from generals that have trodden a similar path in the past and shudder at their remarks.  Some of them had to do some rather unsavory things, some warn against the consequences of using exactly the method you were thinking of.

You wipe sweat from your brow; is it really worth it to try and guide this plucky new data across such a hideous mine field?  A message from the tower tells you that you have no choice.  Reluctantly you make a plan for the fresh data recruit to cut through the battlefield at a certain point.

Your data makes it through and helps fortify the frontline – it all looks good.  Only three months later does a subtle change you made show up as having a hideous side-effect on the entire outcome of the war.  A halfling managed to make it behind the lines and is causing all kinds of problems.   But your plan was reviewed by one or more generals!  No one saw anything wrong with it!  Until that day when it *all* went wrong and now the Eye of Sauron is focused squarely on you…

If only there was an alarm hooked up… a simple safe guard to indicate there was something amiss. Find a way to test, this story need not be about you!

Allow time for tests

Managers and leads, please allow time for tests in schedules.  Programmers, include extra slack in estimates for writing them.   There are many reasons why – but it all comes back to your definition of done.  Is “it works for me” good enough?  Is it enough to set things up and assume everything will go to plan?  When is something really signed off?

Scott Evil: Wait aren’t you even going to watch them?  They could get away!

Dr. Evil: No, no, no; I’m going to leave them alone and not actually witness them dying.  I’m just going to assume it all went to plan, what?

… and Austin Powers escapes.

Benefits of automated testing

For the coder it is like cover fire.  Every test is another sniper covering some small part of the code for you.  If something flashes up in their scope they can immediately warn of some nasty bug lurking.  That regressive bug never gets a chance to get a foothold.  Each extra test watches out over your software at a different angle and cumulatively they become a very useful force.

A well written test reviews your code – not on how you wrote something like a human reviewer might – just against an expected result.  A rational coder can not argue their code is decent if it fails a good test.

Tests form a kind of living functional documentation that any written form can never hope to compete with.  If a test becomes obsolete it is changed to fit a new requirement or even removed entirely.  Anyone that can write code can understand and write the tests.

ahem Alex Evans :-) ).  As long as the new code satisfies the integration tests and functionally achieves the same thing – have at it.  But before rewriting create some tests to measure performance, then after the rewrite the new work can have proven value.  If another change in the future reduces performance an alarm bell will go off.

Code passing a good set of tests in a branch can give any coder the confidence that a clean get from source control will result in a successful build of useful software.  That could be a pivotal moment for getting together something presentable for that last minute demo.

If you cannot test something, then it is unlikely that the code in question is fully understood.

But I can’t test

I can’t test because I don’t know where to start… there is just so much already there!

  • Some tests are better than none.
  • Try “Tracer Bullets”[2] to shoot down bugs and introduce tests to your game / application:
    • Create a test that tests the bug and fails.
    • Fix the bug, make sure the test now passes.
    • Commit and resolve.
    • Now if anyone regresses the behavior, your test will catch that right away.
    • Test coverage is cumulative – more bugs fixed in this way increase coverage at valuable pain points.

I can’t test because I don’t know what I expect yet.  I’m still figuring that out.

  • Prototype to investigate the problem domain.
  • It would be helpful to set a time box around the research period (a “spike” [4]).
    • Setting a time limit will allow you to :
      • Step back before going too deep down a rabbit hole.
      • Concentrate on estimating the size of the problem domain.
  • Concentrate on figuring out what to test – the tests are likely to be usable even if the prototype code is not.

I can’t test it because there is no one expected state.

  • Just think creatively while writing the test.  For example :
    • If a test is for randomness, then obviously there is no one predictable state.  But there can be a fitness function that defines how random something should be.
    • If looking for problems in a scene screenshot, perhaps a test could ensure a reserved color used for missing textures is not present.

I can’t test it because I cannot isolate the behavior to test

  • This could indicate:
    • Possibly blurring of responsibility with highly dependent and coupled code (perhaps too general).
    • Code could be reaching through objects, violating the principle of least knowledge.
    • Architecturally a lot of cruft is required to access useful behavior.
    • Code has been added to workaround a prior issue rather than refactored for a new requirement, adding to the cruft.
  • Rework to be testable if you can, try to isolate and retire if you cannot.Rework to be testable if you can, try to isolate and retire if you cannot.

I can’t do all these tests, it would increase build time

  • Keep tests in separate project or solution, you can run them only before committing to source control.
  • Ensure each test does not take long to run, mock where necessary.
  • Partition tests that must run for a long time.  Perhaps only run them on the build server.
  • Partition tests per module so that only particular module and integration tests are executed.   This gives the coder the choice to only run the tests they need.

I can’t write tests, I just do not have the time

  • This one is the hardest thing to overcome if there is no management buy-in to testing.  As a programmer at any level try your best to campaign for more scheduled time.
  • Writing tests gets quicker as a programmer gains experience (just arrange, act and assert one thing in each test).
  • Once tests start proving there are fewer regressions being reported in the bug database, it is an easier sell.  Gather any time saving metrics you can.

Summary

To many game developers I am sure tests don’t seem very rock & roll… not something they may have considered to be part of their role in the industry.  So I expect many do not test their work.  Writing elegant tests can present interesting engineering challenges and can be a good way to understand something that is new to you.  There is satisfaction to be had when all your tests pass after a big change, or when you realise a test saved you from an obscure bug or bad design choice.

Testing is no silver bullet, but try to think of it as an investment against that same bug being reopened many times later on.  Wouldn’t you rather spend time on polishing something else then finding and squashing the same bug again?

Further Reading

Paul’s personal blog.

Clinton Keith’s book is written for game developers that want to think about different ways of prioritizing and scheduling tasks during game development.  It includes many reasons for testing and many interesting stories from the front line.  (For the purposes of disclosure he was also kind enough to acknowledge me for helping him out a little bit).

The other books listed are not specifically about game development but software excellence in general.

twitter.  Everything in this article is Paul’s opinion alone and does not necessarily reflect his employers views. Copyright ©2011, Paul Evans.