A few articles here on AltDevBlog have mentioned focus testing, and it’s one of those things that development teams rarely embrace enough. The concept is simple enough: get someone from the outside world into your studio to play your game, watch them, take notes, give them tea and cake, get them to sign a non-disclosure act so they can’t talk about it on the web and then discuss what you’ve learnt. Action the key points, ignore the stupid ones, refine the game. Find another willing subject, repeat.
Naturally, it’s not that simple in practice. The biggest hurdle that a game faces during its development is that it’s not really ready for external testing to take place until very near the end, at which point it’s too late to be able to change much. Bugs get in the way and skew the testing, we already know that doesn’t work / animations / effects / UI / etc aren’t in yet / we don’t have any time to change anything anyway. The person you got in for testing was an idiot that didn’t know what a console was.
I used to believe all of that. If you still believe it, this post is here to tell you that you’re wrong. It’s never too early to focus test, and if the test is run properly it will be invaluable: it will make your game better and it will save you money.
The lesson
Rogue Trooper, though I didn’t realise it at the time. In fact, at that time, I fully believed the paragraph above: I’d made a number of games, I’d been a Lead Designer several times and I was confident in my skills and the team. I was, almost undeniably, a pain in the ass to work with – mostly because of my refusal to delegate. At no point was this truer that with the level design: having made countless levels for Doom, Quake and then all the games that I’d worked on I truly believed I knew everything there was to know about making a good level. I did the initial white boxes for almost every level in Rogue Trooper and, because of that, found it very difficult to relinquish control and look at them with the critical eyes that I should’ve been using as the Lead Designer.
Nowhere was this more evident than in the first few steps of the game. Here’s a (very rough) recreation of the initial layout of the start area to the game:
The design seems pretty simple and the expectation was that the tutorial would be overlaid onto this and players would do the following:
The path back to the start was put in purely as a method for getting players back to the start and forcing them to jump over it because I wanted to teach climb next. Jumping is a key mechanic in the game, not because it’s a platform game but because it’s useful for diving out of the way of enemy fire. I wanted to teach it to players early on.
At this point in development we had the basic feature set for the game in place and running on PS2 – we’d done a green light demo to get the main development funded and had most of Rogue’s fundamental moves worked out, and we had very basic AI. The game was a long way off being finished but you could get around, climb, use cover and shoot enemies. We had ways of putting debug text on screen: we could have created a rudimentary tutorial. Basically: we should have put the white boxes into focus testing.
We didn’t for the most of the reasons I mentioned in the second paragraph.
What we did do was get team members to play through the levels, and do fairly lengthy technical analysis on the layouts with the lead programmer to ensure he was happy that we weren’t going to blow the PS2 up when we started layering art on. Refinements were made and we carried on: an artist spent several weeks making the detailed geometry, textures and lighting. A designer spent several weeks polishing up the gameplay and animators started creating the cinematics required to tell the story. Lots of money got spent.
Once it was all polished, and the mechanics of the game had been refined, we finally entered focus testing. The result was, at the time, heart sinking. Instead of doing what the development team had been doing, and what I expected, most people instead did this:
And not just once. Not just twice. They just did this. Round and round. Some people got it, but not the majority.
Some of it was due to the tutorial triggers only firing once – if someone missed the instruction on how to jump the first time they never saw it again. So we changed that and put it back into focus testing. The results were a little better, but not much. One guy spent 15 minutes trying to work out how to get out of the trench. My reaction at the time was “you stupid idiot!!!!” (note: there may have been more swearing than this). Only later did I come to realise, and come to understand that, if a design can’t be understood, it is usually not the person at fault: it is the design that is wrong. (This is an important lesson: learn it well for it will make you a better designer.)
There were plenty of other issues with the tutorial as well, some which were quick fixes and some similar to to this that required changes to the art. Which then had knock-on effects to the animations and so on. If we’d tested the level when it was in the white box phase it would have taken 10 minutes to fix. By the time we came to fix it, it took well over a man-week.
Using focus testing right
So, with such an arsenal of excuses at our disposal, how do you go about using focus testing right? Well, the clue is in the name: concentrate on a single thing that you want to test. Is it a mechanic? A layout? A boss battle? Whatever it is, make sure that only the thing you want to test is exposed to the people you get involved. Take the testing of a mechanic for example: don’t create huge environments or spend time making something pretty, but instead concentrate on the core elements. Spend time working on the controls, any animations, sound and visual effects that are required. Look at how you’re teaching that mechanic – how is it going to be introduced to the player?
Looking back on the experience from Rogue Trooper, we were trying to teach the player how to jump before they could walk. Which in turn was systematic of our approach to making the game in the first place – we spent a lot of effort on lots of mechanics to give the player choices, then presented them with an action game. Gears of War came out around the same time and did two things – cover based shooting and active reload. Epic ensured that their key mechanics were honed to perfection and then built the game around them. Valve do the same with their games, and it’s also why Call of Duty has become so huge.
So when people ask you to keep adding more features, focus test them. Point out that less is more – time spent on polishing key mechanics will always produce a better game over one with many. Try to follow these simple steps when creating something new:
- Decide what you want testing.
- Create a prototype for that test.
- Allocate sufficient time to organising the test.
- Ensure the people that need to see the results are involved.
- Keep a track of all the problems raised.
- Respond to the key issues.
In the day-to-day deluge of tasks and information that flies around, it will seem hard to achieve this. It’s something that should really happen at the start of the project. Once a large team has come on to create content and flesh out the rest of the experience, it’s kind of too late. I know that’s an ideal world scenario that rarely happens, but when it does great games are made.