One of my least favourite things about programming is debugging reported bugs with little or no information. You know the ones I mean, ‘I was testing the game and went to make a cup of tea, when I came back it had crashed. Could not repeat’. During the final weeks these bugs seem to come in thick and fast and are hard to track down quickly.
When the testing department is on the other side of the world, getting more information is hard. Time differences mean a reply can take a whole day. The build is often stale by then, leaving the team unsure if it has been fixed or not.
For these situations dump files are your best friend. Most platforms have them these days and they are easy to use. That trickiest bit is setting up your build process. I will cover Windows and Xbox 360 builds in this article, mainly because they are the hardest to setup. I will also mention a bit about other platforms.
Building your project
For all platforms you must build the project with debug information enabled. For Windows/Xbox this is pretty simple. In the project properties set C/C++->General->Debug Information Format to either Program Database or Program Database for Edit and Continue. Also set Linker->Debugging->Generate Debug Info to Yes. For the others (Apple, Sony etc) you typically need to just use the –g flag.
While here you may as well generate a MAP file as well. This is in the Linker->Debugging window for Microsoft (-Wl,-Map,output.map for most other compilers).
Generating Dump Files
Most platforms these days have a dump feature. Some platforms do full dumps but they can be fairly large. 512MB can take a while to download, even with good internet connections. Fortunately, mini Dumps are often just as good and only a couple MB at most.
For the Windows apps I add this code to the WinMain function:
SetUnhandledExceptionFilter(ExceptionFilter);
Then I add this code somewhere in the project as well
LONG WINAPI ExceptionFilter (struct _EXCEPTION_POINTERS *pExceptionInfo)
{
HANDLE hFile = CreateFile(<FileNameNormallyWithDateTimeAndBuild>, GENERIC_WRITE, FILE_SHARE_WRITE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
_MINIDUMP_EXCEPTION_INFORMATION ExInfo;
ExInfo.ThreadId = ::GetCurrentThreadId();
ExInfo.ExceptionPointers = pExceptionInfo;
ExInfo.ClientPointers = NULL;
MiniDumpWriteDump(GetCurrentProcess(), GetCurrentProcessId(), hFile, MiniDumpNormal, &ExInfo, NULL, NULL);
CloseHandle(hFile);
return EXCEPTION_EXECUTE_HANDLER;
}
This code won’t fire when a debugger is attached but works perfectly on testers’ machines.
The only difference with a mini dump is that it only dumps the basic information. Registers, threads, stacks. Full dumps include the whole of the system memory as well. Handy in some situations but only when you suspect a memory problem. These are generally easier to track down with a custom allocator (see my other series here).
Get the testers to attach these dump files to their crash reports.
Backup the information
You need to keep a back up of your executable and PDB files. For Microsoft builds you really only need the PDB’s but the executable may come in handy. Do the same with the MAP files. Make this part of your build process. If you have a build machine this is the ideal place to do this. The space requirements are not massive and they don’t need to be stored on a fast machine.
The Symbol Server
A dump file in itself is pretty pointless to anyone that can’t read assembly. Unfortunately, the number of people on the dev team that can do this is normally few and far between these days. Even if you do understand assembly, I find a wall of code, stack addresses and raw values a bit pointless, when I can get better information with no extra work involved.
What I really want is for any one on the team to be able to do debug these files. Fortunately, Microsoft has a tool called the symbol server which can help. It is free and you can download it here:
\\Buildmachine\SymServer /t "<appname>" /v "<version string>"
This registers the information with the server and prepares it for Visual Studio.
Visual Studio Setup
You need to point Visual Studio to the server by going to Tools->Options->Debugging->Symbols. Add your network symbol server path to the .pdb locations box. If you don’t have it already I recommend adding