As one of the older guys in the studios I work in (This is for #AltDevBlogADay right? For grumpy old game devs?), I occasionally hear young programmers whinging about how hard debugging is. About about how their debugger doesn’t integrate quite the way they would prefer it to, about the complexity of multi-threaded debugging or about how the froth on their latte isn’t quite the right consistency. When I hear this type of conversation I saunter over with my strong black coffee, introduce myself, tell them to pull their pants up and then tell them about a job I once had and a debugging session I once survived.

 

This was in a time before I worked in the game industry, back in the early to mid ’90s. It was my first job where I was being actually paid to program – I’d been there for a couple of years writing code for mining and some visualisation applications for defence – you know, helping big organisations to destroy both the earth and the humans on it. It was fun stuff – I was working on Silicon Graphics machines and writing 3D graphics in OpenGL – it was as close to writing games as I could get without actually writing games.

Then I was moved onto an unusual new task, one that involved porting some PC code to a Silicon Graphics Indy. Now that in itself wasn’t that unusual. The unusual bit of this job was that it was code for a program that was to be run in an abattoir. It was designed to take snap shots of the halved carcasses of the cattle as they moved down the production line, hung by their hooves on a hook, and then analyse those snap shots and determine meat grades, highlight any abnormalities in the flesh (tumors, severe bruising or the like) and then file the images and associated data to be used for stored cataloging. The code was awful – written by physicists (I can say that, I used to be one. Physicists tend to believe that you don’t need to be taught to code, that you just need to transcribe the maths into code and it will all just work. Anyway, I digress…) The code was full of hardcoded floats, painfully copied out to far too many decimal places, and obscure mathematics written with complete disregard of the physical limitations of floating point numbers on that platform. It was impossible to debug, but I was assured that it was working.

 

 

The Indy that was to be used wasn’t hardy enough to be on the slaughter floor where the camera was (apparently they weren’t designed to be immersed in blood), so that was kept in a concrete room not too far away. This room had only one window, and just outside that window was where the newly sloughed cow skin was taken and boiled in preparation for tanning. The smell from that neighbouring room was unpleasant, to say the least. This concrete room with the computer in it was the room where I had the fragrant pleasure of programming from.A small podium was built on the slaughter floor, onto which the camera was mounted. On the side of the podium (accessible from the ground) was a small LCD display with a keypad on it. This display would show the image of the carcass when it was captured and then flash up analysed results as a text overlay as it was saved to disk in the other room. That LCD and keypad also acted as an interface to the application (connected to the Indy via 20 or 30 meters of cable) – I could tweak the program settings from there as the carcasses moved past.

Just behind the camera’s podium (to my right as I faced the LCD) was where the cows were sliced in twain by a large mustachioed guy with a chainsaw. Yes, there was splattering blood, grinding bones and lots of noise – but that wasn’t an issue. As the cow was cut in two, the insides of that cow *usually* ended up on a conveyor belt which shipped the steaming internals off to somewhere unknown. Occasionally, the stomach missed the belt or was split by the chainsaw – usually resulting in me having to leap to safety as the semi digested contents splashed to the ground about me. Or, I’d end up buying new shoes.

The workers there seemed to take great pleasure in making me as bloody and gore splattered as possible. As the carcasses rolled past on their hooks in front of the computer’s camera, there was a guy who would ensure that if the carcass was too damaged that it would be pushed off onto another track where it could be trimmed further (if possible) before returning it to the track with the other meat. This other track happened to pass through exactly where I would stand in front of the LCD, so if I wasn’t paying attention then I’d be hit by 300kg of still quivering steak (this only happened once, to the great amusement of those around me).

 

So, yes, this was an unusual work environment for a programmer. To make it even more of a challenge, I would have to start work at 4am with the rest of the guys on the slaughter floor.  I’d do my 8 hours there, checking readings, collecting data, doing a bit of coding in the stinky room, dodging carcasses and bursting stomachs on the slaughter floor and then drive the hour or so back to my office and code for a few more hours – 12+ hour days were fairly normal (and no, I didn’t get overtime).

I lost count of the number of weeks that I did this for. At some stage I was involved in a 5 car pile up on my way to the abattoir, with me being car number 5. I was fine, but my car and the $14,000 SGI Indy on the passenger seat didn’t fare so well. The Indy worked, but not for long – the mother board had cracked when the Indy hit the car floor. Luckily it was still under warranty and Silicon Graphics replaced it for us (don’t tell them it was due to the car crash – they might want their money back).

As the code matured I was needed less at the abattoir, so I could drop in at the start of the day and install a new build, then head back to my office to code more or work on other tasks. There was no internet connection to the abattoir, so the only way to change anything was to drive there with the code on a 3.5″ floppy. There were two technicians who were trained to start up the computers and set things running, but they weren’t programmers and had no understanding of the OS or anything other than what we’d trained them for.

One day I was at the office and received a call that the software was crashing consistently. I managed to duplicate the problem locally, and it was a quick and easy fix. But I didn’t want to do the 2 hour round trip drive to the abattoir to install it, so I figured that I could “Airport ’75″ the tech guys and talk them through making the code changes required, compile it and set it running. I mean, how hard could it be?

 

The first problem I had was that the mobile (cell) phone didn’t work in the stinky concrete room – there was no signal at all. So, I directed Technician ‘B’ (a guy with no computing skills) to be the phone guy and he had to stand at the doorway of the stinky room and relay my instructions to Technician ‘A’ who would type in what he was told. This started fine – we managed to get emacs up and running from a console window which was a great start.

Then the fun started.

“You need to press Control X Control F directory_name slash filename.cpp”

“Press what?”

“The control key – bottom left corner of the keyboard. Hold it down and then press the ‘x’ key then the ‘f’ key, then let go of the control key and type in the file name… What can you see on the screen now? Ok, now press Control Meta Cokebottle…”

…and so on…

 

This would have been hard enough at the best of times; it was like coding via Chinese Whispers – a noisy environment via a conduit who had no idea what I was talking about relaying my instructions to someone who had on the vaguest notion of what I was talking about. I had to direct the remote programmer to the correct point in the correct line in the correct file, make the changes required and then repeat until the code *should* have been fixed. Fortunately, it was an easy fix that required only a couple of changes – it only took us about half an hour to make the necessary code changes.

Compilation was easy, “Hold the Alt key and press Enter and that will build it. Wait until its stops printing and then read me what you see on screen”. I was hoping for success first time.

“What all of it?” came the delayed relayed reply.

“Shit”

I spent the next hour deciphering the relayed compilation errors, reverse engineering them to try and figure what could possibly be the problem in the first place and then directing the remote programmer to the point in the file that might have been the problem in order to fix it. It was fun in almost exactly the same way that drunkenly falling over and smashing face first into concrete is fun – a moment of blissful free fall when you think you might have gotten away with it, followed by the realisation that this will take a long time to fix.

But fix it I did, and it only took slightly longer then it would have for me to drive there and back. Which in my mind was a win.

So next time you’re feeling hard done by because you have to use printfs to debug some code, or you’re having trouble understanding what a particular error relates to, just remember. It could be worse. Much worse. At least you’re not standing ankle deep in blood and bile while you’re doing it.

And what happened to the program I ported? Well I ended up proving that the original code never worked reliably. It was too heavily dependent on the ambient lighting conditions  and would give different readings depending on the time of day.

So, what was your worst debugging session?

(This article and others like it can be found on my blog - Seven Degrees of Freedom)