Comments on: Do or do not " “Something has to be fixed” but not necessarily by you and not necessarily immediately." In my experience, that usually means "no one can be arsed to fix this". Once you get a group of these problems forming it gets hard to keep track of whose fault they were and they persist into the game until QA puts a bug in about it. It might be that the person whose fault it is never sees the problem. As a multiplayer programmer, I've seen it so often where other programmers working on single player use "I've only got one dev kit" as an excuse to never test multiplayer before committing code, and break something they'll never see until someone goes and kicks their ass about it. Keeping on top of issues keeps them under control. The more errors you let people skip, the more unstable the system is and you wonder what you can trust any more. You might get errors that look like they're your fault but actually caused by something completely different that everyone else has been ignoring. ” “Something has to be fixed” but not necessarily by you and not necessarily immediately.”

In my experience, that usually means “no one can be arsed to fix this”. Once you get a group of these problems forming it gets hard to keep track of whose fault they were and they persist into the game until QA puts a bug in about it. It might be that the person whose fault it is never sees the problem. As a multiplayer programmer, I’ve seen it so often where other programmers working on single player use “I’ve only got one dev kit” as an excuse to never test multiplayer before committing code, and break something they’ll never see until someone goes and kicks their ass about it. Keeping on top of issues keeps them under control. The more errors you let people skip, the more unstable the system is and you wonder what you can trust any more. You might get errors that look like they’re your fault but actually caused by something completely different that everyone else has been ignoring.

]]>
By: TomF/2011/02/25/do-or-do-not-2/#comment-1082 TomF Sun, 27 Feb 2011 06:42:46 +0000 I was going to say that I wish GCC/MSVC had an option to tell you if you miss an enum in a switch statement. I do the same thing you do with an "unset" value. Then i found that GCC does have it! Its -Wswitch-enum and you can make it an error with -Werror=switch-enum. Don't know if this helps, haven't even tried it myself but it could be useful. I was going to say that I wish GCC/MSVC had an option to tell you if you miss an enum in a switch statement. I do the same thing you do with an “unset” value. Then i found that GCC does have it! Its -Wswitch-enum and you can make it an error with -Werror=switch-enum. Don’t know if this helps, haven’t even tried it myself but it could be useful.

]]>
By: TomF/2011/02/25/do-or-do-not-2/#comment-1066 TomF Sat, 26 Feb 2011 18:47:06 +0000 Let's not get hung up on what the word "assert" means. There's that nice taxonomy by Chris Hargrove above, and whatever you call those four things, I'd hope we can all agree they're all valuable tools to have? Let’s not get hung up on what the word “assert” means. There’s that nice taxonomy by Chris Hargrove above, and whatever you call those four things, I’d hope we can all agree they’re all valuable tools to have?

]]>
By: Pete Cooper/2011/02/25/do-or-do-not-2/#comment-1059 Pete Cooper Sat, 26 Feb 2011 11:29:33 +0000 Yes, we actually have three different kinds of asserts for just that purpose: <pre lang="cpp"> // The "normal" kind of assert. // Note that printf(...) syntax is used for the message. XASSERT(textures.has(key), "No such texture %s", key.to_cstring()); // For when you don't need a message. // (Because it is clearer than XASSERT(condition, "").) XENSURE(buffer.size() > 0); // Because assert(false) sucks! XERROR("Unknown alignment enum %i", alignment); </pre> The last one is for example used in default: clauses for switch statements that should be exhaustive. The X is just there because too many third party libraries #define ASSERT, so we add an X factor. Yes, we actually have three different kinds of asserts for just that purpose:

// The "normal" kind of assert.
 
  // Note that printf(...) syntax is used for the message.
 
  XASSERT(textures.has(key), "No such texture %s", key.to_cstring());
 
   
 
  // For when you don't need a message.
 
  // (Because it is clearer than XASSERT(condition, "").)
 
  XENSURE(buffer.size() &gt; 0);
 
   
 
  // Because assert(false) sucks!
 
  XERROR("Unknown alignment enum %i", alignment);

The last one is for example used in default: clauses for switch statements that should be exhaustive.

The X is just there because too many third party libraries #define ASSERT, so we add an X factor.

]]>
By: Dylan Cuthbert/2011/02/25/do-or-do-not-2/#comment-1054 Dylan Cuthbert Sat, 26 Feb 2011 07:21:36 +0000 "To Assert" means "To State As True." You shouldn't be fixing anything. If you can (and want to) recover from a condition, print an error message or a warning and then carry on; assert is the wrong tool for that job. Writing a mechanism to skip asserts is basically saying that the preconditions of your code can be violated if you want to. That's not healthy for anyone. In my view, assertions are exceptions, and they are never handled. “To Assert” means “To State As True.” You shouldn’t be fixing anything. If you can (and want to) recover from a condition, print an error message or a warning and then carry on; assert is the wrong tool for that job.

Writing a mechanism to skip asserts is basically saying that the preconditions of your code can be violated if you want to. That’s not healthy for anyone.

In my view, assertions are exceptions, and they are never handled.

]]>
By: tomp/2011/02/25/do-or-do-not-2/#comment-1047 tomp Sat, 26 Feb 2011 02:49:52 +0000

Although I would say that your asserts can offer information on /why/ the crash is about to happen. E.g. if your assert macro includes the option to put a useful message in there like, “Array needs to be sorted before BinarySearch is called on it.” Also, “assert(false)” sucks.

]]>
By: Paul Evans/2011/02/25/do-or-do-not-2/#comment-1039 Paul Evans Fri, 25 Feb 2011 22:51:00 +0000 Great comments Chris and Tom! It has been a great discussion so far - I think a few other blog entries are going to pop up about asserts :0) Great comments Chris and Tom! It has been a great discussion so far – I think a few other blog entries are going to pop up about asserts :0)

]]>
By: Paul Evans/2011/02/25/do-or-do-not-2/#comment-1035 Paul Evans Fri, 25 Feb 2011 22:39:58 +0000 >Permanently burying asserts deep inside your code could end up causing more problems than they seek to solve. Asserts embedded in methods are useful while debugging, but should rarely be committed to source control. I disagree strongly. Many software systems are not "write once use many times", they are in a constant state of flux, and assumptions made in code can cause bugs if they're violated by future maintenance/refactoring/tuning. Asserts will help catch these violations of assumptions earlier, but only if they're checked in. Like TomF says, the one that is not checked in could cause hours of debugging time. This is extremely important when working on large software systems (which I consider modern games to be) and where the ownership of the code is distributed across a large set of people or teams. The use of assertions therefore allows a soft-contract during development for the definition of interfaces between modules/teams and allows team members the flexibility to construct software without requiring specific knowledge in all domains of the software. And all done without imposing external test requirements on everyone, which quite frankly is a pipe dream; in my experience it is difficult to hit high-coverage ratios in a purely external test scenario. Got a problem with artists hitting asserts all the time? your assert infrastructure could be augmented to optionally: i] auto-open a remote debugging connection. ii] auto-email system to the owner of the assert with a link to remote debugging connection info. iii] send a callstack dump. iv] send other information (which artist/user, dev kit, which level running, etc...) v] auto-file a bug in your DB if it doesn't already exist. Producer/coder can evaluate, instance the bug and assign to artist, close as dup etc. if there are missing assets etc. eg. severity = {WARNING, FATAL}... owner = {PROGRAMMER_X, PROGRAMMER_Y, ARTIST_LEAD, etc..} assertAndNotify(severity, owner, cond); // e-mails owner and CCs the lead, files bug in DB etc. Each artist machine could be configured to skip warnings owned by programmers, but e-mail + bug DB actions can be done regardless of the severity/owner. Moreover, an influx of bug DB/e-mail will provide strong quality indication of your software, and motivation to schedule "bug bash" days. >Permanently burying asserts deep inside your code could end up causing more problems than they seek to solve. Asserts embedded in methods are useful while debugging, but should rarely be committed to source control.

I disagree strongly. Many software systems are not “write once use many times”, they are in a constant state of flux, and assumptions made in code can cause bugs if they’re violated by future maintenance/refactoring/tuning. Asserts will help catch these violations of assumptions earlier, but only if they’re checked in. Like TomF says, the one that is not checked in could cause hours of debugging time.

This is extremely important when working on large software systems (which I consider modern games to be) and where the ownership of the code is distributed across a large set of people or teams. The use of assertions therefore allows a soft-contract during development for the definition of interfaces between modules/teams and allows team members the flexibility to construct software without requiring specific knowledge in all domains of the software. And all done without imposing external test requirements on everyone, which quite frankly is a pipe dream; in my experience it is difficult to hit high-coverage ratios in a purely external test scenario.

Got a problem with artists hitting asserts all the time? your assert infrastructure could be augmented to optionally:
i] auto-open a remote debugging connection.
ii] auto-email system to the owner of the assert with a link to remote debugging connection info.
iii] send a callstack dump.
iv] send other information (which artist/user, dev kit, which level running, etc…)
v] auto-file a bug in your DB if it doesn’t already exist. Producer/coder can evaluate, instance the bug and assign to artist, close as dup etc. if there are missing assets etc.

eg.
severity = {WARNING, FATAL}…
owner = {PROGRAMMER_X, PROGRAMMER_Y, ARTIST_LEAD, etc..}
assertAndNotify(severity, owner, cond); // e-mails owner and CCs the lead, files bug in DB etc.

Each artist machine could be configured to skip warnings owned by programmers, but e-mail + bug DB actions can be done regardless of the severity/owner.

Moreover, an influx of bug DB/e-mail will provide strong quality indication of your software, and motivation to schedule “bug bash” days.

]]>
By: TomF/2011/02/25/do-or-do-not-2/#comment-1031 TomF Fri, 25 Feb 2011 20:36:52 +0000 I agree. I basically only use asserts to tell me why the code was about to crash horribly anyway, e.g. null pointer dereference. I agree. I basically only use asserts to tell me why the code was about to crash horribly anyway, e.g. null pointer dereference.

]]>
By: Chris Hargrove/2011/02/25/do-or-do-not-2/#comment-1029 Chris Hargrove Fri, 25 Feb 2011 18:50:24 +0000 Violently agree with this. I am used to working on very small game teams with possibly only 1 programmer and asserts are my main weapon against myself. I constantly find myself thinking "Ok, is it safe to assume XYZ here? ...*investigates... I'm pretty sure it is." only to discover much later that there was in fact some situation where it was not a safe assumption. This happens all the time, even when I'm the original author of ALL of the code involved. In these situations, adding an assert for the condition in question, possibly along with a comment explaining to myself what my thought process is when adding it, has saved me countless hours of debugging because it allows me to easily jump back in time if the assert ever fires and pick up where I left off evaluating the assumption. Then I can fix the code, amend the assert, amend the comment with new information about this situation I failed to foresee - lather, rinse, repeat. I try to write code to be so eager to assert and crash the game that everything has to be perfect for anything to work at all. When I fix a bug, I almost always add an assert that would have caught that bug (hopefully sooner if possible) as a sort of regression insurance. As long as your team is 1- willing and able to immediately investigate, thoroughly understand and then address any asserts that do fire and 2- willing to actually build and test new content before committing it to source control, this methodology has always worked out quite well. Having said that, I fully realize that the reality of working on a large game team (which I have done in the past) can be quite different and using the same approach would often not be feasible. To each his own. Violently agree with this. I am used to working on very small game teams with possibly only 1 programmer and asserts are my main weapon against myself. I constantly find myself thinking “Ok, is it safe to assume XYZ here? …*investigates… I’m pretty sure it is.” only to discover much later that there was in fact some situation where it was not a safe assumption. This happens all the time, even when I’m the original author of ALL of the code involved. In these situations, adding an assert for the condition in question, possibly along with a comment explaining to myself what my thought process is when adding it, has saved me countless hours of debugging because it allows me to easily jump back in time if the assert ever fires and pick up where I left off evaluating the assumption. Then I can fix the code, amend the assert, amend the comment with new information about this situation I failed to foresee – lather, rinse, repeat.

I try to write code to be so eager to assert and crash the game that everything has to be perfect for anything to work at all. When I fix a bug, I almost always add an assert that would have caught that bug (hopefully sooner if possible) as a sort of regression insurance. As long as your team is 1- willing and able to immediately investigate, thoroughly understand and then address any asserts that do fire and 2- willing to actually build and test new content before committing it to source control, this methodology has always worked out quite well.

Having said that, I fully realize that the reality of working on a large game team (which I have done in the past) can be quite different and using the same approach would often not be feasible. To each his own.

]]>
By: TomF/2011/02/25/do-or-do-not-2/#comment-1026 TomF Fri, 25 Feb 2011 15:42:09 +0000 I make asserts "non-fatal" and "once-only" where-ever possible. This comes from the perspective of working on a large team and having an non-critical error in data take down an entire level is unacceptable. Asserts will tell you "Something has to be fixed" but not necessarily by you and not necessarily immediately. For instance, an animator would export an enemy with a missing jump animation. An assert will go off to say an enemy is missing an animation (and hopefully give all the information necessary for identifying and fixing the issue). This should not stop (a large number of) devs from continuing their own tasks that may relate to a level with that enemy in it. It is an assert, and WILL be fixed, but we don't stop all our developers from working while they wait for an animator to be found and re-tasked to fix the issue. Also, if we are doing a publisher build, and an assert slips through the QA process, it is better to say "just skip that, its non-fatal" than - "oh sorry, that build is useless. please wait for a new one". I don't think there is ever a good reason for an assert for be fatal if it could "easily" have been non-fatal. That is, if the code can easily recover without causing "structural damage" to the code or impacting on performance, then it should recover. It will save development time in the end. I make asserts “non-fatal” and “once-only” where-ever possible. This comes from the perspective of working on a large team and having an non-critical error in data take down an entire level is unacceptable. Asserts will tell you “Something has to be fixed” but not necessarily by you and not necessarily immediately. For instance, an animator would export an enemy with a missing jump animation. An assert will go off to say an enemy is missing an animation (and hopefully give all the information necessary for identifying and fixing the issue). This should not stop (a large number of) devs from continuing their own tasks that may relate to a level with that enemy in it. It is an assert, and WILL be fixed, but we don’t stop all our developers from working while they wait for an animator to be found and re-tasked to fix the issue.

Also, if we are doing a publisher build, and an assert slips through the QA process, it is better to say “just skip that, its non-fatal” than – “oh sorry, that build is useless. please wait for a new one”.

I don’t think there is ever a good reason for an assert for be fatal if it could “easily” have been non-fatal. That is, if the code can easily recover without causing “structural damage” to the code or impacting on performance, then it should recover. It will save development time in the end.

]]>
By: Paul Evans/2011/02/25/do-or-do-not-2/#comment-1024 Paul Evans Fri, 25 Feb 2011 15:14:02 +0000 Violently disagree. In my opinion good code should be 1/3rd code, 1/3rd comments and 1/3rd asserts - though I admit I'm a little more assert-happy than most coders I know. They have saved me countless hours of debugging, and so often when I eventually track down a bug in other people's code I can point to a place I would have put an assert that would have found it in five seconds. If you're getting annoyed with asserts then you need to make your asserts more usable, useful and friendly. As people have pointed out, if you're going to use asserts for non-fatal errors then you need something more sophisticated. I have a blog post slow-cooking on this subject which one day I will finish. Probably. Violently disagree. In my opinion good code should be 1/3rd code, 1/3rd comments and 1/3rd asserts – though I admit I’m a little more assert-happy than most coders I know. They have saved me countless hours of debugging, and so often when I eventually track down a bug in other people’s code I can point to a place I would have put an assert that would have found it in five seconds.

If you’re getting annoyed with asserts then you need to make your asserts more usable, useful and friendly. As people have pointed out, if you’re going to use asserts for non-fatal errors then you need something more sophisticated. I have a blog post slow-cooking on this subject which one day I will finish. Probably.

]]>
By: Dave Herod/2011/02/25/do-or-do-not-2/#comment-1021 Dave Herod Fri, 25 Feb 2011 14:42:38 +0000 *headdesk* Sorry, the above should read: User: “It crashed.” Me: “That’s odd… what did you do?” User: “Nothing. I just loaded the level and it crashed.” Me: “Er… well, can you try that again and we’ll see if it’s reproducible.” <I watch in horror as user loads level and skips about 30 asserts warning that something is horribly broken and about to crash the game, followed by exactly that happening> User: “See, it just crashes!” ...sill me for forgetting that < and > would get read as tags... *headdesk*
Sorry, the above should read:

User: “It crashed.”
Me: “That’s odd… what did you do?”
User: “Nothing. I just loaded the level and it crashed.”
Me: “Er… well, can you try that again and we’ll see if it’s reproducible.”
<I watch in horror as user loads level and skips about 30 asserts warning that something is horribly broken and about to crash the game, followed by exactly that happening>
User: “See, it just crashes!”

…sill me for forgetting that < and > would get read as tags…

]]>
By: Ben Carter/2011/02/25/do-or-do-not-2/#comment-1019 Ben Carter Fri, 25 Feb 2011 14:10:21 +0000 Oh, and I assume those asserts in those games are special asserts that were supposed to be in production code RELEASE_ASSERT or whatever - and they are *definitely* the evil - as pointed out by some folks (or soon to be pointed out by some folks), the idea is that all these asserts will have caught all the problems during development and testing, and you should be able to do a production build with them all removed. I still prefer real unit/integration tests for that though... Oh, and I assume those asserts in those games are special asserts that were supposed to be in production code RELEASE_ASSERT or whatever – and they are *definitely* the evil – as pointed out by some folks (or soon to be pointed out by some folks), the idea is that all these asserts will have caught all the problems during development and testing, and you should be able to do a production build with them all removed.

I still prefer real unit/integration tests for that though…

]]>
By: Rob Ashton/2011/02/25/do-or-do-not-2/#comment-1014 Rob Ashton Fri, 25 Feb 2011 11:51:01 +0000 [...] This post was mentioned on Twitter by Mike Acton, frogameleon. frogameleon said: RT @mike_acton: #AltDevBlogADay Do or do not Lee and Dave - totally agree that the assert is for the developer, but during development more than just one developer will be looking at the code. Asserts should not be for the user - though even other developers of your team end up being "users" of your code. I would say that an assert *is* useful for debugging though Lee. For example you can very quickly make a change, run your code and only attach the debugger on the assert - important as under some conditions attaching a debugger early can really slow things down. Committing that message to yourself deep in the source for you and everyone in your team I am unsure about - there are better ways of communicating error. Dave you are right about it being a team culture thing; warnings and asserts should be first class failures. Unless you have that culture in place though - if something will be skipped it will be. Asserts are not exceptions, so unless they completely fail the option is there. Data sanitation via asserts though? Surely a exception or predictable error code returned for misuse of a method would be better than an assert firing only under certain build conditions? Though I do think asserts at those entry points make far more sense than scattering them to the winds throughout the code - at least your error messages are close to where the bad data was first received. As Steve McConnel says: "Sometimes the best defence is a good offense. Fail hard during development so that you can fail softer during production". If asserts are used there should be no ignoring them. Lee and Dave – totally agree that the assert is for the developer, but during development more than just one developer will be looking at the code. Asserts should not be for the user – though even other developers of your team end up being “users” of your code.

I would say that an assert *is* useful for debugging though Lee. For example you can very quickly make a change, run your code and only attach the debugger on the assert – important as under some conditions attaching a debugger early can really slow things down. Committing that message to yourself deep in the source for you and everyone in your team I am unsure about – there are better ways of communicating error.

Dave you are right about it being a team culture thing; warnings and asserts should be first class failures. Unless you have that culture in place though – if something will be skipped it will be. Asserts are not exceptions, so unless they completely fail the option is there.

Data sanitation via asserts though? Surely a exception or predictable error code returned for misuse of a method would be better than an assert firing only under certain build conditions? Though I do think asserts at those entry points make far more sense than scattering them to the winds throughout the code – at least your error messages are close to where the bad data was first received.

As Steve McConnel says: “Sometimes the best defence is a good offense. Fail hard during development so that you can fail softer during production”. If asserts are used there should be no ignoring them.

]]>
By: Niklas Frykholm/2011/02/25/do-or-do-not-2/#comment-1010 Niklas Frykholm Fri, 25 Feb 2011 10:37:10 +0000 I have to say I completely disagree with the whole "crying wolf" idea. If your game is constantly firing asserts and people are just going "whatever, it always does that" and ignoring and skipping past them, then there's a problem with your team - the solution is not "don't use so many asserts", it's to get your team to actually work as a team. If you run the game and an assert goes off, you should either fix the problem immediately or go find the person who's responsible and get them to fix it - otherwise the person who put the assert there might not even know it happens. The same goes with your point about compiler warnings. Turn on "treat warnings as errors" and people will be forced to fix the problem as it happens, so they don't build up and important mistakes get lost in a sea of spam. The whole point of an assert is to catch a mistake as early as possible and force someone to look at it before things get out of hand. Brushing the problem under the carpet and having some default behaviour in case of a mistake just makes that mistake get overlooked, and even worse, have knock on effects leading to more serious issues, which are more difficult to debug because the root cause of the problem happened earlier. If I use a piece of someone else's code incorrectly, I'd hope it would have asserts in to kick my ass and tell me what I've done wrong, rather than just merrily pretend everything's fine, give me bad results and cause me grief later on. I have to say I completely disagree with the whole “crying wolf” idea. If your game is constantly firing asserts and people are just going “whatever, it always does that” and ignoring and skipping past them, then there’s a problem with your team – the solution is not “don’t use so many asserts”, it’s to get your team to actually work as a team. If you run the game and an assert goes off, you should either fix the problem immediately or go find the person who’s responsible and get them to fix it – otherwise the person who put the assert there might not even know it happens. The same goes with your point about compiler warnings. Turn on “treat warnings as errors” and people will be forced to fix the problem as it happens, so they don’t build up and important mistakes get lost in a sea of spam.

The whole point of an assert is to catch a mistake as early as possible and force someone to look at it before things get out of hand. Brushing the problem under the carpet and having some default behaviour in case of a mistake just makes that mistake get overlooked, and even worse, have knock on effects leading to more serious issues, which are more difficult to debug because the root cause of the problem happened earlier. If I use a piece of someone else’s code incorrectly, I’d hope it would have asserts in to kick my ass and tell me what I’ve done wrong, rather than just merrily pretend everything’s fine, give me bad results and cause me grief later on.

]]>
By: Tweets that mention Do or do not » #AltDevBlogADay -- Topsy.com/2011/02/25/do-or-do-not-2/#comment-1008 Tweets that mention Do or do not » #AltDevBlogADay -- Topsy.com Fri, 25 Feb 2011 09:50:39 +0000 #fb [...]

]]>
By: Lee Packham/2011/02/25/do-or-do-not-2/#comment-1007 Lee Packham Fri, 25 Feb 2011 09:37:14 +0000