Hey guys,

I'm interested in starting a discussion about testing game code. I want to get some information from fellow game developers on their thoughts on testing, so I have a few questions:

1) Do you use any sort of formal testing approach? Such as test-driven development (where you write a test case before implementing any actual code)? Unit testing, where you test discrete bits of functionality? etc? Or do you consider testing to be an unnecessary expenditure of resources for your particular project?

2) If you do use a testing approach, what are the challenges that you have faced in trying to test features for your game?

3) What tools do you leverage for testing your software? Are there any limitations present, or features that could be added to make your life easier?

I'll share my thoughts on these issues.

1) I use unit tests to test functions that exhibit a discrete item of functionality. For example, certain string parsing functions with border cases would be useful functions to unit test. I'm implementing a UI and it helps to have test cases like "well, if the textbox has such and such text, after the user inputs X text with the cursor at Y, then the output text should result in Z". Testing general cases, as well as border cases where is at Y weird places (before text, after text, out of bounds areas) is useful and gives you a better understanding and confidence of the code, IMHO. I'll use test-driven development RARELY if I feel I have not defined an interface well enough, and want to see what it looks like in code. Test-driven development makes me uncomfortable because I don't have my intellisense to save me typing, haha..  (because I haven't implemented the thing i'm testing yet). But it is cool to help figure out how you want your class/object/feature to interact and be used in code.

2) Testing graphical features has been an issue. I can't really use NUnit to test graphical features. I can't regression test my rendering pipeline, etc. Generally I just have test cases that I show on screen to check if my graphical features are working. This works OK, i guess.

3) I use NUnit to for my unit testing. I don't really have a tool for my graphical tests - i just follow along the same line of writing a test case and displaying it on screen (like in Benjamin Nitschke's book)

I just thought it'd be cool to see what other developers are doing! I'm working on an XNA engine, and I want to make testing an integrated part of the engine to try and speed up development time. I'm trying to figure out what improvements can be made to testing processes in games, how valuable testing is in developing games,  and what would help developers test features and improve reliability & code confidence.

Thanks for your input!
Bryan

Preface: my unit tests so far cover helper classes and such only and am by no means an expert for TDD and friends :]

bp:

2) Testing graphical features has been an issue. I can't really use NUnit to test graphical features. I can't regression test my rendering pipeline, etc. Generally I just have test cases that I show on screen to check if my graphical features are working. This works OK, i guess.

I've been thinking about this a bit (but did not apply these thoughts yet) and I guess you could mock the graphics device and friends to ensure the proper values are set/passed instead of rendering and review the results manually.

Another possibility would be to render to a render target and compare its bits to those of a reference image. This approach might require using the reference rasterizer to ensure that the images look the same on an GPU, resulting in quite some run time of the tests. Also, this could be too strict to be applicable in a game environment.