There should be a way for XNA to emulate an Xbox 360 or on the Xbox 360 run through the some automated tests to make sure things are not broken. For instance the testing program can ask the tester to enter data on how to reach a menu, or exit the game and record those user inputs and then try the inputs on sevreal configurations (no HDD, memory card, diff. screen rezes). It could also capture the image on the screen to make sure all text is in the safe area.


How would everybody like a program or system that can do this, some programs spam other programs with random input to make sure they don't crash and that is the same princible here. I am considering building something like this, perhaps it could just wrap the game... IDK really, it could really help with the review process and quality control of games.

Maybe we can iron out some specs. here and some community members might want to take the project up... Just thinking... sue* me if it is a bad idea :/

*This does not mean that you can actually sue me if the idea sucks.

-Bradleat

I dislike the idea of automatic testing. I think a person should be present to observe everything for themselves. A machine trying to mimic a user is bound to miss all sorts of things. It would lead to a lapse in quality control rather than an increase if more and more relied on it. I'm not even sure it's possible to make something like this. Sure you could tell a machine to essentially run a "macro" on the game to get into various menus and whatnot, but how is it really going to verify that anything is kosher?

I believe you get more automation ability on the dev kit xboxes that you get if you're a "real" game developer for Xbox.

The closest you can get right now is to use the Deploy actions from MSBuild, or perhaps just use the CCGAME unpacker to deploy, but that doesn't quite get you to the point of running. I suppose you can write a VBScript macro for Visual Studio itself, and do the deploy/run from with DEVENV.EXE, but then you have to have the GUI running to run your automated process.

Once running, the program needs to have built-in self-tests. Either you can put test vectors in a data file, and have the input.read() function in your game read from a file instead of from a controller, or you can vector the input.read() function to the network and send the data you want returned from the PC. I think the file would be easier.
And, yes, this means you have to abstract the controller reading (which most people do anyway), and you can't actually test physically removing a controller or MU; the closest you can get is emulating it by returning IsConnected as false (which isn't quite the same thing).

I dislike the idea of automatic testing.

Automated testing saves so much time during development. Person testing also has its place, but it's not something you can afford to do after every build (or every night, or whatever). The good thing about automation is that it allows you to find bugs fast after they have been introduced, when they are easy to find and fix. If you have to wait for a software tester to run through a test plan and get to the part that broke, that will take much longer, and will be much more error prone.
So, save the human tester time and money for the things that humans actually are good at testing, and machines cannot test!

I was thinking of something that just runs down the checklist that is in place during peer review... something that helps to streamline the process. For instance, it could make sure that the game runs in all view modes, etc. Things like that, make sure that no random errors break the game. It would require changing input devices to something else, but a system that sits on top of the game (taking screen shots for user valadation) might be a good thing. I wouldn't want to make anything that degrades peer review quality, etc but just something to help a developer out until it is ready. For instance, instead of having to track bugs for a game that spur from common mistakes the system can catch them before they manage to creep into the SVN code.

I am not a programmer for my team atm and thusly I have been letting my skills slip. Can someone point me in the right direction?

MS just needs to buy Cyberdyne Systems and pump a ton of cash into them. Then we can use SkyNet to test everything in Peer Review. :-)

<serious>
Reading controller inputs from a text file would work pretty well, but everything depends on already having wrapped input, storage, video, etc. Most people don't. Even I'm guilty of this when I'm in the last "get it over with" stages of a project when it's easier to just hack something in than use an OO approach. So, judging by the number of fails we see in Peer Review, that wouldn't work for the majority of games as an automated Reviewer. As an internal dev tool it might be useful, but I think the time required to set everything up correctly so it would work far outweighs the time of spending a few minutes testing it yourself every day...

True, The guys in the XNA team could do this best. They could just have a review version of the framework that can fake the controller inputs without having to wrap the game to do it. There might be a way to do this on the computer... but IDK. My main thing though is testing the technical parts... making sure the game doesn't crash if the resoultion is X, Y, or Z.

There's a difference between driving your own game, and driving other people's games. Driving other people's games is hard. However, driving your own game only requires one small feature addition to the XNA Game Studio Connect: the ability to specify a command-line option to the unpacker to make it start the game after it's un-packed. After that, you can automate the rest yourself. (Well, you can't drive the Dashboard, so testing resolutions still can't be automated).

Given that this is a simple, common-sense feature request, I filed a Connect request for it. Vote early, vote often!

i voted for it, that should make things easier

jwatte:

driving your own game only requires one small feature addition to the XNA Game Studio Connect: the ability to specify a command-line option to the unpacker to make it start the game after it's un-packed.

You mean like xnapack unpack foo.ccgame /run?

Sweet!

Where's the documentation for xnapack? I tried googling, and even bing-ing, but didn't find anything good.

What's funny about this thread is that I'm researching Automated Testing (even asked Shawn a couple of questions, one that was kinda mentioned here) for my university. There are alot of hurdles that you have to cross in order to do automated testing and specifically what tests you want to perform. You also have to write testable code. You can write all of a pong game within the Game class with no extra methods but you wouldn't be able to test it in a reasonable way. Hopefully I get the approval of my advisor to create a tool to help with some things that will help with development. There's not that much information on Automated Testing for Video Games out there, most of that stuff is like in house stuff. I found a couple articles on it but nothing in depth, unfortunately, makes my life harder.

jwatte:

Where's the documentation for xnapack? I tried googling, and even bing-ing, but didn't find anything good.

Ooh ooh ooh...

http://lmgtfy.com/?q=xnapack&l=1

Sorry, but I've always wanted a good excuse to do that  :-)

Bradleat:

There should be a way for XNA to emulate an Xbox 360 or on the Xbox 360 run through the some automated tests to make sure things are not broken. For instance the testing program can ask the tester to enter data on how to reach a menu, or exit the game and record those user inputs and then try the inputs on sevreal configurations (no HDD, memory card, diff. screen rezes). It could also capture the image on the screen to make sure all text is in the safe area.


How would everybody like a program or system that can do this, some programs spam other programs with random input to make sure they don't crash and that is the same princible here. I am considering building something like this, perhaps it could just wrap the game... IDK really, it could really help with the review process and quality control of games.

Maybe we can iron out some specs. here and some community members might want to take the project up... Just thinking... sue* me if it is a bad idea :/

*This does not mean that you can actually sue me if the idea sucks.

-Bradleat

mpipe:

The best way to test a program is with people, since people will be the ones using it. 

For certain things, yes, people are needed. For other things, automated testing is the best way. Would you want a person to keep pressing buttons to get to a certain part of a game just to confirm that it loads or would you want a program that can do that for you? Automated testing has more pros than cons.

UberGeekGames:

Reading controller inputs from a text file would work pretty well, but everything depends on already having wrapped input, storage, video, etc. Most people don't. Even I'm guilty of this when I'm in the last "get it over with" stages of a project when it's easier to just hack something in than use an OO approach. So, judging by the number of fails we see in Peer Review, that wouldn't work for the majority of games as an automated Reviewer.

Can't you just make dummy #using headers and override the XNA framework itself?

you wouldn't be able to test it in a reasonable way

You clearly haven't made friends with Reflection.Emit yet :-)
Although on Xbox, you don't have Emit. However, for a thesis, automated testing for Windows XNA games is probably good enough.

Shawn: I googled "xnapack documentation," and then I bing-ed it, and all I got on the first page hits were forum threads and release note references -- no MSDN hits (which are the only hits I bother with for documentation :-)

If, however, I google only "xnapack," then perversely the MSDN link is the top... Shoulda known!

Running "xnapack" without any arguments displays pretty good usage info, too.