A good selection of tests for everyone to keep in mind during game development and during peer review.I hadn't even thought of removing the storage devices during play but of course, it might happen sometime.

EDIT: After additional testing, I've very pleased that whilst I had not consciously considered some of these tests, my design meant that my own game Artoon did not crash at any of these. I still have to add some dialogs to inform the player however.

I wouldn't fail a game for crashing if a storage device is removed in the middle of play, but I would provide that as feedback if I noticed it - for the submitter's choice as to whether he would want to fix it or not.

arrogancy:

I wouldn't fail a game for crashing if a storage device is removed in the middle of play, but I would provide that as feedback if I noticed it - for the submitter's choice as to whether he would want to fix it or not.

I haven't remembered to test it, but I would. It's no different than handling situations like a user signing out while a search for a network session is in progress. There is a strong possibility of errors being involved and it is the developer's job to handle them. Same thing here. Many people assume they can hit save and pull a memory card right then and there. At the simplest level, all a developer has to do to support this is monitor their StorageDevice.IsConnected property and, if it goes to false ever, simply show the storage UI again to get a new storage device. Nothing really fancy needed and it prevents your game from trying to save to a disconnected storage device.

There's a lot of little edge cases that I ran into. I'll give one that I thought was kind of obscure - if you have an "unlock full game" menu option, you need to check that the gamer profile that is choosing the option has LIVE capability and allows content (there's a property somewhere on SignedInGamer.Privileges that says this).  An exception will be thrown and crash your game unless you handle it or check that property before calling Guide.ShowMarketplace().  The best thing to do is probably hide that menu option if the user doesn't have the ability to purchase content.  I just popped up a message that says "sorry, your profile doesn't allow you to purchase content.".  It's up to you how to implement it.   

ScrumThorax:

if you have an "unlock full game" menu option, you need to check that the gamer profile that is choosing the option has LIVE capability and allows content (there's a property somewhere on SignedInGamer.Privileges that says this). 

That's interesting information. You mean GamerPrivileges.AllowPurchaseContent, right?
I didn't know this one. Since the only gamer tags I test with are normal live-enabled gamertags (with and without CC-membership), I wouldn't have found this during my testing either.
How would one go about for creating a Gamertag where this privilege is false? Is there something in the dashboard settings somewhere, where you can forbid purchasing to a gamertag (via parental settings, maybe)?
Is the exception also thrown during the simulated trial mode purchase?

Doc

@Doc - yes, AllowPurchaseContent.  The exception that gets thrown is the same i think for both having a profile that cannot connect to LIVE and for a profile that is not allowed to purchase content. 

You have to create a local account on your xbox.  I can't remember how to do it, though, i think it is pretty straightforward.  I made that account a long time ago when I first purchased my xbox and couldn't get it connected to the internet.  (Btw- i played a few games with that account and my achievements are not transferrable to my LIVE account which i created after it - so i had to redo them!). 

I found having this local account was a good idea to also test sign out handling.  If you have one live account and that's what you sign into your game with (and it is your creator's club account), you can't sign out because XNA game studio connect requires the signed in to LIVE account.  In fact, I wonder if people are handling sign out correctly because it is hard to test for a peer reviewer....

Spyn Doctor:

How would one go about for creating a Gamertag where this privilege is false? Is there something in the dashboard settings somewhere, where you can forbid purchasing to a gamertag (via parental settings, maybe)?

That or just make a local account. If an account isn't a LIVE account, that privilege will be set to false.

Is the exception also thrown during the simulated trial mode purchase?

Not sure. I'll give it a test and let you all know.

Edit: Yep, the exception is thrown when you call Guide.ShowMarketplace if the player index passed is not a signed in gamer with a LIVE account and the privileges to purchase content.

If I am not signed into LIVE and try to run XNA Game Studio Connect, I get a Code 7 error ("A connection to Xbox LIVE could not be established"). If I sign in to my LIVE enabled account and then start my game, and then sign out during gameplay, I also get a Code 7 error (and the game forcefully exits). Is it possible to try {} catch {} this Code 7 error so that a more graceful exit is possible? If so, how do you do it?

That isn't an error your users are likely to see. That error is because if you sign out of LIVE, then XNA Game Studio Connect cannot validate your CC subscription and therefore terminates.

That's one I tested last night myself, and couldn't really think of a good way to handle.  Eventually I just gave up trying, and let the Xbox error speak for itself.

You may choose to reject games as "unplayable" based on those criteria (and I may choose to test for those criteria), but I think those criteria are harsher than the general intent behind the review.

I would expect to pick up a controller (which may not be controller 1), and play the game through, and not see any forbidden content. If I can do all that without crashing, then I would pass the game. My system is fairly standard: HDTV, 6.1 sound, guitar and wireless controllers, and a MU + hard disk, so chances are most people who did reasonable testing will work out on my system.

jwatte:

You may choose to reject games as "unplayable" based on those criteria (and I may choose to test for those criteria), but I think those criteria are harsher than the general intent behind the review.

I want to stress that all the test cases lists are only meant to be used as reason for a rejection if they expose a crash or other reason that makes the game unplayable (important info outside of titlesafe area, etc.). In so far, I think that the test cases do indeed reflect the intent behind the review (i.e. expose crashes or other reasons for unplayability [is that a word, even?]), at least in so far as I interpret the possible reasons for a rejection that are given on the peer review form. However:

I would expect to pick up a controller (which may not be controller 1), and play the game through, and not see any forbidden content. If I can do all that without crashing, then I would pass the game. My system is fairly standard: HDTV, 6.1 sound, guitar and wireless controllers, and a MU + hard disk, so chances are most people who did reasonable testing will work out on my system.

I also didn't mean to suggest that all these test case have to be tested by every peer reviewer. In my opinion too, it would certainly be totally acceptable to do a peer review like you describe it: Just start the game normally, play through it, then pass it if everything works and nothing "forbidden" is found. Pushing it through the stress testing is of course only an option. Some reviewers may feel like it, some may not. Or sometimes I might feel like it, other times not. Actually, some reviewers may not even be able to perform all test cases (have no MU, or no 2nd controller, etc.), which should certainly not stop them from doing a review!
My list here is only meant as a suggestion. Sort of a loose collection of vulnerable issues that may expose games to crash scenarios that can give reviewers ideas for their own testing. Or, even better, developers ideas for testing their games before submitting them.

Doc

Spyn Doctor:

Or, even better, developers ideas for testing their games before submitting them.

That is the most important part of this whole thread.... if developers stressed their games first then we wouldn't feel the need to.

The ZMan:

Spyn Doctor:

Or, even better, developers ideas for testing their games before submitting them.

That is the most important part of this whole thread.... if developers stressed their games first then we wouldn't feel the need to.

That's exactly what I've gotten out of this thread.  Essential scenarios where the average user may experience problems with my game.  It's caused a few more days of internal testing before I plan to submit for playtesting by the community, but I think that's a good thing.  Bugs caught early means the community is never exposed to them.  And for that alone I'm thankful for all of this thread.

Nick Gravelyn:

That isn't an error your users are likely to see. That error is because if you sign out of LIVE, then XNA Game Studio Connect cannot validate your CC subscription and therefore terminates.

Is there a way to test what would happen in the real scenario?

I also noticed that the same error occurs if I do the following

1. Connect two controllers where my live account is signed in on controller one and no account signed in to controller two
2. Start the game
3. Press start using controller two (this sets controller two as the active controller and puts up a SignIn window)
4. Select guest in the SignIn window

The same thing happens if I choose my (only) live enabled profile, but that actually makes sense because that would cause that profile to get signed off for a short while. The scenario described above though makes absolutely no sense in my head. Am I doing something wrong or does everybody have this problem when signing in guest nowadays? Seems that a lot of new error that didn't occurr before has come up now with xna 3.0.

I know, pushing a thread is not good style. But there have recently been a lot of games submitted for review which produced crashes when tested against various items on my little "evil" list, that I thought it might be a good idea to get this thread a bit closer to the top again, where newcomers have a chance to see it.

Use these test cases to test your own game before submission please! (Of course in addition to all the normal testing you do anyway...)

Doc

Spyn Doctor:

I know, pushing a thread is not good style. But there have recently been a lot of games submitted for review which produced crashes when tested against various items on my little "evil" list, that I thought it might be a good idea to get this thread a bit closer to the top again, where newcomers have a chance to see it.

Use these test cases to test your own game before submission please! (Of course in addition to all the normal testing you do anyway...)

Doc

Hahaha!

goob job! Keep pushing this list. It is the holy grail for me, now.
if I were informed of this list before submitting my game now I was not still waiting for reviews. My fault obviously.

I have another test idea.
Try repeatedly saving and loading games and such, in one of my projects, after a few loads and saves, there would be an error in the XML document that held the save data.
This caused an exception that puzzled me for a while. I eventually just put the document into a string and looked at it in debug, it turned out that there were extra things on the end that
messed up the document and crashed the game.
For example:

</playerStats>>playerStats>

that is what the last line looked like for me.
I fixed it by deleting the file before rewriting it. But that is a problem with the XML serializer that might need to be looked at.

Antigroup:

I have another test idea.
Try repeatedly saving and loading games and such, in one of my projects, after a few loads and saves, there would be an error in the XML document that held the save data.
This caused an exception that puzzled me for a while. I eventually just put the document into a string and looked at it in debug, it turned out that there were extra things on the end that
messed up the document and crashed the game.
For example:

</playerStats>>playerStats>

that is what the last line looked like for me.
I fixed it by deleting the file before rewriting it. But that is a problem with the XML serializer that might need to be looked at.

I think you are experiencing the same problem I had with my high score list. See this thread.

This thread should be more visible, perhaps as a sticky in the FAQ or Game Review/Playtest forums. It´s a great list of use cases to test again.

In addition to the list of test cases for reviewers, I've now also started to list a few "often encountered but easily forgot" caveats that can manifest as ugly bugs in a game (and have already caused troubles even for such high-profile games as Weapon of Choice or CarneyVale Showtime). See the first post in this thread.
The caveats are however not so much meant for reviewers (as they are hard to test for), but more for the benefit of developers who use my little checklist for testing/verifying their own games.

Doc

Spyn Doctor:

Caveat 1: Floating point number parsing.

What is the easiest way to test this? Can I just go to my Console Settings and change the Locale to Spain (my normal Locale is the United States), or do I need to change the Language setting to Spanish?

Does this problem with floating point numbers apply to XML serialization or only to specific float.ToString() calls?

I always test this by changing the language setting of the Xbox, so I assume that the language setting is the one that affects the floating point parsing. But changing the language on the Xbox is simple to do, and even if the dashboard then appears in a language that you may not understand, there's usually no problem navigating around (to the game library, to start a game for testing), as I guess all of us here know the layout of the dashboard/NXE quite well by now.

I can't answer the second question, as I'm not using the XML serialization for my games. If no one else chimes in with an answer, I would suggest testing it with a simple test serialization.

Doc

Spyn Doctor:

I always test this by changing the language setting of the Xbox, so I assume that the language setting is the one that affects the floating point parsing.

When I change my Locale, the numbers displayed on my Xbox take on a different form (like 48,6 GB remaining on my hard drive instead of 48.6). I will test setting the Language as well and will also test the XML serialization question.

It appears that XML serialization is not affected by the language/locale issue.