3/18: Clarified the MU/Device no fails for sorrupt and full devices and added best practises. See http://forums.xna.com/forums/t/49693.aspx

See also the 'Evil checklist of ways to fail a game' http://forums.xna.com/forums/t/19525.aspx
See also the 'Legal FAQ' for legal reasons you should and should not fail for http://forums.xna.com/forums/t/17830.aspx
See also the 'language FAQ' for language/translation reasons to fail/not fail http://forums.xna.com/forums/t/26198.aspx

This FAQ is a list of things that historically we don't fail a game for in peer review. Yes "confusing or frustrating" is vague, but if you fail for one of these things under that phrase then you will be mocked at best, possibly vilified and worse get a ban if people complain or you persist in doing it.

Many of these things are things that you SHOULD fix because it makes your game look lower quality and could annoy some users enough not to buy it. But that's your problem - don't come complaining when your sales suck. Many developers would love to see any or all of these as feedback in playtest so don't hold back there. Just don't fail for it in peer review.

DO NOT FAIL FOR

• Not having an exit option - users can always quit using the guide 
• Using "Return to Arcade" as an exit option. Prefer "Exit" though "Return to Dashboard", "Return to Game Library" or "Return to Indie Games" are OK too. 
• Leaving in the default 'back'=quit code
• Not having a pause/Not using start for pause/not pausing when Guide is pressed/not pausing when controller is disconnected.
• Not supporting controller switching during the game. A single press start screen is all that is required.
• Legal reasons beyond those in the legal FAQ http://forums.xna.com/forums/t/17830.aspx
• Forcing a storage device to be selected
• Forcing a user to log in/play with a logged in controller (though test what happens if you sign out AFTER the game has started and ensure the game can be played from any controller)
• Code 7 crashes. These are caused by logging out the account with the Creators club membership. This crash never occurs in non-review play
• Spelling or grammar - point it out but its the developer problem if they want to release a poor quality game
• Lack of instructions or controller hints on screens and menus unless they are so convoluted the game/screen couldn't be used with a reasonable amount of guesswork e.g. you are totally stuck on the first screen because you had no clue that you had to hold down both triggers and use the right thumbstick to move.  In these cases post to the forum or look at other reviews - if everyone is having the same confusion it might be worth a fail. If others are not confused then don't fail. Note that we occasionally see a game where guessing the controls and gameplay is part of the game. Developers should call this out in advance so that they don't fail for it.
• Not using thumbstick AND dpad for navigation
• Not using 'standard' buttons or controls
• IN GENERAL - confuse and frustrate is related to technical issues. If the instructions, control scheme or the gameplay is confusing or frustrating but its as the developer intended then just ignore the game. You can't fail a game because you think it's too difficult, boring, stupid or ugly.  You can't fail a game because you think it's not a game! Feel free to send the developer a comment on any of these and ignore the game but its not a fail.
• Not providing a way to upgrade to the full version of the game.
• Memory Units/Device Selector (but see some best practises below):
  • Undefined/odd/annoying/confusing behaviour after pulling a MU or selecting a full or corrupted device. If it crashes then fail, but anything besides that is the developers decision.
  • Pulling the MU that the game is stored on. No XNA or XBLIG can survive that.
  • Mandatory device selection - though it is annoying to some
• Mandatory login - also annoying to some
• Vomit doesn't count as 'excretion' - poop does. We've not debated if urination is.... have fun with that one.

SPECIAL CASES:

• Some xboxes with some games under some unknown conditions exhibit very long deployment and loading times. If you find a game that appears to have very long loading times then don't fail right away. Post to or read the peer review forum and see if anyone else is seeing this. If everyone is seeing it then its OK to fail for. However if its just you then its likely that you have one of those special xboxes. See discussion here http://forums.xna.com/forums/p/27019/148171.aspx - the issue is known to Microsoft and we hope a fix will happen at some point. The issue only occurs when sending a .ccgame through connect. It doesn't happen in marketplace and it doesn't happen under the VS debugger.
• A small numer of testers report black screens at game startup when reviewing a game through GS connect. A restart of the xbox usually clears this up. It appears to be random and not related to the game or XNA framework. Its not been reported with any marketplace release. If you see this and it doesn't repro every time then check the forum to ensure its not just you.

OTHER ADVICE (especially to new reviewers):

• Read through the previous review comments and/or playtest comments (click the number under the developers name to find all their threads including the playtest one). You may find that your fail reason has already been discussed.
• If in doubt post first - other reviewers will help out
• If really in doubt then spend some time playtesting only. Note in playtest what you would fail for and people will correct you and you can learn the ways of the peer review process.

MU/Device usage recommendations:
While you can cover all the failure cases with a simple try/catch and an 'infinite' loop back to the device selector its not the most user friendly experience. Consider these best practises.

• If the user cancels the selector it means they don't want you to fill up their drive. Show a dialog explaining what features they will lose but allow them to reselect or continue without a device. If your game truly cannot continue then give them the option to quit here rather than force them to choose a device to see your menu
• If the selected device is full then show a dialog explaining why this device cannot be selected with the option to reselect. Ensure they can bring the guide up to go to the dashboard if they need to tidy up
• If the selected device is corrupted..... well right now this is so hard to test we are not 100% sure you can tell the difference between a corrupted drive and one that just got pulled. So display a message that the device is not available with the option to reselect or quit. Ensure the guide is available in case they want to try to format or delete the corrupted file

Please reply with other suggestions as you see them in the forum. If its already on the evil checklist as a fail then it will NOT be reconsidered here. If you can't provide a link to where its been discussed or happened then I'm not going to preemptively include random things people can think of as the list will get too big.

That's a good list ZMan. If one of these things popped up in my game I would most likely *want* it to be failed so I can fix it. Sometimes these things don't get picked up until towards the end of the review process and by the time the developer sees the comment the game has completed review and has gone live. They can ofcourse do an update but by the time the update has gone through the game would have had its moment in the spotlight (new releases), and a bunch of those potential customers would be lost. Ofcourse most of it should already have been identified in playtest so I suppose most people who put their game in review with these things are already aware of them and just don't want to fix them.

Some of these points are deemed as acceptable even in XBLA, for example some game erquire that you play with a logged in controller. If you log out the game will pause until you log back in again. So these items would be more likey a concious design decision made by the developer and not just a flaw that they're too lazy to fix.

I should point out that Code 7 crashes can (and for me, often do) occur during non-review play. I have some sketchy network equipment that tends to kick the bucket whenever it's least convenient, and for several community games that I've purchased this instantly exits. It royally stinks near the end of a level.

I have a request.  Would it be possible to make an addition to the not-so-evil list to clarify one particular case of MU removal?

The consensus here: http://forums.xna.com/forums/p/41855/247254.aspx seems to be that it is not absolutely necessary to tell the player if an MU has been removed, so long as of course your game doesn't crash or become unplayable.

I agree it makes sense to say that best practice is to be polite and tell the player the MU has been removed (perhaps it got dislodged accidentally), but as the thread above shows, the opinion seems to be that this is not a fail issue.

Might this be added to the above list?

One small suggestion.  The "Credits" screen surely does not matter if it remains in the native language?  The user might not understand a few of the words on the left of my name.  Not an issue for me, so therefore not an issue?  I've had an otherwise 100% icon driven app failed for this, which seems daft to me.

Dave Daniel:

One small suggestion.  The "Credits" screen surely does not matter if it remains in the native language?  The user might not understand a few of the words on the left of my name.  Not an issue for me, so therefore not an issue?  I've had an otherwise 100% icon driven app failed for this, which seems daft to me.

No. Any language used in the game on any screen has to be claimed. No exceptions.