How funny, thats the solution I came up with last night too... and I just made a new sticky for it... I'm popping out now but I'll copy yours over to mine when I get back... an comments on my list would be great too http://forums.xna.com/forums/t/30487.aspx

Cool! A separate sticky list is even better (and I like the name for the "Not So Evil checklist..." :-).

I was a bit unsure anyway, if it was so good an idea to include the "not-to-fail-for" reasons with the "fail" reasons in one list (from a "didactical" point of view)...

Doc

Figured you didn't want discussion in the sticky, some I posting here. You should not FAIL for requiring non-standard controllers like guitars, drums, big button pads, etc. If the developer wants to do this & target a much smaller audience that should be his/her choice. But supporting a regular controller would be nice so that they at least see what the game is about.

Tommy McClain

Spyn Doctor:

Big Daddio:

require a signed in player but then allow the game to be started with a non signed in controller (I guess that would be confusing and frustrating)

I'm trying to decide if this is something to add to my evil list, but I'm not 100% sure I understand what you mean. Could you please elaborate?

Doc

This was the situation. I was signed in on a controller. The game did not tell me I needed to have a signed in player. So as normal I picked up another controller and started the game. I couldn't control my player. The only way I knew that I needed a signed in player was there was my gamertag was on one of the player labels. The signed in controller was vibrating as my player was getting damaged in game. Os the game first assumed I had a signed in gamertag, and second allowed me to start on any controller even though I needed to start on the controller with the signed in gamertag. If a game requires a signed in tag, it will should prompt to sign in on a non signed in controller.

Spyn Doctor:

So I've tentatively added a section with "Things you should NOT fail a game for" to my evil list with items that I think there is a community consensus about that they are not reasons for a failure.

This is a great idea. I think we need to have some "issues" that people have failed for called out.

So if I understand this correctly, then you hade your CC-profile signed in on #1 and no profile on #2, and you started the game with #2, but the game automatically assumed that you were using #1 (where the profile was signed in) instead of #2 which you used to start the game.
If that's correct, then this is now covered in the new 1c sub-case I mentioned above (which I added after Zmans interpretation of your problems, so I guess he understood you better than I did ;-).

Big Daddio:

Spyn Doctor:

So I've tentatively added a section with "Things you should NOT fail a game for" to my evil list with items that I think there is a community consensus about that they are not reasons for a failure.

This is a great idea. I think we need to have some "issues" that people have failed for called out.

Just for the record: This section is now gone again from the "evil list", since ZMan made an extra-sticky for it (the "not so evil list", see above).

Doc

Hi All,

After be inundated by a flood of literally two emails, I've come to the conclusion that people around these parts aren't as interested in the peer review process/improving games as they might make out.

First addition: Some people test for crashes for removing media during file IO. This is left as an open question in the evil list, so it'd be worth us lot figuring out if this is reasonabe or not.
Second addition: Testing for custom soundtracks playing properly.

In relation to the first addition, I'd like to query another issue. If you're testing for crashes by removing media mid-IO, do you also test for the game being able to cope with corrupted/half-written save data? This raises an interesting further question of how should you deal with such an instance? Number one rule of usamajility is never destroy the user's work, but the chances of someone having a Datel device to retrieve their save are pretty slim.

Your thoughts? Overall, pretty disappointed by the lack of feedback on edge cases. At least we've established either the community (including some MVPs) care more in arguments than in normal business, or that the evil list covers pretty much all edge-case tests.

Most developers are only interested in the review process when it slows down the release of their masterpiece. At that point they yell and scream about how broken the system is until their game passes and then we never hear from them again. This is the reason why playtest is farily empty and reviews are slow.

I personally think that removing a MU at any time should not crash or cause an issue. Saldy I only have one MU and I keep my profile on it so thats one test don't try.

From what the team have told us all read/write on the 360 is transacted - you either get a succesful file or you don't so in theory there should never be a broken file.

This MVP didn't email you becuase I think that the evil checklist, the all new not so evil checklist and the best practises documents alread do a great job of covering most of the cases that have been discussed in the last 3 months. I don't think we need another list.

Deej:

the evil list covers pretty much all edge-case tests

Deej:

First addition: Some people test for crashes for removing media during file IO. This is left as an open question in the evil list, so it'd be worth us lot figuring out if this is reasonabe or not.

In relation to the first addition, I'd like to query another issue. If you're testing for crashes by removing media mid-IO, do you also test for the game being able to cope with corrupted/half-written save data? This raises an interesting further question of how should you deal with such an instance? Number one rule of usamajility is never destroy the user's work, but the chances of someone having a Datel device to retrieve their save are pretty slim.

Generally it should be quite hard to corrupt or have half-written saves because the StorageContainer buffers. It doesn't actually write anything to disk until you call Dispose. So to get corrupted data, you'd pretty much have to pull the MU out during the execution of the Dispose method, which technically could happen. In general, I say that all code that has to do with the loading/saving should have try/catch blocks and handle all cases.

Second addition: Testing for custom soundtracks playing properly.

If a game claims custom soundtrack support, this is a must have test case. If the game music plays over the user selected music, the game does not support custom soundtracks, so selecting that as a feature is misrepresenting a game which is a fail.

Nick Gravelyn:

Generally it should be quite hard to corrupt or have half-written saves because the StorageContainer buffers. It doesn't actually write anything to disk until you call Dispose. So to get corrupted data, you'd pretty much have to pull the MU out during the execution of the Dispose method, which technically could happen. In general, I say that all code that has to do with the loading/saving should have try/catch blocks and handle all cases.

Aaah, didn't know that. I'm sure I've seen this happen on my 360, but I could just be getting confused with various test runs on the PC too.

Test 2 also mentions "Verify that the game handles this gamertag sign-out correctly." What do you guys reckon correct handling of this should be? Might be worth seeing if there's a consensus and adding it to the description, to save ambiguity.

Also, apologies for earlier lariness!

Deej:

Test 2 also mentions "Verify that the game handles this gamertag sign-out correctly." What do you guys reckon correct handling of this should be? Might be worth seeing if there's a consensus and adding it to the description, to save ambiguity.

That's probably entirely game-dependent.  One game might only use the signed-in gamertag for a leaderboard entry, while another might use it for specific storage issues, or a plethora of other reasons.

Most games seem to dump you back to the 'press start' screen which might annoy some folk... someone has mentioned that some XBLA games pause and ask you to log back in. Though how you handle if they log in as a different user might be interesting if you have a per user storage container in use. I don't think anyone has relly come up with a nice way to always handle this - but its probably a fairly rare thing to do especially in a single player game. As long as it doesn't crash when I do it then I'm sort of happy.

I suspect it comes dwn to why you require them to be logged in.. if its to save things then you must get them to log back in with the same account, or at the very least tell thm that the game will end if they don't. If its just to get a name/picture then dont worry about it and use 'anonymous' and a general tag picture. If its to play a live enabled game then do whatever XBLA live enabled games do.

Deej:

Test 2 also mentions "Verify that the game handles this gamertag sign-out correctly." What do you guys reckon correct handling of this should be? Might be worth seeing if there's a consensus and adding it to the description, to save ambiguity.

The only requirement to me is that it not crash. How it handles it past that is up to the game to decide.

The ZMan:

This MVP didn't email you becuase I think that the evil checklist, the all new not so evil checklist and the best practises documents alread do a great job of covering most of the cases that have been discussed in the last 3 months. I don't think we need another list.

I, not a MVP, also did the same for the same reason. The Evil Check List already has more than I'm willing to test on it. I felt it was enough. Sure there are other things that can go wrong but they aren't very common.

Patrick

I did mail Deej. Thanks for the offer man. It seems like we have a nice list now after all, lets see if anyone uses it.

Deej:

Some people test for crashes for removing media during file IO. This is left as an open question in the evil list, so it'd be worth us lot figuring out if this is reasonabe or not.

I left this intentionally vague for two reasons: First, I didn't want the test cases to be too explicit/detailed (they already are detailed enough), because as I said above somewhere, I fear that if the test cases appear as too detailed, they may give the impression that they already cover everything, and that you simply have to test all cases, and then you have tested for all problems there ever could be. Which is of course obviously not true. This could negatively impact the behavior of both developers (who could incorrectly assume that if they pass the evil list tests, they will be guaranteed to also pass review) and for testers (who could incorrectly assume that the test cases from the evil list are the only things they have to test, and thus stop being creative when reviewing games).

The second reason I left this vague is, because it would be extremely difficult to deliberately pull the MU in the middle of an I/O. Granted, reading/writing stuff from/to the MU is relatively slow, but compared to human reaction times it's still very fast. Even if you programmed the game yourself, I would bet that it would be very difficult for you to hit exactly the right moment to pull the MU in the middle of an I/O (without using strategically placed breakpoints or something similar). And for a game you didn't program, i.e. where you don't know the inner workings, trying this would really be like a lottery. You would never know if you did indeed pull the MU in the middle, and not maybe right before or after the IO. So if I spelled out a test case like "pull the MU in the middle of an I/O-operation", I would only create a nightmare situation for testers who attempted to follow this test case and who then would try to hit the I/O moment without really knowing if they ever did.

But I have now removed the "(although not necessarily in the middle of a storage operation)" part from that test case, because it was probably more confusing than helpful, with the same argument as above, only reversed: If the reviewer can't know when/if a storage operation is in progress to explicitly pull the MU at that moment, he also can't know if it is in progress to avoid pulling the MU at that moment. So essentially, whenever the reviewer decides to pull the MU, the game shouldn't crash or exhibit any other failworthy defects.

Deej:

Testing for custom soundtracks playing properly.

and

Nick Gravelyn:

If a game claims custom soundtrack support, this is a must have test case. If the game music plays over the user selected music, the game does not support custom soundtracks, so selecting that as a feature is misrepresenting a game which is a fail.

I've added a new test case for this and other "capabilities-claims" checks:

Test case 11: Verify claimed capabilities. When a game is submitted, the creator checks a set of "capabilities", for example how many players the game supports, if it supports co-op play, the supported resolutions, if it supports "Custom SoundTracks", etc. Check if all these claims are indeed true (as far as this is possible for you).
Tip: To check if "Custom SoundTracks" are indeed supported, try playing some music through the Xbox (from a CD, your music library, etc.) while the game is running. The game must then automatically switch off its own music and only play sound effects.

Nick Gravelyn:

Deej:

Test 2 also mentions "Verify that the game handles this gamertag sign-out correctly." What do you guys reckon correct handling of this should be? Might be worth seeing if there's a consensus and adding it to the description, to save ambiguity.

The only requirement to me is that it not crash. How it handles it past that is up to the game to decide.

I agree, which is also the reason why I left he wording in the test case so vague. Also for similar reasons as above: There are so many possible ways how a game could handle the sign-out case, that it would be impossible for the test case to mention them all (and even more impossible to try to sort them into "this is OK" and "this merits a fail" categories). So the rather vague text should make the reviewer aware that he has to think a bit for himself to decide if a certain behavior is "correct" or not.

Doc

One issue ,that I think is not well covered by the evil checklist, happened with my game and I dropped off review because many people told me that it was a fail reason.

It's related to multicontroller issues. Basically, in that case my game got every connected controller as a player, so if you had three controllers but were only using one, the game would start with three players.

In my opinion, this case is not explicitly defined in the evil checklist and it should. "Test the game allows non-playing connected controllers".

hi, when you press start to play a community game and then press the guide button and then pull out the MU, the game crashes to dashboard with "game stopped working, please download again" message. Since this isnt a code 4 should we just ignore this crash?

Does that work for every game or just a game that is using the MU? Is your gold membership stored on that MU? Otherwise seems odd to see why it would crash when you remover a MU.

Edit:
Just tried it with QuadTrix. My profile is on the MU so I expected issues. 
It says 'Please reinsert memory unit' - I did - the game continued. Using #2, not logged in. #1 logged in with gold. No crash

Tried with towers of Cedrick. Much the same thing.

So seems to me that its not a generic issue. Which game are you seeing this in and what is your setup. Are you running games from review through connect or in marketplace? Are they trial or purchased?

it happened for every game i tried, i was using ccgame files i downloaded. did you pull out the MU while the guide was still up? I too got the message to reinsert memory card, maybe I exited the guide without reinserting MU.

Yes I followed your instructions. Try it again and specify the game and exaclty the steps you did and whcih controller etc you are using.

i cant at the moment since my xbox is in for repairs.

I don't understand Test Case 6. I can't even run my game on my 360 without being logged into a Live account either in trial or normal mode. Why is that? What do I need to look at in order to figure this out?

I tried running the XNA creators connect and debugging my application and then bringing up the panel and logging out. This causes my game to go away right away and I get a code 7 error that a connection to Live cannot be established.

Where do I look? I am not doing any storage or network play. I put try catch around all places that I call Guide and even around my call to GamerServicesComponent.

I cannot debug this to find the crash because I can't start the XNA connect application without being logged into a Live account so how do I debug it?

Any suggestions?

You need multiple controllers to test this. Start your game with a controller that is not logged in (but leave one of the other controllers logged in with your CC account) and test the marketplace. If your game requires a logged in controller at startup then log in as a silver account and then half way through press the guide button and then log out.

I tend to test with controller #1 logged out, controller #2 logged in with my CC account and then I play the game on controller #3. That way you catch bugs for logged out controllers and also bugs where developers assumed #1 is always logged in.

I just figured this out a little while after posting. Thank you for the reply.

What kinds of things require a signed in silver account? What function calls specifically, if you don't mind. Is there a list somewhere? For example what kind of an account do I need to simply call Guide.IsTrialMode? Can a local player or even not signed in at all call that function? How about where I create the GamerServices object and add it to Components?

Any tips or links would be great.