Regarding the most recent requests to add this-and-that special case / edge case to the evil list:
Even though the evil list has long ceased to be "my evil list" and has since evolved to be the "community's evil list", I still feel a bit of fatherly responsibility for it ;-), so I seem to post this here every few months, hoping to get my "baby" on the right path again... :-)
The evil list can not be - and should not attempt to be - a complete list of all cases that lead to a failure. If the list becomes too fine grained, specifying and describing every single special case and edge case, then this will have three effects, all of which will be detrimental to the review process:
First, developers will start to assume that everything they need to test for is described on the evil list. So they will start to test
only the cases on the evil list, thinking that if they pass all tests, their game is guaranteed to pass review. They will stop being creative and testing the myriad of possible test scenarios that the evil list can never cover (even if it would attempt it). So we would soon arrive at a point where a developer thinks he has fully tested his game and in reality is overlooking serious problems, simply because the problems are slightly different than the examples given on the evil list. And worse, if such a developer then fails peer review for such an overlooked problem, he may be of the opinion that the fail was incorrect - after all, the evil list didn't mention it, right?
Second, peer reviewers will start to assume the same, i.e. that all necessary test cases are already described on the evil list, so they will only ever test those cases, but never any other possible error scenarios. So they will pass games thinking they tested them well, when in reality they didn't.
One might think that the solution to both problems would be to add more and more test cases to the evil list, and make it more and more specific. I don't think so, because then we will at some point encounter the third detrimental effect:
At some point, the list would become so overburdened, lengthy and detail obsessed (rivaling the XBLA TCRs), that it would become useless because it would be too hard to use - and people would start ignoring it. That's a typical danger of bureaucracy - once you start making rules, there's always arguments for adding more rules (and often those are very sensible arguments, so everyone thinks it's a good idea). Up until the point where the rules smother the process and everything either grinds to a halt or the rules are thrown out as a whole to start over from scratch.
We should be careful not to bloat the evil list to that point!
Instead, remember that the evil ist is
not a complete list! It
intentionally is not complete! In the very introduction of the list I wrote:
Note, that these test cases are meant as a guideline and inspiration only, as a basis for your own testing procedure. They are not meant to be followed slavishly and are not meant as a complete set of tests. A game may very well fail review even if it passes all tests mentioned here, if the reviewer finds some other problem! So be creative when testing and don't rely solely on this list!
and I think this should still be your guide when working with this list.
Now with that in mind, look at test case 1 which, in my opinion, has already become bloated in this fashion: It deals with multiple controller support and originally had only 3 sub-cases: 1a)-1c). (Now we have 1a)-f) and funnily enough an additional 1d) and 1e) for multiplayer scenarios - so 1d) and 1e) are actually present twice).
The original test case for multi-controller scenarios was two simple sentences:
Start the game with the second (or third/fourth) controller. Verify that the game is playable in various scenarios
and this test case then continued with the phrase "
for example:" before the sub-cases a)-c) were listed, which in turn closed with the phrase "
...or other multi-controller scenarios...".
See what I mean? The test case was intentionally vague, simply giving a few short examples for possible multi-controller scenarios to illustrate it's meaning. With the intention, that you, as either the developer or peer reviewer, would realize that you have the responsibility of coming up with possible multi-controller test-scenarios
yourself.
Since then sub-cases d)-f) have been added, including such a detail (down to technical implementation level) that there's an impression of completeness and exhaustiveness - which of course is only an illusion. Resulting in less motivation for the developer/reviewer to think about possible scenarios that are
not listed (and there are still a lot! - the recent problem with a controller grabbing a profile from another controller just being one of them).
So I postulate: Instead of thinking up more and more sub- and edge-cases to add to the evil list, we should become more aware that the evil list is only a guide, but that the responsibility of thinking up suitable test cases (for our own games or those we review) is still our own!
Actually, while I'm on the topic of test case 1: 1d)-1f) aren't even test cases, but descriptions of possible pit-falls that you need to avoid as a developer. This mixes test-case descriptions with pitfall/caveat descriptions. In my initial version of the list I had attempted to avoid these mixes, which is why I had the additional "caveats" section after the "test cases" section.
I think it would probably be a good idea to remove test-cases 1d)-1f) again and instead consolidate them into a new caveat description, and in test case 1 simply point to that new caveat, thus cleaning up the test case again. I'll do that, if there are no objections?
Doc
