George Clingerman:

If a game is using a signed in profile for a control I don't have in my hand, I personally consider that a fail as well.

I agree with George (and Pino). My usual method for peer review is to have the gamertag with my CCO-membership logged in on controller #1. Then I switch off that controller and use controller #2 (without a profile signed in) to start the game. Now, if during playing the game I would have to see that the game is using my CCO-gamertag even though I have explicitely picked up and activated the non-profile controller, I would immediately fail the game. If the game requires a logged in profile for play, and I have no profile on my controller, then the game is welcome to present me with a login dialog. And if I cancel that, it is equally welcome to tell me that I can't play it then. It is however not welcome to make this decision for me and simply pick whatever other profile it can find. And even though I would call this confusing&frustrating enough to fail on those grounds alone, I don't even think that it would be necessary to invoke that rather vague rule. For me, that's a clear technical defect, as I would assume that the game is somehow mixing up PlayerIndexes. So it's a clear fail in the "game defect" category.

Doc

I think no game should ever use a profile that is connected to another controller. If Microsoft wanted this functionality they would not tie a Gamertag to a controller. Ignoring children or anything else it, is simply wrong to do regardless. It also doesn't work, what if I have controllers #1 and #2 signed into Gold and use #3 not signed in but want to use Controller #2's Gamertag?

I don't even understand how one can argue. Microsoft has a system of one profile for each controller. If you use a controller that implies you want to use that controllers profile. If they want to use another controller's profile they can use the built in system to do it. If anyone was actually interested in that kind of feature Microsoft would have the Xbox handle it. It wouldn't be rocket science for them to add a swap profile option, clearly no one is interested in it.

If a game uses the wrong profile I consider that a bug even if it was intentional, just like a game that intentionaly draws important data outside of the title-safe region will fail.

Patrick

Spyn Doctor:

George Clingerman:

If a game is using a signed in profile for a control I don't have in my hand, I personally consider that a fail as well.

I agree with George (and Pino). My usual method for peer review is to have the gamertag with my CCO-membership logged in on controller #1. Then I switch off that controller and use controller #2 (without a profile signed in) to start the game. Now, if during playing the game I would have to see that the game is using my CCO-gamertag even though I have explicitely picked up and activated the non-profile controller, I would immediately fail the game. If the game requires a logged in profile for play, and I have no profile on my controller, then the game is welcome to present me with a login dialog. And if I cancel that, it is equally welcome to tell me that I can't play it then. It is however not welcome to make this decision for me and simply pick whatever other profile it can find. And even though I would call this confusing&frustrating enough to fail on those grounds alone, I don't even think that it would be necessary to invoke that rather vague rule. For me, that's a clear technical defect, as I would assume that the game is somehow mixing up PlayerIndexes. So it's a clear fail in the "game defect" category.

Doc

I'm not sure if you read my post directly before yours.

If I was a normal user and picked up a non signed in controller and it said "you have to be signed in with a Live enabled gamertag" my reaction would be "Uhg, that's annoying.", find my controller signed in with the profile, likely a controller the game isn't playable with like a guitar, sign out, then sign in on my other controller and play. It's a hassle.

Saying it's making the decision to use your profile and play on live for you sounds ridiculous to me. The majority of people have 1 Live profile and use it to play and when they navigate to play on xbox live they want to play on xbox live with that profile. It's not like the game is forcing you to use the profile, you can press the guide button and sign in, it displays your gamertag on the screen in the game, you can just press B then sign in and use your profile if you want to use another.


But if the majority of people really think this is EVIL instead of CONVENIENT it needs to be added to the evil checklist, because there clearly isn't universal agreement.

I don't think anyone has pointed this out yet: If a not-logged in controller can use the profile of a different controller (to be convenient), then Rock Band would fail!

Just about every time I play rock band with my guitar, the drums take my xBox Live account. Probably because of the port it is plugged into. If I want to control the session with my guitar, I have to log out the drums and log in with the guitar.

I think that sort of makes the standard. If a AAA game makes you log out and in with a different controller then Indie games should as well.

Star Gaming Network:

If anyone was actually interested in that kind of feature Microsoft would have the Xbox handle it. It wouldn't be rocket science for them to add a swap profile option, clearly no one is interested in it.

So does that mean Guide.ShowSignIn(1, false) is a mistake on Microsofts part?
That lets you sign in to a profile that is signed in on another controller. Haven't tested to see if it swaps the profiles, but it does log out the profile from the original controller and log the profile into the controller taking over the profile.

VDrackus:

That lets you sign in to a profile that is signed in on another controller. Haven't tested to see if it swaps the profiles, but it does log out the profile from the original controller and log the profile into the controller taking over the profile.

That swaps the profile's controller: is the one thing causing the Code 7 if you try that while reviewing a game. And in my case, if any of my children tries that will be prompted two passwords: one for the profile and one for the LIVE connection. Just stealing my profile via software will allow to skip the two passwords violating my settings.

VDrackus:

So does that mean Guide.ShowSignIn(1, false) is a mistake on Microsofts part?
That lets you sign in to a profile that is signed in on another controller. Haven't tested to see if it swaps the profiles, but it does log out the profile from the original controller and log the profile into the controller taking over the profile.

Yes it is. It only happens in Peer Review and does actually swap the GamerTags ( note that it actually signs it out and in again so if it is the CC Gamertag the game will crash with a code 7). On the Marketplace version, the used Gamertags are greyed out and cannot be selected. I have no idea why it is different but it is. You really just don't question the Guide, you accept that it is evil and move on.

Patrick

hotshot 10101:

I don't think anyone has pointed this out yet: If a not-logged in controller can use the profile of a different controller (to be convenient), then Rock Band would fail!

Just about every time I play rock band with my guitar, the drums take my xBox Live account. Probably because of the port it is plugged into. If I want to control the session with my guitar, I have to log out the drums and log in with the guitar.

I think that sort of makes the standard. If a AAA game makes you log out and in with a different controller then Indie games should as well.

Nope, that is the behaviour of ANY game if you have the auto-login on the profile, which is a mess when using the hub (Rock Band docet). In that case you have chosen to have your profile automatically assigned to the first controller, your choice, your problem. Here we're talking about something quite different.

Star Gaming Network:

It only happens in Peer Review and does actually swap the GamerTags ( note that it actually signs it out and in again so if it is the CC Gamertag the game will crash with a code 7). On the Marketplace version, the used Gamertags are greyed out and cannot be selected. I have no idea why it is different but it is.

Interesting, I stopped using that call because I didn't want players logging in with profiles on other controllers accidently. Using Guide.ShowSignIn(4, false) shows the logged in profiles greyed out so I went with that instead. I didn't realize that when it gets on the marketplace ShowSignIn(1, false) would correctly have the profiles greyed out.
Wish they would fix that then. I opened up a connect about it and haven't gotten a response as of yet.

I'm with Pino on this.  A controller should only use the profile signed in on that controller.  It seems that Microsoft has gone the extra mile to keep unrated content out of the hands of those who shouldn't have it.  Anything that decreases those protections should not be allowed.

Considering it seems more people think this should be a fail it should be added to the Evil Checklist.
Probably under test case 1, 2, or in it's own test case.

PinoEire:

If we allow this behaviour the next step could be a messenger like indie game with built-in input boxes and search fields to cheat the system in full... I imagine the box art's slogan: "your parents limit your LIVE account? don't worry... there is another way...".

Well.. well...

looks like I had a vision: I've just failed Ultimate Arcade Chat Rooms which does exactly that: allows a non signed in controller to chat with people not in the friend list using any Gold Profile connected.

I think that in test case 6 there should be added information that the game must display message informing player he needs to log in if the buttons weren't hidden. It may save some people unnecessary reattemps to peer review. Now it just says it can't crash. It was discussed here: http://forums.xna.com/forums/p/35890/207111.aspx.

Not so evil checklist proposal:

Errors and crashes are bad. A game should be failed for them. However there are times when I can't in good conscience fail a game for a crash or error that is not something I can reproduce.
It is disheartening when I see games that are failed for an error that can't be reproduced by the reviewer that discovered it or anyone else.

For me, If you can reproduce the error, fail it, let the dev know exactly how to reproduce it, etc. If I can't reproduce the error I let the dev know that I encountered it, was not able to reproduce it, to be on the lookout for it and that I will not fail because of it. Technically I'm suppose to fail for that, but it just doesn't feel right to me.

So, I propose text to be added to the not so evil checklist that states If the error or issue is not reproducible do not fail for it, but let the dev know about the error or issue that was discovered. (Preferably with as much detail as possible).

VDrackus:

Not so evil checklist proposal:

Errors and crashes are bad. A game should be failed for them. However there are times when I can't in good conscience fail a game for a crash or error that is not something I can reproduce.
It is disheartening when I see games that are failed for an error that can't be reproduced by the reviewer that discovered it or anyone else.

For me, If you can reproduce the error, fail it, let the dev know exactly how to reproduce it, etc. If I can't reproduce the error I let the dev know that I encountered it, was not able to reproduce it, to be on the lookout for it and that I will not fail because of it. Technically I'm suppose to fail for that, but it just doesn't feel right to me.

So, I propose text to be added to the not so evil checklist that states If the error or issue is not reproducible do not fail for it, but let the dev know about the error or issue that was discovered. (Preferably with as much detail as possible).

I second this as I already behave so and hope that most of us do the same. A good phrasing could be "If you cannot tell step by step how to reproduce an error then do not fail the game".

Cheers,
Pino

I would go along with this as long as it truly is not reproducible. If it happens to you over and over again, but you just can't figure out what triggers it, then that might still be considered a fail. If it happens to more than one person, but neither can figure out when or where (think random crashes), then that is almost definately a fail.

Since our games are multi-threaded it can sometimes be hard to create exact steps to reproduce a crash that depends on a lot of variables.

On the other hand I think you basic concept of (it crashed once, but never again and never for anyone else) should not be failed for.

I didn't see anything addressing this issue here..  In the evil list it says that any controller has to go through the menu, but in my experience I don't want to do that.  Now with that said any controller can come in and press star the first one to do so is player one.  But we are only allowing player one to navigate the menus.  The reason is if anybody has had played games with kids they like to press buttons so I only want to have player one to be able to pass through the menus.  So the question is, is that going to make my game fail peer review?  Its an easy task to make is so it uses any controller to pass the menus and everything else is navigated by any controller.

hotshot 10101:

I would go along with this as long as it truly is not reproducible. If it happens to you over and over again, but you just can't figure out what triggers it, then that might still be considered a fail. If it happens to more than one person, but neither can figure out when or where (think random crashes), then that is almost definately a fail.

Since our games are multi-threaded it can sometimes be hard to create exact steps to reproduce a crash that depends on a lot of variables.

On the other hand I think you basic concept of (it crashed once, but never again and never for anyone else) should not be failed for.

I am wary about letting games that crashed even just once pass. If one reviewer by chance happened to crash the game what are the chances the potentially thousands of people downloading the game who would likely play it more than in a review will come across the crash?

You should fail it yourself if you were able to crash it and give as much information as you can on the crash. The other people reviewing it should be on the look out for the crash but if they can't find the crash they can still pass it. If two or more people got it to crash it's a real problem and each of them will have failed it so it will fail review.

If it really is a super rare crash that only one person was able to find one time they could resubmit the game without fixing it and you probably would not no or prove the crash it still exists.

Meta: The Microsoft guidelines state that the user needs to be able to pick up any controller to play the game. I think in that context it means that the first controller to start the game needs to be any controller and you covered that.

I do also think, however, that if you are not going to allow the 2nd, 3rd etc players control the menu you should state that somewhere to avoid confusion by the users.

Meta:

I didn't see anything addressing this issue here..  In the evil list it says that any controller has to go through the menu, but in my experience I don't want to do that.  Now with that said any controller can come in and press star the first one to do so is player one.  But we are only allowing player one to navigate the menus.  The reason is if anybody has had played games with kids they like to press buttons so I only want to have player one to be able to pass through the menus.  So the question is, is that going to make my game fail peer review?  Its an easy task to make is so it uses any controller to pass the menus and everything else is navigated by any controller.

Here is some background:

2 player game

on start screen any controller hits start that controller becomes player one

then in menu only player one can navigate the menu

then choose character screen it says press start on second controller that controller is then set to player 2

after player 2 is selected (made available) they can do everything player one can except navigate the menus.  Ie skip through the high score and credit screens

If someone has a guitar connected as controller 1 they can't play the game which is why it is considered a fail.

If you want only one controller controlling the menu you are allowed to have the first screen of your game say "Press Start" then whichever controller presses start controls the menus throughout the rest of the game.
You can't just assume controller 1.

Loco: What if the crash was the result of YOUR xBos having a problem? I remember when Valcary (spelling) was being reviewed, one of the players ended up having crashes right and left during playtest. The dev tried and tried to reproduce and couldn't. No other players could repo either. It ended up that the testers xBox was screwed. Once he got it fixed (can't remember the problem) Valcary didn't crash anymore. Valcary had not been updated between when it crashed and when it didn't for this user. They fixed their xBox and played the same version and it quit crashing. In a case like this what is the dev to do?

I would rather fail peer review 10 times than have my name attached to a game which crashes!  I'm minded to agree with LoclPuyo.  If you find it, fail it.  If it's real others will fail it too and the dev will have to address it.  Multithreading is the same as any other "tool" we have in the framework - you have to use it right or it will screw up or crash. 

Regards,
Mike

There was a time I would have faught for this more. That time was back when I thought that a single fail could knock you out of review. Since it takes more than one fail to knock you out, I am ok with it. If it crashes for you it fails.

Now if you seem to fail a lot of games because of this you might check your xBox for problems ;-)

hotshot 10101:

Loco: What if the crash was the result of YOUR xBos having a problem? I remember when Valcary (spelling) was being reviewed, one of the players ended up having crashes right and left during playtest. The dev tried and tried to reproduce and couldn't. No other players could repo either. It ended up that the testers xBox was screwed. Once he got it fixed (can't remember the problem) Valcary didn't crash anymore. Valcary had not been updated between when it crashed and when it didn't for this user. They fixed their xBox and played the same version and it quit crashing. In a case like this what is the dev to do?

In the rare cases that happens only the 1 person with the problem would have failed it, games don't fail peer review until 2 people fail them. So the system has it covered.

From my understanding it only takes 2 fails in the same category to fail the game. What if another reviewer deemed that your game has a code defect because it doesn't work how they think it should (ie, no exit for your game, game continues after purchasing the game, etc). That's 1 strike on the code defect category. Then a reviewer comes across an error that they can't reproduce. Strike 2 on the code defect category and your game is out of there. The system is unforgiving in that regard.

If other reviewers are finding similar errors with your game then they need to fail it. But if no one else, not even the one reporting the error can reproduce it after spending half an hour or so on it, then it should not be a fail in my opinion.

If your game is crashing a lot and other reviewers are encountering crashes as well then yes, the reviewer should fail it.

I want my game to be bug/error free as well the next dev, but I also don't want to be chasing a white rabbit for months on end for something no one is able to reproduce and especially if it is a special circumstance, like the storage device being removed during saving, that will rarely (if ever) happen.

If more than one person encounters a similar error that is hard to reproduce then the game should be failed, but I also need to know how they got the error so I can have a better idea of narrowing it down. That is not always something that is done.

Newly released AAA games crash, some crash a lot. The rare crashes eventually get fixed after hundreds of thousands of players play the game and a few of them find the rare crashes and report them. When you have more than one person with the same issue it is much easier to reproduce and narrow it down.

We do not have that many players to review our games.

You are entitled to fail a game if it crashes once, but I simply cannot in good conscience do the same. I will however report the issue to the developer.