We are constantly rejected on controller issues. We are much gave up by now. Our game was finished in late August and it is still being rejected every time we submit.
We tried submitting to play testing 3 times but have not got any clear technical reviewing.
This is the last fail message we got:Sorry, had to Fail your game.
 
"A code 4 error occurs right after you purchase the game using controller 4 and you are prompted to sign in. I used an xbox live silver account. I am able to reproduce the error every time using different controllers.
 
Steps to reproduce
-------------------
launch the game with controller 1 or any other controller (no profile signed in). Controller 2 has the main cc profile signed in.
Press start using controller 1
choose Buy Game from main menu
Press A on the Please Login! screen
Choose a different Xbox live account than your main one. I chose an Xbox live silver account
Choose Yes, Simulate purchase
 
The result is a Code 4 error.
 
 
The issue only occurs if you are prompted to sign in to an xbox live account and choose to simulate purchase right after. If an xbox live profile (silver or gold) is already signed into the controller, the error does not occur."

I mean how is it possible to take care of every single possible way to crash a game with this controller stuff? I dont have 4 different accounts to test everything. I believe its been over 10 fails by now. And I kept fixing every fail reason every single time, and people just keep coming up with something new and original. This looks to me more like a game I (and my poor artist) are playing with peer testers "find a way to crash a game"...

(On a side note: why such a "testing" even needed? We are not coding a software for a spaceship launch. Its a single player game, it passes Evil Chevklist controller reqs...)

Can you advise anything I can do in this situation? I see all kinds of games, even these screen savers and massage apps make it to market with no problems at all.

p.s. I am sorry if I was abit harsh there. You must understand it is very discouraging to see the fruit of 3 months of hard work never even make it out there...
 

If one of the peer reviewers found the bug without 4 accounts then I assume you could have found it too.

It seems like your issue is that when a bug is found you fix that ONE bug, wait 7 days and resubmit. You are using the peer reviewers as testers not as reviewers. If you spent the 7 days restesting your game you might have found all of these yourself. Peer review is a validation of your testing not the actual testing.

And BTW if its crashes with a code 4 it most certainly doesn't pass the evil checklist... I think Crash==fail is right at the top of the list.

There's over 600 games made it through - do you think yours is the exception or do you think that maybe, just maybe you could have tested a little harder?

The ZMan:

If one of the peer reviewers found the bug without 4 accounts then I assume you could have found it too.

It seems like your issue is that when a bug is found you fix that ONE bug, wait 7 days and resubmit. You are using the peer reviewers as testers not as reviewers. If you spent the 7 days restesting your game you might have found all of these yourself. Peer review is a validation of your testing not the actual testing.

And BTW if its crashes with a code 4 it most certainly doesn't pass the evil checklist... I think Crash==fail is right at the top of the list.

There's over 600 games made it through - do you think yours is the exception or do you think that maybe, just maybe you could have tested a little harder?

testing is all we've been doing for past 3 months...

Mihalych:

testing is all we've been doing for past 3 months...

And yet apparantly peer reviewers are still finding code 4 crashes - trivial ones given the above repro case.. If it is a game called 'make the game crash and fail' you are really not making it very ahrd for the peer reviewers.

Making a single player game handle all the combinations of controller and buying is trivial - I think it took Geoerge less than a day to code it into Kissy Poo and I can test all the scenarios in under half an hour by logging in and out and trying each controller. You must be doing some really odd stuff if you can't test it that quickly and find the issues.

One other thing I've noticed is a lack of playesting and peer reviewing on your part (unless you are using another account). If you had spent time doing those you would have undoubtedly learned much more about the types of test cases that are important to test for.

I think trying hard enough, you can crash pretty much any game or application. As I quoted that last guy, he used 2 different accounts to purchase a game and mess with it.. well he can probably pull out memmory card at right moment or something else like that and get a code 4. Endless possibilities.

I dont have a premium account, so I can review other games (that is not an excuse just stating)

I honestly dont have a clue how test every possible aspect of controller-code 4 related issues.
p.s. less than a day to code it into Kissy Poo...  very good game, working controller too.

[The correct game is Gooney Crew Day 1 http://forums.xna.com/forums/t/37566.aspx submitted by Black 59 Raxzor]

Mihalych:

I think trying hard enough, you can crash pretty much any game or application.

Go crash Kissy Poo. In 3 playtests AND review nobody crashed it once. Oh yeah there were crashes while we coded it but from day 1 it supported multiple controllers/logins and purchasing and we tested the hell out of it (including givieng the controllers to many under 5's for hours at a time). And if you do I will thank you... I certainly won't post complaints to the forum making excuses for my bugs. I'm ashamed of every one of them.

Mihalych:

As I quoted that last guy, he used 2 different accounts to purchase a game and mess with it..

Standard test.... and many users only have one gold account. Silver accounts are free to create so that setup will take you about 5 minutes to recreate and not cost you a penny. Seems like you also force a login when they buy and he had to use the silver account becuase the gold account was logged in elsewhere. Seriously this is not even a complex test case. I'm not sure what you have been testing for 3 months but spending an hour in the review forums and reading the evil checklist better would have given you this scenario.

Mihalych:

I dont have a premium account, so I can review other games (that is not an excuse just stating)

Complaining about the people who give their time for free when you don't show them the same respect is really not how the community works.. and like I said if you had been doing it then you would learn. I see rece does do some reviews - if he is a developer on your game then I fail to see how he didn't know about these test cases. We've been failing games for these reasons for a year now.

Mihalych:

well he can probably pull out memmory card at right moment or something else like that and get a code 4

Not if you have excpeption handing in the right places... have you tested yanking the MU at all possible places yourself? If not then your game has no place in peer review yet.



Bottom line here is that when someone finds a bug in your game, especially a trivial one that every other game out there has solved you say 'Thanks for spending your time helping me test' and not claim that its impossible to test. Remember every bug found now is one a customer didn't find - they are not out to get you, they are not being malicious they are doing their job (for free) and they are finding your mistakes and omissions.

My other partner is an artist. He knows about coding just about as much as nothing. I dont know who is this rece guy, my artist has a different account name. I cant also see the link to the game you posted since again I dont have premium acc.

Well if you just want to make me look bad you make a good job at that. I understand all the stuff about free time and that I should be thankful etc. etc. But most people do it for either rep, for fun, or to just free up the queue to their game. Nobody is forcing them to sit out nights reviewing either. If I had premium I would be doing reviews too and not expect to get cake for that.

But anyway that was not the point of my original post was it?

rece is the guy who submitted DotError - thats the game we are talking about right? This is the link in his sig http://iamrece.com/dotError
Sorry I had the wrong game, I found a game that had a very similar fail to what you described so I thought it was that.
The correct game is Gooney Crew Day 1 http://forums.xna.com/forums/t/37566.aspx submitted by Black 59 Raxzor

I'm not trying to make you look bad. When you submit a game to Indie Games you are submitting a game for customers to buy. You are no longer a guy coding for his buddies and to look cool. You are a professional developer. That means you need to accept there is a cost - creators club memberships, controllers to test all the scenarios, MU to test those becuase thats what your customers have. Go get a premium membership so that you can at least talk to the people who are reviewing your game instead of having someone else do it for you.

The point of your original post is implying that the reviewers should not be failingyour game, that they are somehow out to get you when they are doing exactly what is expected of them. You have bugs in your code, trivial bugs that should not be in any game in peer review and you want to blame someone else for finding them.  Bugs are the fault of the developer and nobody else. Suck it up and fix them instead of posting in here. I'm 100% sure I'm annoying you telling you this but thats what it takes to be a serious professional developer. Remember 600+ games already got this right...

Whether you look good or bad is determined by how you behave in this community not by me pointing out the obvious.

The ZMan:

Gooney Crew Day 1

GoonyCru: Day One.

The ZMan:

The point of your original post is implying that the reviewers should not be failingyour game, that they are somehow out to get you when they are doing exactly what is expected of them.

Dont put words in my mouth I never said that.
What I said was that I am having countless controller issues that I am trying to resolve and failing to see the end of it. I was looking for advice.
Mr ZMan if you have nothing better then give me "look others did it why cant you" , "l2code nub", and"quit whining" advices, then I suggest we end this discussion.

Ok I paraphrased..  Here's what you said..

I mean how is it possible to take care of every single possible way to crash a game with this controller stuff? I dont have 4 different accounts to test everything. I believe its been over 10 fails by now. And I kept fixing every fail reason every single time, and people just keep coming up with something new and original. This looks to me more like a game I (and my poor artist) are playing with peer testers "find a way to crash a game"...

None of your fail reasons are new and original - its the same old reasons we've been failing games for a year. You dont need 4 accounts for any of those issues. Iff 600+ games can take care of them then logic says its possible and 'find a way to crash a game" is exactly what peer reviewers are supposaed to do - well actually its your job first.

My advice was really just to try to help you not go through it all again... but I guess you'll just keep on failing until the reviewers find all the bugs you can't be bothered to find yourself.


But here's the technical advice - there's only 2 samples of code you need both of which have been discussed many times in these fourms

1. How to detect and use the current controller
Scan all 4 controllers for (start) and remember the one that pressed it. Use that player index and no other EVERYWHERE else in your code. 

2. How to properly test if someone can buy a game - I suspect this is your issue.
http://xnawiki.com/index.php?title=How_do_I_tell_if_a_player_can_buy_a_game%3F

I can be frustrating to deal with many of these small, unlikely to occur issues that cause crashes, but they do need to be fixed... and it should not be that hard. Zman has some good links there for you to look at. Most of the crashes with controllers that I found had to do with non-signed in controllers trying to do things that only signed in users could and users attempting to purchase items without having a gold account. Find those spots in your code and make sure to check if they will cause an error before performing those opperations and you should be fine.

The memory until removed stuff is easy enough to fix too... just check and make sure you have a valid save container before saving or loading from it.

Mihalych:

I mean how is it possible to take care of every single possible way to crash a game with this controller stuff?

My software testing professor would probably tell you that it is impossible to find 100% of the bugs in a piece of software.  But he would also tell you that is possible to get to a point where further testing would seem futile, as no other bugs are being found.  Your game doesn't seem to have arrived at this point, as reviewers are easily finding bugs in it.  That usually indicates a flaw in the testing process being used.

Mihalych:

I dont have 4 different accounts to test everything. I believe its been over 10 fails by now. And I kept fixing every fail reason every single time, and people just keep coming up with something new and original. This looks to me more like a game I (and my poor artist) are playing with peer testers "find a way to crash a game"...

This is probably the flaw in your testing methodology.  For one, you don't have the tools necessary to fully test the software.  It isn't expected that reviewers have all the hardware needed to fully test games, but it is expected that developers have everything.  Like has been previously stated, a game in peer review is expected to be "perfect".  So, the reviewers 'will' subject your game to all kinds of crazy edge case scenarios.  So, get the controllers you need to test all 4 controller slots.  You should have at least one gold account for testing.  You can use local and silver accounts for the other 3 slots in testing.

There is a form of risk based testing called quick tests that is exactly what these edge cases are.  They are a series of easy to perform tests that will alert the testers if software is vulnerable to certain classifications of bugs.  If you fail a quicktest, then your code is likely to have more bugs of the same type.  If you pass all the quick tests, then it is likely that you designed the code well enough to avoid other bugs in the same area and further testing may not be worth the time.  So, the fact that you keep failing for controller issues means that the testers are likely to target more and more controller based edge cases.  You will have to make sure your controller handling is emaculate.  It also means that other things are probably not going to get tested as hard until your contoller code gets passed.  You might find yourself even more frustrated when you stop getting failed for controller issues and start getting failed for MU issues, for example.

You have a list of these quick tests in the evil checklist.  You have even more examples of quicktests in the review forums, where people are posting detailed information about how other games are failing.  I really recommend getting a premium account and spending some time just reading the playtest and peer review game thread.  Find out what people are getting failed for and test your game with the same conditions.  And get creative with those tests.  If you see that the test isn't a multiple controller test in itself, but could be applied to multiple controllers, then try that.  Try every variation that you can think of. 

Mihalych:

(On a side note: why such a "testing" even needed? We are not coding a software for a spaceship launch. Its a single player game, it passes Evil Chevklist controller reqs...)

A bug is a bug.  If you have one and it is reproducable, you shouldn't pass review.  Pure and simple.  You might not think your software is as important as a space shuttle launch, but the gamers who demo and purchase your software won't be any less annoyed by your game crashing, even if it only once in a blue moon.  The peer review process is also an imperfect system of volunteers.  The standards have to be extremely high in order for it to work.

The evil checklist is a good list of possible bugs, but it isn't the end all list.  It isn't even an official list.  The user that originally posted it, did so to help the reviewers find common issues.  It just happened to be very useful, so the mods stickied it and the community has been adding things to it bit by bit. 

Mihalych:

Can you advise anything I can do in this situation? I see all kinds of games, even these screen savers and massage apps make it to market with no problems at all.

Peer reviewers are getting better at their job.  Like I said earlier, it is impossible to find 100% of the bugs.  It is possible that some of these earlier games got through peer review with bugs similar to the ones you have now.  But, that doesn't mean your game deserves to get through.  If anything, those other games should be pulled (and they can be if enough folks complain about issues through the report channels).  It is going to become harder and harder to get untested games through peer review as reviewers become more and more armed with possible tests to perform.

My only big piece of advice is what I said earlier.  Get all the tools you need.  Those tools are the hardware needed to test all permutations of setups (4 regular controllers, special controllers like guitars, MUs, etc), the accounts needed (gold and premium memberships) and the list of tests that you can use.   The bug list is going to be a dynamic resource.  Start with the evil checklist and expand it as far as your imagination can take it.  Then go through every playtest and peer review thread that you can and find fail reasons that are not listed in the evil checklist and expand upon them as far as your imagination can take them.

Ok if you have had 10 people fail you don't you think there might be something wrong with your so called testing?

Mihalych:

(On a side note: why such a "testing" even needed? We are not coding a software for a spaceship launch. Its a single player game, it passes Evil Chevklist controller reqs...)

Any code 4 at all is an instant fail. Its the first thing on the Evil Checklist. And no matter how many times you have put your games up it will always be a fail.
Making a game handle all controllers properly is easy to do, you don't even have to support controller change (I always do).
Just search for those issues in the forums and you will find many solutions...

Also memory card crashes and all others can all be fixed.
We want to raise the quality bar not lower it which is what people like you are trying to do.

Just spend an hour and fix the bugs in your game, also how many people have you got to test your game that havn't worked on it? I always get about 10 of my freinds to test my games (2 at a time) and tell them to try and break it. You should do the same by the sounds of it.

I think the main point here is that these games are still products that people will be spending their hard earned money on. Imagine buying a car that turned out it wouldn't go if you had a passenger in the back seat. You wouldn't be very happy, and I don't think an explanation of "well hey, it's not a spaceship launch" would comfort you very much. I think everyone here knows how much of a dull job it is to make sure all the fiddly bits work just so, when you really want to get your game out there, but at the same time, you don't want to release your game only to be plagued by dissatisfied customers who have had it crash on them. There's plenty of advice around on how to test all the different controller combinations, so you might be best making a testing plan and methodically working through it. Sit down with a piece of paper, and try to think of every combination of events / controllers, then work through your list and try each one.

Danthekilla:

We want to raise the quality bar not lower it which is what people like you are trying to do.

Well its good to see that you want to make better game than those but you are not trying very hard. 

The reason people are getting annoyed here is because you don't seem to understand that testing this is very very easy (It is in no way complex enough to need to hire someone) All you need is an xbox and a memory card (which is the apsolute bare minimum to even try and make a game with anyway) you already have an xbox so go and buy a $15 memory card (it will cost you much less than paying someone to find your bugs).

And once you have one why don't you start with the case you just thought of?
 

Mihalych:

"What happens if I press A on 4th controller logged in with my grandmas account with my left foot, pull out MU, and do a mamba dance"

Try that. If it crashes fix it, if not try your right foot or maybe the 3rd controller.

Do this until you actually can't think of any more (you said you have been testing the game for 3 months, but it sounds like you ment to say 3 minutes)
The testing the happens in peer review tests about 1% of the stuff you should have tested yourself and it is not by any means an actual quality assurance program (these cost $20,000+)

If you buy a memory card and whatever else you need and spend just 1 day on it you will have your game fixed and ready to go in no time at all.

Mihalych:

I mean how is it possible to take care of every single possible way to crash a game with this controller stuff?

You could start with more robust exception handling.

Jim Perry:

Mihalych:

I mean how is it possible to take care of every single possible way to crash a game with this controller stuff?

You could start with more robust exception handling.

I absolutely agree here with using Nick Gravelyns exception handling technique.  I'm sure I saved weeks, if not MONTHS of bug testing due to it.  It helps narrow down exactly when and why there was a crash.  It doesn't tell you everything but it tells you WAY MORE than a popup message stating "Code 4".

And the best part:  You can make use of it RIGHT NOW, even though your project is 'complete'.  It's so easy to add.

Hi Mihalych,

It seems like you really are trying to test your game. Maybe you´re new to (XNA) programming or just not as good at it as some of the more experienced people here on the forums, or maybe you´ve just had a streak of very bad luck where you didn´t catch all these errors. I can see how it must suck to be failed so many times.

Here are some suggestions:

- Post a list of your test cases in a forum thread and ask for advice. I bet the community will help you out with some other things to test that you hadn´t thought of. For example, making multiple silver accounts is very simple and costs no money.
- Use Playtest and specifically say that you are having difficulty finding all the bugs in the game and would like people to do some hardcore peer review testing. It may take a while to find someone willing to go through their whole checklist and do a full write up (maybe even a few weeks) but in playtest, especially if you ask for it, people will be much more inclined to do a full test of your game and write down all the fails they can find. In review most reviewers just stop when they find one thing.
- Ask for and take a look at some of other people´s code for handling multiple controllers, purchasing the game, and those difficult things. There have been some links posted in this thread already. "Try Catch" blocks are your friend!

And at the end, think of it this way, it may be a bummer that your game has been failed 10 times, but thanks to the system you have removed TEN bugs from the game! So if you do get past peer review it will be a lot better for it. :)

Good luck!

Mihalych:

I have no experience with "what happens if I click on 4th controller logged in with my grandma account with my left foot at the right moment, pull out MU, and do a mamba dance" thing.

Testing well is hard - actually harder than development.  The case you describe is a combination test case.  Cover all the individual combinations first.  Can I use controller 1, 2, 3, and 4?  Can I use my creators club account, my grandma's account, a silver account, a local account, no account?  Do both my right foot and left foot work?  Does it work with and without an MU?  Does it work if I do a mamba, a samba, or a rhumba?

You'll find a lot of bugs just touching once on each possibility without worrying about combinations.

After you touch every feature, look at where problems occurred and try combinations in those areas.

The previous advice to playtest and review games yourself is really good.  You get practice testing (it's a different mindset) and get to see the common pitfalls.  While reviewing I learned about a guitar controller bug that would have failed my game and was able to fix it before review.

Keep a list of your test cases and rerun them before submitting your game for playtest or review.  It takes a while, but it's worth it.  Add to the list as new problems are dicovered.  My current list is appended.

Hope this helps.

1.1       Creator’s Club checklist

·         Evil Checklist

·         Not-So-Evil Checklist

1.2       Feature test

·         All onscreen buttons

·         All menu selections

·         Play games with all settings

·         Play all sound clips

·         Test all input options (both thumbsticks, D-pad)

·         Trial mode

·         Normal mode

·         Network game

·         Make sure network host options override client options

1.3       Fresh machine

·         Test trial mode on machine with no saved options

·         Test non-trial mode single player game on machine with no saved options

·         Test non-trial mode local game on machine with no saved options

·         Test non-trial mode Xbox Live host game on machine with no saved options

·         Test non-trial mode Xbox Live client game on machine with no saved options

·         Test non-trial mode System Link host game on machine with no saved options

·         Test non-trial mode System Link client game on machine with no saved options

1.4       Networking

·         Test with mid-range and poor network quality settings

·         Make sure code recovers from any network exceptions

·         Disconnect DSL router during network game

·         Make sure player leave picture changes propagate to clients

Steve Roe:

Disconnect DSL router during network game

Wouldn't this just throw a Code7 error, since your XNA CC account would also be disconnected from the internet?

To the OP: a lot of the fails you're talking about are for code that should only be written once and then reused for every scenario, eg privilege checking code when buying the game and wrapping IO operations in try/catch blocks. I agree with the above posters - testing your game is almost as difficult as writing it, but once you have code that works and only use that code instead of rewriting the same thing for every screen you'll use it in, 90% of possible fail reasons are removed. The rest is going through checklists and creatively trying to break the game.

If any of my games are failed due to a crash, I'm truly grateful. I'd much rather spend another week polishing and testing then have a gamer run into it, and not only potentially losing that sale, but also having them think less of me as a developer and the service as a whole, even if it is so unlikely to occur it would only happen to that one person.

I feel your pain. Aardvark had a lot of controller issues too. I think the frustration is in that there are plenty of people peer reviewing and not many people playtesting. If a game fails playtest, you can upload a new version immediately. But if if fails peer review, even if it's a 3 second fix, you have to start all over. I would run my game through playtest for 3 weeks and get no feed back at all and then fail peer review. It is enormously frustrating. Testing is difficult and testing your own game is even harder. You'll play the game over and over and never find an issue that a playtester would find in 10 minutes. Aardvark would probably still be in peer review if it wasn't for the few people testing, and one extremely helpful tester in particular (thanks again PinoEire!).

My suggestion, which didn't prove popular last time I mentioned it, is for everyone to try to give back. If someone tests or has tested your game, see what they are up to and see if you can return the favor. This will encourage everyone to test many different games. Find the games that aren't popular. Find the games that no one is testing. But it only works if more people try to give back. If we don't all help each other, the community is kind of pointless.

Hang in there if there's enough of us with the "give back" mentality we can make the process smoother.

With regards to the costs of development:

Three extra Xbox controllers will cost you $40 a piece wired (probably less if you buy them from a used game store instead of new retail.) That's $120 maximum.

A memory unit will cost $30.

One year of premium membership costs $100.

That's $250 total. Assuming you sell your game at the 80 MS point level (~$1), it would only take 357 sales to recoup your costs of development.

...and if you don't think at least 357 people would find your game to be worth the cost of a bottle of Coke, why would you release it?

If you are worried about the fiscal costs of Xbox development then it might be better to release your game for PC first and see what kind of reaction it gets. But in reality, XBLIG development is stupidly cheap, easily the least money it's ever cost to develop for a console. A WiiWare development license costs $2000, and ever since they ditched Linux, PS3 doesn't even have a solution for independent developers.

UberGeekGames:

Steve Roe:

Disconnect DSL router during network game

Wouldn't this just throw a Code7 error, since your XNA CC account would also be disconnected from the internet?

True - that's a pointless test.  Went overboard in that category after I hit some networking bugs, should delete that test case.