I'm sorry but is anyone else sick of this.

Had a game Forum thread absolutely full of passes and generally good feedback... only to receive my rejection email today which was from two fails I couldn't even respond to because they don't "have to" write in the forum thread.

Not only this but because it doesn't even tell me who failed the game I can't even ask for clarification & further feedback in the forum. Peer Review is supposed to be "useful to development". Writing in a box saying "This fails" and then not posting anything in the forum is about as much use to game development as Jack Thompson is to a lawsuit.

At very least the notification of the fail should be posted to the forum... so I'm not waiting for that last bit on the progress bar to go only to fail for what seems to be no reason until an email from nameless failers come through.

As annoying as failing to make the cut is... this is not annoyance at people who fail games... failing is necessary... But at least pop a message on the forum...

Time for another week long wait :/

If the failer didn't specify a reason (ie: "this fails.") then you should seek the official channels and report this behaviour. You may not know who did this, but MS will.

Write an email to Creators@microsoft.com, and they'll be able to tell you exactly what checkboxes were checked by the ones who failed. Of course, they won't be able to tell you their names...

Catalin Zima:

Of course, they won't be able to tell you their names...

No, but if a reviewer is just putting "This fails" in the textbox in their review they should be smacked around a bit by the team. That's just unacceptable and that person should be reviewing.

Jim Perry:

No, but if a reviewer is just putting "This fails" in the textbox in their review they should be smacked around a bit by the team. That's just unacceptable and that person should be reviewing.

Of course, but the XNA guys take care of these things privately

Catalin Zima:

Of course, but the XNA guys take care of these things privately

This is true. We prefer to stay quiet. Like the mafia.

I'll look into this. Thanks for the heads up.

xxoo
~kathleen sanders
XNA Community Manager

ExecutiveIguanaStudios:

Writing in a box saying "This fails"

Ok let me clarify... no official rules have been broken here (The XNA Mafia don't need to get involved) :P... I believe my wording may create some confusion,

In the above statement I meant writing in the box for failure... not a problem with the description of the fault per se.

More a problem that after reviewing and failing the game, that no Forum feedback has been left. It leaves creators like ourselves sat watching the thread seeing the Bar keep raising and waiting for the release of our game, when suddenly despite no fails listed in the forum, the game bombs out of review and tells me it has been rejected.

I saw no reason why this would be until after quite some time an email arrives with descriptions from 2 people failing the game.

The failure of the game is not my concern, the people failing the game had a fair point, and it will be ammended.

The problem I have is that if you do fail a game Leave a message in the forum post. It allows direct contact between reviewers and creators, and it means affirmitive action can be taken to fix problems better and more in tune with what the player is expecting.

Not quite as bad are posts like this that I see..

**Failed Code 4***
If Memory Unit Removed
Game Crashes

Now forum posts like this at least notify a creator in advance that the game is in trouble, however if the game only crashes in this circumstance then I presume the player has actually played the game without causing this crash.

Why not then leave comment on the actuall game play and comment on improvements you think could be made to that? Also as you've listed this fault, is it the only fault you found... or did you stop reviewing as soon as you saw this one.

If the comment read more like this...

**Failed Code 4***
If Memory Unit Removed
Game Crashes

I did enjoy X part of the game but feel that it could be
 improved a little by adding more Y and a little less Z.

The above was the only crash I found during a thorough review.

I believe the above to be a much more helpful post. After all I was lead to believe it was a Peer review, not a Peer list the first fault I found and leave out any further testing. If you worked in QA and your idea of a thorough run through was stopping at the first error and giving no other feedback you wouldn't have a Job very long.

I think thats my rant over.... :)

--------------------------------------------------------------------------------------------------
On a completely off tangent sidenote, are there any PC games to test on this site, I'd sorely like to be part of the development process of other games as well, but albeit I'm the member of the team most active on the forums, I'm also the member who doesn't own an Xbox(shameful I know, you can guess where my first games moneys will be going).

I noted that for "Playtest" PC is an option, but just haven't seen any PC games in there.

Your rant just isn't valid. These guys aren't being paid to QA your game. They're doing it for free on their own time. If I find a single error in Peer Review, I stop. It's a failing offense and it shouldn't be there.

I do agree that in Playtest it's useful to give more feedback, but this isn't Playtest, it's Peer Review. In Peer Review you're not looking for feedback (in fact, we discourage people from leaving detailed feedback in the Peer Review forums). When you submit to Peer Review you're saying your game is gold and that you aren't changing a single thing and that YOUR OWN QA staff (this might be your brother, aunts, uncles, yourself and your buddies or maybe you actually pay for people to QA) has already vetted and verified the game.

You need to change how you're looking at Peer Review. The reviewer was doing the expected action here... 

The only thing I will agree on is that posting your fail in the forum does give that line of communication that a developer may need. So yes, I would encourage all developers to post their failing bugs here for that reason and to give other reviewers the chance to verify the bug and fail it as well.

ExecutiveIguanaStudios:

If the comment read more like this...

**Failed Code 4***
If Memory Unit Removed
Game Crashes

I did enjoy X part of the game but feel that it could be
 improved a little by adding more Y and a little less Z.

The above was the only crash I found during a thorough review.

I believe the above to be a much more helpful post.

The whole "... but feel that it could be improved..." shouldn't even be a factor in peer review. The improvement stuff should have been handled in playtest. Your game should be as perfect in being bug-free and gameplay balance and fun as it can be.

I think that George has a valid point with the "find one bug and quit". If even one bug is found in peer review it shouldn't have been there in the first place. Peer review is to verify that nothing is wrong with the game, not find bugs. If a bug is found you need to test more.


As for the PC games, there are none. The playtest and peer review pipelines are for 360 games. Some people do make PC versions available somewhere and post links but that's not what this site is for.

Here's a case where being a good forum member comes in handy at least for how *I* review games.

If you are the sort of developer who thinks playtest is optional because they are better than everyone else, if you are the sort of developer who rarely shows their face in the forums other than to submit a game, if you are the sort of developer who trolls/argues/causes trouble, if you are the sort of developer who uploads shovelware once a week, if you are the sort of developer who is on their 5th upload to peer review because of trivial bugs/bugs on the checklist, if you are the sort of developer who cries like a baby when they get a fail THEN when I review your game I will stop as soon as I find the first fail. Its not worth my time to help you beyond that.

If you have a 5000 reputation, help people, try hard to make good quality games, use playtest properly, treat fails as bugs that you should have found and deal with them humbly THEN when I review your game I'm likely to play beyond the first bug and try to find as many as I can. In addition if I missed your game in playtest I will likely give you the feedback you want.


Everyone reviews differently but stop after a first fail is very valid for the reasons Jim and George outlined.

Yes I agree its more helpful if people post in the forums but not everyone is comfortable with this. Developers can help with this by not ranting like someone murdered their family when they get a fail. Yes occasionally there is a mistake and the mods/microsoft will deal with those as they happen.

A for PC games - I don't know why more people don't. We released our game outside the peer review forums to anyone with even a trial account and a PC build (with a timeout) and we got good feedback from that. Its linked in the sig, though unless you have young kids its probably not your cup of tea.

George Clingerman:

Your rant just isn't valid. These guys aren't being paid to QA your game. They're doing it for free on their own time. If I find a single error in Peer Review, I stop. It's a failing offense and it shouldn't be there.

I do agree that in Playtest it's useful to give more feedback, but this isn't Playtest, it's Peer Review. In Peer Review you're not looking for feedback (in fact, we discourage people from leaving detailed feedback in the Peer Review forums). When you submit to Peer Review you're saying your game is gold and that you aren't changing a single thing and that YOUR OWN QA staff (this might be your brother, aunts, uncles, yourself and your buddies or maybe you actually pay for people to QA) has already vetted and verified the game.

You need to change how you're looking at Peer Review. The reviewer was doing the expected action here... 

The only thing I will agree on is that posting your fail in the forum does give that line of communication that a developer may need. So yes, I would encourage all developers to post their failing bugs here for that reason and to give other reviewers the chance to verify the bug and fail it as well.

I agrree Playtest Should be the place for this kind of thing but i've found it to be fairly redundant... I did post the games for playtest but just got mad at seeing the game attract very little attention.

I've seen many games in playtest for a week or more with nobody playing them(or nobody leaving feedback anyway). Peer review just seems the fastest option. (And the Last Game I posted was gold) no bugs found just a difference of opinion as regards trials, however I had a few more tentative passes so it should probably be addressed anyway.

However in light of your statement i'll throw the game up for Playtest over the next week to see if I get any response this time.

Zman:

If you are the sort of developer who thinks playtest is optional because they are better than everyone else, if you are the sort of developer who rarely shows their face in the forums other than to submit a game, if you are the sort of developer who trolls/argues/causes trouble, if you are the sort of developer who uploads shovelware once a week, if you are the sort of developer who is on their 5th upload to peer review because of trivial bugs/bugs on the checklist, if you are the sort of developer who cries like a baby when they get a fail THEN when I review your game I will stop as soon as I find the first fail. Its not worth my time to help you beyond that.

If you have a 5000 reputation, help people, try hard to make good quality games, use playtest properly, treat fails as bugs that you should have found and deal with them humbly THEN when I review your game I'm likely to play beyond the first bug and try to find as many as I can. In addition if I missed your game in playtest I will likely give you the feedback you want.

I'm sorry if i'm reading you wrong but i'm a little confused... are you talking hypothetically here or is that me personally.

ExecutiveIguanaStudios:

I'm sorry if i'm reading you wrong but i'm a little confused... are you talking hypothetically here or is that me personally.

I apply this to pretty much every game *I* review so its a general answer, if I'd reviewed Life I would have followed what I normally do. In this case you did go to playtest before and I did miss it so likley I would have given you more feedback. However I don't see you as a particularly active member of the community so its possible I would have fast failed if I had found a bug too. Then there's the 'how much time do I have' and the 'how much fun is this game' factor... sometimes I just don't have the time or the interest to play beyond the first fail.

ExecutiveIguanaStudios:

I agrree Playtest Should be the place for this kind of thing but i've found it to be fairly redundant... I did post the games for playtest but just got mad at seeing the game attract very little attention.

I've seen many games in playtest for a week or more with nobody playing them(or nobody leaving feedback anyway).

Yet another reason to be more active and visible in the forums - checkout the forum for KissyPoo - we got lots. George and I are well known, do a lot of reviews and playtests etc etc. We also twitter, facebook, pop into IRC and drum up support for our playtest that way.

This process needs your peers... if your peers don't know who you are then your playtests and reviews will mostly be done by other people trying to revieew games as fast as they can so THEIR games get attention. They are very likely to fail fast and have little incentive to playtest or help with your game.

Do you playtest/review under a different account? Otherwise the whole rant is a little hypocritical....

Well I'd love to be more active than just submit games believe me, I think this place is brilliant.

However we are currently a 2 man team. The artist (Who also owns the team Xbox,which breaks a lot) has unfortunately currently got pretty heavy time constraints for the time being.

I'm the programmer, (Yes it's me to blame for the bugs :P). I sadly do not own an xbox, and my request to set up a charitable trust to buy me one was sadly turned down by her majesties government :(.

So at the moment I will admit, we are a bit lacking on the review front ourselves. We are expecting this to change shortly with time constraints being lesser on my artist within the coming months and I am hopefully to have ammassed the sum of monies required for my very own xbox (THATS ONE EACH :D) this christmas.

I'd also like to be more active in the forum, and I think i'm getting there slowly, at the moment i'm a little reluctant to dig in and try to help with a lot of issues at the moment due to my programming really being in its infancy. I will possibly update and have a look at twitter and IRC and things, but I do have a lot of work to do for University at the present time.

--------------

To shorten it and put it simply, the games we are submitting are ones that i created in my first year at uni that we are upgrading to conform to Xbox live and earn us a little extra money for christmas(For my up and coming Xbox Hopefully). We are currently working on a more complex game as well that we expect to begin playtesting the beginings of fairly shortly.

We are generally quite good with the community aspect and were very heavily involved with the Red faction Mapping and Modding community, and I'm sure we will soon be contributing here, but it may be a couple of months before we have the time to sit and play through some games, and really start making a contribution other than our own (And Frankly Fantastic) games :P.

Trust me when my Xbox arrives I don't see myself doing much more with it other than playtest and reviewing games... I love that stuff :P

I think the point in short was, don't complain about something you yourself aren't even doing. It makes you look bad.

There's no requirement to post on the forums. Looking at the reviews though there were fairly indepth reasons for the fail which you should have received in email. I can't tell right now if this title went into Playtest, if not then it may have been caught then...

Doesn't the submitter receive an email with the fail reasons that were submitted by the reviewer? That is what I understand from reading Pheel's post. When I fail a game i write out in the where did this occur section exactly what was wrong etc. Do they get my actual response or is it rewritten by someone behind the scenes? As if it really matters to me I was just curious. as you can tell I am not afraid to also post in the forums why I failed, which sadly has at times felt like I opened a hornets nest.

I am perplexed as to why people will not write in the forums for fails or passes for that matter. I had my game pass with like two responses in the forums. I know I want people to see that I am reviewing their games so that when the time comes, and they see my game in review they will remember I reviewed them.

Yes whatever the reviewer types in the fail box is emailed. This thread was confused becuase we thought the failer had typed 'game failed' into the box instead of a useful description... we eventually worked out that was not the case. The developer was wanting to get more feedback and also to have a dialog with the reviewer and the ghost reviewers don't allow that.


People don't post for many reasons, some folk feel that developers overreact to fails. Some folk feel they might get picked on if they fail for the wrong reasons (though if you do this you will get caught whether you post or not). Some people may just not feel comfrtable posting in public.

The ZMan:

Some people may just not feel comfrtable posting in public.

Which I don't understand. It's not like this is really that public. :|

Hi guys,

I decided to post into this 3d because in the past I downloaded that game called Life and after playing it for a while I decided to not submit a review it because I found it boring. Is this, in your opinion, a wrong conduct? I know I'm a little OT but I was just curious.

That said, being only two working on a game isn't an excuse to do not reviews/playtest. I have , for istance,a full life with lots of things to do (xna developing, daily job, charity work, my niece babysitting, my cats and dogs to feed and take care of, friends, some social life and some gaming session) but in this mess of things I try to find some time to review/playtest something, as my reputation says I do not playtest/review tons of games but I try something every month for sure.

edit:
several time I was so slow in playtesting/reviewing that the game got approved or pulled out... LOL!

kittie:

I decided to post into this 3d because in the past I downloaded that game called Life and after playing it for a while I decided to not submit a review it because I found it boring. Is this, in your opinion, a wrong conduct?

Not submitting a passing review is never wrong. You're under no obligation to pass any game, especially one you thought was boring. I can't count the number of games that I looked at and never submitted a review because I couldn't fail it. A vast majority of the games on the service are crap IMO and I can't imagine why people are buying them.

kittie:

Hi guys,

I decided to post into this 3d because in the past I downloaded that game called Life and after playing it for a while I decided to not submit a review it because I found it boring. Is this, in your opinion, a wrong conduct? I know I'm a little OT but I was just curious.

That said, being only two working on a game isn't an excuse to do not reviews/playtest. I have , for istance,a full life with lots of things to do (xna developing, daily job, charity work, my niece babysitting, my cats and dogs to feed and take care of, friends, some social life and some gaming session) but in this mess of things I try to find some time to review/playtest something, as my reputation says I do not playtest/review tons of games but I try something every month for sure.

edit:
several time I was so slow in playtesting/reviewing that the game got approved or pulled out... LOL!

I'm sorry you found the game boring and voicing your displeasure with it in the actuall game thread would have been nice rather than out here to throw some slander of my game accross the forum, seems like an off topic swipe to me, when the initial purpose of this thread was simply to voice displeasure at the fact a game that seems to be passing, fails for reasons that the developer isn't made aware of until after the fact.

I'm also none to appreciative of the "look how many things I can do, why can't you attitude", which seems to be a very 'Holier than thou' attitude which you shouldn't really take when you don't know the full circumstances of my team. Also I think the reason I gave, "I do not own an Xbox" is a pretty good one.

-----

Back to topic, Could a system be implemented where a forum post is made annonymously as soon as a fail reason is given? Or at very least an email sent to the developer. As a pose to the system waiting for the game to be completey failed before dropping the rejection letter in my inbox.

It would help both developers and the peer review system I feel.

For example:

I test my game, see no bugs and put the game up for Peer review.

A fail Post is made, that I must aggree is a valid fail. If so I might as well then pull the game and begin working on the required bug fixes. That way as soon as the bug is discovered the game is pulled and it stops wasting the time of

A/ Myself, waiting for the game to be released rather than fixing it.
B/ Peer reviewers who are really wasting their own time, if someone has already discovered a bug and just not posted it.

In the current setup the first of the two fails that forced my game to be pulled could have been made as soon as the game became available. I can't know this because, I receive no notification of the first fail until the second fail is posted (Possibly Days Later) and the game is pulled.

I suppose this is more of a gripe with the system than the Peer reviewers themselves but i'm hoping you can see my point in how this would be beneficial.

Zman:

Yes whatever the reviewer types in the fail box is emailed. This thread was confused becuase we thought the failer had typed 'game failed' into the box instead of a useful description... we eventually worked out that was not the case. The developer was wanting to get more feedback and also to have a dialog with the reviewer and the ghost reviewers don't allow that.

It's true whatever the reviewer types in the fail box is emailed would probably be enough not to create confusion. However whatever the first failing reviewer puts in the box isn't sent as an email until a second reviewer has failed the game.

This waste many peoples time because the game should probably be pulled after the first fail and it could be days before a second fail actually arrives. Or Possibly even worse. One person may find a simply horrific crash that nobody else does, If that person doesn't write in the thread and the game then gets passed by all other reviewers a faulty game could then be released on to the system.

Which is bad news for everyone.

Pretty much of us here all agree with you - posting in the forums is preferred, your reasoning makes sense too. But right now its not mandatory. Adding a request to http://connect.microsoft.com/ is the right way to change things.

Kitt1e was just asking about if not leaving feedback is allowed - that was on topic for where the thread was going. There's no slander in there and you are the one who started with the list of reasons you can't playtest. A statement like that is going to get a response - almsot nobody in these forums does this full time.

Please tone down the attitude... remember my list of 'how not to get your game reviewed and tested'?

ExecutiveIguanaStudios:

Could a system be implemented...

Any system could be implemented. Feel free to make a suggestion.

ExecutiveIguanaStudios:

This waste many peoples time because the game should probably be pulled after the first fail...

Not necessarily. What if it's a bad fail (which has happened many times). There's valid reasons why the system is as it is.

ExecutiveIguanaStudios:

and it could be days before a second fail actually arrives.

So? That's the developers problem since there shouldn't be any fails when a game is submitted for peer review.

Seems like a lot of nitpicking over nothing.

Jim Perry:

So? That's the developers problem since there shouldn't be any fails when a game is submitted for peer review.

I think you've missed my point. As we all well know, some bugs do escape and come out in Peer review in many titles that have been submitted the service. However it's not just a problem for the Developer. Peer reviewers have limited time to play and test games due to whatever other personal things they have going on in their life.

What I mean by pulled after the first fail is... If the developer agrees it's a legitimate fail and the game should not be released in that state. Then the developer should pull it themselves. However if no feedback is left in the forum after the first fail the developer does not know that a fault has been found. This can mean that the game can be sat on the peer review for a considerable amount of time and both Peer reviewers and developers are unaware of an already discovered fail and will waste time reviewing a game that already has a bad fault rather than reviewing one that is more ready for the service.

I would like to thankyou for the link though and will be making a suggestion to the effect of the following..

I feel it would be useful to both the Developers and Peer reviewers if after the first failing of a game is made that some form of communication is sent to the developer so that problems may be addressed more quickly than waiting for the second fail to come in before sending the Developer any notification.

I believe this will cut the amount of time faulty games are stood on the Peer Review servive and will also be beneficial to any developer wanting to make the changes required to the game.

The Message could be sent to the developer over any of these possible options...

• Making it Mandatory rather than optional to leave a forum post after a game has been reviewed.
• Sending an Email to the Developer as soon as the first fail has occured rather than waiting for a game to completely fail before sending one.
• Having the option to view the fail reports on the games info page.

If any of the above three changes were made I believe it would be beneficial, to both developers and Peer reviewers. I strongly believe that this small change will benefit the running of the XNA community on the whole.

Just thought I'd write it out here to get some feedback on it before sending it down the official channel. Do you think that it is an acceptable suggestion?

*EDIT*
I just noticed that you and Zman have sent me two different links for official suggestions to be submitted. Which of the two would it be best to submit such a request to. Or should it be send to both?

They end up at the same link... Follow Jim's it has full instructions but you will ultimalty use http://connect.microsoft.com/