I thought I would put thread up discussing feedback on the Creators Club. I put a post up on my blog here:

 http://blog.spouting-tech.com/thepensivegamer/2008/06/xna-creators-cl.html

But I've posted it below as well.

-------------------------------------------

My immediate suggestions:

• Don't require games to be in a final state before it can be downloaded; hobbyists need feedback for incomplete games. Introduce the idea of a 'feedback' (think alpha/beta) release that is available to premium members. In this state premium members aren't performing a review for acceptance for download by the entire Live community but instead performing a test so feedback can be given to the developer. When creating a 'feedbk' release the developer can place a note requesting what in particular they would like feedback on.
• Allow games to be downloaded from the dashboard when in the 'feedback' or 'review' states. The need to download to a PC and then to a 360 seems unnecessary.
• Allow a convenient way for premium members to give feedback to the developer during 'feedback' and 'review' states. Developers currently have to create a root forum post under a particular community forum; it would be nice to see this more integrated with the game profile page and have feedback actively solicited from premium members who have downloaded the game.
• For publicly released games allow any Live user that has played the game to give a review/rating of the game. I personally like the idea of rating various components of the game (i.e., story, graphics, game play) as well providing an overall value. This is a feature that, in my opinion, should be available on ALL content on the Marketplace.
• Add a 'reputation'-style system for users that provide feedback and ratings. In essence this allows developers to rate the feedback given to their games. The reputation should be focused on how constructive the criticism is. Instead of a 'star' rating (as seen on Gamertags), perhaps it is word-based such as 'Helpful', 'Not Constructive', etc. The reputation is meant to discourage users from leaving non-constructive messages like 'yer game sux' while also minimizing the risk of reputation retaliation when a developer doesn't like a review.
• The user profile should include the list of games developed and reviewed/rated. Any feedback or review given to other games should be shown so other users can easily browse and read the feedback/ratings.
• Allow any game, regardless of state, to be viewable from the Creator's Club web site. For example, I cannot view game details when a game is in the 'In Process' state.

Now on the longer term side, I would suggest:

• Enhance the game and premium member profile pages. The information on the current pages is sparse. Allow the developers to enhance them. I would heavily consider a member controlled cross between a wiki and blog. The pages should be easily constructed; the developer can choose widgets to display. Such widgets could include picture/video galleries, blog entries, release schedules, top reviews (though all reviews should always be accessible, in my opinion) etc. Naturally some of the information should be structured so that it can be viewed in the Xbox dashboard or in a mobile front-end (think Live Anywhere).
• Provide a source code repository system similar to SourceForge ; perhaps some form of coupling into CodePlex or MSDN's Code Gallery (without forcing projects to be open source).
• Provide an easy way to solicit help. This could evolve into a job-board style mechanism.
• Expand the Creators Club to include libraries (graphics, shader libraries, etc).

Bottom-line: I think the idea of community games is fantastic. But I would love to see the whole community be engaged more. XNA Creators Club has the potential to provide a fantastic social experience for creating games, let alone playing them.

You know what I'm going to say.... its my favourite thing to post right after 'thread locked' and 'No legal advice in the fourms'

To ensure that the right people see this make sure to post it to http://connect.microsoft.com - in your case maybe separate issues are needed.

Sure, I expected that. I'd be more than happy to file a bunch of suggestions but my first goal here is to initiate a discussion. There is nothing better than putting some things down and letting the community discuss. Singular ideas from a person may be good but with feedback and discussion from others, it can grow into something bigger and even more interesting. Yes Microsoft Connection does allow for the community to discuss BUT the location is a step away (people generally aren't going there is frequently as these forums) and if you look at filed bugs the back-and-forth from the communicating is light when compared to many threads here.

Anyhow, if new ideas spawn from this I'll gladly file some additional suggestions in; I'd love to see the Creators Club really blossom!

 

Here's my long winded response to everything you said.

Talvish:

Don't require games to be in a final state before it can be downloaded; hobbyists need feedback for incomplete games. Introduce the idea of a 'feedback' (think alpha/beta) release that is available to premium members. In this state premium members aren't performing a review for acceptance for download by the entire Live community but instead performing a test so feedback can be given to the developer. When creating a 'feedbk' release the developer can place a note requesting what in particular they would like feedback on.

I like the idea of getting feedback, but I don't think it should go out to the public. The fact is that this would require lots and lots of peer review because each time it goes out you have to be sure there is nothing illegal or inappropriate in it. I could see value in having the Creator's Club website allow you to tag releases as feedback where CC subscribers could then download it from the website, deploy to the 360, and then give feedback using the website without the intention of publishing to the marketplace, but involving the public would require lots of review which would quickly become impractical and annoying to reviewers. Plus this could affect the public view of the channel. If Joe User downloads a bunch of unfinished games, he may start associating the Community Games channel with unfinished games and then stop downloading. Even with a separate section you run this risk.

Allow games to be downloaded from the dashboard when in the 'feedback' or 'review' states. The need to download to a PC and then to a 360 seems unnecessary.

This is because you cannot write a review from the Xbox 360. In addition the review process is only for Creator's Club subscribers to avoid people gaming the system (e.g. I upload a game and have 30 of my buddies just say it's good even if I'm breaking every rule). It's a control thing and a logical thing. I don't think this needs to change.

Allow a convenient way for premium members to give feedback to the developer during 'feedback' and 'review' states. Developers currently have to create a root forum post under a particular community forum; it would be nice to see this more integrated with the game profile page and have feedback actively solicited from premium members who have downloaded the game.

Agreed.

For publicly released games allow any Live user that has played the game to give a review/rating of the game. I personally like the idea of rating various components of the game (i.e., story, graphics, game play) as well providing an overall value. This is a feature that, in my opinion, should be available on ALL content on the Marketplace.

Also agreed. User ratings is what will save this system I think. It will help users quickly identify what's good and what's bad which will hopefully prevent that dreaded case of Joe User downloading 30 terrible games and then deciding that all XNA games are terrible, thereby ignoring the whole channel.

Add a 'reputation'-style system for users that provide feedback and ratings. In essence this allows developers to rate the feedback given to their games. The reputation should be focused on how constructive the criticism is. Instead of a 'star' rating (as seen on Gamertags), perhaps it is word-based such as 'Helpful', 'Not Constructive', etc. The reputation is meant to discourage users from leaving non-constructive messages like 'yer game sux' while also minimizing the risk of reputation retaliation when a developer doesn't like a review.

That would be pretty nice. It could just be something numeric where User A posts a bad comment or review, other users rate that comment or review down, then next time User A posts a comment, it starts at a lowered rating. Then users could filter out any comments or reviews with lowered ratings and thus avoid the bad comment/review. Of course this is a LOT of work, so I wouldn't expect to see it in the first release, but it would be nice in time.

The user profile should include the list of games developed and reviewed/rated. Any feedback or review given to other games should be shown so other users can easily browse and read the feedback/ratings.

I believe that users should be able to see all the games I have developed and reviewed, but I do not think that the actual ratings and comments made during review should be available to anyone but that game's developer.

Allow any game, regardless of state, to be viewable from the Creator's Club web site. For example, I cannot view game details when a game is in the 'In Process' state.

I believe it was stated as a bug, so the team is working on this one.

Now on the longer term side, I would suggest:

Some of the above items would have to be longer term as well because they require a lot of planning, design, and back-end support.

Enhance the game and premium member profile pages. The information on the current pages is sparse. Allow the developers to enhance them. I would heavily consider a member controlled cross between a wiki and blog. The pages should be easily constructed; the developer can choose widgets to display. Such widgets could include picture/video galleries, blog entries, release schedules, top reviews (though all reviews should always be accessible, in my opinion) etc. Naturally some of the information should be structured so that it can be viewed in the Xbox dashboard or in a mobile front-end (think Live Anywhere).

You have to be careful with what content you allow. I would bet money you will never have a blog on this site. Otherwise your content is now Microsoft's responsibility. Simply link your blog in your gamer profile and if people care, they'll go there. Then I still don't like the idea of customizable game pages. I prefer being able to look at any game and know exactly what information I can find and exactly where it will be. Once you start letting people shake things up it becomes harder to find certain information.

Provide a source code repository system similar to SourceForge ; perhaps some form of coupling into CodePlex or MSDN's Code Gallery (without forcing projects to be open source).

Also I doubt eyou will see this. That would require massive amounts of server back-end to accomplish. They would need not only the actual disk space, but the RAM and CPU power to make sure this would function for everyone while using the service. I'm sure there are other source repositories you can find online that don't require open source, but even if not, I don't think this is the job of Microsoft to handle for us.

Provide an easy way to solicit help. This could evolve into a job-board style mechanism.

There already is a Help Wanted section of these forums. There's also the Help Wanted over at GameDev.net. I don't think there is a need for any more functionality.

Expand the Creators Club to include libraries (graphics, shader libraries, etc).

Written by whom, exactly? The XNA team? The community? There's a large list of community links on this site where you can find lots of projects, Ziggyware.com always posts about new libraries and things, and Google searching will yield lots of results as well. I don't think this is the team's job because everyone has their own preference as to how things should work, how things should be optimized, and so on. I think this is one of those things where it sounds incredibly useful, but then would just disappoint most users. Not to mention that if the team did it, that would be additional time away from enhancing all the other stuff (framework, Game Studio, and Community Games) and would be a huge resource hog in terms of time and money because they would have to have it all go through the full Microsoft review process.

Bottom-line: I think the idea of community games is fantastic. But I would love to see the whole community be engaged more. XNA Creators Club has the potential to provide a fantastic social experience for creating games, let alone playing them.

Me too. Good thing we're still in the beta so hopefully that end-user experience on the Xbox is improved. I think that the website and developer side is pretty good for the time being (reviews need to have a mandatory minimum time on them to allow for more reviews), but that end-user experience will make or break this. User reviews, categories, a section for recent games, highest rated games, and more would all enhance this for the end-user and make them feel much more in control and more likely to return to download more games.

My thoughts on your thoughts. :)

 

Talvish:

• Don't require games to be in a final state before it can be downloaded; hobbyists need feedback for incomplete games. Introduce the idea of a 'feedback' (think alpha/beta) release that is available to premium members. In this state premium members aren't performing a review for acceptance for download by the entire Live community but instead performing a test so feedback can be given to the developer. When creating a 'feedbk' release the developer can place a note requesting what in particular they would like feedback on.

I disagree. The pipeline should be for finished games only. If a developer wants feedback on an alpha or beta version he should use his own space. The Live Marketplace and soon the XNA Community Games section are already jammed up enough that MS is semi-pulling the less-than-popular games. We don't need more items clogging up things. That would mean additional filtering if a person doesn't want to see the alphas/betas. Sure the default filter could handle this, but it would also require development time to add that filter. People are complaining enough about the speed of things getting released (I'm not one of them, I think the team has done a ton of work given the time XNA GS has been alive).

 

Talvish:

• Allow games to be downloaded from the dashboard when in the 'feedback' or 'review' states. The need to download to a PC and then to a 360 seems unnecessary.

The ability to only download games at a certain time during this beta is a little frustrating. The need to download to a PC first is a good thing (see my first comment about jamming the Marketplace).

Talvish:

• Allow a convenient way for premium members to give feedback to the developer during 'feedback' and 'review' states. Developers currently have to create a root forum post under a particular community forum; it would be nice to see this more integrated with the game profile page and have feedback actively solicited from premium members who have downloaded the game.

I agree. Remember this site is still in beta so the only way to provide feedback is by the functionality that's in place. MS has already stated members will get their own little space.

Talvish:

• For publicly released games allow any Live user that has played the game to give a review/rating of the game. I personally like the idea of rating various components of the game (i.e., story, graphics, game play) as well providing an overall value. This is a feature that, in my opinion, should be available on ALL content on the Marketplace.

As long as the ability to navigate the reviews is easy on the user, I agree. The review areas for submission that's in place now is a nice way of seeing pieces. The additional parts for things like story, graphics, etc could be added to the list of areas that are now being rated.

Talvish:

• Add a 'reputation'-style system for users that provide feedback and ratings. In essence this allows developers to rate the feedback given to their games. The reputation should be focused on how constructive the criticism is. Instead of a 'star' rating (as seen on Gamertags), perhaps it is word-based such as 'Helpful', 'Not Constructive', etc. The reputation is meant to discourage users from leaving non-constructive messages like 'yer game sux' while also minimizing the risk of reputation retaliation when a developer doesn't like a review.

This gets into a very subjective area. What's to stop a person from providing a "Not Constructive" feedback when it actually is.  Abuse of any reputation system is possible. Personally, I don't care who the person is that provides feedback as long as it helps make the game better.

 

Talvish:

• The user profile should include the list of games developed and reviewed/rated. Any feedback or review given to other games should be shown so other users can easily browse and read the feedback/ratings.

I personally don't care what games a person has reviewed, just what they've done so I can find a particular person's games.

 

Talvish:

• Allow any game, regardless of state, to be viewable from the Creator's Club web site. For example, I cannot view game details when a game is in the 'In Process' state.

Agreed.

 

Talvish:

• Enhance the game and premium member profile pages. The information on the current pages is sparse. Allow the developers to enhance them. I would heavily consider a member controlled cross between a wiki and blog. The pages should be easily constructed; the developer can choose widgets to display. Such widgets could include picture/video galleries, blog entries, release schedules, top reviews (though all reviews should always be accessible, in my opinion) etc. Naturally some of the information should be structured so that it can be viewed in the Xbox dashboard or in a mobile front-end (think Live Anywhere).

I agree. Again, remember this is a beta.

 

Talvish:

• Provide a source code repository system similar to SourceForge ; perhaps some form of coupling into CodePlex or MSDN's Code Gallery (without forcing projects to be open source).

While I agree in theory, this should be an item that gets the "implement if free time allows" label. I'd love to see some integrated team-based collaboration system.

Talvish:

• Provide an easy way to solicit help. This could evolve into a job-board style mechanism.

There's already a Help-Wanted/Offered forum. I think this is sufficient. What more is needed?

Talvish:

• Expand the Creators Club to include libraries (graphics, shader libraries, etc).

I agree, but they shouldn't be in the Marketplace. Again, an "as time allows" type item IMO.

Hey Nick, my further comments...

Nick Gravelyn:

Talvish:

Don't require games to be in a final state before it can be downloaded; hobbyists need feedback for incomplete games. Introduce the idea of a 'feedback' (think alpha/beta) release that is available to premium members. In this state premium members aren't performing a review for acceptance for download by the entire Live community but instead performing a test so feedback can be given to the developer.

I like the idea of getting feedback, but I don't think it should go out to the public. The fact is that this would require lots and lots of peer review because each time it goes out you have to be sure there is nothing illegal or inappropriate in it. I could see value in having the Creator's Club website allow you to tag releases as feedback where CC subscribers could then download it from the website, deploy to the 360, and then give feedback using the website without the intention of publishing to the marketplace, but involving the public would require lots of review which would quickly become impractical and annoying to reviewers. Plus this could affect the public view of the channel. If Joe User downloads a bunch of unfinished games, he may start associating the Community Games channel with unfinished games and then stop downloading. Even with a separate section you run this risk.

Ya, we don't want this public, so if I wasnt' clear ,when in a feedback state the item isn't available for ALL live members, just the premium members of the Creators Club. In other words, peer reviews don't play a role. If there is no way to get feedback reviews on the 360 directly (more on that below), then it would be no different then how folks now have to download the game package and deploy via Visual Studio.

Nick Gravelyn:

Allow games to be downloaded from the dashboard when in the 'feedback' or 'review' states. The need to download to a PC and then to a 360 seems unnecessary.

This is because you cannot write a review from the Xbox 360. In addition the review process is only for Creator's Club subscribers to avoid people gaming the system (e.g. I upload a game and have 30 of my buddies just say it's good even if I'm breaking every rule). It's a control thing and a logical thing. I don't think this needs to change.

I understand the notion of not being able to give feedback on the 360 BUT in order to get recognized that you gave feedback in the system you need to go back to the website. The goal here is to ease getting feedback and review releases into people's hands. Again, this would only be for premium members of the Creators Club, so any gaming already exists in the current model, and only those members would be able to download. 

If MS doesn't want the early versions accessible via dashboard (again controlled so they can only be seen by premium members, so shouldn't be a legal issue any more than it is to allow downloading on the PC website) then the other option is to build a Visual Studio extension that allows you to see feedback/review releases easily, download and deploy them and give feedback (right in Visual Studio). That would also give what I was looking for ... ease getting it into the hands for testers/reviews and ease getting feedback back out.

Nick Gravelyn:

The user profile should include the list of games developed and reviewed/rated. Any feedback or review given to other games should be shown so other users can easily browse and read the feedback/ratings.

I believe that users should be able to see all the games I have developed and reviewed, but I do not think that the actual ratings and comments made during review should be available to anyone but that game's developer.

If you mean that users cannot see feedback for games that are not fully released ... in my opinion it is a bit of a toss-up. I think ratings should really only be for final products (not feedback releases), but feedback itself is not that different than having a forum that people can discuss in. I'm a huge fan of being transparent. That said, I would hate people being discouraged to build a game because some unkind person left horrible remarks...but even in the forums there is a risk of that.

Nick Gravelyn:

Enhance the game and premium member profile pages.<cut>.

You have to be careful with what content you allow. I would bet money you will never have a blog on this site. Otherwise your content is now Microsoft's responsibility. Simply link your blog in your gamer profile and if people care, they'll go there. Then I still don't like the idea of customizable game pages. I prefer being able to look at any game and know exactly what information I can find and exactly where it will be. Once you start letting people shake things up it becomes harder to find certain information.

I agree that there needs to be something 'standard', perhaps they can only edit/add to certain sections. But in terms of blog, in essence I am saying that you could in fact treat this as Live Spaces but around the gaming community (think social network for game developers). Not everyone would use it, I definitely agree. But for the new guy starting off it may be convenient to have it all in one place.

Nick Gravelyn:

Expand the Creators Club to include libraries (graphics, shader libraries, etc).

Written by whom, exactly? The XNA team? The community? There's a large list of community links on this site where you can find lots of projects, Ziggyware.com.

These are libraries created by anyone, just like the games are created by anyone. I agree there are community sites like Ziggyware that link over, but it does make it a convenient place to find libraries that a developer may want to have/use. Libraries could be rated, similiar to games. Games that use libraries could, on their profile page, optionally link back and give kudos to those libraries etc. All part of the community process.

Anyhow, that is some of the thinking behind what I wrote if that helps clarify or stirs any additional thoughts. Best part of the community, being able to discuss it! :)

Jim Perry:

Talvish:

• Don't require games to be in a final state before it can be downloaded; hobbyists need feedback for incomplete games. Introduce the idea of a 'feedback' (think alpha/beta) release that is available to premium members. In this state premium members aren't performing a review for acceptance for download by the entire Live community but instead performing a test so feedback can be given to the developer. When creating a 'feedbk' release the developer can place a note requesting what in particular they would like feedback on.

I disagree. The pipeline should be for finished games only. If a developer wants feedback on an alpha or beta version he should use his own space. The Live Marketplace and soon the XNA Community Games section are already jammed up enough that MS is semi-pulling the less-than-popular games. We don't need more items clogging up things. That would mean additional filtering if a person doesn't want to see the alphas/betas. Sure the default filter could handle this, but it would also require development time to add that filter. People are complaining enough about the speed of things getting released (I'm not one of them, I think the team has done a ton of work given the time XNA GS has been alive).

Talvish:

• Allow games to be downloaded from the dashboard when in the 'feedback' or 'review' states. The need to download to a PC and then to a 360 seems unnecessary.

The ability to only download games at a certain time during this beta is a little frustrating. The need to download to a PC first is a good thing (see my first comment about jamming the Marketplace).

Hey Jim. It is obvious I wasn't clear on the fact that this games are only available to Premium Members of the Creators Club. So if MS didn't make the feedback/review releases of games accessible via the Dashboard, it would be done like it is now, available for download off the PC website. Now if they WERE to make it available via the Dashboard it would require a UI update. Seeing feedback/review releases in the dashboard would only be available for those that are Premium Members of the Creators Club and could be under a different heading. That would minimize clutter in the list of games and make it obvious that this is an incomplete and potentially unstable game that is being retrieved. Personally I would love to see a tighter integration between the dashboard/Live and those that are Premium Creators Club members. That is one example, but I would also love to see a 'badge' or something in my GamerCard too :).

Jim Perry:

Talvish:

• Expand the Creators Club to include libraries (graphics, shader libraries, etc).

I agree, but they shouldn't be in the Marketplace.

Right, this would never show up in the Marketplace.

Jim Perry:

Talvish:

• Provide an easy way to solicit help. This could evolve into a job-board style mechanism.

There's already a Help-Wanted/Offered forum. I think this is sufficient. What more is needed?

Given current volumes, things are sufficient. Once you start getting a lot of people you need a easy way to find people that have skills, and an easy way to find what skills people are looking for. Trying to browse through a bunch of forum posts is hard when there are a lot of people. So I would start by allowing people to indicate what skills they happen to have as part of their profile. Then allow searching the members for people that have certain skills. Next is having a structured way of posting what type of skills you are looking for for a project. Then other users can search the posts and potentially subscribe (e.g. via RSS) to requests for help for certain skills.