Hmm... I guess I assumed that because the new dash supported enumerating xna framework 2.0 games, that it would also include the XNA GS Connect, or that said would be published when this dash hit, but I don't see it anywhere. Are you guys really holding off on that until 2.0 rtms? Couldn't it be relased as a beta and then updated when the rtm hits?

(If I bash my head on my desk one more time I think I might miss the 2.0 launch through being unconscious)

No GS Connect until 2.0 launches, no beta of it on the 360. Doing a beta and an RTM version would mean more work getting it through certification twice, which in turn would delay when RTM actually came out.

That reaction was a tad over-dramatic. Do they not have a path for beta code to be released through marketplace? Do successive betas of, say, COD each need to be recertified? Seems a bit obtrusive.

If you'd read the same question as many times as I had you'd perhaps understand the dramatics. Any code, beta or otherwise is gonna have to go through certification. Map packs, patches, probably even picture packs and themes will need to be certified. So I would say its a pretty safe bet that any updates to the COD4 beta that there were were certainly all certified separately.

Yeah, it does seem a bit obstrusive, and I suspect Epic Games will run into some walls in their attempts to get 'modded' maps etc on the 360 version of UT3 before it releases next year, but at the end of the day its all to ensure the experience is as easy as possible. No crashes, no bugs, no panicking when somehow the beta version of connect causes the 360 to inadvertantly delete all content from a memory device... you get the idea :).

EatenOne:

Do they not have a path for beta code to be released through marketplace?

We do, but the nature of our product is such that we probably won't ever release a public beta for it. A big part of our console runtime is isolating your game from the rest of the system in a virtual sandbox. I don't think many people here are comfortable with the idea of releasing a beta or "incomplete" sandbox.

EatenOne:

Do successive betas of, say, COD each need to be recertified?

Yes.

 

Anyone have any info as to how these certifications are performed? Code review? QA Testing? Doesn't sound practical for a game of any significant size. Not sure how patches ever get released at all if a full certification is required each time a game is patched. Is automated unit and integration testing in vogue in the game development circles yet?

That was the answer that I was anticipating. But since using that sandbox requires a connection to LIVE, its not as if people would be able to avoid updating said sandbox as security patches were released. So it doesn't seem like any exploits would remain valid for long. The same risks will exist once the product is fully rtmed and certified. Oh, well ;-) guess I need to wait a bit to try out all these fancy LIVE api's on the real deal. Thanks for the answers.

EatenOne:

Anyone have any info as to how these certifications are performed? Code review? QA Testing?

The standard certification process involves a mixture of QA testing along with some automated verification that uses specially instrumented versions of the system software. It's a laborious and expensive process.

For products that are deemed to have a particularly high security risk, of which the XNA Framework most certainly is one, we also perform additional checks such as threat modelling and code review. That makes it even more laborious and more expensive than usual.

EatenOne:

Doesn't sound practical for a game of any significant size. Not sure how patches ever get released at all if a full certification is required each time a game is patched.

It's not at all easy. But on the plus side, this gives developers an incentive not to make mistakes! On a platform like Windows where you can quickly and easily put out any patches you like, it's easy to think "hey, we have a bunch of bugs, but whatever, we'll ship anyway and then fix them next month". On a console you have to factor in that you are going to have to pay a lot of $$$ to certify that patch, so maybe it would make more sense to just hold back the release and do things properly in the first place :-)

EatenOne:

Is automated unit and integration testing in vogue in the game development circles yet?

I don't believe most game developers make much use of automated testing. Within the XNA Framework team, we do have very thorough automated unit tests. That's a separate thing from the certification process, though. The unit tests prove to us that our code works as expected, but the certification team will perform their own independent tests to convince themselves that it performs as they require. The system would be too easily exploited if the same team that made the product were also responsible for writing the tests that are used to certify it!

Jamie Fristom of XNA and Schizoid fame is one guy i know who blogs on unit testing for games http://www.gamedevblog.com/

For a view of cert you can read this article... the article is really trying to paint a bad light on the employment of the folk who do the certification but you can read between the lines and see some of the cert process.

http://www.seattleweekly.com/2007-07-11/news/testing-video-games-can-t-possibly-be-harder-than-an-afternoon-with-xbox-right.php