All,
In the summer of 2004 we had rudimentary networking in Blockrealm . We submitted the game to the connected systems developer competition and the competition judges were able to move and place blocks in a connected world. In addition, voice over IP support (implemented by Anthony) allowed players to communicate to one another. This simple networking functionality was achieved through DirectPlay. Six months later Directplay was deprecated-- DirectPlay, it was said, would be soon replaced by a .NET Windows version of LIVE for XBOX. Three years later this promise has come to fruition; however, there are limitations present in the Xna.Net API that did not exist in DirectPlay. I would like to share with you the things I have learned and, based on this knowledge, what I think the best approach is going forward.
It is said that in building an MMO, the networking infrastructure is as equally difficult as building the 3D engine. I personally do agree, and to address this difficulty we have used technologies that are easy to use but uncommon to the game industry. Specifically, we used AmazonS3’s distributed storage network to store mass amounts of game data. Our use of this storage network allows us to handle player data, player-to-player content sharing and news dissemination. Similarly, using the simple IRC protocol we designed a solution for scalable player-to-player chat. IRC servers run by Amazon EC2 may be connected for increasing scalability to support hundreds of thousands of users.
While the aforementioned functionality is complete, one area of MMO architecture we have not addressed is real-time player interaction. For example, what happens if a player moves a block and a new player logs in and receives stale world state from AmazonS3? What system orchestrates monster AI and attack sequences? We put things on hold and entered a holding pattern to see what functionality would be provided through XBOX Live.
XBOX Live via Xna.Net offers a .NET centric API for building network and online games. In terms of ease-of-use there is no better solution available to us and equally important, this is a solution that runs on Windows and the XBOX 360. However, Xna.Net has limitations which in theory prevent its use in a massively multiplayer game. Here are the obstacles as I see them:
1) The Xna.Net host support a maximum of 31 players. While this number could change in the future, for the duration of the XNA 2.0 lifespan this number is unlikely to change. In the XNA forums one developer writes, “I think the problem is that for every 10,000 "I'm going to write an MMO" posts, you might get one or two who actually can make MMOs. They are nothing trivial and therefore not something the XNA team should waste time supporting.” To this end, the XNA team does not want to build or support a networking API that could conceivably serve as a foundation for an MMO architecture. The prevailing logic is that if we want this functionality we should use System.Net. However, System.Net does not run within the XBOX360 XNA security sandbox. To use System.Net means to abandon hope of using player profiles, the marketplace, and player-to-player chat services associated with LIVE. In short, if we use XNA.Net the 31 player limitation and peer-to-peer LIVE architecture is something we have to live with.
2) Only one game server instance may run per physical machine. One approach to overcome the 31 player limit is to virtualize our servers and run 1 to N game server instances on our rented game servers at 1&1; however, because of the one game server limitation in Xna.Net this is not a viable solution.
3) XNA System.Link connections do not allow connection of computers outside a single subnet (local LAN). The XNA System.Link connection type is ideal for what we are trying to accomplish. However, it does not allow connections across the Internet.
4) To use XNA LIVE to enable peer-to-peer connections across the Internet our players must have the following (as of 12/14/2007):
1) An XBOX 360 to register for a LIVE Silver or Gold Membership and
2) a LIVE Silver or Gold Membership to purchase an XNA Creators Club Membership.
3) An XNA Creators Club Membership ($99/year) to enable LIVE networking for a game title.
4) An XBOX Gold Membership ($59/year) for LIVE networking across the Internet, and lastly
5) The fee or subscription model we require on-top of aforementioned services.
For the typical XBOX player requirements 1 and 2 are reasonable; 3 is not tenable. For a Windows player only requirements 2 and 5 are marginally realistic expectations for end users. Rumor is that the XNA/LIVE team is working on a content delivery system that removes some of these obstacles. However, there is no mention of schedule or an anticipated date for completion. Not even a triple AAA Windows title could succeed in forcing users to jump through the hoops necessary to play a game with Internet connectivity.
5) LIVE connectivity requires that a title completes quality requirements (TSRs) established by Microsoft. We would have no alpha or beta of a networked version of our game; instead, we would have to wait until Micrsooft approves a near-complete version of the title for use on LIVE. Our ability to release is determined by whether our game meets playability standards set by Microsoft. Titles such as Habbo Hotel and Second Life, while immensely successful MMO games, would likely not pass such metrics. I do not expect our game to be any different due to the evolving nature of MMO quality, gameplay, and complex
ity.
So these are the obstacles to using XNA.Net. If you can believe it, I still think we should use Xna.Net. What I propose here is that we continue use of Amazon Web Services and supplement it with XBOX Live for real-time player-to-player interaction.
Many of you know that at present our game worlds are generated client-side using a unique hash. This integer determines the shape of the terrain, location of trees, presence of mines, quarries, shops, and common area. We support as many worlds (“Jungle, Forest, Desert, Swamp, Barren, etc”) as values exist in a 32-bit integer (4294967920). We know that even with our portal system only a small fraction of worlds can or should exist. To date, we have a limit set at 1000 world areas. So the first step to using Xna.Net is that we scale back the size of each world so that its size is appropriate to 31 players. This means that we need to reintroduce the artificial “non-passable” mountains we were using earlier this year.
How would this look in the code? The NetworkSession class in Xna.Net allows a game host to specify custom properties. For example, when a server is created it can broadcast custom information stored in the form of integers. (Convenient for us, isn’t this?)
So when a player enters a world or walks through a portal to a world, the client searches to see if a game is in progress with the said Int32 hashcode. If the network session exists the player can join an in-progress game, connect to AmazonS3 to download the player-built world, and then begin play. If the network session does not exist the client serves as the host and create a game ad-hoc with permissions for users to join the game in-progress. This approach is deterministic and easily accomplished using our current game architecture. When a player exits one world through a portal and enters another we use the Xna.Net host migration feature to transfer the host to a new peer transparently. If no peer exists the world is saved to AmazonS3 and the session ends.
There are corner cases to this use case that will cause problems, none of which are insurmountable. A. What happens if the last host loses connection before writing to Amazon S3 (accidently or purposefully)? B. What happens when a player tries to walk through a portal to a world with 31 players? C. Is Pandora’s box opened by running critical game logic client-side – how will this be exploited by users?
A. We modified the byte stream so that we send as little information as possible to represent a world. I think Mihail has it very close to the theoretical minimum. 1MB of data can represent complete ephemeris for over 40,000 objects (position, rotation, type, and owner ID). A world with 5,000 blocks is large. 10,000 blocks is down-right crowded. So if we assume 10,000 to be an average maximum (but not a statistical outlier) we can plan on 250KB of data being written to S3. We can have this data written out asynchronously in a separate thread fairly easily. We’ll need to store the user interaction delta since the last save; network command journaling in Suva3D should make this possible.
B. I suggest that we disable the shimmering texture in the portal when passage is not allowed. The difficulty is that only one Xna.Net connection is allowed, so we’ll need to store a number of players value in the region metadata we store on S3. Peers can update this value periodically using the NetworkSession.PlayerCount value so that we don’t have to worry about race conditions.
C. We can limit the return on investment for hackers by setting up a dedicated server with our web services to facilitate secure transactions (i.e. adding/removing things from player inventory and shops. Too bad). Or better, we can use this:
AWS DB.
As an aside, here are some great things we’ll be able to do with Xna.Net. Player positions will be sent using UDP for non-reliable non-ordered delivery. Block manipulation will be sent using reliable ordered delivery. For region ownership and the player coat-of-arms picture we can use their gamer image. Lastly, By limiting each world area to 31 players and using XNA.Net for player and block ephemeris our old networking design is simplified.
To learn more about Xna.Net here are useful links:
http://msdn2.microsoft.com/en-us/library/bb975801.aspx
http://forums.xna.com/thread/35546.aspx
http://forums.xna.com/thread/34455.aspx
http://forums.xna.com/thread/35530.aspx
Separately, Matt you’ll be glad to know that they added IsButtonUp and IsButtonDown for the XNA controller. What classes do we need to clean up now to remove or own implementation of this?
http://msdn2.microsoft.com/en-us/library/bb975648.aspx
Lastly, Mihail, here is change log for XNA 2.0. After we finish up on the current round of bugs I see us porting over to XNA 2.0 while I ready general release of Suva3D. Looks pretty good, doesn’t it? The places where we *have* to change code will throw compile errors.
http://msdn2.microsoft.com/en-us/library/bb975648.aspx
Questions, thoughts?
Shaun Tonstad
1-877-456-SUVA, Ext. 3
Suva Interactive (www.suva3d.com)
