Thought I'd just a little bit of light investigation into what sort of headers the different types of SendDataOptions had, really just out of curiousity rather than a need to critically cut down the amount of packets I'm sending.

I "think" it looks a bit like this:

None - 9 bytes
Reliable - 9 bytes
InOrder - 11 bytes
ReliableInOrder - 11bytes

Assuming my numbers are correct, does that mean that sending something Reliable is as 'bad' as it is sending it with None? Obviously sending something InOrder requires a stall (of sorts?) if one particular packet gets delayed while others don't, but is there anything inherent about Reliable and ReliableInOrder that makes them worse than using their unreliable counterparts?

I haven't looked into what our packet structure is, but if I had to guess, the InOrder option needs to send a sequence number, which is the increase in packet size.  The reliable flag is already accounted for in the 9 bytes.  If you're trying to gauge traffic size, realize that setting reliable would seem to require an ack packet the other way.  While this doesn't increase the size of the sending packet, it does increase the amount of traffic between the two peers.

 

---

Mitch Walker
Program Manager - XNA Game Studio

How are you measuring the header sizes?

These will be quite variable, because we do some very aggressive bit packing, often combining status flags and merging related pieces of information, so there isn't always a direct correlation between send mode and header size: it all depends on the specifics of how well we were able to encode that particular combination of states. For instance the acknowlegements for a reliable packet might or might not take up any extra space, depending on how many header bits happened to be free in some other packet that already happened to be travelling in the same direction as the acknowledgement.

As an overview, though:

• InOrder is very cheap. This just adds a sequence number to the packet. The recipient tracks the most recent packet they have seen, and if one arrives with an older sequence number, they discard it. No extra memory usage, no extra latency, minimal extra bandwidth for the sequence number.

• Reliable is somewhat cheap. Again this adds a sequence number to the packet, but now the sender must also store a copy of the packet data, which costs some extra memory. The recipient acknowledges the packet by sending back a return saying "yeah, I got the data", which may or may not require a separate packet depending on whether any other data happened to be flowing back from the recipient toward the sender (if so, the ack will be merged with that existing packet, making it very cheap). The sender removes the packet from their pending queue as soon as they get the ack, or resends if they don't get an ack within some period of time. The bandwidth usage is a little higher than for InOrder (due to the acks) and the memory usage is higher (due to the queue of packets still awaiting confirmation) but again there is no extra latency.

• ReliableInOrder is the worst. Not only does this have to queue up packets on the sender (as for Reliable), it also has to queue them on the recipient. Lets say we sent packets 1, 2, 3, but the recipient got 2 and 3 while 1 was dropped. It can't deliever them to the game yet, so it just has to hang onto them while waiting for 1 to arrive. After 1 arrives, it can deliver that, then pull 2 and 3 out of the queue and deliver those as well. So you pay an extra memory cost for this "can't deliver yet because the order was wrong" queue on the receiving end, and also an extra latency cost since packets 2 and 3 might have to just sit around waiting while 1 makes its way over the wire. Effectively, all packets end up being limited to the worst latency of any of them.
  

Note: the above is true in principle, but somewhat of an oversimplification (there are many subtleties in how to most efficiently handle different permutations of reliable acks, packet merging, etc). It's true enough to give you a good idea of the expense of each option, though.

---

--
XNA Framework Developer
blog - homepage

Thanks for the explanations of each, should come in very handy. As for measuring the headers I was rather crudely sending 1 byte per frame (60fps), querying BytesPerSecondSend (and Recieved on the recipient), subtracting the 60 bytes from it, then dividing the remainding number of "those byte weren't me!" by 60 (fps). They all tended to come out almost bang on whole numbers, and the ones I expected to be more expensive came out so, so I guessed that these might be vaguely reliable figures.

The 'issue' I'm having at the moment is managing to get anywhere near the sort of throughput I expected from System Link between a PC and the 360. I invisaged being able to get at least a megabyte per second between two, but so far I'm struggling to get over 40-50KB/s. I'm poking around the network at the moment, but the 360 as a media center extender managed 4-5mbit and is right at the top of the network tuning graph, (ethernet), so I'm struggling to understand quite why it takes 14 seconds to send (an admittedly huge) 500,000 byte chunk of data between the two, in either direction. Of course this comes with the disclaimer that I'm not trying for game data at the moment, simply want to push an image between the two, but whether I send lots of small packets or 1 large one on the press of a button, I'm just unable to get any sort of decent transfer speed between the two.

 

Calling NetworkSession.Update more often than 60 times per second might help. Our stack is really tuned for game data, though. We didn't do much testing of this sort of not-practical-for-an-actual-game scenario, so I don't know how well the protocol will scale to that amount of data.

---

--
XNA Framework Developer
blog - homepage

What is strange so far is that at the standard 60fps I can roughly seem to send/receive about 60KB/s, letting it go at 1500fps on the PC and 5000fps on the 360 allows up to about 320KB/s, not a proportional increase, but fairly sizeable. The bandwidth is obviously there, it just doesn't seem to want to use it at the lower frame rate. I tried dotting a few extra session updates around and about, but they don't really seem to have any effect. It's hard to understand (even though it isn't optimised for such scenarios) how with spare CPU time, spare bandwidth and enough calls to Update that it doesn't just send the data... but then thats probably I couldn't write my own library using System.Net.

I've been talking a little to someone on IRC from MS (I hope, or he's had me fooled!) about this, but I thought I'd keep the thread up-to-date.

I enlisted Cryovat's help in testing out my sample application to see how near to the ~320KB/s we could get over the Internet rather than in System Link, and lo and behold, there is absolutely no difference between the two in terms of throughput. He's situated on a rather well-linked university, and I have enough downstream to cope with whatever he could throw upstream, and so the application maxed out at 314KB/s. I went back and checked my transfer speeds for 360 -> PC over system link, and it too can only send up to 314KB/s, the very same number... As much as I can appreciate the fact that this scenario isn't well tested, and perhaps with good reason, being able to get the same throughput over 6000 miles and a trip under an ocean as I can over a 10 metre ethernet cable does make me smile.

My test has simply been to throw a 1 million byte array at send data at the press of the Y button and see how long it takes the BytesPerSecondSent/Received to return to 0, and what it peaks at. Modifying the size of this array has zero effect since presumably the Live API is breaking it up into sanely sized packets and reforming it on the other end once it has all arrived, so I can't imagine I can do anything there to better the throughput.

Increasing/decreasing calls to session.Update had little to no effect, and only served to lower the flatout frame rate when over-called.

Interestingly the same test run with the PC sending data to the 360 doesn't work nearly as well as the other way around. 39KB/s from PC->360 is as much as I can muster, 314KB/s obviously in the other direction.

My worry is that either a) I'm doing something so silly I'm gonna be laughed at from now until Zday 2008, or b) what I'm doing is fine, and in network heavy games with a client/server topology we have a server that starts nearing whatever "limit" there is in place, ~39KB/s in the case of a PC hosting. I've not been able to test this with another more than two 'players' though, so if it were 39KB/s to each player then it's not likely to be an issue for people that aren't doing silly things like me. It's a fairly easy repro, but I've got a link to what I've been using so far, it isn't pretty but it works.

http://aj.uwcs.co.uk/coding/NetworkTestGame.rar

I just open it twice in two instances of VS, set one to startup the 360 version, one for the x86, they automatically create if there are no sessions or join session 0 if there is one. Whichever is the Host will starting printing out "Host sent = x KB/s", at which point you can press Y to send 1MB of data to the other, the output to the console for the host shows bytes sent. For now though I'm off to sleep, hence the rather comprehensive post. Thanks.

Hey guys. I'm kind of pulling a resurrection spell here, but can anyone chime in as to whether or not there is some inherent limit to the transfer over system link using LIVE? This thread popped up in IRC chat and now I'm curious to know if there is an official (or maybe not-so-official) conclusion on the matter.

---

---

Nick Gravelyn -- Microsoft XNA MVP
XNA Wiki | Zune Games

When this issue first came up I discussed it with the guys who implemented the underlying reliable UDP protocol. Their initial reaction was "huh? That's way outside the kind of scenarios we ever designed for or tested for, as our stuff is all built for multiplayer Internet gaming" but they were surprised the throughput was as low as it seems to be.

It appears the bandwidth can be improved if you update the session more frequently (perhaps create a background thread that just pumps NetworkSession.Update as fast as possible), but I don't understand enough about the underlying issue to know why that makes a difference.

I think the fundamental problem is that the network protocol uses some heursitics to detect available bandwidth and back-off if things are becoming too conjested, and those heuristics were never carefully tweaked for fast local network situations, so they just fall into a "hey, this is faster than we ever expected to see on the Internet" case which, although fine for gameplay, is way less than what should theoretically be available over a LAN.

---

--
XNA Framework Developer
blog - homepage

This problem is just so infuriating because it's so random. I fully understand the argument about heuristics and them being biased towards gameplay data, but there are just so many things that don't make sense.

I dug out my network bandwidth testing app today and modified it so that basically the update method is just a big while loop that updates the session, checks for me pressing the Y button (to send 1MB of data), and prints out debug information, then calls base.Update. Draw method never gets called, as Update gets called first, and the while(true) loop keeps it there. One session.Update call per loop, 12000 loops per second on the PC, 9000 per second on the 360.

If it were purely heuristics that were keeping it back, I see little reason why the PC as host sending data to the 360 should cap out at exactly 39KB/s, while the other way round we get 314KB/s... it's the same cable, it works both ways round. Unless for some reason my gigabit port is flooding the 360 and there's issues with the way it responds to that (?) but I thought that would be covered at a lower level than the networking API? Unfortunately I don't have a 10/100 port to test it with at the moment, but I could try that later.

As I'm typing this I've set it off running again, and it's permenantly stuck now at 25KB/s, in a rut that it can't get out of, just perpetually stuck at 25KB/s after having run at 39KB/s for a few minutes.

I've also just noticed the QualityOfService information on AvailableSessions for the first time, average/minimum round trip times are both 1ms, and estimated upstream and downstream bytes per second are both 1250000 (1.25MB/s). Quite how it manages to calculate that, but then completely fail to use it... Assuming the estimate hasn't come from Random.GetNext then surely this is a great heuristic to be using as to whether there is available bandwidth, especially when on SystemLink mode which will be almost completely uanffected by network conditions (discounting wireless interference while using the wireless network adapter).

Again, I can fully understand it being optimised for game data, and as a result of this there being "some sort" of cap as a result, but the phrase "results may vary" has never been more true. More bandwidth one way than the other, random getting stuck in ruts at lower than '39' speeds, extra updates having a beneficial effect up until a certain point then no benefit at all etc. It just isn't predictable what it is going to do in any given situation and that's almost worse than it not working at all... I find it really hard to believe currently that it really does work under 'game data' conditions, but those are hard to replicate without actually writing a game.

 

This is what happens when you use software outside the parameters it was designed and tested for...

The Xbox platform as a whole (not just the XNA Framework) has a very different network stack to a PC operating system.

I find it really hard to believe currently that it really does work under 'game data' conditions, but those are hard to replicate without actually writing a game.

The existence of a rather large number of successful Xbox LIVE games suggests that this does in fact work ok...

---

--
XNA Framework Developer
blog - homepage

What code are you using to calculate KB/s?

What strikes me as odd is the variation in speed.  At those speeds, a drop from 39 KB/s to 25 KB/s is quite a percentage drop.


This is all idle speculation, with the caveat that I have not used XNA networking much:

I wonder if it has anything to do with the XnaLiveProxy.exe process.  The fact that the Xbox is uploading much faster (~ 10x) than downloading almost makes it seem like the PC outbound buffers are slow.  My thought is: if the XNA Framework on Xbox is communicating with the Xbox networking layer more-or-less directly, while the XNA Framework on Windows needs to go through another process, it's possible there are some buffering delays on the Windows side.  Perhaps what's happening is the Xbox is able to send/receive data at more or less the native rate, but the PC is being bottlenecked by inter-process communication.  The XNA Framework needs to push data to the XnaLiveProxy.exe process somehow, and that crosses process boundaries.  Depending on how buffering is being done here, a series of send requests on Windows could result in several buffer copies from your game process to the XnaLiveProxy.exe process, while received data from the Xbox could be buffered in the XnaLiveProxy.exe process and sent to the game process all at once.

Just a thought.

I suspect I'm going to have to make a repro that shows a genuine problem within the "parameters it was designed and tested" for, so here goes I guess.

The test I setup was for the host and client to connect, the host then sends a single 512 byte message once per frame, 60 times per second until 5000 messages have been sent, this obviously takes 84 seconds. The host stores how many milliseconds after the start of the 1st packet each packet is sent, and the client does the same. Using the standard while(gamer.isdatavailable) process, the buffer is completely empty every time the client finishes a frame, all data is read.

http://aj.uwcs.co.uk/XNA/Networking/xnaNetwork1.PNG

What this graph shows is the host (blue) sending the packets at a rate of 60 per second, perfectly straight from the start of the game until the end at 83501ms since the start of the game. The client gets the packets no more than about 20ms later (due to there only being 60 opportunities per second to receive a packet). There are glitches along the way where fewer than 60 packets per second are received, but it seems to make an attempt to catch up as more arrive in subsequent frames, however the general trend from around 3000 packets onwards seems obvious.

The client is draining the incoming buffer, that much I know, the host however is buffering the data and there's a large delay in it getting sent, and by the last few packets the delay is 7 seconds. The red line finishes early because I sent the data with SendDataOptions.None, 99 packets were lost in transit somewhere between my PC and the 360.

Now, 512bytes per frame at 60fps is 30KB/s, so still a little on the high side, but I'll certainly try and demonstrate this at 256 byte messages, I've already observed this. When I tried 2048 bytes per packet, the buffering was almost immediate, the test never completed as it threw an exception mid way through saying the buffer had got full and that I should send less data.

2048 I will agree is too much per frame if you want to send data every frame, 512 is borderline but it wouldn't have taken much longer for the buffer to fill and crash. 256 and 128 are 15KB/s and 7.5KB/s respectively, both surely well within the sort of parameters the API was tested for?

You may suggest that games don't and shouldn't be sending an update every frame, and that it should be less frequent, but surely this is far far nearer to the intended use of the API than my original "throw 1MB at it" tests?

If there are multiple processes involved on the PC, then thread scheduling may get in the way.

Try this: Insert a Thread.Yield() or Thread.Sleep(1) call in the middle of your loop; especially on the PC. This ought to give enough time to the other threads to funnel all the data where it needs to go. You could also try a Thread.Sleep(10), to see whether there's some big resource usage.

---

--
Jon Watte, Direct3D MVP
kW X-port 3ds Max .X exporter
14 days after getting my RROD box back, it's going back for service again. Grr.

My program isn't mulithreaded in any way. I tried playing about with the process priorities for my program, xnatrans/xnaproxy, but nothing seems to have any great effect. Thanks for the input though.

Adam Miles:

My program isn't mulithreaded in any way. I tried playing about with the process priorities for my program, xnatrans/xnaproxy, but nothing seems to have any great effect. Thanks for the input though.

Your program may not be, but the XNA framework most likely is. Not to mention that the XnaProxy is a separate process from your program. Insert the Thread.Sleep() calls and measure. Worst case, things don't get better, and you take them out again. Best case, you learn something.

---

--
Jon Watte, Direct3D MVP
kW X-port 3ds Max .X exporter
14 days after getting my RROD box back, it's going back for service again. Grr.

Given that my application is doing almost no work, at a fixed time step, there should always be plenty of CPU left over for anything. But no, sleeping didn't help at all.

The IPC mechanism we use to communicate with the proxy process is very efficient (roughly comparable to a kernel call on Xbox). The threading model is explicitly cooperative, so there's no risk of the scheduler starving things when they need to run.

I'm sorry that you aren't seeing the bandwidth you would like here (as always, please send feedback on this if it's a problem for anyone out there: the more we hear about an issue, the higher priority it gets...) but please believe me when I say this networking stack will work fine as long as you stick within the limits of what is possible for multiplayer Internet gaming. It's just not tuned for higher bandwidth local network applications.

---

--
XNA Framework Developer
blog - homepage

If the results from these experiments are accurate, 30 kb/s is prettty limiting for games... Especially if you have to share between lots of players. 30 Kb/s hasn't been 'high bandwidth' since the early 90's...