Hello all!

I'm beginning a port of an iPhone game this week (G, by Soma Games, if you're interested) and had an idea about how to speed up the game startup. In my last (and first ever finished) game RebelCard, I loaded everything at the beginning of the title, in LoadContent. But I had a lot of big background images to load, and so it took ages of black screen before it finally started up. (about 2 mins). I was keeping everything in a class called GraphicsBundle that I would just pass around, it just was a container full of textures.

Well, I wanted to avoid that for this game - it also has a lot of big images (pretty, hand drawn backround art, and I figure, why not do it HD?)

So I had an idea, and I was wondering if anyone could tell me any potential flaws in it or whatnot before I execute it.

I'm thinking at the beginning of the game only load the stuff for the menu, the game objects, all the small stuff. Then as the main menu/intro screen goes, I could start a new thread and load more content that way. Using this content loading thread, I could load all the big stuff I could ever care about and then give it to the level when it needs it. For example, I could load and prep the next level in this thread prior to it being displayed and eliminate loading times.....

I'm just wondering, when I load a texture, does it go directly into VRAM? (not that I'll load hundreds of megs right away, just curious). And can I do this whole async thing with one ContentManager, or should I create a second one?

Thanks in advance.

Generally I'd say you should only load content as you need it anyway. Loading content in a separate thread sounds like it would be fine, you just need to make sure that you're either guaranteed to have the resource ready before you actually need it, or that your game will run properly if you're using an object that doesn't have everything fully loaded for it already.

Another thing that might be nice, is to limit how many milliseconds the load thread can run each time it is called, this can prevent hitches in the game when loading stuff on the fly. If you need to load a large image in the middle of gameplay, it might be nice to load part of that image each frame, although I'm not sure how easy that is with XNA's content pipeline. At the very least, if you request multiple assets and it goes past the max milliseconds allowed, it could store the requests in a queue and begin loading the assets again on the next frame.

Lord Ikon:

you just need to make sure that you're either guaranteed to have the resource ready before you actually need it, or that your game will run properly if you're using an object that doesn't have everything fully loaded for it already.

The thing to be careful of here is to make sure you have proper locking and/or use interlocked operations when you hand data off from the loading thread to the main thread. It's not good enough just to store the object you loaded into a field, then have the main thread check for this field being null. You need anyone who accesses this shared field to do so inside a lock (but take care not to hold that lock for any longer than absolutely neccesary!) or use an atomic operation such as Interlocked.Exchange each time you access it.

Lord Ikon:

If you need to load a large image in the middle of gameplay, it might be nice to load part of that image each frame, although I'm not sure how easy that is with XNA's content pipeline. At the very least, if you request multiple assets and it goes past the max milliseconds allowed, it could store the requests in a queue and begin loading the assets again on the next frame.

I don't understand what the point would be of trying to manually limit the loading thread, or why you would tie this to the framerate of the main game thread. The whole point of using a second thread is that these can run independently (ideally on a separate CPU core, assuming you set your thread affinity in a sensible way), and the OS will schedule between them as necessary. The very nature of parallel execution will allow each load operation to extend over however many frames it needs to finish the IO operation.

You can *almost* make this work right by wrapping a lock(GraphicsDevice) {} around your Draw function:


Then wrap a lock() around your threaded loading code, too:


Unfortunately, the game loop will still use the graphics device outside of Draw(), to set up the device, and to call Present(). I found this out when generating light maps on demand in the background -- sometimes the screen and the render targets for the light map would fight for the device.

Instead, you have to override BeginDraw() and EndDraw() and emulate lock as described in the C# specification:


You can still use lock() to exclude other users around each call to ContentManager.Load<>.

You could, conceivably, use some object other than the GraphicsDevice for your synchronization, but I feel that, because that's the object that you are contending for, it's the best object to synchronize on.

Note that the threaded loading StateManagement sample doesn't do this right (!) although if all you do is load textures (no render calls or other state changes) then it's likely to be good enough.

jwatte:

Note that the threaded loading StateManagement sample doesn't do this right (!) although if all you do is load textures (no render calls or other state changes) then it's likely to be good enough.

Creating graphics resources (as happens when you load content) is internally threadsafe, so there's no need to synchronize your loading and rendering code in this way.

Shawn Hargreaves:

The thing to be careful of here is to make sure you have proper locking and/or use interlocked operations when you hand data off from the loading thread to the main thread. It's not good enough just to store the object you loaded into a field, then have the main thread check for this field being null. You need anyone who accesses this shared field to do so inside a lock (but take care not to hold that lock for any longer than absolutely neccesary!) or use an atomic operation such as Interlocked.Exchange each time you access it.

I'm curious - why is that necessary? I do that all the time and haven't had any problems.

On a related note, in The Virtual Store Channel, there is a single blank box art texture that is loaded and applied to all of the game boxes. I then start a thread that loads all of the game box art, and changes each game box's texture to the newly loaded one as it gets them. I haven't had any problems doing this - is there a better way?

Shawn Hargreaves:

The thing to be careful of here is to make sure you have proper locking and/or use interlocked operations when you hand data off from the loading thread to the main thread. It's not good enough just to store the object you loaded into a field, then have the main thread check for this field being null.

Could you explain what you mean in more detail? I would've thought this would work fine, assuming nothing else modifies the value. Eg. it's null until it's loaded, then it's a texture, and nothing else ever modifies it. I thought this way I could easily load content on threads, but now you have me worried :-/

Hmm... I find that if I create any graphics resources on a background thread (which happens when I load with the ContentManager), my app will crash if I run it under PIX, or use the debug DirectX runtime (from what I recall... it's been a while).  It runs fine otherwise.
I ensure that calls to the Load method are serialized.

UberGeekGames:

Shawn Hargreaves:

The thing to be careful of here is to make sure you have proper locking and/or use interlocked operations when you hand data off from the loading thread to the main thread. It's not good enough just to store the object you loaded into a field, then have the main thread check for this field being null. You need anyone who accesses this shared field to do so inside a lock (but take care not to hold that lock for any longer than absolutely neccesary!) or use an atomic operation such as Interlocked.Exchange each time you access it.

I'm curious - why is that necessary? I do that all the time and haven't had any problems.

Welcome to the strange and wonderful world of multicore programming :-)

Your CPU does not necessarily execute instructions in the same order you write them in your C# code, for many reasons:

• The compiler or JIT optimizer might reorder your code, swapping the order or two or more instructions
• The CPU might decide to execute instructions in a different order, depending on what execution units it has free at any given moment
• Even if the instructions are executed in linear order, their results may not be immediately written back to memory, but may be stored in a write accumulator or output cache for an indeterminate amount of time
• Even if changed data is written out to memory in a timely fashion, a second CPU core may not notice the changes if it already has that memory location in its own local cache, and doesn't realize it needs to go back and fetch a new copy of the data from main memory

A simple way to run into trouble:

Thread A:

flag = true;

Thread B:

if (flag) Foo();

It is possible that A may set the flag to true, but B never calls Foo. Huh? This could occur if thread B had already loaded the flag value into a local cache line or CPU register, and had been optimized to not bother re-loading the value from main memory every time it was checked, so it never notices that thread A has changed the value.

A more subtle problem:

Thread A:

Level tempLevel = new Level();
tempLevel.Model = Content.Load<Model>(...);
tempLevel.Sky = Content.Load<Model>(...);
loadedLevel = tempLevel;

Thread B:

if (loadedLevel != null)
{
    loadedLevel.Model.Draw();
}

This code could fail in many strange and mysterious ways, if thread A does not execute the instructions in the same order they are written. For instance, it might store the new level object into the loadedLevel memory address before it stores the loaded model references into the Model and Sky fields. If that happens, thread B would see that loadedLevel is no longer null, and proceed to access its Model field, and get a null reference exception.

These problems are rare, because you will only see a failure if the two threads happen to run exactly the wrong instructions at exactly the right times, so your program may run fine for years in spite of subtle threading bugs, but they can and will happen eventually unless you do something to prevent them.

There are several ways to write robust threading code. The simplest, and what I would recommend for the majority of situations especially if you are new to parallel programming, is to make sure that if a given variable or object is touched by more than one thread, every access to it is guarded inside a lock region (obviously you need to lock on the same object every place you access this shared data from). If you have nested locks, there are some subtleties to avoid deadlocks, which can most easily be avoided by not nesting locks inside each other.

Another approach which can sometimes be easier or more efficient is to use WaitHandle events. Setting or waiting on a handle creates a memory barrier, which ensures everything before and after this barrier is fully committed to memory and visible to other CPU cores before execution can continue, so this can be a good pattern for things like transferring a bunch of data between threads: first one thread owns the data while loading it, then sets a wait handle to indicate the loading is complete, and when another thread sees this handle has become set, that means it now owns the data that was just loaded.

In a handful of situations where you want very efficient simultaneous access to shared data, you can use lock-free constructs such as the C# volatile keyword and Interlocked.* atomic operation helper methods. Here be dragons! Lock free threaded programming is extremely subtle and hard to get right. Here's a good place to start if you really want to learn this stuff.

UberGeekGames:

On a related note, in The Virtual Store Channel, there is a single blank box art texture that is loaded and applied to all of the game boxes. I then start a thread that loads all of the game box art, and changes each game box's texture to the newly loaded one as it gets them. I haven't had any problems doing this - is there a better way?

That's a fine approach, as long as you use proper locking or atomic operations every time you access the shared value.

Shawn Hargreaves:

UberGeekGames:

On a related note, in The Virtual Store Channel, there is a single blank box art texture that is loaded and applied to all of the game boxes. I then start a thread that loads all of the game box art, and changes each game box's texture to the newly loaded one as it gets them. I haven't had any problems doing this - is there a better way?

That's a fine approach, as long as you use proper locking or atomic operations every time you access the shared value.

Nope - I just set the texture to its new value when it's loaded, and then it gets changed when it needs to be drawn, like this:

Apparently that's a bad idea.

I'll go read up on lock and other thread safety - dang, just when I thought I knew about multithreading, someone has to come along and rain on my parade. :-) Makes me wonder how much code I have in my other released games and projects that might spontaneously fail at any time...

Assuming that your GameBox instances are always created before you start the loading thread, and the gameBoxes list can never be changed while this thread is still running, you could add locks like so:


If you add or remove new boxes on the fly, you would also need locks around any use of the gameBoxes list.

Wasn't somebody working on writing a super-fancy ContentManager derivative that could handle asynchronous background loads?  I remembered seeing the link in somebody's signature, but I can't remember which user it was...

Thanks for the pointer - it doesn't look quite as intimidating as I first thought. The only potential snag is that it's slightly more complicated than I made out. I have two static lists of GameBoxes in the GameBox class - one that has every game loaded (they are deserialzied from XML to get the text information before the textures are loaded) and one that has the list that is trimmed down with whatever sort options the user selects, but more locks should fix that. I guess I'll be seeing a lot of lock statements in my code from now on!

Creating graphics resources (as happens when you load content) is internally threadsafe, so there's no need to synchronize your loading and rendering code in this way.

When you prepare the resources using render targets, it is!
As I said: if all you do is load canned textures, it's good enough without the exclusive lock.

May be over kill but what about a double buffered messaging system described in this article here
http://www.ziggyware.com/readarticle.php?article_id=221

Shawn Hargreaves:

A simple way to run into trouble:

Thread A:

flag = true;

Thread B:

if (flag) Foo();

It is possible that A may set the flag to true, but B never calls Foo. Huh? This could occur if thread B had already loaded the flag value into a local cache line or CPU register, and had been optimized to not bother re-loading the value from main memory every time it was checked, so it never notices that thread A has changed the value.

How does a lock stop this from happening? Is everything inside a lock always read from main memory?

Shawn Hargreaves:

A more subtle problem:

Thread A:

Level tempLevel = new Level();
tempLevel.Model = Content.Load<Model>(...);
tempLevel.Sky = Content.Load<Model>(...);
loadedLevel = tempLevel;

Thread B:

if (loadedLevel != null)
{
    loadedLevel.Model.Draw();
}

This code could fail in many strange and mysterious ways, if thread A does not execute the instructions in the same order they are written. For instance, it might store the new level object into the loadedLevel memory address before it stores the loaded model references into the Model and Sky fields. If that happens, thread B would see that loadedLevel is no longer null, and proceed to access its Model field, and get a null reference exception.

Wow, this scares me. It seems really obvious now, but I've written loads of code like this in the past and not thought about that. I bet there's not a single programmer at my company that knows this either :-(

Shawn Hargreaves:

Assuming that your GameBox instances are always created before you start the loading thread, and the gameBoxes list can never be changed while this thread is still running, you could add locks like so:

void ThreadedTextureLoading()

{

forach(GameBox gb in gameBoxes)

{

lock (gb)

{

gb.Texture = content.Load<Texture2D>(gb.TexturePath);

}

}

}

void LoadContent()

{

TextureLoadingThread = new Thread(new ThreadStart(ThreadedTextureLoading));

TextureLoadingThread.Start()

}

void Draw()

{

foreach(GameBox gb in gameBoxes)

{

lock (gb)

{

DrawGameBox(gb.Texture);

}

}

}

If you add or remove new boxes on the fly, you would also need locks around any use of the gameBoxes list.

I understand in your original sample that the instructions might not be in the same order, so loadedLevel mayb become not null before the Model is loaded. But in the above example, I don't understand how it would fail. He's assigning dummy textures to all gameBoxes at startup, then the thread is just changing the reference to new ones. Is the problem here the same as your first example (never being read from main memory)?

jwatte:

Creating graphics resources (as happens when you load content) is internally threadsafe, so there's no need to synchronize your loading and rendering code in this way.

When you prepare the resources using render targets, it is!
As I said: if all you do is load canned textures, it's good enough without the exclusive lock.

Now I'm seriously confused. Does this mean, for my purposes, I don't need to go lock-happy to ensure safe operation for 1,000 years to come? Wouldn't the absolute worst case scenario be that a texture doesn't get set until sometime later in the program?

Thanks for the reply Shawn!

Just reading that article (well, skimming) I found that lock() essentially stops all other locks from executing on the particular object (/variable?) until that code is done.

That's handy! But say I put a lock into the drawing code for the level, and I then load the resources (behind the scenes, when the scores are being displayed). If it's not done in time, won't the level appear to freeze (well, actually freeze?)

Could I do it by using a public boolean? set it to false, Load the next level in advance, and then when it's loaded, flag it to true? Then when the level is over, I can just set texture1 to equal newloadedtexture, etc, etc (or just swap simulation1 to equal newloadedsimulation?)

I guess I would definately need locks if I was only using one texture object at a time, eh? Am I getting it right?

Shawn Hargreaves:

Lord Ikon:

you just need to make sure that you're either guaranteed to have the resource ready before you actually need it, or that your game will run properly if you're using an object that doesn't have everything fully loaded for it already.

The thing to be careful of here is to make sure you have proper locking and/or use interlocked operations when you hand data off from the loading thread to the main thread. It's not good enough just to store the object you loaded into a field, then have the main thread check for this field being null. You need anyone who accesses this shared field to do so inside a lock (but take care not to hold that lock for any longer than absolutely neccesary!) or use an atomic operation such as Interlocked.Exchange each time you access it.

Lord Ikon:

If you need to load a large image in the middle of gameplay, it might be nice to load part of that image each frame, although I'm not sure how easy that is with XNA's content pipeline. At the very least, if you request multiple assets and it goes past the max milliseconds allowed, it could store the requests in a queue and begin loading the assets again on the next frame.

I don't understand what the point would be of trying to manually limit the loading thread, or why you would tie this to the framerate of the main game thread. The whole point of using a second thread is that these can run independently (ideally on a separate CPU core, assuming you set your thread affinity in a sensible way), and the OS will schedule between them as necessary. The very nature of parallel execution will allow each load operation to extend over however many frames it needs to finish the IO operation.

I was thinking that the loading times could be limited in the case where you're supporting a single core machine. If you have an asset that is going to take 40ms to load, and you want to run around 60fps, you want to only give it so many milliseconds per frame to load to keep the framerate decent.

Danny Tuppeny:

How does a lock stop this from happening? Is everything inside a lock always read from main memory?

Any time you acquire or release a lock, this inserts a memory barrier. The compiler, jitter, CPU, and memory subsystem all know about memory barriers, and will take care to never reorder any memory reads or writes across them, and to perform whatever synchronization is necessary to make sure that multiple CPU cores all have the same view of cache.

There are actually a couple of flavors of memory barrier (read barriers vs. write barriers) which are important to understand if you are writing lock-free code (in which case you must manually insert the right kind of barrier in the appropriate places), but if you keep it simple and just take out locks around any cross-thread data accesses, this provides both a read and write barrier in one operation.

Danny Tuppeny:

I understand in your original sample that the instructions might not be in the same order, so loadedLevel mayb become not null before the Model is loaded. But in the above example, I don't understand how it would fail. He's assigning dummy textures to all gameBoxes at startup, then the thread is just changing the reference to new ones. Is the problem here the same as your first example (never being read from main memory)?

What if some of the instructions involved in loading these new textures got reordered? In an extreme case, the load function might end up allocating some memory to store the new texture, then storing the address of this newly allocated memory into the gameBox object, and only then storing the actual data that was just loaded into the texture object.

Basically any time you do something like:

1. Create, initialize, or edit an object
2. Hand the resulting object off to a different thread

you need a write barrier in between stage 1 and stage 2, to make sure the modified object state is fully visible to the other thread before you hand the object over to it.

Sparkworker:

That's handy! But say I put a lock into the drawing code for the level, and I then load the resources (behind the scenes, when the scores are being displayed). If it's not done in time, won't the level appear to freeze (well, actually freeze?)

Yes, it will :( You need to make sure you take locks for the absolute minimum time you can get away with. There's no point multithreading if your render thread is always sat waiting to obtain a lock. My suggestion would be that your background thread does all its stuff, then uses a lock just to copy the references into variables used by the main thread. However, having read Shawn's post I'd question my multi-threading ability! :-(

Sparkworker:

Could I do it by using a public boolean? set it to false, Load the next level in advance, and then when it's loaded, flag it to true?

Seemingly not. I thought you could, but check out Shawn's first sample. The boolean could be loaded into cache and then always read from there, so your main thread will always see the flag as false.

:-(

UberGeekGames:

jwatte:

Creating graphics resources (as happens when you load content) is internally threadsafe, so there's no need to synchronize your loading and rendering code in this way.

When you prepare the resources using render targets, it is!
As I said: if all you do is load canned textures, it's good enough without the exclusive lock.

Now I'm seriously confused. Does this mean, for my purposes, I don't need to go lock-happy to ensure safe operation for 1,000 years to come? Wouldn't the absolute worst case scenario be that a texture doesn't get set until sometime later in the program?

You still need a lock when you hand the loaded texture off from one thread to another. You just don't need a lock around the entire time you are loading it. Jon was saying that if your loading involved operations like render to texture (which involves submitting draw calls to the GPU) you would need to lock the entire load operation, and also lock your main draw function, to make sure they didn't both try to use the GPU at the same time.

Since you're not doing any rendering at load time, you can call Content.Load outside of a lock, then just take out the lock around the specific instruction that hands the loaded texture off to a different thread.

• Rule of thumb: any time you transfer data between threads, or access an object that is shared by more than one thread, you need a lock
• More advanced rule: if you fully understand memory barriers and atomic operations, it is possible to transfer data between threads without locks, but this is subtle and extremely hard to get right

Danny Tuppeny:

Sparkworker:

Could I do it by using a public boolean? set it to false, Load the next level in advance, and then when it's loaded, flag it to true?

Seemingly not. I thought you could, but check out Shawn's first sample. The boolean could be loaded into cache and then always read from there, so your main thread will always see the flag as false.

Exactly. Or the boolean might get written out to main memory while the loaded data was still in the local cache of the CPU core that loaded it, in which case the main thread would see the boolean go true before the data was actually ready.

Hence, you need a lock. But you don't need to lock the entire loading code (that would be pointless, as it would prevent the loading running at the same time as the main game). Just lock around the specific place where you transfer data between the two threads (and make sure you lock in both the loading thread and the main thread, any time you access this piece of shared data).

One of the hardest things about writing efficient multithreaded code is working out ways to move data around while using the smallest possible number of locks, and holding those locks for the shortest possible amount of time. It's all too common to spend a ton of time splitting a game over many threads, only to find that all the time you gained through running things in parallel, you immediately lost again through wasting time with millions of lock statements! A good design is one that does as much work as possible using only thread-local or immutable data structures, then only transfers data between threads at rare and well defined intervals.

Lord Ikon:

Shawn Hargreaves:

Lord Ikon:

If you need to load a large image in the middle of gameplay, it might be nice to load part of that image each frame, although I'm not sure how easy that is with XNA's content pipeline. At the very least, if you request multiple assets and it goes past the max milliseconds allowed, it could store the requests in a queue and begin loading the assets again on the next frame.

I don't understand what the point would be of trying to manually limit the loading thread, or why you would tie this to the framerate of the main game thread. The whole point of using a second thread is that these can run independently (ideally on a separate CPU core, assuming you set your thread affinity in a sensible way), and the OS will schedule between them as necessary. The very nature of parallel execution will allow each load operation to extend over however many frames it needs to finish the IO operation.

I was thinking that the loading times could be limited in the case where you're supporting a single core machine. If you have an asset that is going to take 40ms to load, and you want to run around 60fps, you want to only give it so many milliseconds per frame to load to keep the framerate decent.

Even on a single core machine, the operating system will preemptively schedule between threads, interrupting one to give others a chance to run. It's quite rare that you need to do this manually.

Shawn Hargreaves:

Exactly. Or the boolean might get written out to main memory while the loaded data was still in the local cache of the CPU core that loaded it, in which case the main thread would see the boolean go true before the data was actually ready.

Wow, it gets worse :)

Thanks for all the info, it's really opened my eyes. Your time spent in the forums answering these questions is really appreciated :-)

I found a few more interesting posts on MSDN on this subject:

Lockless Programming Considerations for Xbox 360 and Microsoft Windows
Synchronization and Multiprocessor Issues

Tomorrow I'm going to quiz my colleagues ;-)