I am getting an error when I add more than 16.7Million (unsigned 24 bit limit) objects in to a list class.

According to the documentation the index of a List<T> is int, which I understood to be int32 and therefore has 2.1Billion positive and negative limits.

I don't need the answer to this question to resolve any particular problem, I can re-write my code a more efficient way to get round it, however I am curious and could not get an answer from Google or the MSDN documentation (It's probably there I just could not find it, as is normal when I search MSDN.)

Regards

I thought the element count limit was the positive range of an Int32. But of course if you run out of memory first, you'll get an OutOfMemory exception.

What error are you getting?

Interesting question.

Can you post the exact wording of the error you are getting?

I've already re-written my code to trap for and avoid it.  I've gone back to try to reproduce the error but at the moment I cannot get the Index Out of Range error I can only get Out of Memory errors.  It could just be coincidental that this error happens when I reach an index of just over 16.777 Million!

Both machines I run it on have 2GB of RAM although one is XP and one is Vista.

You have answered my question though, the limit should be 2.1Billion, so the problem I need to solve (have apparently already solved) is not related to a List<T> limit.

Many thanks

The real question is why the heck would you even be trying to add that many items to a list?!?

Jim Perry:

why the heck would you even be trying to add that many items to a list?!?

Exactly! If you are doing that then you are probably doing something wrong.

Jim Perry:

The real question is why the heck would you even be trying to add that many items to a list?!?

I thought someone would ask that.

I'm creating an Octree.  To get small enough bounding spheres for character to scenery collision to feel right I split the scenery models triangles in to small pieces while creating the Octree, fairly normal practice as far as I can tell although the exact method of splitting varies by author.  My small pieces are then guaranteed to fit inside my smallest size Octree bounds.

Prior to saving the Octree I de-duplicate and generally reduce the result to only a small number of points per Octree bound.
This is all done on the PC and only for creating the map file.  It is never calculated in game.

My small test scenes worked in a single pass in just a few seconds, however as soon as I used a larger model, including two large triangles for the ground, I hit this problem.

I can reduce the quality of the Octree and it all works but I'm not happy with the distance that this allows a player to approach objects.

I am currently in the process of re-writing the code to create the Octree in batches of triangles, rather than all at once.

Any good advice on creating and using Octrees would be welcome as I've spent a lot of time getting a playable result I'm happy with but my initial version clearly does not scale well!

Many thanks for your interest.

I think that the fairly normal practice is not to split models into tris even if that means letting large objects occupy multiple leaf nodes.
You can't really render the scene on a per triangle basis, you render meshes. Therefore the only thing you care about is wether a mesh is inside a visible leaf or not.
Not sure what you mean by "this is all done on the PC and only for creating map file", seems you use the oct to precalculate something?

JCBDigger:

Any good advice on creating and using Octrees would be welcome as I've spent a lot of time getting a playable result I'm happy with but my initial version clearly does not scale well!

Are you running out of memory in the preprocessing only?  As in, if you managed to finish the precessing, the final result would likely fit into memory just fine?

If that is the case, then don't store your values in a list.  Instead store them to a file.  As long as you can guaruntee the object size, you can still seek directly to any 'index' that you desire for that.  A file can handle up to 4 gigs of data on a 32bit machine.  And if you have to go larger than that, then you can just use a 2nd file.  It will be much slower, but since this is just preprocessing, that shouldn't be too large of a concern.

If your objects are dynamic in size, then it will still work.  You will just lose the direct seek option and will have to parse everytime to get where you want.  Of course, that may slow you down to the point of non-usability, though.

Eckish:

Are you running out of memory in the preprocessing only? 

Yes.

Thank you for the idea.  On my forth attempt I have managed to create the Octree using smaller batches of data rather than in one go.

It turns out that if I have 16.7M triangles and recurse down to the eigth level (each level having 8 times the cubes of the previous level) I always run out of memory but 7 levels works.  The 7th level was not quite enough detail, so to get the 8th level I now add the triangles from one model at a time and tidy up the memory as I go.

The tip about using an intermediary file is a very good idea should I hit a similar problem that won't ever fit in to memory.

Thanks

ERiba:

Not sure what you mean by "this is all done on the PC and only for creating map file"?

I had not included enough detail in the earlier posts to explain exactly what I was doing.  I am creating a 3D First Person Shooter with Over the Shoulder view.  The same sort of view used in Gears or GRAW.

The other posters description of "preprocessing" is a good description.
If I try to do the processing on the Xbox, the load time for each level is over 20 minutes.  Preprocessing the Octree and saving it in the level map file, the load time on the Xbox is reduced to less than one minute.

ERiba:

Therefore the only thing you care about is whether a mesh is inside a visible leaf or not.

The Octree is used for collision not for rendering although your comments have given me an idea of how I could use the Octree instead of the view frustum for mesh culling prior to rendering.  (Sorry, side tracked, but thanks.)

The end result of the Octree calculation does not store the triangles it stores a single point anywhere in the boundingBox of each occupied Octree. 
Each occupied Octree is just a boundingSphere.  I use that for sphere on sphere collision, and the detail is enough to be able to nicely move round a complex shape in 3D and for it to feel right to the player.

I also use the Octree for projectile collision and do a ray on sphere intersection to get the point very close to the point of impact for a bullet.  In addition to the sphere of the Octree I store a link back to the original triangle in the original model, just a pair of ints, model, triangle.  Typically any Octree only contains one sometimes two references to triangles.  Even on the Xbox the system is efficient enough that I can do a final ray on triangle intersection and get the exact point of impact and orientation of the scenery so that I can display a perfectly positioned decal to represent a bullet impact.  Plus of course a particle effect for the dust and debris.

The Octree had worked well on my small test scenes and it's only when I started creating larger test scenes that I encounterred the problem creating it.

Regards

JCBDigger:

I am creating a 3D First Person Shooter with Over the Shoulder view.  The same sort of view used in Gears or GRAW.

Nitpicking here, but that would be a 3D Third Person Shooter.   ;)

This is how the english language deteriorates.

WolRon:

Nitpicking here, but that would be a 3D Third Person Shooter.   ;)
This is how the english language deteriorates.

:-)

Ahh, good try, but no.

The First person relates to the control system, not just the view.

I guess I should say, 3D First person control shooter with over the shoulder third person view... 

... but I'm not going to. :-)

Regards


PS. I have never been able to get the hang of third person controls, it's like remote control cars, I could not do those either.
[PPS. I edited this to make it shorter.]