File streams store results into byte [], however I want the destination to be CustomStruct [].  I know I can force structures to have no "air".  But I cannot convert byte [] into CustomStruct [] with managed code.  Is using "unsafe" the only sane way of solving this, via converting the "pointer to CustomStruct" into a "pointer to byte" to satisfy the stream readers?  Speed is important, so XML and other non-binary text storage are unacceptable.

Update: It looks like even using unsafe code, it's difficult to convert "byte*" into "byte[]".  Also, I don't like using unsafe code since it can cause deployment issues, which I am unsure of what that entails on the Xbox 360.  I am going to code around this using safe code.  Uglier and slower, but apparently necessary.  Why I cannot just read a data stream of bytes into an array of custom structures is beyond me, as it's the same thing as using a BinaryReader and reading in one native data type at a time, which is allowed, but is foolishly slow.

Thanks for your time.

This wouldn't work? Just use a loop for each item in the array.

That is exactly what I want.  Thanks Jim!!

Actually, it's not exactly what I want.

I have a large array of a small struct.  Doing multiple reads on small amounts of data is slow.  I'd prefer to read it all in at once, straight into an array of struct.

I haven't tried it, but maybe if you create a Collection class that holds the array of structs, and mark the Collection class as Serializable, and serialize/deserialize the Collection object?

Jim Perry:

This wouldn't work? Just use a loop for each item in the array.

This requires:

using System.Runtime.Serialization; // IFormatter
using System.Runtime.Serialization.Formatters.Binary; // BinaryFormatter

But "Formatters" isn't available in the Xbox 360 build:

"The type or namespace name 'Formatters' does not exist in the namespace 'System.Runtime.Serialization' (are you missing an assembly reference?)"

BinaryFormatter is not supported on the Xbox 360.
BinaryWriter and BinaryReader are supported on the Xbox 360, but they appear limited to reading/writing single types one at a time.

JasonD:

Doing multiple reads on small amounts of data is slow. 

Have you timed this, or are you guessing?

My experience is that BinaryReader is usually pretty fast.

JasonD:

I'd prefer to read it all in at once, straight into an array of struct.

You can't do that in .NET.

You could read into a large byte array, then use unsafe code to copy those bytes over your array of the other type.

BinaryReader and BinaryWriter are fine, but you'll have to serialize the struct fields individually.  A bit of a pain but not a big deal if you only have a few types to serialize. 

You could make a more generic system with reflection, but I wouldn't bother unless you have dozens of different types to serialize.

Don't worry much about small reads, all the C# readers are buffered and read in reasonable sized chunks.  You can always read the whole file and then use a memory stream if you're really worried about I/O speed.

Shawn Hargreaves:

JasonD:

Doing multiple reads on small amounts of data is slow. 

Have you timed this, or are you guessing?
My experience is that BinaryReader is usually pretty fast.

I am guessing based on past experience (i.e. DOS days) when I timed reading in files 1 byte (or 20 bytes) at a time vs. the entire file at once, with disk caching in effect.  I know how the hardware works and the software caches, and reading in one structure at a time is horribly slower than just reading the whole thing in.  I would be surprised if this didn't continue to be true today, since I don't think much has changed in hard drive hardware or in software caching.  Consider I am designing for worst case scenario of 12 million such structures.  Every bit of speed matters.  1 microsecond gained per structure = 12 seconds faster read.

Shawn Hargreaves:

JasonD:

I'd prefer to read it all in at once, straight into an array of struct.

You can't do that in .NET.
You could read into a large byte array, then use unsafe code to copy those bytes over your array of the other type.

I think I need to read the whole thing into byte[].  If I can quickly convert this over to my own structure, then I'll be happy.  If I can't, it may actually be faster to read more information from the HD, say from an XML file, and let the internal .NET framework do its magic.  I'd prefer to do the magic myself, in pure assembly, if that were possible -- I want to get as close to the hardware as possible.  12 million structures to be read in means every bit of speed matters.  I would compress the data to reduce file size if I the decompression time is faster than the time gained via smaller file size read.

fr3shme4t:

You can always read the whole file and then use a memory stream if you're really worried about I/O speed.

I am concerned with I/O speed reading 200 MB+ and the CPU speed of processing of 12,000,000 such structures.  I just want raw access, as close to the hardware (and software) as I can get, so I can do it as fast as possible.  There is no need to read one structure at a time when I already know I'm reading the whole lot in.  I don't want to deserialize using generic code (even though I know it's fast), when I could use custom compression and decompression to reduce file size and potentially decompress faster than deserialization.  This is not one of those things where you do it the quick way and then profile to find the slow spots.  I already know it's too slow.  I want to do it as fast as possible right from the get go.  If the result is faster than it needs to be, I can make use of that and do even more with the data.  So fastest speed is the first step in my endeavor.

JasonD:

I would be surprised if this didn't continue to be true today, since I don't think much has changed in hard drive hardware or in software caching.

One hugely important thing that has changed is that CPU speeds have increased an order of magnitude faster than hard drive speeds.

Also, the chance of getting true I/O parallelism between CPU and HD is much greater than it was in the days of DOS, thanks to smart HD controllers with internal cache, readahead buffers, and widespread DMA support.

So, these days you are more likely to be bottlenecked by actual device IO speed, and less likely to be bottlenecked by driver work or other CPU side data processing.

JasonD:

If I can't, it may actually be faster to read more information from the HD, say from an XML file, and let the internal .NET framework do its magic.

I guarantee you that reading XML will be slower than pretty much any way you could possibly try to read a binary file.

I hadn't considered smart HD controllers with internal caches & read-ahead buffers, etc.  I want to read the data into memory as fast as the HD reveals it, and just because it reads ahead, which speeds up the next read, does this speed up the next 1,000,000 reads?  Or is it still faster to just read all 200,000,000 bytes in one go?  I'm not sure; I need to test this.  Reading all 200 MB directly into byte [] is as fast as it gets considering only the HD, however I must decompress the data, thus reading one structure at a time, and decompressing it while the HD controller is waiting for more info, may be the fastest overall time for both reading + decompression.

Either way, my problem is converting byte [] --> MyStruct [] as fast as possible.  Can I pass File.Read() a "pointer to MyStruct []" and not bother with software conversion?  Why waste CPU time if I can read it directly into the structure array that wants the data?

JasonD:

Or is it still faster to just read all 200,000,000 bytes in one go?

As always with performance questions, this depends on the details of the hardware you are running on, specifics of your workload, and on what else is going on at the same time.

In general, though, IO code on a modern computer is likely to be spinning the CPU while it waits for the order-of-magnitude slower hardware device. That being so, spending additional CPU cycles to process the data is often not a concern, and may even be free as it can overlap with what would otherwise have been a subsequent hardware IO wait time. And if your processing code reduces the size of the data in any way (even using quite expensive CPU computations for fancy compression algorithms), that is likely to be a major perf win.

JasonD:

Can I pass File.Read() a "pointer to MyStruct []" and not bother with software conversion?

You cannot. The .NET file read calls take a byte array, not a pointer or an array of some other type.

Like I said earlier: "you could read into a large byte array, then use unsafe code to copy those bytes over your array of the other type".

Shawn Hargreaves:

if your processing code reduces the size of the data in any way (even using quite expensive CPU computations for fancy compression algorithms), that is likely to be a major perf win.

This is precisely what I am thinking.  My only real concern with this is to do the compression in a way such that I can update partial sections of the data file.  I don't want to re-write 200 MB if only 5% of it has changed (although maybe a 200 MB write is rather quick?  I'll have to test that.)

Shawn Hargreaves:

The .NET file read calls take a byte array, not a pointer or an array of some other type.

I already know this, already asked, and you already answered.  Sorry.  I guess I am struggling to find a better way.  I am used to C++ and assembly and I can do whatever I want when I want.  I know C# runs on the CLR and currently there's no way to bypass that to get at the IL or the assembly/machine code.  But C# can call other functions, say, from C++, which could have assembly in them.  Is there anyway for C# on the Xbox 360 to do such a thing?  Is there any way to bypass the C# mechanisms that force programmers to "be nice"?

JasonD:

s there anyway for C# on the Xbox 360 to do such a thing?

Not possible.  Sorry if you don't like that answer, but you're wasting your time looking for such a thing.

The right way to do this is to read into a byte array, after which you can do whatever you like with unsafe code.

Understood.

Thanks for your help Shawn, as always.