A friend of mine was talking about an application he wrote, and mentioned that reading the data file was a bit painful. He used the System.IO.BinaryReader class, and called the relevant method for each and every field. It probably went something like this:
static MyStruct[] ReadStructArray1(Stream file, int count)
{
MyStruct[] results = new MyStruct[count];
using (BinaryReader reader = new BinaryReader(file))
{
for (int i = 0; i < count; ++i)
{
results[i].field1 = reader.ReadInt32();
results[i].field2 = reader.ReadInt32();
// etc...
}
}
return results;
}
The drawback is you have to write many lines of code, keeping them in sync with the structure definition, and it is inefficient. You won't be able to get anywhere near the sequential disk read rate with this approach. I came across this post by Eric Gunnerson, which outlines a technique that at least removes the repetitive lines, at the cost of unsafe code:
static unsafe MyStruct[] ReadStructArray2(Stream file, int count)
{
MyStruct[] results = new MyStruct[count];
int structSize = sizeof(MyStruct);
byte[] buffer = new byte[structSize];
for (int i = 0; i < count; ++i)
{
if (file.Read(buffer, 0, structSize) != structSize)
throw new InvalidDataException();
fixed (byte* pBuffer = buffer)
{
results[i] = *(MyStruct*)pBuffer;
}
}
return results;
}
This is better, but results in small reads (a struct at a time), and an extra memory copy from the buffer into the struct. This still won't be able to drive the disk at full speed, although should be faster than the first approach, at least if the struct is fairly large and includes a meaningful number of fields. One significant limitation of this solution is that the struct must be unmanaged - i.e. it must be a value type, such as a struct, and all fields must also be value types.
It is relatively easy to extend this to read directly into the array of structures. At first I tried using the various Read methods on Stream or related classes, but they all expect a byte array, instead of a byte*, and I don't believe one can cast a MyStruct[] into a byte[] (which intuitively makes sense - since the size of each element is different between a byte[] and MyStruct[], how would the Length property work? this of course in addition to other problems).
By using P/Invoke to call the Win32 API ReadFile, we can use a byte* for the buffer, which is an easy cast from our MyStruct[]:
static unsafe MyStruct[] ReadStructArray3(SafeHandle file, int count)
{
MyStruct[] results = new MyStruct[count];
fixed (MyStruct* p = &results[0])
{
byte* p2 = (byte*)p;
UInt32 cbReadDesired = (UInt32)(sizeof(MyStruct) * count);
UInt32 cbReadActual = 0;
bool ret = ReadFile(file, p2, cbReadDesired, &cbReadActual, null);
if (!ret || cbReadActual != cbReadDesired)
throw new InvalidDataException();
}
return results;
}
This implementation avoids any extra memory copies, beyond what the OS and disk subsystem need. There are a few points worth discussing in this implementation:
- MyStruct must be unmanaged, just as with ReadStructArray2.
- Use of the "fixed" statement: this was used in ReadStructArray2 also, but I didn't explain it. It is used to keep the runtime from moving the results array in memory, which would obviously cause the ReadFile function to do strange things (corrupting random bits of your program, resulting in crashes or worse).
- SafeHandle - this class was added in .NET 2.0, and is a significantly better way to use OS handles than IntPtr. You can get a SafeHandle for a file from the FileStream.SafeFileHandle property.
- The in-memory layout of the structure must match the on disk layout; in particular padding may be a problem. See System.Runtime.InteropServices.StructLayoutAttribute for a solution to this problem.
The declaration for ReadFile is as follows:
[DllImport("kernel32", SetLastError = true)]
private static extern unsafe bool ReadFile(
SafeHandle hFile,
byte* lpBuffer,
UInt32 nNumberOfBytesToRead,
UInt32* lpNumberOfBytesRead,
System.Threading.NativeOverlapped* lpOverlapped);
Next was a quick test driver that reads a 1GB file. Each approach was run 10 times, the low and high were tossed out, with the average of remaining values listed below:
| ReadStructArray1 | 9650 ms |
| ReadStructArray2 | 3900 ms |
| ReadStructArray3 | 1650 ms |
-randy
Comments