How do you handle memory address alignment for unbuffered io?

Sep 17, 2013 at 3:25 AM
According to the MSDN article:
http://msdn.microsoft.com/en-us/library/windows/desktop/cc644950(v=vs.85).aspx

"•File access buffer addresses for read and write operations should be physical sector-aligned, which means aligned on addresses in memory that are integer multiples of the volume's physical sector size. Depending on the disk, this requirement may not be enforced."

and further.

"Because buffer addresses for read and write operations must be sector-aligned, the application must have direct control of how these buffers are allocated. One way to sector-align buffers is to use the VirtualAlloc function to allocate the buffers. Consider the following:
• VirtualAlloc allocates memory that is aligned on addresses that are integer multiples of the system's page size. Page size is 4,096 bytes on x64 and x86 or 8,192 bytes for Itanium-based systems. For additional information, see the GetSystemInfo function.
•Sector size is typically 512 to 4,096 bytes for direct-access storage devices (hard drives) and 2,048 bytes for CD-ROMs.
•Both page and sector sizes are powers of 2.

Therefore, in most situations, page-aligned memory will also be sector-aligned, because the case where the sector size is larger than the page size is rare.

Another way to obtain manually-aligned memory buffers is to use the _aligned_malloc function from the C Run-Time library. For an example of how to manually control buffer alignment, see the C++ language code example in the Example Code section of WriteFile."

It does not look like you are holding up this end of the contract by looking at your source, you are just passing .NET byte arrays that could be anywhere.
Coordinator
Sep 19, 2013 at 2:32 AM
Edited Sep 20, 2013 at 8:10 PM
This is a very good question, AceHack. To answer your question, the Memory Address alignment is handled by the .Net FileStream API, the ctor override version expecting a block size and a handle created using a Win32 CreateFile API. You will see the actual code how SOP uses these API to take advantage of this low-level feature in Sop/SystemInterface/Win32.cs: UnbufferedOpen method.

Everywhere else in SOP, yes, buffering is done on standard byte array, which is by design. FileStream I/O methods works low-level, transfer the data to/from its private Memory aligned buffer which uses VirtualAlloc and then does direct, unbuffered I/O to/from the disk. I consider this solution by far, the only feasible way to achieve optimal I/O in c#/.Net without imposing unnecessary latencies due to low-level techniques such as memory pinning/virtual alloc, etc... which doesn't work well with CLR if done systemwide.

By isolating system specificality in the low-level I/O methods, SOP not only became portable, but at same time, the design seems kept optimal, 'zero unnecessary latencies.

FYI: you can learn more about the c#/.Net unbuffered I/O using "FileStream + Win32 CreateFile" technique outlined in Jim Grey's research paper, downloadable sample program in this link: http://research.microsoft.com/en-us/downloads/a6f86a1e-0278-4c72-9300-f747251c3bf0/

For the block size computation, they are located in these files:
1.) Sop/OnDisk/File/File.cs:UnbufferedOpen method computes appropriate block size depending on the Store block size(e.g. - 512 kb), typically, the computed block size is larger than 8kb which should be good even for Itanium systems.
2.) Sop/SystemInterface/Win32.cs: UnbufferedOpen method contains fail safe logic enforcing block size to at least 4kb.

I hope I was able to answer your question clearly and completely. Have fun! :)
-Gerardo
Oct 2, 2013 at 6:43 AM
I've read over that paper in the past and it seems he is saying you can use VirutalAlloc in native code to help with memory alignment. I've been digging into reflection on the FileStream class since I made this post and I've found nothing that would indicate FileStream is doing any kind of alignment. Very much the opposite, I've found that FileStream just uses a cast of the normal .NET buffer in unsafe mode to fixed byte * which will pin the array long enough to send to the C++ WriteFile call. I don't see any alignment or VirtualAlloc. I'm really having a hard time understanding how you are doing alignment.

This requirement of memory alignment seems to not be enforced on every drive I've tested, SSDs, Rotating, Raids, I think it's been lucky that this approach works. If you were to have a drive that has the memory alignment requirement then I am almost certain this approach would fail, it should throw an exception so it would not cause any corruption so that's good. It's to bad I don't have a disk that enforces this requirement to test it out on. I've been able to invalidate this memory alignment in C++ as well with no issues to validate my non enforcement theory.

Could you please point me to some reflected .NET source that shows the use of VirtualAlloc or any other alignment methods so I can feel comfortable with this approach? When I've seen others use unbuffered io in the past in .NET it required the use of InPtrs of memory aligned buffers. I really, really hope I'm wrong on this because this approach is super clean and very nice to work with in .NET. Even if I'm not wrong maybe it does not matter since I've been unable to find a drive that even enforces this rule.

Thanks so much.
Coordinator
Oct 7, 2013 at 4:52 AM
Edited Mar 2, 2014 at 6:32 PM
Short Answer: VirtualAlloc is not necessary bcoz of isolated/exclusive use of the internal buffer. Memory alignment is a trivial (c++) ptr arithmetic on the unmanaged, local buffer.