What does O_DIRECT really mean?

by

(This answer pertains to Linux – other OSes may have different caveats/semantics)

Let’s start with the sub-question:

If I open a file with O_DIRECT flag, does it mean that whenever a write(blocking mode) to that file returns, the data is on disk?

No (as @michael-foukarakis commented) – if you need a guarantee your data made it to non-volatile storage you must use/add something else.

What does O_DIRECT really mean?

It’s a hint that you want your I/O to bypass the Linux kernel‘s caches. What will actually happen depends on things like:

  • Disk configuration
  • Whether you are opening a block device or a file in a filesystem
  • If using a file within a filesystem
    • The exact filesystem used and the options in use on the filesystem and the file
    • Whether you’ve correctly aligned your I/O
    • Whether a filesystem has to do a new block allocation to satisfy your I/O
  • If the underlying disk is local, what layers you have in your kernel storage stack before you reach the disk block device
  • Linux kernel version
  • …

The list above is not exhaustive.

In the “best” case, setting O_DIRECT will avoid making extra copies of data while transferring it and the call will return after transfer is complete. You are more likely to be in this case when directly opening block devices of “real” local disks. As previously stated, even this property doesn’t guarantee that data of a successful write() call will survive sudden power loss. IF the data is DMA’d out of RAM to non-volatile storage (e.g. battery backed RAID controller) or the RAM itself is persistent storage THEN you may have a guarantee that the data reached stable storage that can survive power loss. To know if this is the case you have to qualify your hardware stack so you can’t assume this in general.

In the “worst” case, O_DIRECT can mean nothing at all even though setting it wasn’t rejected and subsequent calls “succeed”. Sometimes things in the Linux storage stack (like certain filesystem setups) can choose to ignore it because of what they have to do or because you didn’t satisfy the requirements (which is legal) and just silently do buffered I/O instead (i.e. write to a buffer/satisfy read from already buffered data). It is unclear whether extra effort will be made to ensure that the data of an acknowledged write was at least “with the device” (but in the O_DIRECT and barriers thread Christoph Hellwig posts that the O_DIRECT fallback will ensure data has at least been sent to the device). A further complication is that using O_DIRECT implies nothing about file metadata so even if write data is “with the device” by call completion, key file metadata (like the size of the file because you were doing an append) may not be. Thus you may not actually be able to get at the data you thought had been transferred after a crash (it may appear truncated, or all zeros etc).

While brief testing can make it look like data using O_DIRECT alone always implies data will be on disk after a write returns, changing things (e.g. using an Ext4 filesystem instead of XFS) can weaken what is actually achieved in very drastic ways.

As you mention “guarantee that the data” (rather than metadata) perhaps you’re looking for O_DSYNC/fdatasync()? If you want to guarantee metadata was written too, you will have to look at O_SYNC/fsync().

References

More Related Contents:

  1. Convert DOS line endings to Linux line endings in Vim
  2. What happens to an open file handle on Linux if the pointed file gets moved or deleted
  3. What is double dot(..) and single dot(.) in Linux?
  4. how can i show the size of files in /proc? it should not be size zero
  5. Quickly create a large file on a Linux system
  6. How to get full path of a file?
  7. How can I randomize the lines in a file using standard tools on Red Hat Linux?
  8. Relinking an anonymous (unlinked but open) file
  9. How can I remove the BOM from a UTF-8 file? [duplicate]
  10. Remove a symlink to a directory
  11. Why no output is shown when using grep twice?
  12. “Max open files” for working process
  13. How to read a file line by line and making a new file for each line having same filename as last word of each line? [closed]
  14. How to change permissions for a folder and its subfolders/files in one step
  15. How can I link to a specific glibc version?
  16. Setting up FTP on Amazon Cloud Server [closed]
  17. Using curl to send email
  18. Can I open a socket and pass it to another process in Linux
  19. Get the Perl rename utility instead of the built-in rename
  20. os.Mkdir and os.MkdirAll permissions
  21. Is the UNIX `time` command accurate enough for benchmarks? [closed]
  22. How to remove non UTF-8 characters from text file
  23. Is there really no asynchronous block I/O on Linux?
  24. How to compare two DateTime strings and return difference in hours? (bash shell)
  25. Keycloak https auth page unable to acces
  26. Move all files except one
  27. get string length in inline GNU Assembler
  28. How to calculate CPU utilization of a process & all its child processes in Linux?
  29. How to create a link to a directory on linux [closed]
  30. Find all files with name containing string [closed]

Leave a Comment