How to read huge file in c++

by

There are a couple of things that you can do.

First, there’s no problem opening a file that is larger than the amount of RAM that you have. What you won’t be able to do is copy the whole file live into your memory. The best thing would be for you to find a way to read just a few chunks at a time and process them. You can use ifstream for that purpose (with ifstream.read, for instance). Allocate, say, one megabyte of memory, read the first megabyte of that file into it, rinse and repeat:

ifstream bigFile("mybigfile.dat");
constexpr size_t bufferSize = 1024 * 1024;
unique_ptr<char[]> buffer(new char[bufferSize]);
while (bigFile)
{
    bigFile.read(buffer.get(), bufferSize);
    // process data in buffer
}

Another solution is to map the file to memory. Most operating systems will allow you to map a file to memory even if it is larger than the physical amount of memory that you have. This works because the operating system knows that each memory page associated with the file can be mapped and unmapped on-demand: when your program needs a specific page, the OS will read it from the file into your process’s memory and swap out a page that hasn’t been used in a while.

However, this can only work if the file is smaller than the maximum amount of memory that your process can theoretically use. This isn’t an issue with a 1TB file in a 64-bit process, but it wouldn’t work in a 32-bit process.

Also be aware of the spirits that you’re summoning. Memory-mapping a file is not the same thing as reading from it. If the file is suddenly truncated from another program, your program is likely to crash. If you modify the data, it’s possible that you will run out of memory if you can’t save back to the disk. Also, your operating system’s algorithm for paging in and out memory may not behave in a way that advantages you significantly. Because of these uncertainties, I would consider mapping the file only if reading it in chunks using the first solution cannot work.

On Linux/OS X, you would use mmap for it. On Windows, you would open a file and then use CreateFileMapping then MapViewOfFile.

More Related Contents:

  1. tellg() function give wrong size of file?
  2. How do I rename a file with a string using the rename() function (c++)? [closed]
  3. How to extract a multiline text segment between two delimiters under a certain heading from a text file using C++ [closed]
  4. program to delete certain text which is predefined in another file from a file in c++
  5. Odd error: ‘FILE *’ differs in levels of indirection from ‘FILE’ [closed]
  6. How can I get the list of files in a directory using C or C++?
  7. Fastest way to check if a file exists using standard C++/C++11,14,17/C?
  8. Getting a FILE* from a std::fstream
  9. Do I need to manually close an ifstream?
  10. Read Unicode UTF-8 file into wstring
  11. Create a directory if it doesn’t exist
  12. How to embed a file into an executable?
  13. c++ Read from .csv file
  14. C++ class header files organization
  15. c++ template and header files [duplicate]
  16. Read file names from a directory
  17. Write a file in a specific path in C++
  18. How to write to middle of a file in C++?
  19. std::fstream buffering vs manual buffering (why 10x gain with manual buffering)?
  20. Open QFile for appending
  21. How can I use non-default delimiters when reading a text file with std::fstream?
  22. Replace a line in text file
  23. C++: Where does the ofstream class save the files to?
  24. How to count lines of a file in C++?
  25. counting the number of lines in a text file
  26. Is it possible to use threads to speed up file reading?
  27. Reading a Matrix txt file and storing as an array
  28. Optimal lock file method
  29. Why does wide file-stream in C++ narrow written data by default?
  30. C++ ifstream error using string as opening file path.

Leave a Comment