Mmap() an entire large file

by

MAP_PRIVATE mappings require a memory reservation, as writing to these pages may result in copy-on-write allocations. This means that you can’t map something too much larger than your physical ram + swap. Try using a MAP_SHARED mapping instead. This means that writes to the mapping will be reflected on disk – as such, the kernel knows it can always free up memory by doing writeback, so it won’t limit you.

I also note that you’re mapping with PROT_WRITE, but you then go on and read from the memory mapping. You also opened the file with O_RDONLY – this itself may be another problem for you; you must specify O_RDWR if you want to use PROT_WRITE with MAP_SHARED.

As for PROT_WRITE only, this happens to work on x86, because x86 doesn’t support write-only mappings, but may cause segfaults on other platforms. Request PROT_READ|PROT_WRITE – or, if you only need to read, PROT_READ.

On my system (VPS with 676MB RAM, 256MB swap), I reproduced your problem; changing to MAP_SHARED results in an EPERM error (since I’m not allowed to write to the backing file opened with O_RDONLY). Changing to PROT_READ and MAP_SHARED allows the mapping to succeed.

If you need to modify bytes in the file, one option would be to make private just the ranges of the file you’re going to write to. That is, munmap and remap with MAP_PRIVATE the areas you intend to write to. Of course, if you intend to write to the entire file then you need 8GB of memory to do so.

Alternately, you can write 1 to /proc/sys/vm/overcommit_memory. This will allow the mapping request to succeed; however, keep in mind that if you actually try to use the full 8GB of COW memory, your program (or some other program!) will be killed by the OOM killer.

More Related Contents:

  1. Why does unaligned access to mmap’ed memory sometimes segfault on AMD64?
  2. When should I use mmap for file access?
  3. Unexpected exec permission from mmap when assembly files included in the project
  4. How to access mmaped /dev/mem without crashing the Linux kernel?
  5. Sharing memory between processes through the use of mmap()
  6. Linux shared memory: shmget() vs mmap()?
  7. How to clear input buffer in C?
  8. Correct format specifier for double in printf
  9. How do I work with dynamic multi-dimensional arrays in C?
  10. Is there a 128 bit integer in gcc?
  11. Checking available stack size in C
  12. Where is PATH_MAX defined in Linux?
  13. What is __stdcall?
  14. Recommended gcc warning options for C [closed]
  15. Does C have a standard ABI?
  16. How can I run a C program on Mac OS X using Terminal?
  17. Complex declarations
  18. What is the rationale for fread/fwrite taking size and count as arguments?
  19. Unsigned values in C
  20. Two Equal Signs in One Line?
  21. How to use kbhit and getch (C programming) [closed]
  22. Centering strings with printf()
  23. c99 goto past initialization
  24. How can I fix the scanf to take data into the array?
  25. Pipes, dup2 and exec()
  26. Difference between char a[]=”string”; char *p=”string”; [duplicate]
  27. Is it required that a C Variable Length Array is allocated from the stack?
  28. Function pointer as an argument
  29. Function declaration inside of function — why?
  30. How can I malloc a struct array inside a function? Code works otherwise

Leave a Comment