Address canonical form and pointer arithmetic

by

The canonical address rules mean there is a giant hole in the 64-bit virtual address space. 2^47-1 is not contiguous with the next valid address above it, so a single mmap won’t include any of the unusable range of 64-bit addresses.

+----------+
| 2^64-1   |   0xffffffffffffffff
| ...      |
| 2^64-2^47|   0xffff800000000000
+----------+
|          |
| unusable |      not to scale: this part is 2^16 times as large
|          |
+----------+
| 2^47-1   |   0x00007fffffffffff
| ...      |
| 0        |   0x0000000000000000
+----------+

Also most kernels reserve the high half of the canonical range for their own use. e.g. x86-64 Linux’s memory map. User-space can only allocate in the contiguous low range anyway so the existence of the gap is irrelevant.

Is there a guarantee by the OS that you will never be allocated memory whose address range does not vary by the 47th bit?

Not exactly. The 48-bit address space supported by current hardware is an implementation detail. The canonical-address rules ensure that future systems can support more virtual address bits without breaking backwards compatibility to any significant degree.

At most, you’d just need a compat flag to have the OS not give the process any memory regions with high bits not all the same. (Like Linux’s current MAP_32BIT flag for mmap, or a process-wide setting). That could support programs that used the high bits for tags and manually redid sign-extension.

Future hardware won’t need to support any kind of flag to ignore high address bits or not, because junk in the high bits is currently an error. Intel 5-level paging adds another 9 virtual address bits, widening the canonical high andd low halves. white paper.

See also Why in 64bit the virtual address are 4 bits short (48bit long) compared with the physical address (52 bit long)?

Fun fact: Linux defaults to mapping the stack at the top of the lower range of valid addresses. (Related: Why does Linux favor 0x7f mappings?)

$ gdb /bin/ls
...
(gdb) b _start
Function "_start" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_start) pending.
(gdb) r
Starting program: /bin/ls

Breakpoint 1, 0x00007ffff7dd9cd0 in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) p $rsp
$1 = (void *) 0x7fffffffd850
(gdb) exit

$ calc
2^47-1
              0x7fffffffffff

(Modern GDB can use starti to break before the first user-space instruction executes instead of messing around with breakpoint commands.)

More Related Contents:

  1. Why in x86-64 the virtual address are 4 bits shorter than physical (48 bits vs. 52 long)?
  2. Why can’t kernel code use a Red Zone
  3. Why do x86-64 systems have only a 48 bit virtual address space?
  4. Disable randomization of memory addresses
  5. x86-64 canonical address?
  6. What EXACTLY is the difference between intel’s and amd’s ISA, if any?
  7. Why use RIP-relative addressing in NASM?
  8. Why does Windows64 use a different calling convention from all other OSes on x86-64?
  9. Is there a 128 bit integer in gcc?
  10. Submit to App Store issues: Unsupported Architecture x86
  11. Basic use of immediates vs. square brackets in YASM/NASM x86 assembly
  12. Floating point vs integer calculations on modern hardware
  13. Why are rbp and rsp called general purpose registers?
  14. Why does adding an xorps instruction make this function using cvtsi2ss and addss ~5x faster?
  15. Lost Cycles on Intel? An inconsistency between rdtsc and CPU_CLK_UNHALTED.REF_TSC
  16. Why NASM on Linux changes registers in x86_64 assembly
  17. What’s the difference between “statically linked” and “not a dynamic executable” from Linux ldd?
  18. Compiler using local variables without adjusting RSP
  19. Difference between x86, x32, and x64 architectures?
  20. Is garbage allowed in high bits of parameter and return value registers in x86-64 SysV ABI?
  21. Swift, Strings and Memory Addresses
  22. When should I use size directives in x86?
  23. Count each bit-position separately over many 64-bit bitmasks, with AVX but not AVX2
  24. Why does printf print random value with float and integer format specifier
  25. Pointer arithmetic when void has unknown size [closed]
  26. Handling calls to (potentially) far away ahead-of-time compiled functions from JITed code
  27. x86_64: Is it possible to “in-line substitute” PLT/GOT references?
  28. Why does Math.Exp give different results between 32-bit and 64-bit, with same input, same hardware
  29. Has Hardware Lock Elision gone forever due to Spectre Mitigation?
  30. Force gdb to load shared library at randomized address

Leave a Comment