Can the following code be true for pointers to different things

It is still possible for an allocation to generate a pointer that satisfies the condition despite the pointer not pointing into the region. This will happen, for example, on an 80286 in protected mode, which is used by Windows 3.x in Standard mode and OS/2 1.x.

In this system, pointers are 32-bit values, split into two 16-bit parts, traditionally written as XXXX:YYYY. The first 16-bit part (XXXX) is the “selector”, which chooses a bank of 64KB. The second 16-bit part (YYYY) is the “offset”, which chooses a byte within that 64KB bank. (It’s more complicated than this, but let’s just leave it at that for the purpose of this discussion.)

Memory blocks larger than 64KB are broken up into 64KB chunks. To move from one chunk to the next, you add 8 to the selector. For example, the byte after 0101:FFFF is 0109:0000.

But why do you add 8 to move to the next selector? Why not just increment the selector? Because the bottom three bits of the selector are used for other things.

In particular, the bottom bit of the selector is used to choose the selector table. (Let’s ignore bits 1 and 2 since they are not relevant to the discussion. Assume for convenience that they are always zero.)

There are two selector tables, the Global Selector Table (for memory shared across all processes) and the Local Selector Table (for memory private to a single process). Therefore, the selectors available for process private memory are 0001, 0009, 0011, 0019, etc. Meanwhile, the selectors available for global memory are 0008, 0010, 0018, 0020, etc. (Selector 0000 is reserved.)

Okay, now we can set up our counter-example. Suppose guardArea = 0101:0000 and guardSize = 0x00020000. This means that the guarded addresses are 0101:0000 through 0101:FFFF and 0109:0000 through 0109:FFFF. Furthermore, guardArea + guardSize = 0111:0000.

Meanwhile, suppose there is some global memory that happens to be allocated at 0108:0000. This is a global memory allocation because the selector is an even number.

Observe that the global memory allocation is not part of the guarded region, but its pointer value does satisfy the numeric inequality 0101:0000 <= 0108:0000 < 0111:0000.

Bonus chatter: Even on CPU architectures with a flat memory model, the test can fail. Modern compilers take advantage of undefined behavior and optimize accordingly. If they see a relational comparison between pointers, they are permitted to assume that the pointers point into the same array (or one past the last element of that array). Specifically, the only pointers that can legally be compared with guardArea are the ones of the form guardArea, guardArea+1, guardArea+2, …, guardArea + guardSize. For all of these pointers, the condition ptr >= guardArea is true and can therefore be optimized out, reducing your test to

if (ptr < (guardArea + guardSize))

which will now be satisfied for pointers that are numerically less than guardArea.

Moral of the story: This code is not safe, not even on flat architectures.

But all is not lost: The pointer-to-integer conversion is implementation-defined, which means that your implementation must document how it works. If your implementation defines the pointer-to-integer conversion as producing the numeric value of the pointer, and you know that you are on a flat architecture, then what you can do is compare integers rather than pointers. Integer comparisons are not constrained in the same way that pointer comparisons are.

if ((uintptr_t)ptr >= (uintptr_t)guardArea &&
    (uintptr_t)ptr < (uintptr_t)guardArea + (uintptr_t)guardSize)

Leave a Comment