The POSIX standard (IEEE 1003.1:2008) prototypes open()
as:
int open(const char *path, int oflag, ...);
The section describing the behaviour of O_CREAT
doesn’t say what will happen if you omit the necessary third argument, which means the behaviour is undefined – anything is possible.
In practice, the use of part of the stack that was intended to be stack frame or return address or something similar is quite likely – unto a reasonable approximation, that can be considered a random integer.
The POSIX 2008 standard has some interesting new (and useful) flags for open()
, including:
O_FDCLOEXEC
to specify close-on-exec at open.O_DIRECTORY
to specify that the file must be a directory.O_NOFOLLOW
to specify not to chase symlinks.