How can it be that writing to physical disk is WAY faster than writing to the “screen” (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?
Congratulations, you have just discovered the importance of I/O buffering. 🙂
The disk appears to be faster, because it is highly buffered: all Python’s write() calls are returning before anything is actually written to physical disk. (The OS does this later, combining many thousands of individual writes into a big, efficient chunks.)
The terminal, on the other hand, does little or no buffering: each individual print / write(line) waits for the full write (i.e. display to output device) to complete.
To make the comparison fair, you must make the file test use the same output buffering as the terminal, which you can do by modifying your example to:
fp = file("out.txt", "w", 1) # line-buffered, like stdout
[...]
for x in range(lineCount):
fp.write(line)
os.fsync(fp.fileno()) # wait for the write to actually complete
I ran your file writing test on my machine, and with buffering, it also 0.05s here for 100,000 lines.
However, with the above modifications to write unbuffered, it takes 40 seconds to write only 1,000 lines to disk. I gave up waiting for 100,000 lines to write, but extrapolating from the previous, it would take over an hour.
That puts the terminal’s 11 seconds into perspective, doesn’t it?
So to answer your original question, writing to a terminal is actually blazingly fast, all things considered, and there’s not a lot of room to make it much faster (but individual terminals do vary in how much work they do; see Russ’s comment to this answer).
(You could add more write buffering, like with disk I/O, but then you wouldn’t see what was written to your terminal until after the buffer gets flushed. It’s a trade-off: interactivity versus bulk efficiency.)