Fastest way to write huge data in file

by

Two major reasons for observed “slowness”:

  • your while loop is slow, it has about a million iterations.
  • You do not make proper use of I/O buffering. Do not make so many system calls. Currently, you are calling write() about one million times.

Create your data in a Python data structure first and call write() only once.

This is faster:

t0 = time.time()
open("bla.txt", "wb").write(''.join(random.choice(string.ascii_lowercase) for i in xrange(10**7)))
d = time.time() - t0
print "duration: %.2f s." % d

Output: duration: 7.30 s.

Now the program spends most of its time generating the data, i.e. in random stuff. You can easily see that by replacing random.choice(string.ascii_lowercase) with e.g. "a". Then the measured time drops to below one second on my machine.

And if you want to get even closer to seeing how fast your machine really is when writing to disk, use Python’s fastest (?) way to generate largish data before writing it to disk:

>>> t0=time.time(); chunk="a"*10**7; open("bla.txt", "wb").write(chunk); d=time.time()-t0; print "duration: %.2f s." % d
duration: 0.02 s.

More Related Contents:

  1. How do I re.search or re.match on a whole file without reading it all into memory?
  2. How do I check whether a file exists without exceptions?
  3. Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?
  4. String formatting: % vs. .format vs. f-string literal
  5. When should I (not) want to use pandas apply() in my code?
  6. Flask raises TemplateNotFound error even though template file exists
  7. How to measure elapsed time in Python?
  8. How to read specific lines from a file (by line number)?
  9. Python convert csv to xlsx
  10. What is the perfect counterpart in Python for “while not EOF”
  11. Link ATLAS/MKL to an installed Numpy
  12. What is the time complexity of popping elements from list in Python?
  13. Python list directory, subdirectory, and files
  14. What does ‘wb’ mean in this code, using Python?
  15. How to save a list to a file and read it as a list type?
  16. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  17. How do you read a file into a list in Python? [duplicate]
  18. Fast arbitrary distribution random sampling (inverse transform sampling)
  19. Mass string replace in python?
  20. Python: get a dict from a list based on something inside the dict
  21. Why is variable1 += variable2 much faster than variable1 = variable1 + variable2?
  22. Check if object is file-like in Python
  23. Python built-in sum function vs. for loop performance
  24. How to read one single line of csv data in Python?
  25. Dictionary vs Object – which is more efficient and why?
  26. sum each value in a list of tuples
  27. Why is `with open()` better for opening files in Python?
  28. differences between “d = dict()” and “d = {}”
  29. numpy faster than numba and cython , how to improve numba code
  30. Change the file extension for files in a folder?

Leave a Comment