Speed up writing to files

by

Actually, your problem is not that file.write() takes 20% of your time. Its that 80% of the time you aren’t in file.write()!

Writing to the disk is slow. There is really nothing you can do about it. It simply takes a very large amount of time to write stuff out to disk. There is almost nothing you can do to speed it up.

What you want is for that I/O time to be the biggest part of the program so that your speed is limited by the speed of the hard disk not your processing time. The ideal is for file.write() to have 100% usage!

More Related Contents:

  1. File read using “open()” vs “with open()” [duplicate]
  2. Is doing multiplication multiple times or assigning to a variable faster?
  3. Lazy Method for Reading Big File in Python?
  4. Reading binary file and looping over each byte
  5. open() gives FileNotFoundError/IOError: Errno 2 No such file or directory
  6. pandas loc vs. iloc vs. at vs. iat?
  7. How to write a multidimensional array to a text file?
  8. What is the best way to generate all possible three letter strings?
  9. Why is [] faster than list()?
  10. Read file from line 2 or skip header row
  11. Why is numpy’s einsum faster than numpy’s built in functions?
  12. Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell
  13. What’s a faster operation, re.match/search or str.find?
  14. Cost of exception handlers in Python
  15. Writing a list to a file with Python, with newlines
  16. Most efficient way to forward-fill NaN values in numpy array
  17. [] and {} vs list() and dict(), which is better?
  18. Python Pandas: How to read only first n rows of CSV files in?
  19. python: deque vs list performance comparison
  20. multiprocessing.Pool() slower than just using ordinary functions
  21. How to safely open/close files in python 2.4
  22. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  23. Is & faster than % when checking for odd numbers?
  24. Get difference between two lists with Unique Entries
  25. How to read a single character at a time from a file in Python?
  26. Fastest save and load options for a numpy array
  27. Count() vs len() on a Django QuerySet
  28. What is the fastest way to output large DataFrame into a CSV file?
  29. Treat binary data as a file object?
  30. Unicode encoding for filesystem in Mac OS X not correct in Python?

Leave a Comment