The fastest way to read input in Python

by

numpy has the functions loadtxt and genfromtxt, but neither is particularly fast. One of the fastest text readers available in a widely distributed library is the read_csv function in pandas (http://pandas.pydata.org/). On my computer, reading 5 million lines containing two integers per line takes about 46 seconds with numpy.loadtxt, 26 seconds with numpy.genfromtxt, and a little over 1 second with pandas.read_csv.

Here’s the session showing the result. (This is on Linux, Ubuntu 12.04 64 bit. You can’t see it here, but after each reading of the file, the disk cache was cleared by running sync; echo 3 > /proc/sys/vm/drop_caches in a separate shell.)

In [1]: import pandas as pd

In [2]: %timeit -n1 -r1 loadtxt('junk.dat')
1 loops, best of 1: 46.4 s per loop

In [3]: %timeit -n1 -r1 genfromtxt('junk.dat')
1 loops, best of 1: 26 s per loop

In [4]: %timeit -n1 -r1 pd.read_csv('junk.dat', sep=' ', header=None)
1 loops, best of 1: 1.12 s per loop

More Related Contents:

  1. How can I read inputs as numbers?
  2. What’s the difference between `raw_input()` and `input()` in Python 3?
  3. Time-Limited Input? [duplicate]
  4. How do you input integers using input in Python
  5. print statement inside of input returns with a “none”
  6. Backwards-compatible input calls in Python
  7. User input boolean in python
  8. How to blit a random image from a list Pygame [duplicate]
  9. Calling a python script with input within a python script using subprocess
  10. How do I format a string using a dictionary in python-3.x?
  11. certificate verify failed: unable to get local issuer certificate
  12. How do I resolve a TesseractNotFoundError?
  13. How to test a function with input call?
  14. Finding the Values of the Arrow Keys in Python: Why are they triples?
  15. Print list of lists in separate lines
  16. Type hints when unpacking a tuple?
  17. CSS not loading wrong MIME type Django [duplicate]
  18. What is “pkg-resources==0.0.0” in output of pip freeze command
  19. How to share numpy random state of a parent process with child processes?
  20. How can I print multiple things (fixed text and/or variable values) on the same line, all at once?
  21. Check if rows in one dataframe exist in another dataframe
  22. Multiple keys per value
  23. Python: can’t assign to literal
  24. How to reuse an expression in a comprehension expression?
  25. How to move the background image with keys in pygame?
  26. Split text after the second occurrence of character
  27. How to deal with unicode string in URL in python3?
  28. Search and replace dots and commas in pandas dataframe
  29. A more pythonic way to define an enum with dynamic members
  30. Correct Use Of Global Variables In Python 3

Leave a Comment