Searching for a string in a large text file – profiling various methods in python

by

Variant 1 is great if you need to launch many sequential searches. Since set is internally a hash table, it’s rather good at search. It takes time to build, though, and only works well if your data fit into RAM.

Variant 3 is good for very big files, because you have plenty of address space to map them and OS caches enough data. You do a full scan; it can become rather slow once your data stop to fit into RAM.

SQLite is definitely a nice idea if you need several searches in row and you can’t fit the data into RAM. Load your strings into a table, build an index, and SQLite builds a nice b-tree for you. The tree can fit into RAM even if data don’t (it’s a bit like what @alienhard proposed), and even if it doesn’t, the amount if I/O needed is dramatically lower. Of course, you need to create a disk-based SQLite database. I doubt that memory-based SQLite will beat Variant 1 significantly.

More Related Contents:

  1. How do I profile a Python script?
  2. Which Python memory profiler is recommended? [closed]
  3. Why does Python code run faster in a function?
  4. Most efficient way for a lookup/search in a huge list (python)
  5. Fastest way to check if a value exists in a list
  6. Get difference between two lists
  7. Why does concatenation of DataFrames get exponentially slower?
  8. Roll rows of a matrix independently
  9. Efficient calculation of Fibonacci series
  10. Python string ‘join’ is faster (?) than ‘+’, but what’s wrong here?
  11. Bin elements per row – Vectorized 2D Bincount for NumPy
  12. Why is looping over range() in Python faster than using a while loop?
  13. Vectorized searchsorted numpy
  14. python: read lines from compressed text files
  15. why isn’t numpy.mean multithreaded?
  16. Why is Collections.counter so slow?
  17. Python built-in sum function vs. for loop performance
  18. Python Random Access File
  19. Any way to speed up Python and Pygame?
  20. Are NumPy’s math functions faster than Python’s?
  21. matplotlib savefig performance, saving multiple pngs within loop
  22. Is there a way to circumvent Python list.append() becoming progressively slower in a loop as the list grows?
  23. Speed up writing to files
  24. Efficiently replace elements in array based on dictionary – NumPy / Python
  25. What are the differences between bool() and operator.truth()?
  26. First items in inner list efficiently as possible [duplicate]
  27. improving python code in Monte Carlo simulation
  28. Calculate summary statistics of columns in dataframe
  29. Why is a function/method call in python expensive?
  30. Searching a sorted list? [closed]

Leave a Comment