Read random lines from huge CSV file

by 
import random

filesize = 1500                 #size of the really big file
offset = random.randrange(filesize)

f = open('really_big_file')
f.seek(offset)                  #go to random position
f.readline()                    # discard - bound to be partial line
random_line = f.readline()      # bingo!

# extra to handle last/first line edge cases
if len(random_line) == 0:       # we have hit the end
    f.seek(0)
    random_line = f.readline()  # so we'll grab the first line instead

As @AndreBoos pointed out, this approach will lead to biased selection. If you know min and max length of line you can remove this bias by doing the following:

Let’s assume (in this case) we have min=3 and max=15

1) Find the length (Lp) of the previous line.

Then if Lp = 3, the line is most biased against. Hence we should take it 100% of the time
If Lp = 15, the line is most biased towards. We should only take it 20% of the time as it is 5* more likely selected.

We accomplish this by randomly keeping the line X% of the time where:

X = min / Lp

If we don’t keep the line, we do another random pick until our dice roll comes good. 🙂

More Related Contents:

  1. How do I create a python code that distinguishes between the capital of a state and the state that the capital has?
  2. Writing a Python list of lists to a csv file
  3. Reading a huge .csv file
  4. How to write to a CSV line by line?
  5. Python convert csv to xlsx
  6. How to read one single line of csv data in Python?
  7. Find all CSV files in a directory using Python
  8. Best way to choose a random file from a directory
  9. How to name a file using a value in a list in a dictionary?
  10. How to search and replace text in a file?
  11. How to add pandas data to an existing csv file?
  12. How to ignore the first line of data when processing CSV data?
  13. Change specific value in CSV file via Python
  14. Pandas split column into multiple columns by comma
  15. Remove file after Flask serves it
  16. General Unicode/UTF-8 support for csv files in Python 2.6
  17. Convert commas decimal separators to dots within a Dataframe
  18. How to generate random numbers that are different? [duplicate]
  19. Read data from CSV file and transform from string to correct data-type, including a list-of-integer column
  20. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  21. Re-read an open file Python
  22. How to hash a string into 8 digits?
  23. how to find the owner of a file or directory in python
  24. How to read the last line of a file in Python? [duplicate]
  25. CSV reader behavior with None and empty string
  26. Pandas groupby to to_csv
  27. Set Django’s FileField to an existing file
  28. double quoted elements in csv cant read with pandas
  29. How to return a csv file/Pandas DataFrame in JSON format using FastAPI?
  30. Python: Writing Nested Dictionary to CSV

Leave a Comment