Python random sample with a generator / iterable / iterator

by

While the answer of Martijn Pieters is correct, it does slow down when samplesize becomes large, because using list.insert in a loop may have quadratic complexity.

Here’s an alternative that, in my opinion, preserves the uniformity while increasing performance:

def iter_sample_fast(iterable, samplesize):
    results = []
    iterator = iter(iterable)
    # Fill in the first samplesize elements:
    try:
        for _ in xrange(samplesize):
            results.append(iterator.next())
    except StopIteration:
        raise ValueError("Sample larger than population.")
    random.shuffle(results)  # Randomize their positions
    for i, v in enumerate(iterator, samplesize):
        r = random.randint(0, i)
        if r < samplesize:
            results[r] = v  # at a decreasing rate, replace random items
    return results

The difference slowly starts to show for samplesize values above 10000. Times for calling with (1000000, 100000):

  • iterSample: 5.05s
  • iter_sample_fast: 2.64s

More Related Contents:

  1. Is there a way to ordinate numbers in a list?
  2. Python random.choice same output
  3. Randomly generate 4 or 5 digit Unique Codes in Subsequent Runs of the Script
  4. How to select random key with common values?
  5. What’s the purpose of “send” function on Python generators?
  6. How to get a random value from dictionary?
  7. Weighted random selection with and without replacement
  8. Generate random numbers summing to a predefined value
  9. Reading from a frequently updated file
  10. How to print a generator expression?
  11. How do I build a numpy array from a generator?
  12. Generate a random derangement of a list
  13. yield in list comprehensions and generator expressions
  14. How to get two random records with Django
  15. Weighted random sample in python
  16. Select 50 items from list at random
  17. How to generate a repeatable random number sequence?
  18. Can generators be recursive?
  19. How to make a reproducible data sample in PowerBI using Python?
  20. Handle generator exceptions in its consumer
  21. Choosing a function randomly
  22. Are Generators Threadsafe?
  23. How to generate a random normal distribution of integers
  24. Generating non-repeating random numbers in Python
  25. Weighted random sample without replacement in python
  26. Set.pop() isn’t random?
  27. Python: powerset of a given set with generators [duplicate]
  28. TensorFlow: Non-repeatable results
  29. Can I generate authentic random number with python?
  30. Generate random array of floats between a range

Leave a Comment