How to slice a generator object or iterator?

by

In general, the answer is itertools.islice, but you should note that islice doesn’t, and can’t, actually skip values. It just grabs and throws away start values before it starts yield-ing values. So it’s usually best to avoid islice if possible when you need to skip a lot of values and/or the values being skipped are expensive to acquire/compute. If you can find a way to not generate the values in the first place, do so. In your (obviously contrived) example, you’d just adjust the start index for the range object.

In the specific cases of trying to run on a file object, pulling a huge number of lines (particularly reading from a slow medium) may not be ideal. Assuming you don’t need specific lines, one trick you can use to avoid actually reading huge blocks of the file, while still testing some distance in to the file, is the seek to a guessed offset, read out to the end of the line (to discard the partial line you probably seeked to the middle of), then islice off however many lines you want from that point. For example:

import itertools

with open('myhugefile') as f:
    # Assuming roughly 80 characters per line, this seeks to somewhere roughly
    # around the 100,000th line without reading in the data preceding it
    f.seek(80 * 100000)
    next(f)  # Throw away the partial line you probably landed in the middle of
    for line in itertools.islice(f, 100):  # Process 100 lines
        # Do stuff with each line

For the specific case of files, you might also want to look at mmap which can be used in similar ways (and is unusually useful if you’re processing blocks of data rather than lines of text, possibly randomly jumping around as you go).

Update: From your updated question, you’ll need to look at your API docs and/or data format to figure out exactly how to skip around properly. It looks like skbio offers some features for skipping using seq_num, but that’s still going to read if not process most of the file. If the data was written out with equal sequence lengths, I’d look at the docs on Alignment; aligned data may be loadable without processing the preceding data at all, by e.g by using Alignment.subalignment to create new Alignments that skip the rest of the data for you.

More Related Contents:

  1. Can we Write For loop as a function?
  2. How to append new line to the list
  3. How on earth does this code work
  4. What’s the purpose of “send” function on Python generators?
  5. Reading from a frequently updated file
  6. How to print a generator expression?
  7. How do I build a numpy array from a generator?
  8. How to extract an arbitrary line of values from a numpy array?
  9. If range() is a generator in Python 3.3, why can I not call next() on a range?
  10. Get loop count inside a for-loop [duplicate]
  11. Pandas selecting by label sometimes return Series, sometimes returns DataFrame
  12. How to write a generator class?
  13. Get the nth item of a generator in Python
  14. Reversing a list using slice notation
  15. Python vectorizing nested for loops
  16. Why does next raise a ‘StopIteration’, but ‘for’ do a normal return?
  17. How to append rows in a pandas dataframe in a for loop?
  18. Iteration over list slices
  19. Turn functions with a callback into Python generators?
  20. Handle an exception thrown in a generator
  21. Why python’s list slicing doesn’t produce index out of bound error? [duplicate]
  22. Why does a for-loop with pop-method (or del statement) not iterate over all list elements [duplicate]
  23. How does a for loop evaluate its argument
  24. Why do I get an IndexError (or TypeError, or just wrong results) from “ar[i]” inside “for i in ar”?
  25. interleaving 2 lists of unequal lengths [duplicate]
  26. What is the meaning of list[:] in this code? [duplicate]
  27. Are Generators Threadsafe?
  28. Redo for loop iteration in Python
  29. Python: powerset of a given set with generators [duplicate]
  30. Fast Way to slice image into overlapping patches and merge patches to image

Leave a Comment