Sorting text file by using Python

by

Don’t sort 10 million lines in memory. Split this up in batches instead:

  • Run 100 100k line sorts (using the file as an iterator, combined with islice() or similar to pick a batch). Write out to separate files elsewhere.

  • Merge the sorted files. Here is an merge generator that you can pass 100 open files and it’ll yield lines in sorted order. Write to a new file line by line:

    import operator

def mergeiter(*iterables, **kwargs):
    """Given a set of sorted iterables, yield the next value in merged order

    Takes an optional `key` callable to compare values by.
    """
    iterables = [iter(it) for it in iterables]
    iterables = {i: [next(it), i, it] for i, it in enumerate(iterables)}
    if 'key' not in kwargs:
        key = operator.itemgetter(0)
    else:
        key = lambda item, key=kwargs['key']: key(item[0])

    while True:
        value, i, it = min(iterables.values(), key=key)
        yield value
        try:
            iterables[i][0] = next(it)
        except StopIteration:
            del iterables[i]
            if not iterables:
                raise

More Related Contents:

  1. Pythonic way to check if a list is sorted or not
  2. About Python’s built in sort() method
  3. Quicksort with Python
  4. Why is Bubble Sort implementation looping forever?
  5. heapq with custom compare predicate
  6. Find elements of array one nearest to elements of array two
  7. What is the complexity of the sorted() function?
  8. What algorithm does Python’s built-in sort() method use?
  9. Mergesort with Python
  10. What is this odd sorting algorithm?
  11. How to delete a specific line in a file?
  12. What is the most efficient way of finding all the factors of a number in Python?
  13. How to open a file for both reading and writing?
  14. Is the time-complexity of iterative string append actually O(n^2), or O(n)?
  15. How do I remove/delete a folder that is not empty?
  16. case-insensitive list sorting, without lowercasing the result?
  17. Sort a pandas dataframe series by month name
  18. whitespaces in the path of windows filepath
  19. Python Implementations of Packing Algorithm
  20. How to get all of the immediate subdirectories in Python
  21. Django Admin: Ordering of ForeignKey and ManyToManyField relations referencing User
  22. Efficient method to calculate the rank vector of a list in Python
  23. How to upload File in FastAPI, then to Amazon S3 and finally process it?
  24. Python – When to use file vs open
  25. Where can I find source or algorithm of Python’s hash() function?
  26. Split a list of tuples into sub-lists of the same tuple field [duplicate]
  27. argsort for a multidimensional ndarray
  28. What is the Big O Complexity of Reversing the Order of Columns in Pandas DataFrame?
  29. Python – Removing overlapping lists
  30. Reading a File Line by Line in Python [duplicate]

Leave a Comment