Very large matrices using Python and NumPy

by

PyTables and NumPy are the way to go.

PyTables will store the data on disk in HDF format, with optional compression. My datasets often get 10x compression, which is handy when dealing with tens or hundreds of millions of rows. It’s also very fast; my 5 year old laptop can crunch through data doing SQL-like GROUP BY aggregation at 1,000,000 rows/second. Not bad for a Python-based solution!

Accessing the data as a NumPy recarray again is as simple as:

data = table[row_from:row_to]

The HDF library takes care of reading in the relevant chunks of data and converting to NumPy.

More Related Contents:

  1. Difference between numpy.array shape (R, 1) and (R,)
  2. What are the differences between numpy arrays and matrices? Which one should I use?
  3. numpy matrix vector multiplication [duplicate]
  4. Numpy matrix to array
  5. how does multiplication differ for NumPy Matrix vs Array classes?
  6. Bin elements per row – Vectorized 2D Bincount for NumPy
  7. Convolve2d just by using Numpy
  8. Numpy ‘smart’ symmetric matrix
  9. Convert a 1D array to a 2D array in numpy
  10. numpy get index where value is true
  11. Deprecation status of the NumPy matrix class
  12. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  13. Block tridiagonal matrix python
  14. Iterating over Numpy matrix rows to apply a function each?
  15. how to perform max/mean pooling on a 2d array using numpy
  16. How to get element-wise matrix multiplication (Hadamard product) in numpy?
  17. Why does numpy.linalg.solve() offer more precise matrix inversions than numpy.linalg.inv()?
  18. NumPy array/matrix of mixed types
  19. Replace sub part of matrix by another small matrix in numpy
  20. How to convert a column or row matrix to a diagonal matrix in Python?
  21. Is there a standard solution for Gauss elimination in Python?
  22. How to find linearly independent rows from a matrix
  23. Convert row vector to column vector in NumPy
  24. Numpy select rows based on condition
  25. How to find the pairwise differences between rows of two very large matrices using numpy?
  26. Multiple matrix multiplication
  27. A numpy array unexpectedly changes when changing another one despite being separate
  28. Slice 2d array into smaller 2d arrays
  29. How to create a density plot in matplotlib?
  30. How to do n-D distance and nearest neighbor calculations on numpy arrays

Leave a Comment