Sparse matrix slicing using list of int

by

I think I’ve recreated the csr row indexing with:

def extractor(indices, N):
   indptr=np.arange(len(indices)+1)
   data=np.ones(len(indices))
   shape=(len(indices),N)
   return sparse.csr_matrix((data,indices,indptr), shape=shape)

Testing on a csr I had hanging around:

In [185]: M
Out[185]: 
<30x40 sparse matrix of type '<class 'numpy.float64'>'
    with 76 stored elements in Compressed Sparse Row format>

In [186]: indices=np.r_[0:20]

In [187]: M[indices,:]
Out[187]: 
<20x40 sparse matrix of type '<class 'numpy.float64'>'
    with 57 stored elements in Compressed Sparse Row format>

In [188]: extractor(indices, M.shape[0])*M
Out[188]: 
<20x40 sparse matrix of type '<class 'numpy.float64'>'
    with 57 stored elements in Compressed Sparse Row format>

As with a number of other csr methods, it uses matrix multiplication to produce the final value. In this case with a sparse matrix with 1 in selected rows. Time is actually a bit better.

In [189]: timeit M[indices,:]
1000 loops, best of 3: 515 µs per loop
In [190]: timeit extractor(indices, M.shape[0])*M
1000 loops, best of 3: 399 µs per loop

In your case the extractor matrix is (len(training_indices),347) in shape, with only len(training_indices) values. So it is not big.

But if the matrix is so large (or at least the 2nd dimension so big) that it produces some error in the matrix multiplication routines, it could give rise to segmentation fault without python/numpy trapping it.

Does matrix.sum(axis=1) work. That too uses a matrix multiplication, though with a dense matrix of 1s. Or sparse.eye(347)*M, a similar size matrix multiplication?

More Related Contents:

  1. Populate a Pandas SparseDataFrame from a SciPy Sparse Matrix
  2. Efficiently create sparse pivot tables in pandas?
  3. Pandas sparse dataFrame to sparse matrix, without generating a dense matrix in memory
  4. Directly use Intel mkl library on Scipy sparse matrix to calculate A dot A.T with less memory
  5. Concatenate sparse matrices in Python using SciPy/Numpy
  6. Iterating through a scipy.sparse vector (or matrix)
  7. Scipy.sparse.csr_matrix: How to get top ten values and indices?
  8. How to transform numpy.matrix or array to scipy sparse matrix
  9. sparse 3d matrix/array in Python?
  10. Computing the correlation coefficient between two multi-dimensional arrays
  11. How does condensed distance matrix work? (pdist)
  12. ImportError in importing from sklearn: cannot import name check_build
  13. binning data in python with scipy/numpy
  14. Calculating Pearson correlation and significance in Python
  15. Calculating the area under a curve given a set of coordinates, without knowing the function
  16. In Scipy how and why does curve_fit calculate the covariance of the parameter estimates
  17. How to implement band-pass Butterworth filter with Scipy.signal.butter
  18. Does `anaconda` create a separate PYTHONPATH variable for each new environment?
  19. ImportError: cannot import name NUMPY_MKL
  20. How to add a new row to an empty numpy array
  21. How do I put a constraint on SciPy curve fit?
  22. SciPy Create 2D Polygon Mask
  23. scipy.curve_fit() returns multiple lines
  24. How to calculate cumulative normal distribution?
  25. Capture “Segmentation fault” message for a crashed subprocess: no out and err after a call to communicate()
  26. How to display progress of scipy.optimize function?
  27. Fitting a Weibull distribution using Scipy
  28. Numpy transpose of 1D array not giving expected result
  29. Fit mixture of two gaussian/normal distributions to a histogram from one set of data, python
  30. Speedup scipy griddata for multiple interpolations between two irregular grids

Leave a Comment