Populate a Pandas SparseDataFrame from a SciPy Sparse Matrix

by

A direct conversion is not supported ATM. Contributions are welcome!

Try this, should be ok on memory as the SpareSeries is much like a csc_matrix (for 1 column)
and pretty space efficient

In [37]: col = np.array([0,0,1,2,2,2])

In [38]: data = np.array([1,2,3,4,5,6],dtype="float64")

In [39]: m = csc_matrix( (data,(row,col)), shape=(3,3) )

In [40]: m
Out[40]: 
<3x3 sparse matrix of type '<type 'numpy.float64'>'
        with 6 stored elements in Compressed Sparse Column format>

In [46]: pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel()) 
                              for i in np.arange(m.shape[0]) ])
Out[46]: 
   0  1  2
0  1  0  4
1  0  0  5
2  2  3  6

In [47]: df = pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel()) 
                                   for i in np.arange(m.shape[0]) ])

In [48]: type(df)
Out[48]: pandas.sparse.frame.SparseDataFrame

More Related Contents:

  1. Most efficient way to sort an array into bins specified by an index array?
  2. Efficiently create sparse pivot tables in pandas?
  3. Pandas sparse dataFrame to sparse matrix, without generating a dense matrix in memory
  4. Concatenate sparse matrices in Python using SciPy/Numpy
  5. Parse a Pandas column to Datetime when importing table from SQL database and filtering rows by date
  6. How to transform numpy.matrix or array to scipy sparse matrix
  7. sparse 3d matrix/array in Python?
  8. Convert Pandas dataframe to Sparse Numpy Matrix directly
  9. Moving average or running mean
  10. Restart cumsum and get index if cumsum more than value
  11. What does the term “broadcasting” mean in Pandas documentation?
  12. Dropping infinite values from dataframes in pandas?
  13. ValueError: numpy.dtype has the wrong size, try recompiling
  14. Save / load scipy sparse csr_matrix in portable data format
  15. Find matching rows in 2 dimensional numpy array
  16. Pandas lookup from one of multiple columns, based on value
  17. Difference between data type ‘datetime64[ns]’ and ‘
  18. Selecting Pandas Columns by dtype
  19. Get coordinates of local maxima in 2D array above certain value
  20. pandas distinction between str and object types
  21. Directly use Intel mkl library on Scipy sparse matrix to calculate A dot A.T with less memory
  22. NumPy version of “Exponential weighted moving average”, equivalent to pandas.ewm().mean()
  23. How to share numpy random state of a parent process with child processes?
  24. Fastest pairwise distance metric in python
  25. How to get the indices list of all NaN value in numpy array?
  26. setting spacing between grouped bar plots in matplotlib
  27. Resample time series in pandas to a weekly interval
  28. Scipy.sparse.csr_matrix: How to get top ten values and indices?
  29. How to use numpy to get the cumulative count by unique values in linear time?
  30. Computing cross-correlation function?

Leave a Comment