efficiently convert uneven list of lists to minimal containing array padded with nan

by

This seems to be a close one of this question, where the padding was with zeros instead of NaNs. Interesting approaches were posted there, along with mine based on broadcasting and boolean-indexing. So, I would just modify one line from my post there to solve this case like so –

def boolean_indexing(v, fillval=np.nan):
    lens = np.array([len(item) for item in v])
    mask = lens[:,None] > np.arange(lens.max())
    out = np.full(mask.shape,fillval)
    out[mask] = np.concatenate(v)
    return out

Sample run –

In [32]: l
Out[32]: [[1, 2, 3], [1, 2], [3, 8, 9, 7, 3]]

In [33]: boolean_indexing(l)
Out[33]: 
array([[  1.,   2.,   3.,  nan,  nan],
       [  1.,   2.,  nan,  nan,  nan],
       [  3.,   8.,   9.,   7.,   3.]])

In [34]: boolean_indexing(l,-1)
Out[34]: 
array([[ 1,  2,  3, -1, -1],
       [ 1,  2, -1, -1, -1],
       [ 3,  8,  9,  7,  3]])

I have posted few runtime results there for all the posted approaches on that Q&A, which could be useful.

More Related Contents:

  1. how to do multiple scatter plots with matploatlib
  2. Split (explode) pandas dataframe string entry to separate rows
  3. Binning a column with Python Pandas
  4. NumPy or Pandas: Keeping array type as integer while having a NaN value
  5. Performant cartesian product (CROSS JOIN) with pandas
  6. Best way to join / merge by range in pandas
  7. Numpy “where” with multiple conditions
  8. ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
  9. Dropping infinite values from dataframes in pandas?
  10. How to check if a value is in the list in selection from pandas data frame?
  11. Replacing Pandas or Numpy Nan with a None to use with MysqlDB
  12. shuffling/permutating a DataFrame in pandas
  13. SQL-like window functions in PANDAS: Row Numbering in Python Pandas Dataframe
  14. Cumsum reset at NaN
  15. Pandas lookup from one of multiple columns, based on value
  16. Vectorizing Haversine distance calculation in Python
  17. Set values on the diagonal of pandas.DataFrame
  18. What techniques can be used to measure performance of pandas/numpy solutions
  19. Custom transformer for sklearn Pipeline that alters both X and y
  20. Python Numpy TypeError: ufunc ‘isfinite’ not supported for the input types
  21. Find symmetric pairs quickly in numpy
  22. Why is np.where faster than pd.apply
  23. Unpivot Pandas Data
  24. How to determine whether a column/variable is numeric or not in Pandas/NumPy?
  25. How to resample a dataframe with different functions applied to each column?
  26. Resample time series in pandas to a weekly interval
  27. Python Pandas: remove entries based on the number of occurrences
  28. Convert a column of timestamps into periods in pandas
  29. What is the Big O Complexity of Reversing the Order of Columns in Pandas DataFrame?
  30. How to use numpy to get the cumulative count by unique values in linear time?

Leave a Comment