Most efficient way to forward-fill NaN values in numpy array

by

Here’s one approach –

mask = np.isnan(arr)
idx = np.where(~mask,np.arange(mask.shape[1]),0)
np.maximum.accumulate(idx,axis=1, out=idx)
out = arr[np.arange(idx.shape[0])[:,None], idx]

If you don’t want to create another array and just fill the NaNs in arr itself, replace the last step with this –

arr[mask] = arr[np.nonzero(mask)[0], idx[mask]]

Sample input, output –

In [179]: arr
Out[179]: 
array([[  5.,  nan,  nan,   7.,   2.,   6.,   5.],
       [  3.,  nan,   1.,   8.,  nan,   5.,  nan],
       [  4.,   9.,   6.,  nan,  nan,  nan,   7.]])

In [180]: out
Out[180]: 
array([[ 5.,  5.,  5.,  7.,  2.,  6.,  5.],
       [ 3.,  3.,  1.,  8.,  8.,  5.,  5.],
       [ 4.,  9.,  6.,  6.,  6.,  6.,  7.]])

More Related Contents:

  1. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  2. Efficiently return the index of the first value satisfying condition in array
  3. Convert pandas dataframe to NumPy array
  4. numpy: most efficient frequency counts for unique values in an array
  5. Why is numpy’s einsum faster than numpy’s built in functions?
  6. Numpy sum elements in array based on its value
  7. Frequency counts for unique values in a NumPy array
  8. Numpy: Fix array with rows of different lengths by filling the empty elements with zeros
  9. Ambiguity in Pandas Dataframe / Numpy Array “axis” definition
  10. Vectorizing Haversine distance calculation in Python
  11. Pandas pd.Series.isin performance with set versus array
  12. Python Numpy TypeError: ufunc ‘isfinite’ not supported for the input types
  13. Why is a `for` over a Python list faster than over a Numpy array?
  14. pd.Timestamp versus np.datetime64: are they interchangeable for selected uses?
  15. Fastest save and load options for a numpy array
  16. Numpy isnan() fails on an array of floats (from pandas dataframe apply)
  17. Why is Numpy much faster at creating a Zero array compared to replacing the values of an existing array with zeros?
  18. Getting days since last occurence in Pandas DataFrame?
  19. Efficient dot products of large memory-mapped arrays
  20. Creating a numpy array of 3D coordinates from three 1D arrays
  21. Split Python sequence (time series/array) into subsequences with overlap
  22. Find column name in pandas that matches an array
  23. Reshape an array in NumPy
  24. Suppressing scientific notation in pandas? [duplicate]
  25. how to read certain columns from Excel using Pandas – Python
  26. load csv into 2D matrix with numpy for plotting
  27. Filling in date gaps in MultiIndex Pandas Dataframe
  28. How does the axis parameter from NumPy work?
  29. numpy faster than numba and cython , how to improve numba code
  30. Interweaving two NumPy arrays efficiently

Leave a Comment