Fastest way to grow a numpy numeric array

by

I tried a few different things, with timing.

import numpy as np

  1. The method you mention as slow: (32.094 seconds)

    class A:

    def __init__(self):
        self.data = np.array([])

    def update(self, row):
        self.data = np.append(self.data, row)

    def finalize(self):
        return np.reshape(self.data, newshape=(self.data.shape[0]/5, 5))

  2. Regular ol Python list: (0.308 seconds)

    class B:

    def __init__(self):
        self.data = []

    def update(self, row):
        for r in row:
            self.data.append(r)

    def finalize(self):
        return np.reshape(self.data, newshape=(len(self.data)/5, 5))

  3. Trying to implement an arraylist in numpy: (0.362 seconds)

    class C:

    def __init__(self):
        self.data = np.zeros((100,))
        self.capacity = 100
        self.size = 0

    def update(self, row):
        for r in row:
            self.add(r)

    def add(self, x):
        if self.size == self.capacity:
            self.capacity *= 4
            newdata = np.zeros((self.capacity,))
            newdata[:self.size] = self.data
            self.data = newdata

        self.data[self.size] = x
        self.size += 1

    def finalize(self):
        data = self.data[:self.size]
        return np.reshape(data, newshape=(len(data)/5, 5))

And this is how I timed it:

x = C()
for i in xrange(100000):
    x.update([i])

So it looks like regular old Python lists are pretty good 😉

More Related Contents:

  1. Most efficient way to map function over numpy array
  2. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  3. numpy: most efficient frequency counts for unique values in an array
  4. Why is numpy’s einsum faster than numpy’s built in functions?
  5. Roll rows of a matrix independently
  6. Numpy sum elements in array based on its value
  7. Efficiently return the index of the first value satisfying condition in array
  8. Bin elements per row – Vectorized 2D Bincount for NumPy
  9. Vectorized searchsorted numpy
  10. Frequency counts for unique values in a NumPy array
  11. Most efficient way to forward-fill NaN values in numpy array
  12. Numpy: Fix array with rows of different lengths by filling the empty elements with zeros
  13. Pandas pd.Series.isin performance with set versus array
  14. Why is numpy.array so slow?
  15. Efficiently detect sign-changes in python
  16. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  17. Efficient way to take the minimum/maximum n values and indices from a matrix using NumPy
  18. why isn’t numpy.mean multithreaded?
  19. Numpy vs Cython speed
  20. numpy float: 10x slower than builtin in arithmetic operations?
  21. Fastest save and load options for a numpy array
  22. No speedup when summing uint16 vs uint64 arrays with NumPy?
  23. Python Numpy Data Types Performance
  24. how to speed up a vector cross product calculation
  25. Getting days since last occurence in Pandas DataFrame?
  26. What is the most efficient way to check if a value exists in a NumPy array?
  27. Efficient dot products of large memory-mapped arrays
  28. Creating a numpy array of 3D coordinates from three 1D arrays
  29. Split Python sequence (time series/array) into subsequences with overlap
  30. numpy faster than numba and cython , how to improve numba code

Leave a Comment