UPDATE: this post answered is now superseded by this new one (which take into account the updates of the question) providing an even faster code based on a different approach. Step 1: better algorithm First of all, building a k-d tree runs in O(n log n) time and doing a query runs in O(log n) … Read more
The loop cannot be avoided, but it can be parallelized using numba‘s njit: from numba import njit, prange @njit def dynamic_cumsum(seq, index, max_value): cumsum = [] running = 0 for i in prange(len(seq)): if running > max_value: cumsum.append([index[i], running]) running = 0 running += seq[i] cumsum.append([index[-1], running]) return cumsum The index is required here, assuming … Read more
As we will see the behavior is dependent on which numpy-distribution is used. This answer will focus on Anacoda-distribution with Intel’s VML (vector math library), millage can vary given another hardware and numpy-version. It will also be shown, how VML can be utilized via Cython or numexpr, in case one doesn’t use Anacoda-distribution, which plugs-in … Read more
Here is a more elegant and scalable solution, imo. It’ll work for any nxn matrix and you may find use for the other methods. Note that getMatrixInverse(m) takes in an array of arrays as input. Please feel free to ask any questions. def transposeMatrix(m): return map(list,zip(*m)) def getMatrixMinor(m,i,j): return [row[:j] + row[j+1:] for row in … Read more
1. Passing extra arguments through scipy.integrate.quad The quad docs say: If the user desires improved integration performance, then f may be a scipy.LowLevelCallable with one of the signatures: double func(double x) double func(double x, void *user_data) double func(int n, double *xx) double func(int n, double *xx, void *user_data) The user_data is the data contained in … Read more
The support for Dictionary has now been added in Numba version 43.0. Although it quite limited (does not support list and set as key/values). You can however read the updated documentation here for more info. Here is an example import numpy as np from numba import njit from numba import types from numba.typed import Dict … Read more
Series.astype converts all date-like objects to datetime64[ns]. To convert to datetime64[D], use values to obtain a NumPy array before calling astype: dates_input = df[“month_15”].values.astype(‘datetime64[D]’) Note that NDFrames (such as Series and DataFrames) can only hold datetime-like objects as objects of dtype datetime64[ns]. The automatic conversion of all datetime-likes to a common dtype simplifies subsequent date … Read more
Summary Using a loop-based approach with single pass and copying, accelerated with Numba, offers the best overall trade-off in terms of speed, memory efficiency and flexibility. If the execution of the condition function is sufficiently fast, two-passes (filter2_nb()) may be faster, while they are more memory efficient regardless. Also, for sufficiently large inputs, resizing instead … Read more
set_index on merging column does indeed speed this up. Below is a slightly more realistic version of julien-marrec’s Answer. import pandas as pd import numpy as np myids=np.random.choice(np.arange(10000000), size=1000000, replace=False) df1 = pd.DataFrame(myids, columns=[‘A’]) df1[‘B’] = np.random.randint(0,1000,(1000000)) df2 = pd.DataFrame(np.random.permutation(myids), columns=[‘A2’]) df2[‘B2′] = np.random.randint(0,1000,(1000000)) %%timeit x = df1.merge(df2, how=’left’, left_on=’A’, right_on=’A2′) #1 loop, best of … Read more
There are arguably at least two errors in that posted code: This can’t possibly be a correct range check: if x >= C.shape[0] and y >= C.shape[1]: In order for us to decide that a particular thread in the grid not do any loading activity, we require either that x is out of range or … Read more