What is the performance impact of non-unique indexes in pandas?

by

When index is unique, pandas use a hashtable to map key to value O(1). When index is non-unique and sorted, pandas use binary search O(logN), when index is random ordered pandas need to check all the keys in the index O(N).

You can call sort_index method:

import numpy as np
import pandas as pd
x = np.random.randint(0, 200, 10**6)
df1 = pd.DataFrame({'x':x})
df2 = df1.set_index('x', drop=False)
df3 = df2.sort_index()
%timeit df1.loc[100]
%timeit df2.loc[100]
%timeit df3.loc[100]

result:

10000 loops, best of 3: 71.2 µs per loop
10 loops, best of 3: 38.9 ms per loop
10000 loops, best of 3: 134 µs per loop

More Related Contents:

  1. pandas loc vs. iloc vs. at vs. iat?
  2. When should I (not) want to use pandas apply() in my code?
  3. Does pandas iterrows have performance issues?
  4. How are iloc and loc different?
  5. How to convert index of a pandas dataframe into a column
  6. Replace values in a pandas series via dictionary efficiently
  7. What is the most efficient way to loop through dataframes with pandas?
  8. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  9. Pandas split DataFrame by column value
  10. Selecting a row of pandas series/dataframe by integer index
  11. Pandas: peculiar performance drop for inplace rename after dropna
  12. Python Pandas: Get index of rows which column matches certain value
  13. Remove index name in pandas
  14. Update row values where certain condition is met in pandas
  15. Most efficient way to forward-fill NaN values in numpy array
  16. Fastest way to sort each row in a pandas dataframe
  17. Get column index from column name in python pandas
  18. Vectorizing Haversine distance calculation in Python
  19. Pandas pd.Series.isin performance with set versus array
  20. Pandas – Compute z-score for all columns
  21. How to avoid pandas creating an index in a saved csv
  22. Python Pandas: Get index of rows where column matches certain value
  23. Selecting pandas column by location
  24. Pandas max value index
  25. Python: Removing Rows on Count condition
  26. Using replace efficiently in pandas
  27. Select the max row per group – pandas performance issue
  28. Split Python sequence (time series/array) into subsequences with overlap
  29. How to efficiently assign unique ID to individuals with multiple entries based on name in very large df
  30. start index at 1 for Pandas DataFrame

Leave a Comment