How to iterate over consecutive chunks of Pandas dataframe efficiently

by

Use numpy’s array_split():

import numpy as np
import pandas as pd

data = pd.DataFrame(np.random.rand(10, 3))
for chunk in np.array_split(data, 5):
  assert len(chunk) == len(data) / 5, "This assert may fail for the last chunk if data lenght isn't divisible by 5"

More Related Contents:

  1. How to display pandas DataFrame of floats using a format string for columns?
  2. Progress indicator during pandas operations
  3. Selection with .loc in python
  4. Coloring Cells in Pandas
  5. Parallelize apply after pandas groupby
  6. How to dynamically update a plot in a loop in IPython notebook (within one cell)
  7. How to read a .xlsx file using the pandas Library in iPython?
  8. How to show a bar and line graph on the same plot
  9. python dask DataFrame, support for (trivially parallelizable) row apply?
  10. how to parallelize many (fuzzy) string comparisons using apply in Pandas?
  11. How to format IPython html display of Pandas dataframe?
  12. How can I pivot a dataframe?
  13. GroupBy pandas DataFrame and select most common value
  14. Find column name in pandas that matches an array
  15. Using the multiprocessing module for cluster computing
  16. Create Empty Dataframe in Pandas specifying column types
  17. How to plot multiple pandas columns
  18. pandas read_html ValueError: No tables found
  19. how to read certain columns from Excel using Pandas – Python
  20. How to slice a pandas DataFrame by position?
  21. Find unique values in a Pandas dataframe, irrespective of row or column location
  22. Select row from a DataFrame based on the type of the object(i.e. str)
  23. Selecting Data between Specific hours in a pandas dataframe
  24. Change the facecolor of boxplot in pandas
  25. How to annotate a seaborn barplot with the aggregated value
  26. How do I use vi keys in ipython under *nix?
  27. Add numpy array as column to Pandas data frame
  28. How to use ast.literal_eval in a pandas dataframe and handle exceptions
  29. Why is NaN considered as a float?
  30. Python Pandas : pandas.to_datetime() is switching day & month when day is less than 13

Leave a Comment