Efficient way to add rows to dataframe

by

I used this answer’s df.loc[i] = [new_data] suggestion, but I have > 500,000 rows and that was very slow.

While the answers given are good for the OP’s question, I found it more efficient, when dealing with large numbers of rows up front (instead of the tricking in described by the OP) to use csvwriter to add data to an in memory CSV object, then finally use pandas.read_csv(csv) to generate the desired DataFrame output.

from io import BytesIO
from csv import writer 
import pandas as pd

output = BytesIO()
csv_writer = writer(output)

for row in iterable_object:
    csv_writer.writerow(row)

output.seek(0) # we need to get back to the start of the BytesIO
df = pd.read_csv(output)
return df

This, for ~500,000 rows was 1000x faster and as the row count grows the speed improvement will only get larger (the df.loc[1] = [data] will get a lot slower comparatively)

Hope this helps someone who need efficiency when dealing with more rows than the OP.

More Related Contents:

  1. How do I select rows from a DataFrame based on column values?
  2. Filter pandas DataFrame by substring criteria
  3. Binning a column with Python Pandas
  4. Aggregation in Pandas
  5. Compare two DataFrames and output their differences side-by-side
  6. Constructing pandas DataFrame from values in variables gives “ValueError: If using all scalar values, you must pass an index”
  7. Selecting a row of pandas series/dataframe by integer index
  8. How do I find the closest values in a Pandas series to an input number?
  9. Remove reverse duplicates from dataframe
  10. How do I count the NaN values in a column in pandas DataFrame?
  11. How to split text in a column into multiple rows
  12. getting the index of a row in a pandas apply function
  13. How to flatten a pandas dataframe with some columns as json?
  14. Tilde sign in pandas DataFrame
  15. Drop columns whose name contains a specific string from pandas DataFrame
  16. Python Pandas How to assign groupby operation results back to columns in parent dataframe?
  17. Pandas won’t fillna() inplace
  18. python Pandas DataFrame copy(deep=False) vs copy(deep=True) vs ‘=’
  19. From password-protected Excel file to pandas DataFrame
  20. How can repetitive rows of data be collected in a single row in pandas?
  21. FastAPI is very slow in returning a large amount of JSON data
  22. Store numpy.array in cells of a Pandas.DataFrame
  23. Python pandas: how to remove nan and -inf values
  24. Remove substring from column based on another column
  25. Python Pandas iterate over rows and access column names
  26. Intersection of two pandas dataframes based on column entries
  27. pandas dataframe with 2-rows header and export to csv
  28. Python Pandas – Lookup a variable column depending on another column’s value
  29. How to stream DataFrame using FastAPI without saving the data to csv file?
  30. Plot pandas dates in matplotlib

Leave a Comment