Large, persistent DataFrame in pandas

by

Wes is of course right! I’m just chiming in to provide a little more complete example code. I had the same issue with a 129 Mb file, which was solved by:

import pandas as pd

tp = pd.read_csv('large_dataset.csv', iterator=True, chunksize=1000)  # gives TextFileReader, which is iterable with chunks of 1000 rows.
df = pd.concat(tp, ignore_index=True)  # df is DataFrame. If errors, do `list(tp)` instead of `tp`

More Related Contents:

  1. How to plot in multiple subplots
  2. Pandas GroupBy.apply method duplicates first group
  3. Best way to join / merge by range in pandas
  4. Pandas: peculiar performance drop for inplace rename after dropna
  5. Could pandas use column as index?
  6. How can I make a barplot and a lineplot in the same seaborn plot with different Y axes nicely?
  7. Why does pandas apply calculate twice
  8. NLTK-based text processing with pandas
  9. Set values on the diagonal of pandas.DataFrame
  10. Pandas: Bar-Plot with two bars and two y-axis
  11. Extracting specific selected columns to new DataFrame as a copy
  12. Get pandas.read_csv to read empty values as empty string instead of nan
  13. Pandas join/merge/concat two dataframes
  14. Assign pandas dataframe column dtypes
  15. pandas replace zeros with previous non zero value
  16. Python – Delete duplicates in a dataframe based on two columns combinations?
  17. Pandas error when using if-else to create new column: The truth value of a Series is ambiguous
  18. How do I make the width of the title box span the entire plot?
  19. Renaming columns in a Pandas dataframe with duplicate column names?
  20. How to edit properties of whiskers, fliers, caps, etc. in Seaborn boxplot
  21. How to determine whether a column/variable is numeric or not in Pandas/NumPy?
  22. Python Pandas: Convert “.value_counts” output to dataframe
  23. Given a pandas Series that represents frequencies of a value, how can I turn those frequencies into percentages?
  24. LabelEncoder: TypeError: ‘>’ not supported between instances of ‘float’ and ‘str’
  25. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  26. Run an OLS regression with Pandas Data Frame
  27. Relative Strength Index in python pandas
  28. Pandas: remove duplicates that exist in any order
  29. Difference between df[x], df[[x]], df[‘x’] , df[[‘x’]] and df.x
  30. Numpy.dtype has the wrong size, try recompiling

Leave a Comment