How to create a large pandas dataframe from an sql query without running out of memory?

by

As mentioned in a comment, starting from pandas 0.15, you have a chunksize option in read_sql to read and process the query chunk by chunk: 

sql = "SELECT * FROM My_Table"
for chunk in pd.read_sql_query(sql , engine, chunksize=5):
    print(chunk)

Reference: http://pandas.pydata.org/pandas-docs/version/0.15.2/io.html#querying

More Related Contents:

  1. SQL / Pandas equivalent
  2. Pandas read_sql with parameters
  3. Pandas DENSE RANK
  4. Pandas left outer join multiple dataframes on multiple columns
  5. Get data from pandas into a SQL server with PYODBC
  6. Python Pandas write to sql with NaN values
  7. Pandas query function not working with spaces in column names
  8. python pandas to_sql with sqlalchemy : how to speed up exporting to MS SQL?
  9. Pandas analogue to SQL MINUS / EXCEPT operator, using multiple columns
  10. Extract the data and modify an existing excel with openpyxl
  11. How to drop rows of Pandas DataFrame whose value in a certain column is NaN
  12. How can I filter lines on load in Pandas read_csv function?
  13. Pandas Groupby Range of Values
  14. Pandas column access w/column names containing spaces
  15. How to get value counts for multiple columns at once in Pandas DataFrame?
  16. Add x and y labels to a pandas plot
  17. Plot different DataFrames in the same figure
  18. How to replace negative numbers in Pandas Data Frame by zero
  19. How to implement sql coalesce in pandas
  20. Subtract a year from a datetime column in pandas
  21. How to calculate correlation between all columns and remove highly correlated ones using pandas?
  22. Python Pandas merge samed name columns in a dataframe
  23. How can I replicate rows of a Pandas DataFrame?
  24. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  25. In Python pandas, start row index from 1 instead of zero without creating additional column
  26. pandas, melt, unmelt preserve index
  27. Conditional row read of csv in pandas
  28. Changing width of bars in bar chart created using seaborn.factorplot
  29. How to iterate over columns of pandas dataframe to run regression
  30. Python pandas – new column’s value if the item is in the list

Leave a Comment