I also used the dataframe’s append function inside a loop and I was perplexed how slow it ran.
A useful example for those who are suffering, based on the correct answer on this page.
Python version: 3
Pandas version: 0.20.3
# the dictionary to pass to pandas dataframe
d = {}
# a counter to use to add entries to "dict"
i = 0
# Example data to loop and append to a dataframe
data = [{"foo": "foo_val_1", "bar": "bar_val_1"},
{"foo": "foo_val_2", "bar": "bar_val_2"}]
# the loop
for entry in data:
# add a dictionary entry to the final dictionary
d[i] = {"col_1_title": entry['foo'], "col_2_title": entry['bar']}
# increment the counter
i = i + 1
# create the dataframe using 'from_dict'
# important to set the 'orient' parameter to "index" to make the keys as rows
df = DataFrame.from_dict(d, "index")
The “from_dict” function: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html