MemoryError when I merge two Pandas data frames

When you are merging data using pandas.merge it will use df1 memory, df2 memory and merge_df memory. I believe that it is why you get a memory error. You should export df2 to a csv file and use chunksize option and merge data.

It might be a better way but you can try this.
*for large data set you can use chunksize option in pandas.read_csv

df1 = pd.read_csv("yourdata.csv")
df2 = pd.read_csv("yourdata2.csv")
df2_key = df2.Colname2

# creating a empty bucket to save result
df_result = pd.DataFrame(columns=(df1.columns.append(df2.columns)).unique())
df_result.to_csv("df3.csv",index_label=False)

# save data which only appear in df1 # sorry I was doing left join here. no need to run below two line.
# df_result = df1[df1.Colname1.isin(df2.Colname2)!=True]
# df_result.to_csv("df3.csv",index_label=False, mode="a")

# deleting df2 to save memory
del(df2)

def preprocess(x):
    df2=pd.merge(df1,x, left_on = "Colname1", right_on = "Colname2")
    df2.to_csv("df3.csv",mode="a",header=False,index=False)

reader = pd.read_csv("yourdata2.csv", chunksize=1000) # chunksize depends with you colsize

[preprocess(r) for r in reader]

this will save merged data as df3.