make pandas DataFrame to a dict and dropna

by

There are many ways you could accomplish this, I spent some time evaluating performance on a not-so-large (70k) dataframe. Although @der_die_das_jojo’s answer is functional, it’s also pretty slow.

The answer suggested by this question actually turns out to be about 5x faster on a large dataframe.

On my test dataframe (df):

Above method:

%time [ v.dropna().to_dict() for k,v in df.iterrows() ]
CPU times: user 51.2 s, sys: 0 ns, total: 51.2 s
Wall time: 50.9 s

Another slow method:

%time df.apply(lambda x: [x.dropna()], axis=1).to_dict(orient="rows")
CPU times: user 1min 8s, sys: 880 ms, total: 1min 8s
Wall time: 1min 8s

Fastest method I could find:

%time [ {k:v for k,v in m.items() if pd.notnull(v)} for m in df.to_dict(orient="rows")]
CPU times: user 14.5 s, sys: 176 ms, total: 14.7 s
Wall time: 14.7 s

The format of this output is a row-oriented dictionary, you may need to make adjustments if you want the column-oriented form in the question.

Very interested if anyone finds an even faster answer to this question.

More Related Contents:

  1. Renaming column names in Pandas
  2. Filtering Pandas DataFrames on dates
  3. What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?
  4. Seaborn Barplot – Displaying Values
  5. How can I map True/False to 1/0 in a Pandas DataFrame?
  6. Pandas create new column with count from groupby
  7. Pandas dataframe with multiindex column – merge levels
  8. How to qcut with non unique bin edges?
  9. Search for “does-not-contain” on a DataFrame in pandas
  10. Pandas select rows and columns based on boolean condition
  11. Pandas: convert dtype ‘object’ to int
  12. Plotting multiple lines, in different colors, with pandas dataframe
  13. How can I subclass a Pandas DataFrame?
  14. What are Python pandas equivalents for R functions like str(), summary(), and head()?
  15. Python pandas cumsum with reset everytime there is a 0
  16. Pip Install Timeout Issue
  17. ‘A value is trying to be set on a copy of a slice from a DataFrame’ error while using ‘iloc’
  18. Converting strings to floats in a DataFrame
  19. how to multiply multiple columns by a column in Pandas
  20. Pandas – select column using other column value as column name
  21. Map dataframe index using dictionary
  22. DataFrame with MultiIndex to dict
  23. get first and last values in a groupby
  24. mean from pandas and numpy differ
  25. Python: Datetime to season
  26. Replace NaN with empty list in a pandas dataframe
  27. How to set the xlim and ylim of a FacetGrid
  28. Problem in combining bar plot and line plot
  29. Plot all pandas dataframe columns separately
  30. Python pandas equivalent for replace

Leave a Comment