Dummy variables when not all categories are present

by

TL;DR:

pd.get_dummies(cat.astype(pd.CategoricalDtype(categories=categories)))
  • Older pandas: pd.get_dummies(cat.astype('category', categories=categories))

is there a way to pass to get_dummies (or an equivalent function) the names of the categories, so that, for the categories that don’t appear in a given dataframe, it’d just create a column of 0s?

Yes, there is! Pandas has a special type of Series just for categorical data. One of the attributes of this series is the possible categories, which get_dummies takes into account. Here’s an example:

In [1]: import pandas as pd

In [2]: possible_categories = list('abc')

In [3]: cat = pd.Series(list('aba'))

In [4]: cat = cat.astype(pd.CategoricalDtype(categories=possible_categories))

In [5]: cat
Out[5]: 
0    a
1    b
2    a
dtype: category
Categories (3, object): [a, b, c]

Then, get_dummies will do exactly what you want!

In [6]: pd.get_dummies(cat)
Out[6]: 
   a  b  c
0  1  0  0
1  0  1  0
2  1  0  0

There are a bunch of other ways to create a categorical Series or DataFrame, this is just the one I find most convenient. You can read about all of them in the pandas documentation.

EDIT:

I haven’t followed the exact versioning, but there was a bug in how pandas treats sparse matrices, at least until version 0.17.0. It was corrected by version 0.18.1 (released May 2016).

For version 0.17.0, if you try to do this with the sparse=True option with a DataFrame, the column of zeros for the missing dummy variable will be a column of NaN, and it will be converted to dense.

It looks like pandas 0.21.0 added a CategoricalDType, and creating categoricals which explicitly include the categories as in the original answer was deprecated, I’m not quite sure when.

More Related Contents:

  1. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  2. How can I one hot encode in Python?
  3. Pandas: Get Dummies
  4. Custom transformer for sklearn Pipeline that alters both X and y
  5. Identifying statistical outliers with pandas: groupby and reduce rows into different dataframe
  6. Converting pandas column of comma-separated strings into dummy variables
  7. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  8. Time Series Analysis – unevenly spaced measures – pandas + statsmodels
  9. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  10. How to compute jaccard similarity from a pandas dataframe
  11. What values are valid in Pandas ‘Freq’ tags?
  12. Format a number with commas to separate thousands
  13. Save Naive Bayes Trained Classifier in NLTK
  14. Python: UserWarning: This pattern has match groups. To actually get the groups, use str.extract
  15. reading multiple files contained in a zip file with pandas
  16. Sorting the order of bars in pandas/matplotlib bar plots
  17. OneHotEncoder categorical_features deprecated, how to transform specific column
  18. How to filter a DataFrame column of lists for those that contain a certain item
  19. Split Python sequence (time series/array) into subsequences with overlap
  20. Keras give input to intermediate layer and get final output
  21. Model help using Scikit-learn when using GridSearch
  22. Pandas combining rows based on dates
  23. Collapsing rows in a Pandas dataframe if all rows have only one value in their columns
  24. Determining the most contributing features for SVM classifier in sklearn
  25. Preserving column order in Python Pandas DataFrame
  26. Convert dates to pd.to_datetime where month could be either a number or month name
  27. start index at 1 for Pandas DataFrame
  28. Multi Index Sorting in Pandas
  29. Pytorch ValueError: optimizer got an empty parameter list
  30. Pandas: select all dates with specific month and day

Leave a Comment