Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting

by

Instead of using pd.get_dummies() you need LabelEncoder + OneHotEncoder which can store the original values and then use them on the new data.

Changing your code like below will give you required results.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
input_df = pd.DataFrame(dict(fruit=['Apple', 'Orange', 'Pine'], 
                             color=['Red', 'Orange','Green'],
                             is_sweet = [0,0,1],
                             country=['USA','India','Asia']))

filtered_df = input_df.apply(pd.to_numeric, errors="ignore")

# This is what you need
le_dict = {}
for col in filtered_df.columns:
    le_dict[col] = LabelEncoder().fit(filtered_df[col])
    filtered_df[col] = le_dict[col].transform(filtered_df[col])

enc = OneHotEncoder()
enc.fit(filtered_df)
refreshed_df = enc.transform(filtered_df).toarray()

new_df = pd.DataFrame(dict(fruit=['Apple'], 
                             color=['Red'],
                             is_sweet = [0],
                             country=['USA']))
for col in new_df.columns:
    new_df[col] = le_dict[col].transform(new_df[col])

new_refreshed_df = enc.transform(new_df).toarray()

print(filtered_df)
      color  country  fruit  is_sweet
0      2        2      0         0
1      1        1      1         0
2      0        0      2         1

print(refreshed_df)
[[ 0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  0.]
 [ 0.  1.  0.  0.  1.  0.  0.  1.  0.  1.  0.]
 [ 1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  1.]]

print(new_df)
      color  country  fruit  is_sweet
0      2        2      0         0

print(new_refreshed_df)
[[ 0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  0.]]

More Related Contents:

  1. How can I one hot encode in Python?
  2. Custom transformer for sklearn Pipeline that alters both X and y
  3. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  4. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  5. Label encoding across multiple columns in scikit-learn
  6. Convert array of indices to 1-hot encoded numpy array
  7. ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
  8. Save classifier to disk in scikit-learn
  9. Accuracy Score ValueError: Can’t Handle mix of binary and continuous target
  10. How to get most informative features for scikit-learn classifiers?
  11. Dummy variables when not all categories are present
  12. Convert array of indices to one-hot encoded array in NumPy
  13. Why is Random Forest with a single tree much better than a Decision Tree classifier?
  14. How to get precision, recall and f-measure from confusion matrix in Python [duplicate]
  15. How to convert a Scikit-learn dataset to a Pandas dataset
  16. Feature/Variable importance after a PCA analysis
  17. Recovering features names of explained_variance_ratio_ in PCA with sklearn
  18. What is exactly sklearn.pipeline.Pipeline?
  19. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  20. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  21. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  22. LabelEncoder: TypeError: ‘>’ not supported between instances of ‘float’ and ‘str’
  23. Can anyone explain me StandardScaler?
  24. Run an OLS regression with Pandas Data Frame
  25. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  26. Stratified Sampling in Pandas
  27. scikit-learn: how to scale back the ‘y’ predicted result
  28. Using Smote with Gridsearchcv in Scikit-learn
  29. Does GridSearchCV perform cross-validation?
  30. Does the SVM in sklearn support incremental (online) learning?

Leave a Comment