Scikit learn – fit_transform on the test set

by

You are not supposed to do fit_transform on your test data, but only transform. Otherwise, you will get different vectorization than the one used during training.

For the memory issue, I recommend TfIdfVectorizer, which has numerous options of reducing the dimensionality (by removing rare unigrams etc.).

UPDATE

If the only problem is fitting test data, simply split it to small chunks. Instead of something like

x=vect.transform(test)
eval(x)

you can do

K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    eval(x)

and record results/stats and analyze them afterwards.

in particular

predictions = []

K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    predictions += rf.predict(x) # assuming it retuns a list of labels, otherwise - convert it to list

print accuracy_score( predictions, true_labels )

More Related Contents:

  1. Getting a low ROC AUC score but a high accuracy
  2. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  3. How to extract the decision rules from scikit-learn decision-tree?
  4. Loss & accuracy – Are these reasonable learning curves?
  5. Save classifier to disk in scikit-learn
  6. How to get most informative features for scikit-learn classifiers?
  7. Predict classes or class probabilities?
  8. How to interpret MSE in Keras Regressor
  9. Why is Random Forest with a single tree much better than a Decision Tree classifier?
  10. What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
  11. Cost function training target versus accuracy desired goal
  12. Why does one hot encoding improve machine learning performance? [closed]
  13. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  14. How to put more weight on certain features in machine learning?
  15. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  16. Early stopping with Keras and sklearn GridSearchCV cross-validation
  17. Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit python
  18. How should I teach machine learning algorithm using data with big disproportion of classes? (SVM)
  19. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?
  20. scikit-learn .predict() default threshold
  21. Will it lead to Overfitting / Curse of Dimensionality
  22. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  23. Finding K-nearest neighbors and its implementation
  24. Many to one and many to many LSTM examples in Keras
  25. What is the difference between loss function and metric in Keras? [duplicate]
  26. How to interpret loss and accuracy for a machine learning model [closed]
  27. Keras: model.evaluate vs model.predict accuracy difference in multi-class NLP task
  28. Understanding concept of Gaussian Mixture Models
  29. What is `lr_policy` in Caffe?
  30. Does the SVM in sklearn support incremental (online) learning?

Leave a Comment