sklearn : TFIDF Transformer : How to get tf-idf values of given words in document

by

You can use TfidfVectorizer from sklean

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from scipy.sparse.csr import csr_matrix #need this if you want to save tfidf_matrix

tf = TfidfVectorizer(input="filename", analyzer="word", ngram_range=(1,6),
                     min_df = 0, stop_words="english", sublinear_tf=True)
tfidf_matrix =  tf.fit_transform(corpus)

The above tfidf_matix has the TF-IDF values of all the documents in the corpus. This is a big sparse matrix. Now,

feature_names = tf.get_feature_names()

this gives you the list of all the tokens or n-grams or words.
For the first document in your corpus,

doc = 0
feature_index = tfidf_matrix[doc,:].nonzero()[1]
tfidf_scores = zip(feature_index, [tfidf_matrix[doc, x] for x in feature_index])

Lets print them,

for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:
  print w, s

More Related Contents:

  1. python kmeans on string
  2. Why does the following code cant get value of n_fold = 1?
  3. How to extract the decision rules from scikit-learn decision-tree?
  4. How to one-hot-encode from a pandas column containing a list?
  5. scikit-learn & statsmodels – which R-squared is correct?
  6. ImportError in importing from sklearn: cannot import name check_build
  7. Random state (Pseudo-random number) in Scikit learn
  8. Error importing scikit-learn modules
  9. How to one hot encode variant length features?
  10. pandas dataframe columns scaling with sklearn
  11. How do I calculate a word-word co-occurrence matrix with sklearn?
  12. Clustering text documents using scikit-learn kmeans in Python
  13. scikit-learn cross validation, negative values with mean squared error
  14. Efficiently create sparse pivot tables in pandas?
  15. ROC for multiclass classification
  16. Passing categorical data to Sklearn Decision Tree
  17. fit-transform on training data and transform on test data [duplicate]
  18. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  19. Getting model attributes from pipeline
  20. Efficiently count word frequencies in python
  21. Sklearn Pipeline: Get feature names after OneHotEncode In ColumnTransformer
  22. How to find the importance of the features for a logistic regression model?
  23. scikit-learn random state in splitting dataset
  24. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  25. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  26. TypeError: cannot perform reduce with flexible type
  27. scikit-learn: how to scale back the ‘y’ predicted result
  28. Using Smote with Gridsearchcv in Scikit-learn
  29. Does GridSearchCV perform cross-validation?
  30. Python import error: cannot import name ‘six’ from ‘sklearn.externals’

Leave a Comment