How do I calculate a word-word co-occurrence matrix with sklearn?

by

Here is my example solution using CountVectorizer in scikit-learn. And referring to this post, you can simply use matrix multiplication to get word-word co-occurrence matrix.

from sklearn.feature_extraction.text import CountVectorizer
docs = ['this this this book',
        'this cat good',
        'cat good shit']
count_model = CountVectorizer(ngram_range=(1,1)) # default unigram model
X = count_model.fit_transform(docs)
# X[X > 0] = 1 # run this line if you don't want extra within-text cooccurence (see below)
Xc = (X.T * X) # this is co-occurrence matrix in sparse csr format
Xc.setdiag(0) # sometimes you want to fill same word cooccurence to 0
print(Xc.todense()) # print out matrix in dense format

You can also refer to dictionary of words in count_model, 

count_model.vocabulary_

Or, if you want to normalize by diagonal component (referred to answer in previous post).

import scipy.sparse as sp
Xc = (X.T * X)
g = sp.diags(1./Xc.diagonal())
Xc_norm = g * Xc # normalized co-occurence matrix

Extra to note @Federico Caccia answer, if you don’t want co-occurrence that are spurious from the own text, set occurrence that is greater that 1 to 1 e.g.

X[X > 0] = 1 # do this line first before computing cooccurrence
Xc = (X.T * X)
...

More Related Contents:

  1. How can I plot a confusion matrix? [duplicate]
  2. Why does the following code cant get value of n_fold = 1?
  3. scikit-learn & statsmodels – which R-squared is correct?
  4. ImportError in importing from sklearn: cannot import name check_build
  5. Find p-value (significance) in scikit-learn LinearRegression
  6. Stratified Train/Test-split in scikit-learn
  7. Numpy ‘smart’ symmetric matrix
  8. How does sklearn.svm.svc’s function predict_proba() work internally?
  9. Error importing scikit-learn modules
  10. Efficiently create sparse pivot tables in pandas?
  11. ROC for multiclass classification
  12. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  13. Block tridiagonal matrix python
  14. How to generalize fast matrix multiplication on GPU using numba
  15. Passing categorical data to Sklearn Decision Tree
  16. Scikit Learn OneHotEncoder fit and transform Error: ValueError: X has different shape than during fitting
  17. Why does numpy.linalg.solve() offer more precise matrix inversions than numpy.linalg.inv()?
  18. Getting model attributes from pipeline
  19. Efficiently count word frequencies in python
  20. NumPy array/matrix of mixed types
  21. Sklearn Pipeline: Get feature names after OneHotEncode In ColumnTransformer
  22. Scikit Learn GridSearchCV without cross validation (unsupervised learning)
  23. Convert row vector to column vector in NumPy
  24. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  25. sklearn : TFIDF Transformer : How to get tf-idf values of given words in document
  26. Using scikit to determine contributions of each feature to a specific class prediction
  27. In sklearn.decomposition.PCA, why are components_ negative?
  28. Pyinstaller ; ModuleNotFoundError: No module named ‘sklearn.utils._cython_blas’
  29. Model help using Scikit-learn when using GridSearch
  30. Determining the most contributing features for SVM classifier in sklearn

Leave a Comment