Using the probabilities of the topics, you can try to set some threshold and use it as a clustering baseline, but i am sure there are better ways to do clustering than this ‘hacky’ method. from gensim import corpora, models, similarities from itertools import chain “”” DEMO “”” documents = [“Human machine interface for lab … Read more
train() expects a sequence of sentences on input, not one sentence. train() only updates weights for existing feature vectors based on existing vocabulary. You cannot add new vocabulary (=new feature vectors) using train().
I use this code to load binary model, then save the model to text file, from gensim.models.keyedvectors import KeyedVectors model = KeyedVectors.load_word2vec_format(‘path/to/GoogleNews-vectors-negative300.bin’, binary=True) model.save_word2vec_format(‘path/to/GoogleNews-vectors-negative300.txt’, binary=False) References: API and nullege. Note: Above code is for new version of gensim. For previous version, I used this code: from gensim.models import word2vec model = word2vec.Word2Vec.load_word2vec_format(‘path/to/GoogleNews-vectors-negative300.bin’, binary=True) model.save_word2vec_format(‘path/to/GoogleNews-vectors-negative300.txt’, binary=False)
Why does the same LDA parameters and corpus generate different topics everytime? Because LDA uses randomness in both training and inference steps. And how do i stabilize the topic generation? By resetting the numpy.random seed to the same value every time a model is trained or inference is performed, with numpy.random.seed: SOME_FIXED_SEED = 42 # … Read more
In recent gensim versions you can load a subset starting from the front of the file using the optional limit parameter to load_word2vec_format(). (The GoogleNews vectors seem to be in roughly most- to least- frequent order, so the first N are usually the N-sized subset you’d want. So use limit=500000 to get the most-frequent 500,000 … Read more
here the procedure to incorporate the fasttext model inside an LSTM Keras network # define dummy data and precproces them docs = [‘Well done’, ‘Good work’, ‘Great effort’, ‘nice work’, ‘Excellent’, ‘Weak’, ‘Poor effort’, ‘not good’, ‘poor work’, ‘Could have done better’] docs = [d.lower().split() for d in docs] # train fasttext from gensim api … Read more
This is actually a pretty challenging problem that you are asking. Computing sentence similarity requires building a grammatical model of the sentence, understanding equivalent structures (e.g. “he walked to the store yesterday” and “yesterday, he walked to the store”), finding similarity not just in the pronouns and verbs but also in the proper nouns, finding … Read more
Do not call .train() multiple times in your own loop that tries to do alpha arithmetic. It’s unnecessary, and it’s error-prone. Specifically, in the above code, decrementing the original 0.025 alpha by 0.001 forty times results in (0.025 – 40*0.001) -0.015 final alpha, which would also have been negative for many of the training epochs. … Read more