One way to do this would be to extract words that occur more frequently in a document than you would expect them to by chance. For example, say in a larger collection of documents the term ‘Markov’ is almost never seen. However, in a particular document from the same collection Markov shows up very frequently. … Read more
I made that wikipedia contraction-to-expansion page into a python dictionary (see below) Note, as you might expect, that you definitely want to use double quotes when querying the dictionary: Also, I’ve left multiple options in as in the wikipedia page. Feel free to modify it as you wish. Note that disambiguation to the right expansion … Read more
Three excellent and mature NLP packages are Stanford Core NLP, Open NLP and LingPipe. There are Ruby bindings to the Stanford Core NLP tools (GPL license) as well as the OpenNLP tools (Apache License). On the more experimental side of things, I maintain a Text Retrieval, Extraction and Annotation Toolkit (Treat), released under the GPL, … Read more
You have basically three options: You can cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient. You can split your text in multiple subtexts, classify each of them and combine the … Read more
They likely use Information Extraction techniques for this. Here is a demo of Stanford’s SUTime tool: http://nlp.stanford.edu:8080/sutime/process You would extract attributes about n-grams (consecutive words) in a document: numberOfLetters numberOfSymbols length previousWord nextWord nextWordNumberOfSymbols … And then use a classification algorithm, and feed it positive and negative examples: Observation nLetters nSymbols length prevWord nextWord isPartOfDate … Read more
I think there is a lot of confusion about which weights are used for what. I am not sure I know precisely what bothers you so I am going to cover different topics, bear with me ;). Class weights The weights from the class_weight parameter are used to train the classifier. They are not used … Read more
Short and dense: http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. However, the two words differ in their flavor. Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope … Read more
This seems like a graph problem. You could try to use networkx: import networkx as nx G = nx.from_pandas_edgelist(df, ‘v1’, ‘v2’) clusters = nx.connected_components(G) output: [{‘be’, ‘belong’}, {‘delay’, ‘increase’, ‘decrease’}, {‘analyze’, ‘assay’}, {‘report’, ‘bespeak’, ‘circulate’}, {‘induce’, ‘generate’}, {‘trip’, ’cause’}, {‘distinguish’, ‘isolate’}, {‘infect’, ‘give’}, {‘prove’, ‘result’}, {‘intercede’, ‘describe’, ‘explain’}, {‘affect’, ‘expose’}, {‘restrict’, ‘suppress’}] As graph: Small … Read more
If you have you data in exactly the same structure as the movie_review corpus in NLTK, there are two ways to “hack” your way through: 1. Put your corpus directory into where you save the nltk.data First check where is your nltk.data saved: >>> import nltk >>> nltk.data.find(‘corpora/movie_reviews’) FileSystemPathPointer(u’/home/alvas/nltk_data/corpora/movie_reviews’) Then move your directory to where … Read more