What is the significance and difference between each file? Train.txt/Dev.txt/Test.txt ? This is standard machine learning terminology. The train set is used to (surprise surprise) train a model. The development set is used to tune any parameters the model might have. What you would normally do is pick a parameter value, train a model on … Read more
Thanks to the link discovered by @Vaulstein, it is clear that the trained Stanford tagger, as distributed (at least in 2012) does not chunk named entities. From the accepted answer: Many NER systems use more complex labels such as IOB labels, where codes like B-PERS indicates where a person entity starts. The CRFClassifier class and … Read more
Use py-corenlp Download Stanford CoreNLP The latest version at this time (2020-05-25) is 4.0.0: wget https://nlp.stanford.edu/software/stanford-corenlp-4.0.0.zip https://nlp.stanford.edu/software/stanford-corenlp-4.0.0-models-english.jar If you do not have wget, you probably have curl: curl https://nlp.stanford.edu/software/stanford-corenlp-4.0.0.zip -O https://nlp.stanford.edu/software/stanford-corenlp-4.0.0-models-english.jar -O If all else fails, use the browser 😉 Install the package unzip stanford-corenlp-4.0.0.zip mv stanford-corenlp-4.0.0-models-english.jar stanford-corenlp-4.0.0 Start the server cd stanford-corenlp-4.0.0 java -mx5g … Read more
You can check the DocumentPreprocessor class. Below is a short snippet. I think there may be other ways to do what you want. String paragraph = “My 1st sentence. “Does it work for questions?” My third sentence.”; Reader reader = new StringReader(paragraph); DocumentPreprocessor dp = new DocumentPreprocessor(reader); List<String> sentenceList = new ArrayList<String>(); for (List<HasWord> sentence … Read more
The Penn Treebank Project. Look at the Part-of-speech tagging ps. JJ is adjective. NNS is noun, plural. VBP is verb present tense. RB is adverb. That’s for english. For chinese, it’s the Penn Chinese Treebank. And for german it’s the NEGRA corpus. CC Coordinating conjunction CD Cardinal number DT Determiner EX Existential there FW Foreign … Read more
Note that this answer applies to NLTK v 3.0, and not to more recent versions. Sure, try the following in Python: import os from nltk.parse import stanford os.environ[‘STANFORD_PARSER’] = ‘/path/to/standford/jars’ os.environ[‘STANFORD_MODELS’] = ‘/path/to/standford/jars’ parser = stanford.StanfordParser(model_path=”/location/of/the/englishPCFG.ser.gz”) sentences = parser.raw_parse_sents((“Hello, My name is Melroy.”, “What is your name?”)) print sentences # GUI for line in sentences: … Read more
You could install and use the nltk library. This provides you with a list of English words and a means to split each line into words: from nltk.tokenize import word_tokenize from nltk.corpus import words english = words.words() with open(‘Dari.pos’) as f_input, open(‘DariNER.txt’, ‘w’) as f_output: for line in f_input: f_output.write(‘ ‘.join(word for word in word_tokenize(line) … Read more
If I understand the correct output format word-O, you can try something like this: words = open(‘filename’).read().split() labeled_words = [word+”-O” for word in words] # And now user your output format, each word a line, separate by tabs, whatever. # For example new lines with open(‘outputfile’,’w’) as output: output.write(“\n”.join(labeled_words))