How to get a Token from a Lucene TokenStream?

by

Yeah, it’s a little convoluted (compared to the good ol’ way), but this should do it:

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.getAttribute(OffsetAttribute.class);
TermAttribute termAttribute = tokenStream.getAttribute(TermAttribute.class);

while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = termAttribute.term();
}

Edit: The new way

According to Donotello, TermAttribute has been deprecated in favor of CharTermAttribute. According to jpountz (and Lucene’s documentation), addAttribute is more desirable than getAttribute.

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.addAttribute(OffsetAttribute.class);
CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class);

tokenStream.reset();
while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = charTermAttribute.toString();
}

More Related Contents:

  1. Is it 2 character search possible in lucence
  2. How to split a string if any chracter is found in java
  3. Scanner vs. StringTokenizer vs. String.Split
  4. What is the easiest/best/most correct way to iterate through the characters of a string in Java?
  5. private final static attribute vs private final attribute
  6. how to get data between quotes in java?
  7. How do I read input character-by-character in Java?
  8. How do you tell if a checkbox is selected in Selenium for Java?
  9. N-gram generation from a sentence
  10. How to do query auto-completion/suggestions in Lucene?
  11. How to loop over a Class attributes in Java?
  12. Tokenizing Error: java.util.regex.PatternSyntaxException, dangling metacharacter ‘*’
  13. How to implement auto suggest using Lucene’s new AnalyzingInfixSuggester API?
  14. Get highest frequency terms from Lucene index
  15. What is the default list of stopwords used in Lucene’s StopFilter?
  16. Exact Phrase search using Lucene?
  17. Unable to implement Struts 2 token interceptor with hyperlink
  18. JWT (JSON Web Token) library for Java [closed]
  19. Stemming English words with Lucene
  20. string tokenizer in Java
  21. Lucene: Multi-word phrases as search terms
  22. Mac User – How do I set CLASSPATHS in Mac (I’m working on a Lucene Demo)
  23. Why 2 return statements are mandatory in equals method
  24. Scanner vs. BufferedReader
  25. How to convert milliseconds to “hh:mm:ss” format?
  26. @OneToMany and composite primary keys?
  27. Load arrayList data into JTable
  28. How to downgrade Java from 9 to 8 on a MACOS. Eclipse is not running with Java 9
  29. JpaRepository caches newly created object. How to refresh it?
  30. java timer task schedule

Leave a Comment