How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?

by

In short:

df['Text'].apply(word_tokenize)

Or if you want to add another column to store the tokenized list of strings:

df['tokenized_text'] = df['Text'].apply(word_tokenize)

There are tokenizers written specifically for twitter text, see http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.casual

To use nltk.tokenize.TweetTokenizer:

from nltk.tokenize import TweetTokenizer
tt = TweetTokenizer()
df['Text'].apply(tt.tokenize)

Similar to:

More Related Contents:

  1. re.sub erroring with “Expected string or bytes-like object”
  2. NLTK-based text processing with pandas
  3. How do I tokenize a string sentence in NLTK?
  4. How to get rid of punctuation using NLTK tokenizer?
  5. Counting the Frequency of words in a pandas data frame
  6. Pandas max value index
  7. Difference between map, applymap and apply methods in Pandas
  8. Python Pandas : group by in group by and average?
  9. Python pandas insert list into a cell
  10. Stacked Bar Chart with Centered Labels
  11. How to replace text in a string column of a Pandas dataframe?
  12. Pandas DataFrame Groupby two columns and get counts
  13. Pandas dataframe get first row of each group
  14. Pandas: conditional rolling count
  15. What does the term “broadcasting” mean in Pandas documentation?
  16. Selection with .loc in python
  17. Add column to dataframe with constant value
  18. Check if string is in a pandas dataframe
  19. Why is Apache-Spark – Python so slow locally as compared to pandas?
  20. Pandas: compare list objects in Series
  21. Python / Pandas – GUI for viewing a DataFrame or Matrix [closed]
  22. Filter string data based on its string length
  23. Converting pandas column of comma-separated strings into dummy variables
  24. Creating an empty Pandas DataFrame, and then filling it
  25. Matplotlib showing x-tick labels overlapping
  26. Check if rows in one dataframe exist in another dataframe
  27. Why doesn’t my pandas rolling().apply() work when the series contains collections?
  28. Pandas custom function to find whether it is the 1st, 2nd etc Monday, Tuesday, etc – all suggestions welcome
  29. Pandas Rolling Computations on Sliding Windows (Unevenly spaced)
  30. DataFrame object has no attribute append

Leave a Comment