Count distinct words from a Pandas Data Frame

by

Use a set to create the sequence of unique elements.

Do some clean-up on df to get the strings in lower case and split:

df['text'].str.lower().str.split()
Out[43]: 
0             [my, nickname, is, ft.jgt]
1    [someone, is, going, to, my, place]

Each list in this column can be passed to set.update function to get unique values. Use apply to do so:

results = set()
df['text'].str.lower().str.split().apply(results.update)
print(results)

set(['someone', 'ft.jgt', 'my', 'is', 'to', 'going', 'place', 'nickname'])

Or use with Counter() from comments:

from collections import Counter
results = Counter()
df['text'].str.lower().str.split().apply(results.update)
print(results)

More Related Contents:

  1. Create Pandas DataFrame from txt file with specific pattern
  2. How can I create a text input box with Pygame?
  3. Pretty-print an entire Pandas Series / DataFrame
  4. why should I make a copy of a data frame in pandas
  5. How to pivot a dataframe in Pandas? [duplicate]
  6. Concatenate rows of two dataframes in pandas
  7. Normalize columns of pandas data frame
  8. How do I sum values in a column that match a given condition using pandas?
  9. multiprocessing in python – sharing large object (e.g. pandas dataframe) between multiple processes
  10. Pandas – Explanation on apply function being slow
  11. Pandas sum by groupby, but exclude certain columns
  12. Is there a simple way to change a column of yes/no to 1/0 in a Pandas dataframe?
  13. Pandas convert string to int
  14. Changing multiple column names but not all of them – Pandas Python
  15. python pandas flatten a dataframe to a list
  16. Convert pandas series from string to unique int ids [duplicate]
  17. Get all combinations of elements from two lists?
  18. How to create a dictionary of two pandas DataFrame columns
  19. Can You Consistently Keep Track of Column Labels Using Sklearn’s Transformer API?
  20. How to extract information between two unique words in a large text file
  21. Normalize data in pandas
  22. In Pandas, how to delete rows from a Data Frame based on another Data Frame?
  23. Python Pandas ValueError Arrays Must be All Same Length
  24. Get part of day (morning, afternoon, evening, night) in Python dataframe
  25. Sorting by absolute value without changing the data
  26. How to sort pandas data frame using values from several columns?
  27. How to install pandas from pip on windows cmd?
  28. Seaborn factor plot custom error bars instead of bootstrapping
  29. Convert hh:mm:ss to minutes using python pandas
  30. How can I fill in a missing values in range with Pandas?

Leave a Comment