Removing non-English words from text using Python

by

You can use the words corpus from NLTK:

import nltk
words = set(nltk.corpus.words.words())

sent = "Io andiamo to the beach with my amico."
" ".join(w for w in nltk.wordpunct_tokenize(sent) \
         if w.lower() in words or not w.isalpha())
# 'Io to the beach with my'

Unfortunately, Io happens to be an English word. In general, it may be hard to decide whether a word is English or not.

More Related Contents:

  1. look through a very large numbers (1.2e+34) in Python
  2. Tuples conversion into JSON with python [closed]
  3. Unable to allocate array with shape and data type
  4. ‘Conda’ is not recognized as internal or external command
  5. ValueError: Wrong number of items passed – Meaning and suggestions?
  6. How to plot multiple pandas columns
  7. Cannot import name ‘CRS’ from ‘pyproj’ for using the osmnx library
  8. Scikit-learn’s LabelBinarizer vs. OneHotEncoder
  9. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  10. How can repetitive rows of data be collected in a single row in pandas?
  11. Where do I call the BatchNormalization function in Keras?
  12. Python pandas groupby aggregate on multiple columns, then pivot
  13. How to remove items from a list while iterating?
  14. Text progress bar in terminal with block characters [closed]
  15. How do I check what version of Python is running my script?
  16. What does the percentage sign mean in Python
  17. pandas dataframe str.contains() AND operation
  18. How to insert a character after every 2 characters in a string
  19. Bin size in Matplotlib (Histogram)
  20. How can I check whether a numpy array is empty or not?
  21. Run local python script on remote server
  22. is there a way to loop over two lists simultaneously in django?
  23. Python script to list users and groups
  24. Add Jar to standalone pyspark
  25. Mass DM bot was working fine and now it wont send messages
  26. How to store an image in a variable
  27. How to find the intersection of two graphs
  28. Size of a Python list in memory
  29. Using VirtualEnv with multiple Python versions on windows
  30. Setting styles in Openpyxl

Leave a Comment