UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xe9’ – -when using urlib.request python3

by

Use a percent-encoded URL:

link = 'http://finance.yahoo.com/news/caf%C3%A9s-growing-faster-than-fast-food-peers-144512056.html'

I found the above percent-encoded URL by pointing the browser at 

http://finance.yahoo.com/news/cafés-growing-faster-than-fast-food-peers-144512056.html

going to the page, then copying-and-pasting the
encoded url supplied by the browser back into the text editor. However, you can generate a percent-encoded URL programmatically using:

from urllib import parse

link = 'http://finance.yahoo.com/news/cafés-growing-faster-than-fast-food-peers-144512056.html'

scheme, netloc, path, query, fragment = parse.urlsplit(link)
path = parse.quote(path)
link = parse.urlunsplit((scheme, netloc, path, query, fragment))

which yields

http://finance.yahoo.com/news/caf%C3%A9s-growing-faster-than-fast-food-peers-144512056.html

More Related Contents:

  1. retrieve links from web page using python and BeautifulSoup [closed]
  2. How to find elements by class
  3. can we use XPath with BeautifulSoup?
  4. Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
  5. Using BeautifulSoup to extract text without tags
  6. Scraping Google Finance (BeautifulSoup)
  7. How to scrape a website which requires login using python and beautifulsoup?
  8. Web scraping program cannot find element which I can see in the browser
  9. BeautifulSoup webscraping find_all( ): finding exact match
  10. Beautiful Soup 4 find_all don’t find links that Beautiful Soup 3 finds
  11. How to get text from span tag in BeautifulSoup
  12. How to scrape only visible webpage text with BeautifulSoup?
  13. How to find tag with particular text with Beautiful Soup?
  14. Can bs4 get the dynamic content of a webpage if requests can’t?
  15. Python regular expression for HTML parsing
  16. How find specific data attribute from html tag in BeautifulSoup4?
  17. Exclude unwanted tag on Beautifulsoup Python
  18. BeautifulSoup: Get the contents of a specific table
  19. Speeding up beautifulsoup
  20. Scrape Dynamic contents created by Javascript using Python
  21. BeautifulSoup returns None even though the element exists
  22. Clicking link using beautifulsoup in python
  23. Beautiful Soup and Table Scraping – lxml vs html parser
  24. Download all pdf files from a website using Python
  25. How to scrape dynamic webpages by Python
  26. How to handle IncompleteRead: in python
  27. Beautiful Soup: ‘ResultSet’ object has no attribute ‘find_all’?
  28. Scraping dynamic content using python-Scrapy
  29. Matching partial ids in BeautifulSoup
  30. BeautifulSoup – modifying all links in a piece of HTML?

Leave a Comment