Fetch a Wikipedia article with Python

by

You need to use the urllib2 that superseedes urllib in the python std library in order to change the user agent.

Straight from the examples

import urllib2
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
infile = opener.open('http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes')
page = infile.read()

More Related Contents:

  1. Changing user agent on urllib2.urlopen
  2. How to download image using requests
  3. What are the differences between the urllib, urllib2, urllib3 and requests module?
  4. How to use Python requests to fake a browser visit a.k.a and generate User Agent?
  5. Downloading a picture via urllib and python
  6. Does python urllib2 automatically uncompress gzip data fetched from webpage?
  7. How to fetch a non-ascii url with urlopen?
  8. A good way to get the charset/encoding of an HTTP response in Python
  9. Python Urllib2 SSL error
  10. How do I prevent Python’s urllib(2) from following a redirect
  11. Python URLLib / URLLib2 POST
  12. Python urllib2 Basic Auth Problem
  13. Python urllib2 with keep alive
  14. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  15. Python urllib2 Progress Hook
  16. Extracting text from script tag using BeautifulSoup in Python
  17. Python urllib over TOR? [duplicate]
  18. Python Requests: Post JSON and file in single request
  19. What should I do if socket.setdefaulttimeout() is not working?
  20. Multiple (asynchronous) connections with urllib2 or other http library?
  21. Tell urllib2 to use custom DNS
  22. opening websites using urllib2 from behind corporate firewall – 11004 getaddrinfo failed
  23. Source interface with Python and urllib2
  24. Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str
  25. Python POST binary data
  26. How to get the URL of a redirect with Python
  27. urllib2 file name
  28. Using urllib2 with SOCKS proxy
  29. How do I send a custom header with urllib2 in a HTTP Request?
  30. urllib2 and json

Leave a Comment