How to use Python requests to fake a browser visit a.k.a and generate User Agent?

by

Provide a User-Agent header:

import requests

url="http://www.ichangtou.com/#company:data_000008.html"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

response = requests.get(url, headers=headers)
print(response.content)

FYI, here is a list of User-Agent strings for different browsers:

As a side note, there is a pretty useful third-party package called fake-useragent that provides a nice abstraction layer over user agents:

fake-useragent

Up to date simple useragent faker with real world database

Demo:

>>> from fake_useragent import UserAgent
>>> ua = UserAgent()
>>> ua.chrome
u'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'
>>> ua.random
u'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'

More Related Contents:

  1. Using python Requests with javascript pages
  2. Web scraping program cannot find element which I can see in the browser
  3. Can bs4 get the dynamic content of a webpage if requests can’t?
  4. How to limit download rate of HTTP requests in requests python library?
  5. Python 3: using requests does not get the full content of a web page
  6. XHR request URL says does not exist when attempting to parse it’s content
  7. How to install packages offline?
  8. How to send a “multipart/form-data” with requests in python?
  9. Changing user agent on urllib2.urlopen
  10. Problem HTTP error 403 in Python 3 Web Scraping
  11. How to send cookies in a post request with the Python Requests library?
  12. How to get Python requests to trust a self signed SSL certificate?
  13. Timeout for python requests.get entire response
  14. How to make python Requests work via socks proxy
  15. Python – Download Images from google Image search?
  16. Passing csrftoken with python Requests
  17. BeautifulSoup webscraping find_all( ): finding exact match
  18. Reading dynamically generated web pages using python
  19. Python requests SSL error – certificate verify failed
  20. Selenium-Debugging: Element is not clickable at point (X,Y)
  21. Why doesn’t requests.get() return? What is the default timeout that requests.get() uses?
  22. Python requests speed up using keep-alive
  23. Change the connection pool size for Python’s “requests” module when in Threading
  24. How can I get all the plain text from a website with Scrapy?
  25. Python regular expression for HTML parsing
  26. Python requests, how to limit received size, transfer rate, and/or total time?
  27. Clicking link using beautifulsoup in python
  28. Beautiful Soup and Table Scraping – lxml vs html parser
  29. Download all pdf files from a website using Python
  30. How to handle IncompleteRead: in python

Leave a Comment