How to rotate various user agents using selenium python on each request

by

First the update 1

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the user-agent easily to prevent Selenium from getting detected.

  • Code Block:

    from selenium import webdriver

driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe')
print(driver.execute_script("return navigator.userAgent;"))
# Setting user agent as Chrome/83.0.4103.97
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
# Setting user agent as Chrome/83.0.4103.53
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

  • Console Output:

    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36

  • Browser Snapshot:

useragent

Legend: 1 – Applicable only to Selenium python clients.

Originally answered Nov 6 ’18 at 8:00

No. When you configure an instance of a ChromeDriver with ChromeOptions to initiate a new Chrome Browser Session the configuration of the ChromeDriver remains unchanged throughout the lifetime of the ChromeDriver and remains uneditable. So you can’t change the user agent when the WebDriver instance is executing the loop making 10 requests.

Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. UserAgent, Session ID, Cookies and other session attributes from the already initiated Browsing Session still you won’t be able to change those attributes of the ChromeDriver.

A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.

Here you can find a relevant discussion on How can I reconnect to the browser opened by webdriver with selenium?

Reference

You can find a couple of relevant detailed discussions in:

More Related Contents:

  1. How to allow or deny notification geo-location microphone camera pop up
  2. Selenium on MAC, Message: ‘chromedriver’ executable may have wrong permissions
  3. How to get rid of the infobar “Chrome is being controlled by automated test software” through Selenium
  4. Undetected Chromedriver not loading correctly
  5. Python Selenium driver.execute_script WebDriverException: Message: unknown error: call function result missing ‘value’
  6. selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service chromedriver.exe while opening chrome browser
  7. ChromeDriver Error: ‘chromedriver’ executable needs to be in PATH [duplicate]
  8. Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed
  9. How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
  10. unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed with ChromeDriver Selenium
  11. How to get attribute of element from Selenium?
  12. ‘Webdrivers’ executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home
  13. How to work with a specific version of ChromeDriver while Chrome Browser gets updated automatically through Python selenium
  14. Error Message: ‘chromedriver’ executable needs to be PATH
  15. find_element_by_* commands are deprecated in selenium
  16. Filling in login forms in Instagram using selenium and webdriver (chrome) python OSX
  17. Unable to use Selenium to automate Chase site login
  18. Clicking on Get Data button for Monthly Settlement Statistics on nseindia.com doesn’t fetch results using Selenium and Python
  19. How to rotate Selenium webrowser IP address
  20. How do you fix the “element not interactable” exception?
  21. ‘list’ object has no attribute ‘get_attribute’ while iterating through WebElements
  22. Access to table objects on Webpage Using Python Selenium
  23. How to open multiple webpages in separate tabs within a browser using selenium-webdriver and python
  24. How to change the User Agent using Selenium and Python
  25. SessionNotCreatedException: This version of ChromeDriver only supports Chrome version 84 using ChromeDriver and Chrome through Selenium and Python
  26. selenium grid listening on node port instead of hub port
  27. How to test non-standard drop down lists through a crawler using Selenium and Python
  28. Python selenium: wait until element is clickable – not working
  29. Handle notifications in Python + Selenium Chrome WebDriver
  30. What is unittest in selenium Python?

Leave a Comment