Chrome browser initiated through ChromeDriver gets detected

by

I have tried to access the url https://www.mouser.co.uk/ with certain chrome.options but did get detected and was redirected to Pardon Our Interruption page.

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.mouser.co.uk")
myElement = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//a[@id='1_lnkLeftFlag']")))
driver.execute_script("arguments[0].click();", myElement)

Now on inspecting the Pardon Our Interruption page you will find the <body> tag contains:

  • The class attribute dist-GlobalHeader
  • The class attribute dist-PageWrap

Which is a clear indication that the website is protected by Bot Management service provider Distil Networks and the navigation by ChromeDriver gets detected and subsequently blocked.

Distil

As per the article There Really Is Something About Distil.it…:

Distil protects sites against automatic content scraping bots by observing site behavior and identifying patterns peculiar to scrapers. When Distil identifies a malicious bot on one site, it creates a blacklisted behavioral profile that is deployed to all its customers. Something like a bot firewall, Distil detects patterns and reacts.

Further,

"One pattern with Selenium was automating the theft of Web content", Distil CEO Rami Essaid said in an interview last week. "Even though they can create new bots, we figured out a way to identify Selenium the a tool they're using, so we're blocking Selenium no matter how many times they iterate on that bot. We're doing that now with Python and a lot of different technologies. Once we see a pattern emerge from one type of bot, then we work to reverse engineer the technology they use and identify it as malicious".

Reference

You can find a couple of detailed discussion in:

More Related Contents:

  1. Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed
  2. WebDriverException: Message: ‘chromedriver’ executable needs to be in PATH while setting UserAgent through Selenium Chromedriver python
  3. unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed with ChromeDriver Selenium
  4. ‘Webdrivers’ executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home
  5. WebDriverException: unknown error: cannot find Chrome binary error with Selenium in Python for older versions of Google Chrome
  6. How to work with a specific version of ChromeDriver while Chrome Browser gets updated automatically through Python selenium
  7. WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 127
  8. Error Message: ‘chromedriver’ executable needs to be PATH
  9. What is the fastest way to open urls in new tabs via Selenium – Python?
  10. Clicking on Get Data button for Monthly Settlement Statistics on nseindia.com doesn’t fetch results using Selenium and Python
  11. How to download XML files avoiding the popup This type of file may harm your computer through ChromeDriver and Chrome using Selenium in Python
  12. TypeError: ‘module’ object is not callable error with driver=webdriver(“C:\\Python34\\Lib\\site-packages\\selenium\\webdriver\\chromedriver.exe”)
  13. SessionNotCreatedException: This version of ChromeDriver only supports Chrome version 84 using ChromeDriver and Chrome through Selenium and Python
  14. How to solve ‘Getting Default Adapter failed’ error when launching Chrome and try to access a webpage using the ChromeDriver using Selenium
  15. WebDriverException: Service U:/Scraping/chromedriver.exe unexpectedly exited. Status code was: 1 while working with Chrome and Python
  16. Selenium – Python – AttributeError: ‘WebDriver’ object has no attribute ‘find_element_by_name’
  17. ChromeDriver Error: ‘chromedriver’ executable needs to be in PATH [duplicate]
  18. How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
  19. Running Selenium with Headless Chrome Webdriver
  20. Unable to use Selenium to automate Chase site login
  21. How to disable CSS in Python selenium using ChromeOptions
  22. ERROR:gpu_process_transport_factory.cc(1007)-Lost UI shared context : while initializing Chrome browser through ChromeDriver in Headless mode
  23. Using Extensions with Selenium (Python)
  24. How do I pass options to the Selenium Chrome driver using Python?
  25. Chromedriver only supports characters in the BMP error while sending Emoji with ChromeDriver Chrome using Selenium Python to Tkinter’s label() textbox
  26. How to get rid of the infobar “Chrome is being controlled by automated test software” through Selenium
  27. Handle notifications in Python + Selenium Chrome WebDriver
  28. How to rotate various user agents using selenium python on each request
  29. How to install Chrome Extension using Selenium & Python
  30. Automation Google login with python and selenium shows “”This browser or app may be not secure””

Leave a Comment