Is there a version of Selenium WebDriver that is not detectable?

by

The fact that selenium driven WebDriver gets detected doesn’t depends on any specific Selenium, Chrome or ChromeDriver version. The Websites themselves can detect the network traffic and can identify the Browser Client i.e. Web Browser as WebDriver controled.

However some generic approaches to avoid getting detected while web-scraping are as follows:

@Antoine Vastel in his blog site Detecting Chrome Headless mentioned several approaches, which distinguish the Chrome browser from a headless Chrome browser.

  • User agent: The user agent attribute is commonly used to detect the OS as well as the browser of the user. With Chrome version 59 it has the following value: 

    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/59.0.3071.115 Safari/537.36

    • A check for the presence of Chrome headless can be done through:

      if (/HeadlessChrome/.test(window.navigator.userAgent)) {
    console.log("Chrome headless detected");
}

  • Plugins: navigator.plugins returns an array of plugins present in the browser. Typically, on Chrome we find default plugins, such as Chrome PDF viewer or Google Native Client. On the opposite, in headless mode, the array returned contains no plugin.

    • A check for the presence of Plugins can be done through:

      if(navigator.plugins.length == 0) {
    console.log("It may be Chrome headless");
}

  • Languages: In Chrome two Javascript attributes enable to obtain languages used by the user: navigator.language and navigator.languages. The first one is the language of the browser UI, while the second one is an array of string representing the user’s preferred languages. However, in headless mode, navigator.languages returns an empty string.

    • A check for the presence of Languages can be done through:

      if(navigator.languages == "") {
     console.log("Chrome headless detected");
}

  • WebGL: WebGL is an API to perform 3D rendering in an HTML canvas. With this API, it is possible to query for the vendor of the graphic driver as well as the renderer of the graphic driver. With a vanilla Chrome and Linux, we can obtain the following values for renderer and vendor: Google SwiftShader and Google Inc.. In headless mode, we can obtain Mesa OffScreen, which is the technology used for rendering without using any sort of window system and Brian Paul, which is the program that started the open source Mesa graphics library.

    • A check for the presence of WebGL can be done through:

      var canvas = document.createElement('canvas');
var gl = canvas.getContext('webgl');

var debugInfo = gl.getExtension('WEBGL_debug_renderer_info');
var vendor = gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL);
var renderer = gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL);

if(vendor == "Brian Paul" && renderer == "Mesa OffScreen") {
    console.log("Chrome headless detected");
}

    • Not all Chrome headless will have the same values for vendor and renderer. Others keep values that could also be found on non headless version. However, Mesa Offscreen and Brian Paul indicates the presence of the headless version.

  • Browser features: Modernizr library enables to test if a wide range of HTML and CSS features are present in a browser. The only difference we found between Chrome and headless Chrome was that the latter did not have the hairline feature, which detects support for hidpi/retina hairlines.

    • A check for the presence of hairline feature can be done through:

      if(!Modernizr["hairline"]) {
    console.log("It may be Chrome headless");
}

  • Missing image: The last on our list also seems to be the most robust, comes from the dimension of the image used by Chrome in case an image cannot be loaded. In case of a vanilla Chrome, the image has a width and height that depends on the zoom of the browser, but are different from zero. In a headless Chrome, the image has a width and an height equal to zero.

    • A check for the presence of Missing image can be done through:

      var body = document.getElementsByTagName("body")[0];
var image = document.createElement("img");
image.src = "http://iloveponeydotcom32188.jg";
image.setAttribute("id", "fakeimage");
body.appendChild(image);
image.onerror = function(){
    if(image.width == 0 && image.height == 0) {
    console.log("Chrome headless detected");
    }
}

References

You can find a couple of similar discussions in:

tl; dr

More Related Contents:

  1. How does chrome driver interact with Chrome browser?
  2. Many process of Google Chrome (32 bit)
  3. ERROR:ssl_client_socket_openssl.cc(1158)] handshake failed with ChromeDriver Chrome browser and Selenium
  4. selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 80
  5. WebDriverException: unknown error: DevToolsActivePort file doesn’t exist while trying to initiate Chrome Browser
  6. NoSuchElementException, Selenium unable to locate element
  7. Selenium Google Login Block
  8. How to run Selenium WebDriver test cases in Chrome
  9. SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81
  10. Why doesn’t ChromeDriver require Chrome or Chromium?
  11. Page load strategy for Chrome driver (Updated till Selenium v3.12.0)
  12. Server terminated early with status 1 error with Selenium for ChromeDriver and Chrome Browser and the log message “Only local connections are allowed”
  13. What is benefit of using ChromeDriver over WebDriver if we are using only Chrome Browser in our Selenium Automation Script
  14. How to turn off w3c in chromedriver to address the error unknown command: Cannot call non W3C standard command while in W3C
  15. Chrome – org.openqa.selenium.WebDriverException: unknown error: cannot get automation extension at driver.manage().window().maximize();
  16. Selenium WebDriverException: unknown error: call function result missing ‘value’ while calling sendkeys method
  17. Cannot get automation extension from timeout: Timed out receiving message from renderer
  18. Selenium how to manage wait for page load?
  19. How to open Chrome browser console through Selenium?
  20. WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 127
  21. Unable to sign into google with selenium automation because of “This browser or app may not be secure.”
  22. How to download XML files avoiding the popup This type of file may harm your computer through ChromeDriver and Chrome using Selenium in Python
  23. Need to find element in selenium by css
  24. How to get rid of the infobar “Chrome is being controlled by automated test software” through Selenium
  25. org.openqa.selenium.WebDriverException: chrome not reachable – when attempting to start a new session
  26. SessionNotCreatedException: This version of ChromeDriver only supports Chrome version 84 using ChromeDriver and Chrome through Selenium and Python
  27. How to execute Selenium Chrome WebDriver in silent mode?
  28. org.openqa.selenium.InvalidCookieDomainException: Document is cookie-averse using Selenium and WebDriver
  29. How to ignore ssl_client_socket_impl.cc(1061)] handshake failed in selenium c# ChromeDriver
  30. Selenium – Python – AttributeError: ‘WebDriver’ object has no attribute ‘find_element_by_name’

Leave a Comment