Selenium and non-headless browser keeps asking for Captcha

by

In the discussion entitled How does recaptcha 3 know I’m using selenium/chromedriver we have discussed about some generic approaches to avoid getting detected while web-scraping. Let’s deep dive.

Headless Browser

A headless browser is a browser that can be used without a graphical interface. It can be controlled programmatically to automate tasks, such as doing tests or taking screenshots of webpages.

Why detect headless browser?

As per @AntoineVastel, headless browsers are used to automate malicious tasks. The most common cases are web scraping, increase advertisement impressions or look for vulnerabilities on a website.

Until an year ago, one of the most popular headless browser was PhantomJS. Since it is built on the Qt framework, it exhibits many differences compared to most popular browsers. It was possible to detect PhantomJS using some browser fingerprinting techniques. Since version 59, Google released a headless version of its Chrome browser. Unlike PhantomJS, it is based on a vanilla Chrome, and not on an external framework, making its presence more difficult to detect. So there are likely other ways to detect Chrome headless.

Detecting Chrome Headless

  • User agent: The user agent attribute is commonly used to detect the OS as well as the browser of the user. With Chrome version 59 it has the following value: 

    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/59.0.3071.115 Safari/537.36

    • A check for the presence of Chrome headless can be done through:

      if (/HeadlessChrome/.test(window.navigator.userAgent)) {
    console.log("Chrome headless detected");
}

  • Plugins: navigator.plugins returns an array of plugins present in the browser. Typically, on Chrome we find default plugins, such as Chrome PDF viewer or Google Native Client. On the opposite, in headless mode, the array returned contains no plugin.

    • A check for the presence of Plugins can be done through:

      if(navigator.plugins.length == 0) {
    console.log("It may be Chrome headless");
}

  • Languages: In Chrome two Javascript attributes enable to obtain languages used by the user: navigator.language and navigator.languages. The first one is the language of the browser UI, while the second one is an array of string representing the user’s preferred languages. However, in headless mode, navigator.languages returns an empty string.

    • A check for the presence of Languages can be done through:

      if(navigator.languages == "") {
     console.log("Chrome headless detected");
}

  • WebGL: WebGL is an API to perform 3D rendering in an HTML canvas. With this API, it is possible to query for the vendor of the graphic driver as well as the renderer of the graphic driver. With a vanilla Chrome and Linux, we can obtain the following values for renderer and vendor: Google SwiftShader and Google Inc.. In headless mode, we can obtain Mesa OffScreen, which is the technology used for rendering without using any sort of window system and Brian Paul, which is the program that started the open source Mesa graphics library.

    • A check for the presence of WebGL can be done through:

      var canvas = document.createElement('canvas');
var gl = canvas.getContext('webgl');

var debugInfo = gl.getExtension('WEBGL_debug_renderer_info');
var vendor = gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL);
var renderer = gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL);

if(vendor == "Brian Paul" && renderer == "Mesa OffScreen") {
    console.log("Chrome headless detected");
}

    • Not all Chrome headless will have the same values for vendor and renderer. Others keep values that could also be found on non headless version. However, Mesa Offscreen and Brian Paul indicates the presence of the headless version.

  • Browser features: Modernizr library enables to test if a wide range of HTML and CSS features are present in a browser. The only difference we found between Chrome and headless Chrome was that the latter did not have the hairline feature, which detects support for hidpi/retina hairlines.

    • A check for the presence of hairline feature can be done through:

      if(!Modernizr["hairline"]) {
    console.log("It may be Chrome headless");
}

  • Missing image: The last on our list also seems to be the most robust, comes from the dimension of the image used by Chrome in case an image cannot be loaded. In case of a vanilla Chrome, the image has a width and height that depends on the zoom of the browser, but are different from zero. In a headless Chrome, the image has a width and an height equal to zero.

    • A check for the presence of Missing image can be done through:

      var body = document.getElementsByTagName("body")[0];
var image = document.createElement("img");
image.src = "http://iloveponeydotcom32188.jg";
image.setAttribute("id", "fakeimage");
body.appendChild(image);
image.onerror = function(){
    if(image.width == 0 && image.height == 0) {
    console.log("Chrome headless detected");
    }
}

These are some of the crucial factors why headless browsers are more prone to get detected.

Outro

More Related Contents:

  1. What is benefit of using ChromeDriver over WebDriver if we are using only Chrome Browser in our Selenium Automation Script
  2. What Is Selenium And What Is WebDriver?
  3. What is the difference between ChromeDriver and WebDriver in selenium?
  4. NoSuchElementException, Selenium unable to locate element
  5. Can Selenium interact with an existing browser session?
  6. What is the difference between cssSelector & Xpath and which is better with respect to performance for cross browser testing?
  7. How to run Selenium WebDriver test cases in Chrome
  8. How to inspect element for Selenium v3.6 as FireBug is not an option any more for FF 56?
  9. Can Selenium WebDriver open browser windows silently in the background?
  10. Is there a version of Selenium WebDriver that is not detectable?
  11. Many process of Google Chrome (32 bit)
  12. Clarification of the cause of mixing Implicit and Explicit waits in Selenium doc
  13. Reached error page: about:neterror when trying to navigate to other tabs if there is a form submit under that tab
  14. Is there a way to use Selenium WebDriver without informing the document that it is controlled by WebDriver?
  15. Need to find element in selenium by css
  16. org.openqa.selenium.ElementNotInteractableException: Element could not be scrolled into view when trying to click a button
  17. What is the exact purpose of Selenium Grid?
  18. Cannot get automation extension from timeout: Timed out receiving message from renderer
  19. How do you click on a checkbox from a list of checkboxes via Selenium/Webdriver in Java?
  20. What is the difference between a CSS selector and XPath? And which is better with respect to performance for cross-browser testing?
  21. org.openqa.selenium.InvalidCookieDomainException: Document is cookie-averse using Selenium and WebDriver
  22. What is the difference between a CSS and XPath selector? And which is better with respect to performance for cross-browser testing?
  23. I need to find an element in Selenium by CSS
  24. org.openqa.selenium.SessionNotCreatedException: Unable to find a matching set of capabilities while initiating Firefox v37 through Selenium v3.11.0
  25. Selenium how to manage wait for page load?
  26. selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 80
  27. Get PID of Browser launched by selenium
  28. Handle windows authentication pop up on Chrome
  29. How does Selenium WebDriver’s isDisplayed() method work
  30. isElementPresent is very slow in case if element does not exist.

Leave a Comment