Reliably detecting PhantomJS-based spam bots

by

I very much share your take on CAPTCHA. I’ll list what I have been able to detect so far, for my own detection script, with similar goals. It’s only partial, as they are many more headless browsers.

Fairly safe to use exposed window properties to detect/assume those particular headless browser:

window._phantom (or window.callPhantom) //phantomjs
window.__phantomas //PhantomJS-based web perf metrics + monitoring tool 
window.Buffer //nodejs
window.emit //couchjs
window.spawn  //rhino

The above is gathered from jslint doc and testing with phantom js.

Browser automation drivers (used by BrowserStack or other web capture services for snapshot):

window.webdriver //selenium
window.domAutomation (or window.domAutomationController) //chromium based automation driver

The properties are not always exposed and I am looking into other more robust ways to detect such bots, which I’ll probably release as full blown script when done. But that mainly answers your question.

Here is another fairly sound method to detect JS capable headless browsers more broadly:

if (window.outerWidth === 0 && window.outerHeight === 0){ //headless browser }

This should work well because the properties are 0 by default even if a virtual viewport size is set by headless browsers, and by default it can’t report a size of a browser window that doesn’t exist. In particular, Phantom JS doesn’t support outerWith or outerHeight.

ADDENDUM: There is however a Chrome/Blink bug with outer/innerDimensions. Chromium does not report those dimensions when a page loads in a hidden tab, such as when restored from previous session. Safari doesn’t seem to have that issue..

Update: Turns out iOS Safari 8+ has a bug with outerWidth & outerHeight at 0, and a Sailfish webview can too. So while it’s a signal, it can’t be used alone without being mindful of these bugs. Hence, warning: Please don’t use this raw snippet unless you really know what you are doing.

PS: If you know of other headless browser properties not listed here, please share in comments.

More Related Contents:

  1. downloading a file that comes as an attachment in a POST request response in PhantomJs
  2. Grab the resource contents in CasperJS or PhantomJS
  3. Understanding the evaluate function in CasperJS
  4. CasperJS/PhantomJS doesn’t load https page
  5. Headless Browser and scraping – solutions [closed]
  6. How can I catch and process the data from the XHR responses using casperjs?
  7. CasperJS bind issue
  8. CasperJS passing data back to PHP
  9. Use a node module from casperjs
  10. How effective is the honeypot technique against spam?
  11. Why doesn’t this.evaluate return DOM nodes correctly?
  12. How do I access an iframe from CasperJS?
  13. getting more information from phantomjs “SyntaxError: Parse error” message
  14. PhantomJS fails to open local file
  15. How to login into a website with CasperJS?
  16. How to scroll down with Phantomjs to load dynamic content
  17. Testing AngularJS with Selenium
  18. Headless browser detection
  19. Python: Selenium with PhantomJS empty page source
  20. Save and render a webpage with PhantomJS and node.js
  21. PhantomJS as web job in Azure
  22. How to properly stop phantomjs execution
  23. console.log doesn’t work in CasperJS’ evaluate with setTimeout
  24. How to download a csv file using PhantomJS
  25. How to Prevent SPAM without CAPTCHAs or a Centrally managed system (e.g. akismet)
  26. Why is my e-mail still being picked up as spam? Using mail() function
  27. How to protect html form from spammers?
  28. Selenium Webdriver PhantomJS C# always opens a cmd window
  29. Hunting cheaters in a voting competition
  30. How can I use persisted cookies from a file using phantomjs

Leave a Comment