Why does headless need to be false for Puppeteer to work?

by

The reason it might work in UI mode but not headless is that sites who aggressively fight scraping will detect that you are running in a headless browser.

Some possible workarounds:

Use puppeteer-extra

Found here: https://github.com/berstend/puppeteer-extra
Check out their docs for how to use it. It has a couple plugins that might help in getting past headless-mode detection:

  1. puppeteer-extra-plugin-anonymize-ua — anonymizes your User Agent. Note that this might help with getting past headless mode detection, but as you’ll see if you visit https://amiunique.org/ it is unlikely to be enough to keep you from being identified as a repeat visitor.
  2. puppeteer-extra-plugin-stealth — this might help win the cat-and-mouse game of not being detected as headless. There are many tricks that are employed to detect headless mode, and as many tricks to evade them.

Run a “real” Chromium instance/UI

It’s possible to run a single browser UI in a manner that let’s you attach puppeteer to that running instance. Here’s an article that explains it: https://medium.com/@jaredpotter1/connecting-puppeteer-to-existing-chrome-window-8a10828149e0

Essentially you’re starting Chrome or Chromium (or Edge?) from the command line with --remote-debugging-port=9222 (or any old port?) plus other command line switches depending on what environment you’re running it in. Then you use puppeteer to connect to that running instance instead of having it do the default behavior of launching a headless Chromium instance: const browser = await puppeteer.connect({ browserURL: ENDPOINT_URL });. Read the puppeteer docs here for more info: https://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api-puppeteerlaunchoptions

The ENDPOINT_URL is displayed in the terminal when you launch the browser from the command line with the --remote-debugging-port=9222 option.

This option is going to require some server/ops mojo, so be prepared to do a lot more Stack Overflow searches. 🙂

There are other strategies I’m sure but those are the two I’m most familiar with. Good luck!

More Related Contents:

  1. How can I pass variable into an evaluate function?
  2. How to run Puppeteer code in any web browser?
  3. How to deal with the captcha when doing Web Scraping in Puppeteer?
  4. Crawling multiple URLs in a loop using Puppeteer
  5. Web-scraping JavaScript page with Python
  6. CasperJS/PhantomJS doesn’t load https page
  7. How to convert raw javascript object to a dictionary?
  8. How do print the console output of the page in puppeter as it would appear in the browser?
  9. Puppeteer – scroll down until you can’t anymore
  10. How to scroll down with Phantomjs to load dynamic content
  11. How do I call a Javascript function from Python?
  12. Puppeteer log inside page.evaluate
  13. How can I handle Javascript in a Perl web crawler?
  14. How to pass a function in Puppeteers .evaluate() method?
  15. How to make Puppeteer work with a ReactJS application on the client-side
  16. Save and render a webpage with PhantomJS and node.js
  17. Puppeteer unable to run on Heroku
  18. Message “Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout”
  19. Scraping data to google sheets from a website that uses JavaScript
  20. puppeteer: wait N seconds before continuing to the next line
  21. Puppeteer doesn’t close browser
  22. How can I capture all network requests and full response data when loading a page in Chrome?
  23. VSCODE: No debug adapter, can not send ‘variables'”
  24. How do you click on an element with text in Puppeteer?
  25. Get elements from page.evaluate in Puppeteer? [duplicate]
  26. How to fix ‘$(…).click is not a function’ in Node/Cheerio
  27. Using Google Apps Script to scrape Dynamic Web Pages
  28. Can I scrape the raw data from highcharts.js?
  29. How can I scrape a page with dynamic content (created by JavaScript) in Python?
  30. Puppeteer wait for all images to load then take screenshot

Leave a Comment