Here is what worked for me. First, I clear all my buildpacks and then I added the puppeteer-heroku-buildpack and the heroku/nodejs one: $ heroku buildpacks:clear $ heroku buildpacks:add –index 1 https://github.com/jontewks/puppeteer-heroku-buildpack $ heroku buildpacks:add –index 1 heroku/nodejs Then, add the following args to the puppeteer launch function: const browser = await puppeteer.launch({ ‘args’ : [ … Read more
There is a headless browser detection test which tests for the following: Does the User-Agent contain the string “HeadlessChrome”? Is navigator.webdriver set? Is window.chrome unset? Does the browser skip asking for permissions (like notifications)? Are browser plugins unavailable? Is navigator.languages unset? If your browser answers any of these questions with yes, then you fail the … Read more
If your code looks like this const button = await page.$(selector); await button.click(); Try changing await button.click(); to await button.evaluate(b => b.click()); The difference is that button.click() clicks using Puppeteer’s ElementHandle.click() which scrolls the page until the element is in view gets the bounding box of the element (this step is where the error happens) … Read more
Your React.js application runs on the client-side (in the browser). Puppeteer cannot run inside that environment as you cannot start a full browser inside the browser. What you need is a server which does that for you. You could ether offer a HTTP endpoint (option 1) or expose your puppeteer Websocket (option 2): Option 1: … Read more
You cannot pass a function directly into page.evaluate(), but you can call another special method (page.exposeFunction), which expose your function as a global function (also available in as an attribute of your page window object), so you can call it when you are inside page.evaluate(): var myFunc = function() { console.log(“lol”); }; await page.exposeFunction(“myFunc”, myFunc); … Read more
Puppeteer v0.13.0 has page.select() method, which does exactly that. You just have to give it the value to select. So, assuming you have an <option value=”my-value”> in your <select>: await page.select(‘#telCountryInput’, ‘my-value’)
I think you can use page.waitForSelector(selector[, options]) function for that purpose. const puppeteer = require(‘puppeteer’); puppeteer.launch().then(async browser => { const page = await browser.newPage(); page .waitForSelector(‘#myId’) .then(() => console.log(‘got it’)); browser.close(); }); To check the options avaible, please see the github link.
This is a reCAPTCHA (version 2, check out demos here), which is shown to you as the owner of the page does not want you to automatically crawl the page. Your options are the following: Option 1: Stop crawling or try to use an official API As the owner of the page does not want … Read more
EDIT: Since puppeteer removed support for puppeteer-web, I moved it out of the repo and tried to patch it a bit. It does work with browser. The package is called puppeteer-web, specifically made for such cases. But the main point is, there must be some instance of chrome running on some server. Only then you … Read more
Update for puppeteer 12, adapted from the current documentation: page.on(‘console’, async (msg) => { const msgArgs = msg.args(); for (let i = 0; i < msgArgs.length; ++i) { console.log(await msgArgs[i].jsonValue()); } }); await page.evaluate(() => console.log(‘hello’, 5)); await page.evaluate(() => console.log({ foo: ‘bar’ })); await page.evaluate(() => console.log([1, 2, 3, 4, 5])); Shows the following … Read more