Crawling multiple URLs in a loop using Puppeteer

by

map, forEach, reduce, etc, does not wait for the asynchronous operation within them, before they proceed to the next element of the iterator they are iterating over.

There are multiple ways of going through each item of an iterator synchronously while performing an asynchronous operation, but the easiest in this case I think would be to simply use a normal for operator, which does wait for the operation to finish.

const urls = [...]

for (let i = 0; i < urls.length; i++) {
    const url = urls[i];
    await page.goto(`${url}`);
    await page.waitForNavigation({ waitUntil: 'networkidle2' });
}

This would visit one url after another, as you are expecting. If you are curious about iterating serially using await/async, you can have a peek at this answer: https://stackoverflow.com/a/24586168/791691

More Related Contents:

  1. How can I pass variable into an evaluate function?
  2. Puppeteer wait until page is completely loaded
  3. Puppeteer page.evaluate querySelectorAll return empty objects
  4. How to run Puppeteer code in any web browser?
  5. How to deal with the captcha when doing Web Scraping in Puppeteer?
  6. Why does headless need to be false for Puppeteer to work?
  7. Web-scraping JavaScript page with Python
  8. How do print the console output of the page in puppeter as it would appear in the browser?
  9. Puppeteer – scroll down until you can’t anymore
  10. Android – Parse JS generated urls with JSOUP
  11. How to scroll down with Phantomjs to load dynamic content
  12. How do I call a Javascript function from Python?
  13. How can I handle Javascript in a Perl web crawler?
  14. puppeteer: how to wait until an element is visible?
  15. How to pass a function in Puppeteers .evaluate() method?
  16. Save and render a webpage with PhantomJS and node.js
  17. Phantomjs page.content isn’t retrieving the page content
  18. Scrape web pages in real time with Node.js
  19. Scraping data to google sheets from a website that uses JavaScript
  20. puppeteer: wait N seconds before continuing to the next line
  21. Puppeteer doesn’t close browser
  22. VSCODE: No debug adapter, can not send ‘variables'”
  23. Get elements from page.evaluate in Puppeteer? [duplicate]
  24. To exceed the ImportXML limit on Google Spreadsheet
  25. Can the browser turned headless mid-execution when it was started normally, or vice-versa?
  26. Can I use puppeteer inside chrome extension
  27. Why I got document is not defined error in puppeteer?
  28. puppeteer wait for page/DOM updates – respond to new items that are added after initial loading
  29. How can I scrape a page with dynamic content (created by JavaScript) in Python?
  30. Puppeteer wait for all images to load then take screenshot

Leave a Comment