Properly batch nested promises in Node

by

In this line:

let images = await Promise.all(batches.map(run_batch));

You are trying to run ALL the batches in parallel which is defeating your chunking entirely.

You could use a regular for loop with await instead of the .map() so you runva batch, wait for it to finish, then run the next batch.

let allResults = [];
for (let batch of batches) {
     let images = await run_batch(batch);
     allResults.push(...images);
}
console.log(allResults);

FYI, you might benefit from any number of functions people have written for processing a large array with no more than N requests in flight at the same time. These do not require you to manually break the data into batches. Instead, they monitor how many requests are in-flight at the same time and they start up your desired number of requests and as one finishes, they start another one, collecting the results for you.

runN(fn, limit, cnt, options): Loop through an API on multiple requests

pMap(array, fn, limit): Make several requests to an api that can only handle 20 at a time

rateLimitMap(array, requestsPerSec, maxInFlight, fn): Proper async method for max requests per second

mapConcurrent(array, maxConcurrent, fn): Promise.all() consumes all my ram

There are also features to do this built into the Bluebird promise library and the Async-promises library.

More Related Contents:

  1. Asynchronous exception handling with bluebird promises
  2. Node JS Promise.all and forEach
  3. using async await and .then together
  4. Using mongoose promises with async/await
  5. Javascript Asynchronous Exception Handling with node.js
  6. JavaScript asynchronous programming: promises vs generators
  7. Why does .json() return a promise?
  8. Issue in returning data retrieved from DB queries called in the loop
  9. Difference between microtask and macrotask within an event loop context
  10. Using async/await with a forEach loop
  11. JavaScript, Node.js: is Array.forEach asynchronous?
  12. How to write asynchronous functions for Node.js
  13. Understanding promises in Node.js
  14. Why does the Promise constructor need an executor?
  15. How can I chain these functions together with promises?
  16. node.js async/await using with MySQL
  17. ‘await Unexpected identifier’ on Node.js 7.5
  18. Node exits without error and doesn’t await promise (Event callback)
  19. Understanding async/await on NodeJS
  20. Nodejs express and promises not doing what I expect
  21. How do I handle errors with promises?
  22. Why is `.catch(err => console.error(err))` discouraged?
  23. How to do promise.all for array of array of promises?
  24. What happens if i reject / resolve multiple times in Kriskowal’s q?
  25. Javascript – if with asynchronous case
  26. How to structure nested Promises
  27. What are asynchronous functions in JavaScript? What is “async” and “await” in JavaScript?
  28. How to use Promise.all with an object as input
  29. Node.js: When to use Promises vs Callbacks
  30. Using Async waterfall in node.js

Leave a Comment