Best way to execute parallel processing in Node.js

by

Should be doable. As a simple example:

// parent.js
var child_process = require('child_process');

var numchild  = require('os').cpus().length;
var done      = 0;

for (var i = 0; i < numchild; i++){
  var child = child_process.fork('./child');
  child.send((i + 1) * 1000);
  child.on('message', function(message) {
    console.log('[parent] received message from child:', message);
    done++;
    if (done === numchild) {
      console.log('[parent] received all results');
      ...
    }
  });
}

// child.js
process.on('message', function(message) {
  console.log('[child] received message from server:', message);
  setTimeout(function() {
    process.send({
      child   : process.pid,
      result  : message + 1
    });
    process.disconnect();
  }, (0.5 + Math.random()) * 5000);
});

So the parent process spawns an X number of child processes and passes them a message. It also installs an event handler to listen for any messages sent back from the child (with the result, for instance).

The child process waits for messages from the parent, and starts processing (in this case, it just starts a timer with a random timeout to simulate some work being done). Once it’s done, it sends the result back to the parent process and uses process.disconnect() to disconnect itself from the parent (basically stopping the child process).

The parent process keeps track of the number of child processes started, and the number of them that have sent back a result. When those numbers are equal, the parent received all results from the child processes so it can combine all results and return the JSON result.

More Related Contents:

  1. Find the version of an installed npm package
  2. E11000 duplicate key error index in mongodb mongoose
  3. NPM Install Error:Unexpected end of JSON input while parsing near ‘…nt-webpack-plugin”:”0’
  4. How to share sessions with Socket.IO 1.x and Express 4.x?
  5. Mongoose: findOneAndUpdate doesn’t return updated document
  6. What does body-parser do with express?
  7. NodeJS – What does “socket hang up” actually mean?
  8. Why does mongoose always add an s to the end of my collection name
  9. What is the difference between res.end() and res.send()?
  10. Angular2 QuickStart npm start is not working correctly
  11. How can I see the SQL generated by Sequelize.js?
  12. Redirecting to previous page after authentication in node.js using passport.js
  13. Test process.env with Jest
  14. How can I get the path of a module I have loaded via require that is *not* mine (i.e. in some node_module)
  15. async await with setInterval
  16. How to send email from my react web application
  17. projection not working with find query
  18. How can I get a list of callbacks in the Node work queue? (or, Why won’t Node exit?)
  19. How can I import an ES module in the Node.js REPL in Node 8?
  20. Send mail via Google Apps Gmail using service account domain wide delegation in nodejs
  21. Browserify – How to call function bundled in a file generated through browserify in browser
  22. PostgreSQL multi-row updates in Node.js
  23. Stream files in node/express to client
  24. Sinon stub function used with destructuring
  25. Appropriate hashbang for Node.js scripts
  26. What is the proper way to check for existence of variable in an EJS template (using ExpressJS)?
  27. ESLint error while trying to deploy firebase functions
  28. Getting 404 when attempting to publish new package to NPM
  29. Express.js Routing error: Can’t set headers after they are sent
  30. body-parser – extended option (qs vs querystring)

Leave a Comment