Is there a limit on the size of a string in JSON with Node.js?

by

V8 (the JavaScript engine node is built upon) until very recently had a hard limit on heap size of about 1.9 GB.

Node v0.10 is stuck on an older version of V8 (3.14) due to breaking V8 API changes around native addons. Node 0.12 will update to the newest V8 (3.26), which will break many native modules, but opens the door for the 1.9 GB heap limit to be raised.

So as it stands, a single node process can keep no more than 1.9 GB of JavaScript code, objects, strings, etc combined. That means the maximum length of a string is under 1.9 GB.

You can get around this by using Buffers, which store data outside of the V8 heap (but still in your process’s heap). A 64-bit build of node can pretty much fill all your RAM as long as you never have more than 1.9 GB of data in JavaScript variables.

All that said, you should never come anywhere near this limit. When dealing with this much data, you must deal with it as a stream. You should never have more than a few megabytes (at most) in memory at one time. The good news is node is especially well-suited to dealing with streaming data.

You should ask yourself some questions:

  • What kind of data are you actually receiving from the user?
  • Why do you want to store it in JSON format?
  • Is it really a good idea to stuff gigabytes into JSON? (The answer is no.)
  • What will happen with the data later, after it is stored? Will your code read it? Something else?

The question you’ve posted is actually quite vague in regard to what you’re actually trying to accomplish. For more specific advice, update your question with more information.

If you expect the data to never be all that big, just throw a reasonable limit of 10 MB or something on the input, buffer it all, and use JSON.stringify.

If you expect to deal with data any larger, you need to stream the input straight to disk. Look in to transform streams if you need to process/modify the data before it goes to disk. For example, there are modules that deal with streaming JSON.

More Related Contents:

  1. how to get POJO object in javascript from a JSON?
  2. How can I print a circular structure in a JSON-like format?
  3. “SyntaxError: Unexpected token < in JSON at position 0"
  4. Is it not possible to stringify an Error using JSON.stringify?
  5. How to parse JSON using Node.js? [closed]
  6. Using Node.JS, how do I read a JSON file into (server) memory?
  7. Unexpected token colon JSON after jQuery.ajax#get
  8. Parse large JSON file in Nodejs
  9. Write / add data in JSON file using Node.js
  10. Calling a JSON API with Node.js
  11. How to convert CSV to JSON in Node.js
  12. Parse XLSX with Node and create json
  13. Encrypt string in PHP and decrypt in Node.js
  14. Re-associating an object with its class after deserialization in Node.js
  15. node.js TypeError: path must be absolute or specify root to res.sendFile [failed to parse JSON]
  16. Combine json arrays by key, javascript
  17. Why is JSON.stringify not serializing prototype values?
  18. Typescript compiler error when importing json file
  19. TypeError: Converting circular structure to JSON in nodejs
  20. Unable to send numbers using res.send() using express with node
  21. Using JSON.stringify on custom class
  22. Cannot create a string longer than 0x1fffffe8 characters in JSON.parse?
  23. What is the best way to limit concurrency when using ES6’s Promise.all()?
  24. getting how many people are in a chat room in socket.io [duplicate]
  25. How to convert HTML to JSON using PHP?
  26. How to unit test a Node.js module that requires other modules and how to mock the global require function?
  27. Where argument of first next() call goes? [duplicate]
  28. Recursive tree search in a nested object structure in JavaScript
  29. JavaScript EventSource SSE not firing in browser
  30. Node JS fs module inside browser

Leave a Comment