Why is json.loads an order of magnitude faster than ast.literal_eval?

by

The two functions are parsing entirely different languages—JSON, and Python literal syntax.* As literal_eval says:

The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.

JSON, by contrast, only handles double-quoted JavaScript string literals (not quite identical to Python’s**), JavaScript numbers (only int and float***), objects (roughly equivalent to dicts), arrays (roughly equivalent to lists), JavaScript booleans (which are different from Python’s), and null.

The fact that these two languages happen to have some overlap doesn’t mean they’re the same language.

Why is json.loads so much faster ?

Because Python literal syntax is a more complex and powerful language than JSON, it’s likely to be slower to parse. And, probably more importantly, because Python literal syntax is not intended to be used as a data interchange format (in fact, it’s specifically not supposed to be used for that), nobody is likely to put much effort into making it fast for data interchange.****

This question seems to imply that ast is more flexible regarding the input data (double or single quotes)

That, and raw string literals, and Unicode vs. bytes string literals, and complex numbers, and sets, and all kinds of other things JSON doesn’t handle.

Are there use cases where I would prefer to use ast.literal_eval over json.loads although it’s slower ?

Yes. When you want to parse Python literals, you should use ast.literal_eval. (Or, better yet, re-think your design so you don’t want to parse Python literals…)

* This is a bit of a vague term. For example, -2 is not a literal in Python, but an operator expression, but literal_eval can handle it. And of course tuple/list/dict/set displays are not literals, but literal_eval can handle them—except that comprehensions are also displays, and literal_eval cannot handle them. Other functions in the ast module can help you find out what really is and isn’t a literal—e.g., ast.dump(ast.parse("expr")).

** For example, "\q" is an error in JSON.

*** Technically, JSON only handles one “number” type, which is floating-point. But Python’s json module parses numbers with no decimal point or exponent as integers, and the same is true in many other languages’ JSON modules.

**** If you missed Tim Peters’s comment on the question: “ast.literal_eval is so lightly used that nobody felt it was worth the time to work (& work, & work) at speeding it. In contrast, the JSON libraries are routinely used to parse gigabytes of data.”

More Related Contents:

  1. Why can’t Python parse this JSON data? [closed]
  2. How to parse data in JSON format?
  3. How to extract multiple JSON objects from one file?
  4. Pandas read nested json
  5. ‘module’ object has no attribute ‘loads’ while parsing JSON using python
  6. Python/Json:Expecting property name enclosed in double quotes
  7. How can I parse (read) and use JSON?
  8. How can I parse (read) and use JSON in Python?
  9. json reading error json.decoder.JSONDecodeError: Invalid \escape
  10. Check if a dictionary contain data true of false, then send a response back
  11. Why am I seeing “TypeError: string indices must be integers”?
  12. Is there a memory efficient and fast way to load big JSON files?
  13. Python: json.loads returns items prefixing with ‘u’
  14. How to convert an XML string to a dictionary?
  15. String to Dictionary in Python [duplicate]
  16. Python – How to convert JSON File to Dataframe
  17. How to compare two JSON objects with the same elements in a different order equal?
  18. How to load all entries in an infinite scroll at once to parse the HTML in python
  19. django-rest-framework 3.0 create or update in nested serializer
  20. Searching text in a PDF using Python?
  21. Python code to remove HTML tags from a string [duplicate]
  22. How do I send a POST request as a JSON?
  23. Python Slice Assignment Memory Usage
  24. Why do I get an error “‘unicode’ object does not support item deletion” when trying to delete values from a JSON object?
  25. How can I split a URL string up into separate parts in Python?
  26. Simple CSV to XML Conversion – Python
  27. Is there a built-in or more Pythonic way to try to parse a string to an integer
  28. Python urlparse — extract domain name without subdomain
  29. Rendering JSON objects using a Django template after an Ajax call
  30. Single versus double quotes in json loads in Python

Leave a Comment