Using regexes, how to efficiently match strings between double quotes with embedded double quotes?

by

A very efficient solution to match such inputs is to use the normal* (special normal*)* pattern; this name is quoted from the excellent book by Jeffrey Friedl, Mastering Regular Expressions.

It is a pattern useful in general to match inputs consisting of regular entries (the normal part) with separators inbetween (the special part).

Note that like all things regex, it should be used when there is no better choice; while one could use this pattern for parsing CSV data, for instance, if you use Java, you’re better off using OpenCSV instead.

Also note that while the quantifiers in the pattern name are stars (ie, zero or more), you can vary them to suit your needs.

Strings with embedded double quotes

Let us take the above example again; and please consider that this text sample may be anywhere in your input:

"He said \"Hello\" to me for the first time"

No matter how hard you try, no amount of “dot plus greedy/lazy quantifiers” magic will help you solve it. Instead, categorize the input between quotes as normal and special:

  • normal is anything but a backslash or a double quote: [^\\"];
  • special is the sequence of a backslash followed by a double quote: \\".

Substituting this into the normal* (special normal*)* pattern, this gives the following regex:

[^\\"]*(\\"[^\\"]*)*

Adding the double quotes around to match the full text gives the final regex:

"[^\\"]*(\\"[^\\"]*)*"

You will note that this will also match empty quoted strings.

Words with dash separators

Here we will have to use a variant on the quantifiers, since:

  • we don’t want empty words,
  • we don’t want words starting with a dash,
  • when a dash appears, it must have at least one letter before another dash, if any.

For simplicity, we will also suppose that only lowercase, ASCII letters are allowed.

Sample input:

the-word-to-match

Let us decompose again into normal and special:

  • normal: a lowercase, ASCII letter: [a-z];
  • special: the dash: -

The canonical form of the pattern would be:

[a-z]*(-[a-z]*)*

But as we said:

  • we don’t want words starting with a dash: the first * should become +;
  • when a dash is found, there should be at least one letter after it: the second * should become +.

We end up with:

[a-z]+(-[a-z]+)*

Adding word anchors around it to obtain the final result:

\b[a-z]+(-[a-z]+)*\b

Other operator variations

The examples above limit themselves to replacing * with +, but of course you can have as many variations as you wish. One ultra classical example would be an IP address:

  • normal is up to three digits (\d{1,3}),
  • special is the dot: (\.),
  • the first normal appears only once, therefore no quantifier,
  • the normal inside the (special normal*) also appears only once, therefore no quantifier,
  • finally the (special normal*) part appears exactly three times, therefore {3}.

Which gives the expresison (decorated with word anchors):

\b\d{1,3}(\.\d{1,3}){3}\b

Conclusion

This pattern’s flexibility makes it one of the most useful tools in your regex toolbox. While many problems exist which you should not use regexes for if libraries exist, in some situations, you have to use regexes. And this will become one of your best friends once you have practiced with it a bit!

Tips

  • It is more than likely that you don’t need (or want) to capture the repeated part (the (special normal*) part); it is therefore recommended that you use a non-capturing group. For instance, use "[^\\"]*(?:\\"[^\\"]*)*" for quoted strings. In fact, had you wanted it, capturing would almost never lead to the desired results in this case, because repeating a capturing group will only ever give you the last capture (all previous repetitions will be overwritten), unless you are using this pattern in .NET. (thanks @ohaal)

More Related Contents:

  1. string pattern matching using perl regex [closed]
  2. Can regular expressions be used to match nested patterns? [duplicate]
  3. How can we match a^n b^n?
  4. Decimal or numeric values in regular expression validation
  5. Carets in Regular Expressions
  6. Regex to split a CSV
  7. How can I match nested brackets using regex?
  8. How to match Cyrillic characters with a regular expression
  9. How do you debug a regex? [closed]
  10. Negative matching using grep (match lines that do not contain foo)
  11. (grep) Regex to match non-ASCII characters?
  12. how to remove folder name from url using htaccess
  13. Regex match one of two words
  14. Create new column in dataframe based on partial string matching other column
  15. How to exclude a specific string constant? [duplicate]
  16. What regex can match sequences of the same character?
  17. What do comma separated numbers in curly braces at the end of a regex mean?
  18. Remove all text between two brackets
  19. Regex look behind in VS Code?
  20. MongoDB reverse regex
  21. Simple AlphaNumeric Regex (single spacing) without Catastrophic Backtracking
  22. Select rows from data.frame ending with a specific character string in R
  23. Regex to match any character including new lines
  24. apache mod_rewrite one rule for any number of possibilities
  25. Splitting String in VBA using RegEx
  26. Difference between regular expression modifiers (or flags) ‘m’ and ‘s’?
  27. Splitting a string on the first space
  28. Regular Expression Word Boundary and Special Characters
  29. ElasticSearch and Regex queries
  30. How to validate email id in angularJs using ng-pattern

Leave a Comment