How to replace paired square brackets with other syntax with sed?

by

It took a little doing, but here:

sed -i.bkup  's/\[\([^]]*\)\]/\\macro{\1}/g' test.txt

Let’s see if I can explain this regular expression:

  1. The \[ is matching a square bracket. Since [ is a valid magic regular expression character, the backslash means to match the literal character.
  2. The \(...\) is a capture group. It captures the part of the regular expression I want. I can have many capture groups, and in sed I can reference them as \1, \2, etc.
  3. Inside the capture group \(...\). I have [^]]*.
    1. The [^...] syntax means any character but.
    2. The [^]] means any character but a closing brace.
    3. The * means zero or more of the preceding. That means I am capturing zero or more characters that are not closing square braces.
  4. The \] means the closing square bracket

Let’s look at the line this is [some] more [text]

  • In #1 above, I capture the first open square bracket in front of the word some. However, it’s not in a capture group. This is the first character I’m going to substitute.
  • I now start a capture group. I am capturing according to 3.2 and 3.3 above, starting with the letter s in some as many characters as possible that are not closing square brackets. This means I am matching [some, but only capturing some.
  • In #4, I have ended my capture group. I’ve matched for substitution purposes [some and now I’m matching on the last closing square bracket. That means I’m matching [some]. Note that regular expressions are normally greedy. I’ll explain below why this is important.
  • Now, I can match the replacement string. This is much easier. It’s \\macro(\1). The \1 is replaced by my capture group. The \\ is just a backslash. Thus, I’ll replace [some] with \macro{some}.

It would be much easier if I could be guaranteed a single set of square brackets in each line. Then I could have done this:

sed -i.bkup 's/\[\(.*\)\]/\\macro(\1)/g'

The capture group is now saying anything between to square brackets. However, the problem is that regular expressions are greedy, that means I would have matched from the s in some all the way to the final t in text. The ‘x’ below show the capture group. The [ and ] show the square brackets I’m matching on:

 this is [some] more [text]
         [xxxxxxxxxxxxxxxx]

This became more complex because I had to match on characters that had special meaning to regular expressions, so we see a lot of backslashing. Plus, I had to account for regular expression greediness, which got the nice looking, non-matching string [^]]* to match anything not a closing bracket. Add in the square brackets before and after \[[^]]*\], and don’t forget the \(...\) capture group: \[\([^]]*\)\]And you get one big mess of a regular expression.

More Related Contents:

  1. Some troubles with using sed and awk [closed]
  2. How can I manipulate text in bash? [closed]
  3. sed substitution with Bash variables
  4. How can I do a recursive find/replace of a string with awk or sed?
  5. How to pass a variable containing slashes to sed
  6. sed search and replace strings containing / [duplicate]
  7. Error when using ‘sed’ with ‘find’ command on OS X: “invalid command code .”
  8. How can I remove the first line of a text file using bash/sed script?
  9. Remove first N lines of a file in place in unix command line
  10. How to address error “bash: !d’: event not found” in Bash command substitution [duplicate]
  11. How to add to the end of lines containing a pattern with sed or awk?
  12. Delete all lines beginning with a # from a file
  13. Why does sed not replace all occurrences?
  14. Insert newline (\n) using sed
  15. How do I escape slashes and double and single quotes in sed?
  16. Error with sed unterminated s command in bash
  17. How to remove double-quotes in jq output for parsing json files in bash?
  18. Use sed to replace all backslashes with forward slashes
  19. sed replace with variable with multiple lines [duplicate]
  20. How to find/replace and increment a matched number with sed/awk?
  21. Convert line endings [duplicate]
  22. Expand variables in sed
  23. sed multiline delete with pattern
  24. Using sed to insert file content
  25. How to match a single quote in sed
  26. Using SED with wildcard
  27. How to insert a new line character after a fixed number of characters in a file
  28. escaping newlines in sed replacement string
  29. How to append a string at end of a specific line in a file in bash [duplicate]
  30. How to delete rows from a csv file based on a list values from another file?

Leave a Comment