How do I find and remove duplicate lines from a file using Regular Expressions? [closed]

by

Regular-expressions.info has a page on Deleting Duplicate Lines From a File

This basically boils down to searching for this oneliner:

^(.*)(\r?\n\1)+$

… And replacing with \1.
Note: Dot must not match Newline

Explanation:

The caret will match only at the start of a line. So the regex engine will only attempt to match the remainder of the regex there. The dot and star combination simply matches an entire line, whatever its contents, if any. The parentheses store the matched line into the first backreference.

Next we will match the line separator. I put the question mark into \r?\n to make this regex work with both Windows (\r\n) and UNIX (\n) text files. So up to this point we matched a line and the following line break.

Now we need to check if this combination is followed by a duplicate of that same line. We do this simply with \1. This is the first backreference which holds the line we matched. The backreference will match that very same text.

If the backreference fails to match, the regex match and the backreference are discarded, and the regex engine tries again at the start of the next line. If the backreference succeeds, the plus symbol in the regular expression will try to match additional copies of the line. Finally, the dollar symbol forces the regex engine to check if the text matched by the backreference is a complete line. We already know the text matched by the backreference is preceded by a line break (matched by \r?\n). Therefore, we now check if it is also followed by a line break or if it is at the end of the file using the dollar sign.

The entire match becomes line\nline (or line\nline\nline etc.). Because we are doing a search and replace, the line, its duplicates, and the line breaks in between them, are all deleted from the file. Since we want to keep the original line, but not the duplicates, we use \1 as the replacement text to put the original line back in.

More Related Contents:

  1. Awk – replace coumn 2 in table 1 from coumn 2 in table 2 based on matching data in column 1 (common between tables)
  2. Reference – What does this regex mean?
  3. How do I match any character across multiple lines in a regular expression?
  4. How do you match only valid roman numerals with a regular expression?
  5. Validating IPv4 addresses with regexp
  6. Regular expression pipe confusion
  7. How to use conditionals when replacing in Notepad++ via regex
  8. Exclude characters from a character class
  9. Find CRLF in Notepad++
  10. Jmeter extracting fields/parsing JSON response
  11. How to back reference “inner” selections ( () ) in a regular expression?
  12. Multi-line regular expressions in Visual Studio Code
  13. Verbs that act after backtracking and failure
  14. mod_rewrite: replace underscores with dashes
  15. Multiline regex to match config block
  16. How is the AND/OR operator represented as in Regular Expressions?
  17. Remove ‘index.php’ from URL with .htaccess
  18. Regex Until But Not Including
  19. Math operations in regex
  20. Multiple words in any order using regex [duplicate]
  21. Split text by columns in PowerShell
  22. How to replace a whole line with sed?
  23. regular expression capture nth match
  24. Replace newlines with literal \n
  25. Regex that matches integers in between whitespace or start/end of string only
  26. Regex to match an ISO 8601 datetime string
  27. How to capture multiple regex matches, from a single line, into the $matches magic variable in Powershell?
  28. Regular Expression usage with ls
  29. Difference between regex quantifiers plus and star
  30. Finding and removing Non-ASCII characters from an Oracle Varchar2

Leave a Comment