Remove duplicates from text file based on second text file

by

There are two standard ways to do this:

With grep:

grep -vxFf removethese main

This uses:

  • -v to invert the match.
  • -x match whole line, to prevent, for example, he to match lines like hello or highway to hell.
  • -F to use fixed strings, so that the parameter is taken as it is, not interpreted as a regular expression.
  • -f to get the patterns from another file. In this case, from removethese.

With awk:

$ awk 'FNR==NR {a[$0];next} !($0 in a)' removethese main
1
5

Like this we store every line in removethese in an array a[]. Then, we read the main file and just print those lines that are not present in the array.

More Related Contents:

  1. How do you extract IP addresses from files using a regex in a linux shell?
  2. Awk – replace coumn 2 in table 1 from coumn 2 in table 2 based on matching data in column 1 (common between tables)
  3. bash regex with quotes?
  4. Find and kill a process in one line using bash and regex
  5. RE error: illegal byte sequence on Mac OS X
  6. Bash Regular Expression — Can’t seem to match any of \s \S \d \D \w \W etc
  7. How to extract text from a string using sed?
  8. Check if a string matches a regex in Bash script
  9. How to match digits in regex in bash script
  10. Regular expressions in a Bash case statement
  11. Regex matching in a Bash if statement
  12. Bash, grep between two lines with specified string
  13. Fully qualified domain name validation
  14. Why is sed not recognizing \t as a tab?
  15. What does ‘\K’ mean in this regex?
  16. Unix grep regex containing ‘x’ but not containing ‘y’
  17. Insert linefeed in sed (Mac OS X)
  18. Does bash support word boundary regular expressions?
  19. how to use a regular expression to extract json fields?
  20. How to match a pattern given in a variable in awk?
  21. Replace a word with multiple lines using sed?
  22. replace a unknown string between two known strings with sed
  23. How can I match a string with a regex in Bash?
  24. Why does BASH_REMATCH not work for a quoted regular expression?
  25. Rename files using a regex with bash [duplicate]
  26. How can I match spaces with a regexp in Bash?
  27. How to change date format in sed?
  28. How to execute a Bash command only if a Docker container with a given name does not exist?
  29. Grep and regex – why am I escaping curly braces?
  30. How to extract domain name from url?

Leave a Comment