How do you filter out all unique lines in a file?

by

Remove duplicated lines:

awk '!a[$0]++' file

This is famous awk one-liner. there are many explanations on inet. Here is one explanation:

This one-liner is very idiomatic. It registers the lines seen in the
associative-array “a” (arrays are always associative in Awk) and at
the same time tests if it had seen the line before. If it had seen the
line before, then a[line] > 0 and !a[line] == 0. Any expression that
evaluates to false is a no-op, and any expression that evals to true
is equal to “{ print }”.

More Related Contents:

  1. What’s the magic of “-” (a dash) in command-line parameters?
  2. Execute command after every command in bash
  3. sh: 1: Syntax error: "(" unexpected error
  4. Iterate over a list of files with spaces
  5. How to declare 2D array in bash
  6. Defining a variable with or without export
  7. What does $@ mean in a shell script?
  8. Better way to rename files based on multiple patterns
  9. Syntax error in shell script with process substitution
  10. Raise error in a Bash script
  11. What is the difference between “#!/usr/bin/env bash” and “#!/usr/bin/bash”?
  12. How to handle more than 10 parameters in shell
  13. How to count lines in a document? [closed]
  14. How to run a shell script on a Unix console or Mac terminal?
  15. How to preserve line breaks when storing command output to a variable?
  16. How to use sed to remove the last n lines of a file
  17. Use sudo without password INSIDE a script
  18. How do I run multiple background commands in bash in a single line?
  19. Round a divided number in Bash
  20. Bash: wait with timeout
  21. Multithreading in Bash [duplicate]
  22. Randomly shuffling lines in Linux / Bash
  23. Difference between shell and environment variables
  24. Replacing Control Character in sed
  25. SUID not working with shell script
  26. Linux shell script to add leading zeros to file names
  27. Colour highlighting output based on regex in shell
  28. bash double bracket issue
  29. Need to remove the count from the output when using “uniq -c” command
  30. How do you run a script on login in *nix?

Leave a Comment