How to delete duplicate lines in a file without sorting it in Unix

by 
awk '!seen[$0]++' file.txt

seen is an associative array that AWK will pass every line of the file to. If a line isn’t in the array then seen[$0] will evaluate to false. The ! is the logical NOT operator and will invert the false to true. AWK will print the lines where the expression evaluates to true.

The ++ increments seen so that seen[$0] == 1 after the first time a line is found and then seen[$0] == 2, and so on.
AWK evaluates everything but 0 and "" (empty string) to true. If a duplicate line is placed in seen then !seen[$0] will evaluate to false and the line will not be written to the output.

More Related Contents:

  1. Enhancement of unix script [closed]
  2. Replace a string in shell script using a variable
  3. How to select lines between two marker patterns which may occur multiple times with awk/sed
  4. Bash tool to get nth line from a file
  5. Why does my tool output overwrite itself and how do I fix it?
  6. Is there a Unix utility to prepend timestamps to stdin?
  7. sed or awk: delete n lines following a pattern
  8. Print lines in one file matching patterns in another file
  9. Date arithmetic in Unix shell scripts
  10. Remove part of path on Unix
  11. Extract version number from a string
  12. shell – remove numbers from a string column [closed]
  13. How to use multiple arguments for awk with a shebang (i.e. #!)?
  14. How can I delete a newline if it is the last character in a file?
  15. Use grep –exclude/–include syntax to not grep through certain files
  16. Delete specific line number(s) from a text file using sed?
  17. Alternative to `sed -i` on Solaris
  18. Copy folder recursively, excluding some folders
  19. How to do a mass rename?
  20. Explain awk command
  21. GROUP BY/SUM from shell
  22. Get last field using awk substr
  23. Uppercasing First Letter of Words Using SED
  24. unix command line execute with . (dot) vs. without
  25. Sed to extract text between two strings
  26. Insert multiple lines into a file after specified pattern using shell script
  27. Trim leading and trailing spaces from a string in awk
  28. Replace whitespace with a comma in a text file in Linux
  29. Awk replace a column with its hash value
  30. Shell script “for” loop syntax

Leave a Comment