Remove line if field is duplicate

by 
awk '{ if (a[$1]++ == 0) print $0; }' "$@"

This is a standard (very simple) use for associative arrays.

More Related Contents:

  1. Remove non-ASCII characters from CSV
  2. Remove multi-line comments
  3. Insert a line at specific line number with sed or awk
  4. delete a column with awk or sed
  5. remove ^M characters from file using sed
  6. How to use awk to print lines where a field matches a specific string?
  7. What is the difference between sed and awk? [closed]
  8. shell – remove numbers from a string column [closed]
  9. sed fails with “unknown option to `s'” error [closed]
  10. How to select lines between two marker patterns which may occur multiple times with awk/sed
  11. How can I do a recursive find/replace of a string with awk or sed?
  12. Bash tool to get nth line from a file
  13. How to delete duplicate lines in a file without sorting it in Unix
  14. How can I delete a newline if it is the last character in a file?
  15. Why doesn’t `\d` work in regular expressions in sed? [duplicate]
  16. Extraction of data from a simple XML file
  17. How to add to the end of lines containing a pattern with sed or awk?
  18. Can I programmatically “burn in” ANSI control codes to a file using unix utils?
  19. Do a tail -F until matching a pattern
  20. What are the differences between Perl, Python, AWK and sed? [closed]
  21. Removing Windows newlines on Linux (sed vs. awk)
  22. Uppercasing First Letter of Words Using SED
  23. Batch script to replace PHP short open tags with
  24. Sed to extract text between two strings
  25. Insert multiple lines into a file after specified pattern using shell script
  26. Extract lines between two patterns from a file [duplicate]
  27. sed gives me “: unexpected EOF (pending }’s) error and I have no idea why
  28. Have sed ignore non-matching lines
  29. Awk replace a column with its hash value
  30. How to delete rows from a csv file based on a list values from another file?

Leave a Comment