Remove duplicate lines without sorting [duplicate]

by

The UNIX Bash Scripting blog suggests:

awk '!x[$0]++'

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.

More Related Contents:

  1. How to mkdir only if a directory does not already exist?
  2. “ls: not found” after running “read PATH”
  3. Global environment variables in a shell script
  4. Meaning of $? (dollar question mark) in shell scripts
  5. What does $$ mean in the shell?
  6. What is the difference between $@ and $* in shell scripts?
  7. What is the $? (dollar question mark) variable in shell scripting? [duplicate]
  8. How does the #! shebang work?
  9. How to add timestamp to STDERR redirection
  10. Remove part of path on Unix
  11. Write byte at address (hexedit/modify binary from the command line)
  12. How to urlencode data for curl command?
  13. The ‘eval’ command in Bash and its typical uses
  14. How do I know the script file name in a Bash script?
  15. How can I shuffle the lines of a text file on the Unix command line or in a shell script?
  16. Getting the last argument passed to a shell script
  17. Is there a way to ‘uniq’ by column?
  18. Need bash shell script for reading name value pairs from a file
  19. ‘\r’: command not found [duplicate]
  20. How can I check the version of sed in OS X?
  21. Can I ssh somewhere, run some commands, and then leave myself a prompt?
  22. How to kill all processes matching a name?
  23. in mac always getting zsh: command not found: [closed]
  24. Ansible Command module says that ‘|’ is illegal character
  25. Is there a “standard” format for command line/shell help text?
  26. Curl to return http status code along with the response
  27. sh read command eats backslashes in input?
  28. Redirecting/storing output of shell into GDB variable?
  29. Execute Shell Script after post build in Jenkins
  30. How do I determine if a web page exists with shell scripting?

Leave a Comment