Parse a csv using awk and ignoring commas inside a field

by 
gawk -vFPAT='[^,]*|"[^"]*"' '{print $1 "," $3}' | sort | uniq

This is an awesome GNU Awk 4 extension, where you define a field pattern instead of a field-separator pattern. Does wonders for CSV. (docs)

ETA (thanks mitchus): To remove the surrounding quotes, gsub("^\"|\"$","",$3); if there’s more fields than just $3 to process that way, just loop through them.
Note this simple approach is not tolerant of malformed input, nor of some possible special characters between quotes – covering all of those would go beyond the scope of a neat one-liner.

More Related Contents:

  1. What’s the most robust way to efficiently parse CSV using awk?
  2. Csv empty when trying to read it
  3. Dealing with commas in a CSV file
  4. Properly escape a double quote in CSV
  5. Is there a way to include commas in CSV columns without breaking the formatting?
  6. How to load jar dependenices in IPython Notebook
  7. How to properly escape a double quote in CSV?
  8. Can you encode CR/LF in into CSV files?
  9. Encoding Error in Panda read_csv [duplicate]
  10. Spark dataframe save in single file on hdfs location [duplicate]
  11. How to export a Hive table into a CSV file?
  12. Extract specific columns from delimited file using Awk
  13. Parsing a CSV file using gawk
  14. Reading csv files with quoted fields containing embedded commas
  15. Split CSV files into smaller files but keeping the headers?
  16. Bash: Parse CSV with quotes, commas and newlines
  17. VBScript to loop through all files in a folder
  18. How to check encoding of a CSV file
  19. How do I use awk under cygwin to print fields from an excel spreadsheet?
  20. Sort CSV file by multiple columns using the “sort” command
  21. Export as csv in beeline hive
  22. UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x8d in position 7240: character maps to
  23. CodedUI test does not read data from CSV input file
  24. Can a CSV file have a comment?
  25. CSV with comma or semicolon?
  26. Replace whitespace with a comma in a text file in Linux
  27. using bash (sed/awk) to extract rows AND columns in CSV files?
  28. Write Spark dataframe as CSV with partitions
  29. How to delete rows from a csv file based on a list values from another file?
  30. Import CSV into google cloud datastore

Leave a Comment