Batch to remove duplicate rows from text file

by

Sure can, but like most text file processing with batch, it is not pretty, and it is not particularly fast.

This solution ignores case when looking for duplicates, and it sorts the lines. The name of the file is passed in as the 1st and only argument to the batch script.

@echo off
setlocal disableDelayedExpansion
set "file=%~1"
set "sorted=%file%.sorted"
set "deduped=%file%.deduped"
::Define a variable containing a linefeed character
set LF=^


::The 2 blank lines above are critical, do not remove
sort "%file%" >"%sorted%"
>"%deduped%" (
  set "prev="
  for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%sorted%") do (
    set "ln=%%A"
    setlocal enableDelayedExpansion
    if /i "!ln!" neq "!prev!" (
      endlocal
      (echo %%A)
      set "prev=%%A"
    ) else endlocal
  )
)
>nul move /y "%deduped%" "%file%"
del "%sorted%"

This solution is case sensitive and it leaves the lines in the original order (except for duplicates of course). Again the name of the file is passed in as the 1st and only argument.

@echo off
setlocal disableDelayedExpansion
set "file=%~1"
set "line=%file%.line"
set "deduped=%file%.deduped"
::Define a variable containing a linefeed character
set LF=^


::The 2 blank lines above are critical, do not remove
>"%deduped%" (
  for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%file%") do (
    set "ln=%%A"
    setlocal enableDelayedExpansion
    >"%line%" (echo !ln:\=\\!)
    >nul findstr /xlg:"%line%" "%deduped%" || (echo !ln!)
    endlocal
  )
)
>nul move /y "%deduped%" "%file%"
2>nul del "%line%"

EDIT

Both solutions above strip blank lines. I didn’t think blank lines were worth preserving when talking about distinct values.

I’ve modified both solutions to disable the FOR /F “EOL” option so that all non-blank lines are preserved, regardless what the 1st character is. The modified code sets the EOL option to a linefeed character.

New solution 2016-04-13: JSORT.BAT

You can use my JSORT.BAT hybrid JScript/batch utility to efficiently sort and remove duplicate lines with a simple one liner (plus a MOVE to overwrite the original file with the final result). JSORT is pure script that runs natively on any Windows machine from XP onward.

@jsort file.txt /u >file.txt.new
@move /y file.txt.new file.txt >nul

More Related Contents:

  1. Variables are not behaving as expected
  2. Single line with multiple commands using Windows batch file
  3. Batch file to delete files older than N days
  4. How to have multiple colors in a Windows batch file?
  5. How do I measure execution time of a command on the Windows command line?
  6. Windows Batch Variables Won’t Set
  7. ‘Pretty print’ windows %PATH% variable – how to split on ‘;’ in CMD shell
  8. Where is “START” searching for executables?
  9. How to verify if a file exists in a batch file?
  10. windows batch SET inside IF not working
  11. Stop and Start a service via batch or cmd file?
  12. Batch files: List all files in a directory with relative paths
  13. How to run a PowerShell script within a Windows batch file
  14. Windows batch script to parse CSV file and output a text file
  15. Using parameters in batch files at Windows command line
  16. Convert a small PS script into a long line in a .BATch file
  17. What is the easiest way to reset ERRORLEVEL to zero?
  18. How to count no of lines in text file and store the value into a variable using batch script?
  19. How to test if a path is a file or directory in Windows batch file?
  20. Rename multiple files in cmd
  21. Split string with string as delimiter
  22. How to change Screen buffer size in Windows Command Prompt from batch script
  23. Delete all files and folders in a directory
  24. windows cmd: problems with for /f with a quoted command with quoted parameters
  25. Remove trailing spaces from a file using Windows batch?
  26. Batch goto loses errorlevel
  27. I need to match or replace an asterisk * in a batch environmental variable using only native Windows commands. Is this possible?
  28. Escape percent signs in given variables
  29. Out of a git console: how do I execute a batch file and then return to git console?
  30. UTF-16 to UTF-8 conversion (for scripting in Windows)

Leave a Comment