How can I split a text file using PowerShell?

by

A word of warning about some of the existing answers – they will run very slow for very big files. For a 1.6 GB log file I gave up after a couple of hours, realising it would not finish before I returned to work the next day.

Two issues: the call to Add-Content opens, seeks and then closes the current destination file for every line in the source file. Reading a little of the source file each time and looking for the new lines will also slows things down, but my guess is that Add-Content is the main culprit.

The following variant produces slightly less pleasant output: it will split files in the middle of lines, but it splits my 1.6 GB log in less than a minute:

$from = "C:\temp\large_log.txt"
$rootName = "C:\temp\large_log_chunk"
$ext = "txt"
$upperBound = 100MB


$fromFile = [io.file]::OpenRead($from)
$buff = new-object byte[] $upperBound
$count = $idx = 0
try {
    do {
        "Reading $upperBound"
        $count = $fromFile.Read($buff, 0, $buff.Length)
        if ($count -gt 0) {
            $to = "{0}.{1}.{2}" -f ($rootName, $idx, $ext)
            $toFile = [io.file]::OpenWrite($to)
            try {
                "Writing $count to $to"
                $tofile.Write($buff, 0, $count)
            } finally {
                $tofile.Close()
            }
        }
        $idx ++
    } while ($count -gt 0)
}
finally {
    $fromFile.Close()
}

More Related Contents:

  1. this code my teacher gave it to us as homework and it does not work with me idk why
  2. How can you use an object’s property in a double-quoted string?
  3. Powershell: passing json string to curl
  4. How can I replace every occurrence of a String in a file with PowerShell?
  5. Function return value in PowerShell
  6. How do I pass variables with the Invoke-Command cmdlet?
  7. Loop through files in a directory using PowerShell
  8. how do I loop through a line from a csv file in powershell
  9. Delete files older than 15 days using PowerShell
  10. Output (“echo”) a variable to a text file
  11. Powershell equivalent of bash ampersand (&) for forking/running background processes
  12. Why does ‘continue’ behave like ‘break’ in a Foreach-Object?
  13. INI file parsing in PowerShell
  14. Parameter interpretation when running jobs
  15. Powershell: Uninstall application by UpgradeCode
  16. Why don’t .NET objects in PowerShell use the current directory?
  17. How to properly -filter multiple strings in a PowerShell copy script
  18. Methods to hex edit binary files via Powershell
  19. How remove accents in PowerShell?
  20. Unable to convert a string to an integer variable from DateTaken attribute on a JPG file
  21. How to Grant permission to user on Certificate private key using powershell?
  22. Using Invoke-Command -ScriptBlock on a function with arguments
  23. How to use PowerShell copy-item and keep structure
  24. Powershell – Reboot and Continue Script
  25. How can I execute an external program with parameters in PowerShell?
  26. How to download the SSL certificate from a website using PowerShell?
  27. Rename file on FTP with PowerShell
  28. Does PowerShell support HashTable Serialization?
  29. In Powershell what is the idiomatic way of converting a string to an int?
  30. Using PowerShell to remove lines from a text file if it contains a string

Leave a Comment