How to process a file in PowerShell line-by-line as a stream

by

If you are really about to work on multi-gigabyte text files then do not use PowerShell. Even if you find a way to read it faster processing of huge amount of lines will be slow in PowerShell anyway and you cannot avoid this. Even simple loops are expensive, say for 10 million iterations (quite real in your case) we have:

# "empty" loop: takes 10 seconds
measure-command { for($i=0; $i -lt 10000000; ++$i) {} }

# "simple" job, just output: takes 20 seconds
measure-command { for($i=0; $i -lt 10000000; ++$i) { $i } }

# "more real job": 107 seconds
measure-command { for($i=0; $i -lt 10000000; ++$i) { $i.ToString() -match '1' } }

UPDATE: If you are still not scared then try to use the .NET reader:

$reader = [System.IO.File]::OpenText("my.log")
try {
    for() {
        $line = $reader.ReadLine()
        if ($line -eq $null) { break }
        # process the line
        $line
    }
}
finally {
    $reader.Close()
}

UPDATE 2

There are comments about possibly better / shorter code. There is nothing wrong with the original code with for and it is not pseudo-code. But the shorter (shortest?) variant of the reading loop is

$reader = [System.IO.File]::OpenText("my.log")
while($null -ne ($line = $reader.ReadLine())) {
    $line
}

More Related Contents:

  1. this code my teacher gave it to us as homework and it does not work with me idk why
  2. How can you use an object’s property in a double-quoted string?
  3. How can I write/create a powershell script verbatim, from another powershell script?
  4. Powershell: passing json string to curl
  5. How can I replace every occurrence of a String in a file with PowerShell?
  6. How do I pass variables with the Invoke-Command cmdlet?
  7. Loop through files in a directory using PowerShell
  8. Difference between ForEach and ForEach-Object in powershell
  9. How to reference .NET assemblies using PowerShell
  10. Any way to double-click on .PS1 file & open them in PowerShell?
  11. Get current user’s credentials object in Powershell without prompting
  12. Progress during large file copy (Copy-Item & Write-Progress?)
  13. How to get started with PowerShell? [closed]
  14. How to convert to UInt64 from a string in Powershell? String-to-number conversion
  15. How to Grant permission to user on Certificate private key using powershell?
  16. Using Invoke-Command -ScriptBlock on a function with arguments
  17. Is there a way to pass serializable objects to a PowerShell script with start-process?
  18. How to use PowerShell copy-item and keep structure
  19. CMD or Powershell command to combine (merge) corresponding lines from two files [duplicate]
  20. How to download the SSL certificate from a website using PowerShell?
  21. Rename file on FTP with PowerShell
  22. Is there an alternative to GUID when using msiexec to uninstall an application?
  23. Can’t install nuget package because of “Failed to initialize the PowerShell host”
  24. What security setting is preventing Remote PowerShell 2.0 from accessing UNC paths
  25. PowerShell Capture Write-Host output
  26. start remote process within the context
  27. How can I elevate Powershell while keeping the current working directory AND maintain all parameters passed to the script?
  28. Are you able to use PtrToStringAuto to decrypt a secure string in Powershell 7 on macOS?
  29. Getting “Can’t find the drive. The drive called ‘IIS’ does not exist.”
  30. Prevent trailing newline in PowerShell Out-File command

Leave a Comment