Bash while read loop extremely slow compared to cat, why?

by

The reason while read is so slow is that the shell is required to make a system call for every byte. It cannot read a large buffer from the pipe, because the shell must not read more than one line from the input stream and therefore must compare each character against a newline. If you run strace on a while read loop, you can see this behavior. This behavior is desirable, because it makes it possible to reliably do things like:

while read size; do test "$size" -gt 0 || break; dd bs="$size" count=1 of=file$(( i++ )); done

in which the commands inside the loop are reading from the same stream that the shell reads from. If the shell consumed a big chunk of data by reading large buffers, the inner commands would not have access to that data. An unfortunate side-effect is that read is absurdly slow.

More Related Contents:

  1. sh: 1: Syntax error: "(" unexpected error
  2. When should I wrap quotes around a shell variable?
  3. How do I test if a variable is a number in Bash?
  4. What does set -e mean in a bash script?
  5. What’s the magic of “-” (a dash) in command-line parameters?
  6. Pseudo-terminal will not be allocated because stdin is not a terminal
  7. Is there a “goto” statement in bash?
  8. Forcing bash to expand variables in a string loaded from a file
  9. Script parameters in Bash
  10. How can I generate new variable names on the fly in a shell script?
  11. Bash script processing limited number of commands in parallel
  12. Avoid gnome-terminal close after script execution?
  13. How can I kill a process by name instead of PID, on Linux? [duplicate]
  14. ‘find -exec’ a shell function in Linux
  15. How to run a shell script on a Unix console or Mac terminal?
  16. Use sudo without password INSIDE a script
  17. Bash Script – iterating over output of find
  18. Round a divided number in Bash
  19. Syntax with pound and percent sign after shell parameter name [duplicate]
  20. Insert multiple lines into a file after specified pattern using shell script
  21. Bash: wait with timeout
  22. Randomly shuffling lines in Linux / Bash
  23. Get ceiling integer from number in linux (BASH)
  24. How can I trim white space from a variable in awk?
  25. Replacing Control Character in sed
  26. SUID not working with shell script
  27. How do you filter out all unique lines in a file?
  28. Counter increment in Bash loop not working
  29. How to pass parameters to a Bash script?
  30. Write a bash shell script that consumes a constant amount of RAM for a user defined time [closed]

Leave a Comment