Read a file line by line and execute commands: 4+ answers
This is because there is not only 1 answer…
- Shell command line expansion
xargs
dedicated toolwhile read
with some remarkswhile read -u
using dedicatedfd
, for interactive processing (sample)- running shell with inline generated script
Regarding the OP request: running chmod
on all targets listed in file, xargs
is the indicated tool. But for some other applications, small amount of files, etc…
0. Read entire file as command line argument.
If your file is not too big and all files are well named (without spaces or other special chars like quotes), you could use shell command line expansion. Simply:
chmod 755 $(<file.txt)
For small amounts of files (lines), this command is the lighter one.
1. xargs
is the right tool
For bigger amount of files, or almost any number of lines in your input file…
For many binutils tools, like chown
, chmod
, rm
, cp -t
…
xargs chmod 755 <file.txt
If you have special chars and/or a lot of lines in file.txt
.
xargs -0 chmod 755 < <(tr \\n \\0 <file.txt)
If your command need to be run exactly 1 time for each entry:
xargs -0 -n 1 chmod 755 < <(tr \\n \\0 <file.txt)
This is not needed for this sample, as chmod
accepts multiple files as arguments, but this matches the title of question.
For some special cases, you could even define the location of the file argument in commands generated by xargs
:
xargs -0 -I '{}' -n 1 myWrapper -arg1 -file="{}" wrapCmd < <(tr \\n \\0 <file.txt)
Test with seq 1 5
as input
Try this:
xargs -n 1 -I{} echo Blah {} blabla {}.. < <(seq 1 5)
Blah 1 blabla 1..
Blah 2 blabla 2..
Blah 3 blabla 3..
Blah 4 blabla 4..
Blah 5 blabla 5..
where your command is executed once per line.
2. while read
and variants.
As OP suggests,
cat file.txt |
while read in; do
chmod 755 "$in"
done
will work, but there are 2 issues:
-
cat |
is a useless fork, and -
| while ... ;done
will become a subshell whose environment will disappear after;done
.
So this could be better written:
while read in; do
chmod 755 "$in"
done < file.txt
But
- You may be warned about
$IFS
andread
flags:
help read
read: read [-r] ... [-d delim] ... [name ...] ... Reads a single line from the standard input... The line is split into fields as with word splitting, and the first word is assigned to the first NAME, the second word to the second NAME, and so on... Only the characters found in $IFS are recognized as word delimiters. ... Options: ... -d delim continue until the first character of DELIM is read, rather than newline ... -r do not allow backslashes to escape any characters ... Exit Status: The return code is zero, unless end-of-file is encountered...
In some cases, you may need to use
while IFS= read -r in;do
chmod 755 "$in"
done <file.txt
for avoiding problems with strange filenames. And maybe if you encounter problems with UTF-8:
while LANG=C IFS= read -r in ; do
chmod 755 "$in"
done <file.txt
While you use a redirection from standard inputfor reading
file.txt`, your script cannot read other input interactively (you cannot use standard input for other input anymore).
3. while read
, using dedicated fd
.
Syntax: while read ...;done <file.txt
will redirect standard input to come from file.txt
. That means you won’t be able to deal with processes until they finish.
This will let you use more than one input simultaneously, you could merge two files (like here: scriptReplay.sh), or maybe:
You plan to create an interactive tool, you have to avoid use of standard input and use some alternative file descriptor.
Constant file descriptors are:
- 0 for standard input
- 1 for standard output
- 2 for standard error.
3.1 posix shell first
You could see them by:
ls -l /dev/fd/
or
ls -l /proc/$$/fd/
From there, you have to choose unused numbers between 0 and 63 (more, in fact, depending on sysctl
superuser tool) as your file descriptor.
For this demo, I will use file descriptor 7:
while read <&7 filename; do
ans=
while [ -z "$ans" ]; do
read -p "Process file '$filename' (y/n)? " foo
[ "$foo" ] && [ -z "${foo#[yn]}" ] && ans=$foo || echo '??'
done
if [ "$ans" = "y" ]; then
echo Yes
echo "Processing '$filename'."
else
echo No
fi
done 7<file.txt
If you want to read your input file in more differents steps, you have to use:
exec 7<file.txt # Without spaces between `7` and `<`!
# ls -l /dev/fd/
read <&7 headLine
while read <&7 filename; do
case "$filename" in
*'----' ) break ;; # break loop when line end with four dashes.
esac
....
done
read <&7 lastLine
exec 7<&- # This will close file descriptor 7.
# ls -l /dev/fd/
3.2 Same under bash
Under bash, you could let him choose any free fd
for you and store into a variable:
exec {varname}</path/to/input
:
while read -ru ${fle} filename;do
ans=
while [ -z "$ans" ]; do
read -rp "Process file '$filename' (y/n)? " -sn 1 foo
[ "$foo" ] && [ -z "${foo/[yn]}" ] && ans=$foo || echo '??'
done
if [ "$ans" = "y" ]; then
echo Yes
echo "Processing '$filename'."
else
echo No
fi
done {fle}<file.txt
Or
exec {fle}<file.txt
# ls -l /dev/fd/
read -ru ${fle} headline
while read -ru ${fle} filename;do
[[ -n "$filename" ]] && [[ -z ${filename//*----} ]] && break
....
done
read -ru ${fle} lastLine
exec {fle}<&-
# ls -l /dev/fd/
5 filtering input file for creating shell commands
sed <file.txt 's/.*/chmod 755 "&"/' | sh
This won’t optimise forks, but this could be usefull for more complex (or conditional) operation:
sed <file.txt 's/.*/if [ -e "&" ];then chmod 755 "&";fi/' | sh
sed 's/.*/[ -f "&" ] \&\& echo "Processing: \\"&\\"" \&\& chmod 755 "&"/' \
file.txt | sh