File not found error when launching a subprocess containing piped commands

by

You have to add shell=True to execute a shell command. check_output is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w and he cannot find it. (no idea why you removed the essential information from the error message).

See the difference between:

>>> subprocess.check_output('date | grep 1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/subprocess.py", line 603, in check_output
    with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
  File "/usr/lib/python3.4/subprocess.py", line 848, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1'

And: 

>>> subprocess.check_output('date | grep 1', shell=True)
b'gio 19 giu 2014, 14.15.35, CEST\n'

Read the documentation about the Frequently Used Arguments for more information about the shell argument and how it changes the interpretation of the other arguments.

Note that you should try to avoid using shell=True since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!).

The documentation for the subprocess module has a little section about replacing the shell pipeline.
You can do so by spawning the two processes in python and use subprocess.PIPE:

date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE)
grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE)
date_proc.stdout.close()
output = grep_proc.communicate()[0]

You can write some simple wrapper function to easily define pipelines:

import subprocess
from shlex import split
from collections import namedtuple
from functools import reduce

proc_output = namedtuple('proc_output', 'stdout stderr')


def pipeline(starter_command, *commands):
    if not commands:
        try:
            starter_command, *commands = starter_command.split('|')
        except AttributeError:
            pass
    starter_command = _parse(starter_command)
    starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE)
    last_proc = reduce(_create_pipe, map(_parse, commands), starter)
    return proc_output(*last_proc.communicate())

def _create_pipe(previous, command):
    proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE)
    previous.stdout.close()
    return proc

def _parse(cmd):
    try:
        return split(cmd)
    except Exception:
        return cmd

With this in place you can write pipeline('date | grep 1') or pipeline('date', 'grep 1') or pipeline(['date'], ['grep', '1'])

More Related Contents:

  1. How do I execute a program or call a system command?
  2. Running shell command and capturing the output
  3. How do I use subprocess.Popen to connect multiple processes by pipes?
  4. How to use `subprocess` command with pipes
  5. Retrieving the output of subprocess.call() [duplicate]
  6. Using sudo with Python script
  7. Python: execute cat subprocess in parallel
  8. live output from subprocess command
  9. Execute terminal command from python in new terminal window?
  10. Multiple pipes in subprocess
  11. Pipe subprocess standard output to a variable [duplicate]
  12. How do I pipe a subprocess call to a text file?
  13. subprocess.call() arguments ignored when using shell=True w/ list [duplicate]
  14. subprocess readline hangs waiting for EOF
  15. link several Popen commands with pipes
  16. Process substitution not allowed by Python’s subprocess with shell=True?
  17. python getoutput() equivalent in subprocess [duplicate]
  18. Checking for interactive shell in a Python script
  19. ‘yes’ reporting error with subprocess communicate()
  20. Understanding python subprocess.check_output’s first argument and shell=True [duplicate]
  21. Python subprocess: how to use pipes thrice? [duplicate]
  22. real time subprocess.Popen via stdout and PIPE
  23. When to use Shell=True for Python subprocess module [duplicate]
  24. Communicate multiple times with a process without breaking the pipe?
  25. Python – how to execute shell commands with pipe, but without ‘shell=True’?
  26. Python: subprocess call with shell=False not working
  27. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  28. Cannot find the file specified when using subprocess.call(‘dir’, shell=True) in Python
  29. What is the subprocess.Popen max length of the args parameter?
  30. Sending ^C to Python subprocess objects on Windows

Leave a Comment