Bash style process substitution with Python’s Popen

by

If pram_axdnull understands "-" convention to mean: “read from stdin” then you could:

p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
          stdin=PIPE, stdout=PIPE)
output = p.communicate(generate_kmers(3))[0]

If the input is generated by external process:

kmer_proc = Popen(["generate_kmers", str(kmer)], stdout=PIPE)
p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
          stdin=kmer_proc.stdout, stdout=PIPE)
kmer_proc.stdout.close()
output = p.communicate()[0]

If pram_axdnull doesn’t understand "-" convention:

import os
import tempfile
from subprocess import check_output

with tempfile.NamedTemporaryFile() as file:
    file.write(generate_kmers(3))
    file.delete = False

try:
    p = Popen(["pram_axdnull", str(kmer), input_filename, file.name],
              stdout=PIPE)
    output = p.communicate()[0]
    # or
    # output = check_output(["pram_axdnull", str(kmer), input_filename, 
                             file.name])
finally:
    os.remove(file.name)

To generate temporary file using external process:

from subprocess import check_call

with tempfile.NamedTemporaryFile() as file:
    check_call(["generate_kmers", str(kmer)], stdout=file)
    file.delete = False

To avoid waiting for all kmers to be generated i.e., to write/read kmers simultaneously, you could use os.mkfifo() on Unix (suggested by @cdarke):

import os
import shutil
import tempfile
from contextlib import contextmanager
from subprocess import Popen, PIPE

@contextmanager
def named_pipe():
    dirname = tempfile.mkdtemp()
    try:
        path = os.path.join(dirname, 'named_pipe')
        os.mkfifo(path)
        yield path
    finally:
        shutil.rmtree(dirname)

with named_pipe() as path:
    p = Popen(["pram_axdnull", str(kmer), input_filename, path],
              stdout=PIPE) # read from path
    with open(path, 'wb') as wpipe:
        kmer_proc = Popen(["generate_kmers", str(kmer)],
                          stdout=wpipe) # write to path
    output = p.communicate()[0]
    kmer_proc.wait()

More Related Contents:

  1. Python subprocess/Popen with a modified environment
  2. Popen waiting for child process even when the immediate child has terminated
  3. How can I specify working directory for popen
  4. Python subprocess.call a bash alias
  5. Intercepting stdout of a subprocess while it is running
  6. Popen with conflicting executable/path
  7. How do I get ‘real-time’ information back from a subprocess.Popen in python (2.5)
  8. subprocess.Popen() error (No such file or directory) when calling command with arguments as a string
  9. How to write to stdout AND to log file simultaneously with Popen?
  10. How do I get all of the output from my .exe using subprocess and Popen?
  11. link several Popen commands with pipes
  12. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  13. running multiple bash commands with subprocess
  14. Process substitution not allowed by Python’s subprocess with shell=True?
  15. Keep a subprocess alive and keep giving it commands? Python
  16. Python output to Console within Subprocess from the child scricpt
  17. Launch a shell command with in a python script, wait for the termination and return to the script
  18. How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out
  19. real time subprocess.Popen via stdout and PIPE
  20. Interacting with bash from python
  21. What’s the difference between subprocess Popen and call (how can I use them)?
  22. How to catch exception output from Python subprocess.check_output()?
  23. python subprocess Popen environment PATH?
  24. How do I eliminate Windows consoles from spawned processes in Python (2.7)? [duplicate]
  25. How can I specify working directory for a subprocess
  26. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  27. How to do multiple arguments with Python Popen?
  28. subprocess.wait() not waiting for Popen process to finish (when using threads)?
  29. Opening a process with Popen and getting the PID
  30. subprocess.Popen in different console

Leave a Comment