Python read file as stream from HDFS

by

You want xreadlines, it reads lines from a file without loading the whole file into memory.

Edit:

Now I see your question, you just need to get the stdout pipe from your Popen object:

cat = subprocess.Popen(["hadoop", "fs", "-cat", "/path/to/myfile"], stdout=subprocess.PIPE)
for line in cat.stdout:
    print line

More Related Contents:

  1. Actual meaning of ‘shell=True’ in subprocess
  2. How to terminate a python subprocess launched with shell=True
  3. Python subprocess readlines() hangs
  4. How to redirect output with subprocess in Python?
  5. Python subprocess get children’s output to file and terminal?
  6. Subprocess changing directory
  7. Blocking and Non Blocking subprocess calls
  8. Launch a completely independent process
  9. Understanding Popen.communicate
  10. How do I pipe a subprocess call to a text file?
  11. Multiple inputs and outputs in python subprocess communicate
  12. Python C program subprocess hangs at “for line in iter”
  13. Popen with conflicting executable/path
  14. How to write to stdout AND to log file simultaneously with Popen?
  15. link several Popen commands with pipes
  16. Python running as Windows Service: OSError: [WinError 6] The handle is invalid
  17. Display realtime output of a subprocess in a tkinter widget
  18. Keep a subprocess alive and keep giving it commands? Python
  19. Python output to Console within Subprocess from the child scricpt
  20. python getoutput() equivalent in subprocess [duplicate]
  21. Why subprocess.Popen doesn’t work when args is sequence?
  22. How to run multiple commands synchronously from one subprocess.Popen command?
  23. Redirect subprocess stderr to stdout
  24. Interacting with bash from python
  25. Event Driven System Call in Python
  26. How do I eliminate Windows consoles from spawned processes in Python (2.7)? [duplicate]
  27. Launch IPython notebook with selected browser
  28. Sending ^C to Python subprocess objects on Windows
  29. Execute Commands Sequentially in Python?
  30. How to install and import Python modules at runtime

Leave a Comment