Read a file in buffer from FTP python

by

Make sure to login to the ftp server first. After this, use retrbinary which pulls the file in binary mode. It uses a callback on each chunk of the file. You can use this to load it into a string.

from ftplib import FTP
ftp = FTP('ftp.ncbi.nlm.nih.gov')
ftp.login() # Username: anonymous password: anonymous@

# Setup a cheap way to catch the data (could use StringIO too)
data = []
def handle_binary(more_data):
    data.append(more_data)

resp = ftp.retrbinary("RETR pub/pmc/PMC-ids.csv.gz", callback=handle_binary)
data = "".join(data)

Bonus points: how about we decompress the string while we’re at it?

Easy mode, using data string above

import gzip
import StringIO
zippy = gzip.GzipFile(fileobj=StringIO.StringIO(data))
uncompressed_data = zippy.read()

Little bit better, full solution:

from ftplib import FTP
import gzip
import StringIO

ftp = FTP('ftp.ncbi.nlm.nih.gov')
ftp.login() # Username: anonymous password: anonymous@

sio = StringIO.StringIO()
def handle_binary(more_data):
    sio.write(more_data)

resp = ftp.retrbinary("RETR pub/pmc/PMC-ids.csv.gz", callback=handle_binary)
sio.seek(0) # Go back to the start
zippy = gzip.GzipFile(fileobj=sio)

uncompressed = zippy.read()

In reality, it would be much better to decompress on the fly but I don’t see a way to do that with the built in libraries (at least not easily).

More Related Contents:

  1. Python FTP get the most recent file by date
  2. Python FTP implicit TLS connection issue
  3. Downloading a directory tree with ftplib
  4. How to get FTP file’s modify time using Python ftplib
  5. Cannot list FTP directory using ftplib – but FTP client works
  6. Checking if object on FTP server is file or directory using Python and ftplib
  7. FTPS with Python ftplib – Session reuse required
  8. Python-FTP download all files in directory
  9. Is it possible to read FTP files without writing them using Python?
  10. Connecting to ‘Explicit FTP over TLS’ in Python (??)
  11. Download large file in python with requests
  12. Can I redirect the stdout into some sort of string buffer?
  13. Python: download a file from an FTP server
  14. How to convert an iterable to a stream?
  15. Python Script Uploading files via FTP
  16. Can you “stream” images to ffmpeg to construct a video, instead of saving them to disk?
  17. Get files names inside a zip file on FTP server without downloading whole archive
  18. Using Python’s ftplib to get a directory listing, portably
  19. How to read the first byte of a subprocess’s stdout and then discard the rest in Python?
  20. Monitor remote FTP directory
  21. TypeError: ‘list’ object is not callable in python
  22. Sqlite / SQLAlchemy: how to enforce Foreign Keys?
  23. Python: URLError:
  24. set environment variable in python script
  25. TensorFlow version 1.0.0-rc2 on Windows: “OpKernel (‘op: “BestSplits” device_type: “CPU”‘) for unknown op: BestSplits” with test code
  26. How to use MySQLdb with Python and Django in OSX 10.6?
  27. Error (-215) size.width>0 && size.height>0 occurred when attempting to display an image using OpenCV
  28. read and write on same csv file
  29. Using Python class as a data container [closed]
  30. How to handle a Button click event

Leave a Comment