Writing to a file on SFTP server opened using Paramiko/pysftp “open” method is slow

by

You open the remote file without buffering. That way, every time the df.to_csv writes to the file, Paramiko/pysftp sends a request to the SFTP server and waits for a response. I do not know internals of df.to_csv, but it’s likely it does one write per line (if not more). That would explain, why the upload is so slow. Particularly, if your connection to the server has high latency.

To enable buffered writes, use bufsize parameter of Connection.open:

with sftp.open(destination_path, 'w+', 32768) as f:

Similarly for reads/downloads:
Reading file opened with Python Paramiko SFTPClient.open method is slow

Obligatory warning: Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp.

More Related Contents:

  1. Python pysftp get_r from Linux works fine on Linux but not on Windows
  2. pysftp vs. Paramiko
  3. Python pysftp put_r does not work on Windows
  4. How to write this code without using pandas?
  5. How to plot in multiple subplots
  6. How can I plot separate Pandas DataFrames as subplots?
  7. Faster way to read Excel files to pandas dataframe
  8. How to reversibly store and load a Pandas dataframe to/from disk
  9. Format y axis as percent
  10. float64 with pandas to_csv
  11. How do I transpose dataframe in pandas without index?
  12. How to select all columns whose names start with X in a pandas DataFrame
  13. Pandas finding local max and min
  14. How to import a text file on AWS S3 into pandas without writing to disk
  15. Append column to pandas dataframe
  16. How do I round datetime column to nearest quarter hour
  17. How to plot and annotate grouped bars in seaborn / matplotlib
  18. Print the raw value of a column alone, in pandas?
  19. Pandas merge two dataframes with different columns
  20. Pandas filter dataframe rows with a specific year
  21. Divide multiple columns by another column in pandas
  22. Access index in pandas.Series.apply
  23. Python Pandas to_sql, how to create a table with a primary key?
  24. Apply function to each cell in DataFrame
  25. Export Pandas DataFrame into a PDF file using Python
  26. Ranking order per group in Pandas
  27. How to edit properties of whiskers, fliers, caps, etc. in Seaborn boxplot
  28. Convert list into a pandas dataframe
  29. Change main plot legend label text
  30. Python: Adding hours to pandas timestamp

Leave a Comment