Slowness about scp.py HOT 11 OPEN

@carolinux are you comparing transfer speed of scp without compression to your code? I made the mistake of not setting client.connect(compress=True) so my transfers were considerably slower compared to my normal usage of scp -C

from scp.py.

Probably a paramiko issue 😢

Have you tried specifying a different buff_size?

from scp.py.

Last line:

scp_cli = scp.SCPClient(ssh_cli.get_transport(), buff_size=2*1024*1024)

from scp.py.

Are you talking about a single file? Multiple small files?

Have this ever been benchmarked?

from scp.py.

single file around 30 MBs.

from scp.py.

def get_ssh_connection():
    host = data_server
    ssh = SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.load_system_host_keys()
    try:
        ssh.connect(data_server, username="root")
    except:
        password = getpass.getpass("SSH password for {}@{} : ".format("root", data_server))
        ssh.connect(data_server, username="root", password=password)
    return ssh


def scp_to_server(local_filenames, remote_directory, skip_if_exists=True):
    ssh_cli = get_ssh_connection()
    scp_cli = scp.SCPClient(ssh_cli.get_transport())

Ok, where would I add the bufsize? Do I need a channel object and if so, how do I give to the SCPClient? (just in case it's convenient for you to answer, I will look at it later myself also)

from scp.py.

Thank you! I will report back when I benchmark it, because I think more people may find it useful.

from scp.py.

Bad news: I didn't perceive a difference one way or the other.

from scp.py.

any update on this, am having a similar issue

from scp.py.

(Oops)

I can't reproduce the issue, so it's pretty hard. It shouldn't be much slower, and if it is, that's probably an issue with Paramiko. But we can run more tests.

from scp.py.

I ran some tests and I really don't see much of a slowdown.

NYC MacBook to GCE: 14.605 / 14.248, French server to GCE: 13.114, 13.294

If you see a really significant difference, do you mind sharing a bit more on your setup? Which OS? Is the machine CPU bound? High latency? Are you running on a very fast link?

from scp.py.