SFTP Incremental Reads + Writes

[emeryalden]

Trying to solve an issue where running .get() on a large file (4GB+) will work at first but appears to eventually exhaust the system's RAM and results in the download freezing.

When using .get(), does asyncssh write data to the destination file as it arrives or does everything need to be collected into memory before it can actually be written to a file?

If it's the latter, is there a way to incrementally download a file from an SFTP and write chunks to the destination file as it goes, allowing the system to then clear some memory space? Or is there another way to recommended approach for instances where file sizes are larger than memory?

[ronf]

All of the get/put/copy and wildcard mget/mput/mcopy functions should write data to the destination as soon as it is read in. The default block size used to be fixed at 16 KB and the max parallel requests at 128, so the most you should need is about 2 MB per file being transferred. If you transfer multiple files at once, this would be the amount needed per file.

With AsyncSSH 2.18.0, the default buffer size can be higher if the server supports the "limit" extension and advertises a larger max read/write size, so on very large files you may see an increase in memory needed (up to something like 512 MB). If this is too much, though, you can either set a smaller block size or reduce the max number of parallel requests, though you may lose some speed depending on how small you make these.

The arguments to control this are block_size and max_requests.

[emeryalden]

Hey @ronf,

Really appreciate your input here. I played around with the block size and max_requests settings, but am still running into difficulties. Upon further investigation, the issue appears not to be related to exhausting RAM but rather on the system failing to rekey with the SFTP server. While the key exchange at the start of the transfer is fine, the one occurring ~1GB into the transfer simply hangs until the connection drops. No exception is raised. I've included the debug logs for the events leading up to this below

DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052262400 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052278784 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052295168 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh:[conn=20, chan=0] Received 32720 data bytes
DEBUG:asyncssh:[conn=20, chan=0] Received 32720 data bytes
DEBUG:asyncssh:[conn=20, chan=0] Received 32720 data bytes
DEBUG:asyncssh:[conn=20, chan=0] Received 20921 data bytes
DEBUG:asyncssh:[conn=20] Requesting key exchange
DEBUG:asyncssh:[conn=20]   Key exchange algs: curve25519-sha256,[email protected],curve448-sha512,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,ecdh-sha2-1.3.132.0.10,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,[email protected],diffie-hellman-group14-sha1,rsa2048-sha256,ext-info-c,[email protected]
DEBUG:asyncssh:[conn=20]   Host key algs: ssh-ed25519
DEBUG:asyncssh:[conn=20]   Encryption algs: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=20]   MAC algs: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1,[email protected],[email protected],[email protected],[email protected],[email protected]
DEBUG:asyncssh:[conn=20]   Compression algs: none,[email protected]
DEBUG:asyncssh:[conn=20] Received key exchange request
DEBUG:asyncssh:[conn=20]   Key exchange algs: curve25519-sha256,[email protected],diffie-hellman-group18-sha512,diffie-hellman-group16-sha512
DEBUG:asyncssh:[conn=20]   Host key algs: ssh-ed25519,rsa-sha2-256,rsa-sha2-512
DEBUG:asyncssh:[conn=20]   Client to server:
DEBUG:asyncssh:[conn=20]     Encryption algs: [email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=20]     MAC algs: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
DEBUG:asyncssh:[conn=20]     Compression algs: none,zlib
DEBUG:asyncssh:[conn=20]   Server to client:
DEBUG:asyncssh:[conn=20]     Encryption algs: [email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
DEBUG:asyncssh:[conn=20]     MAC algs: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
DEBUG:asyncssh:[conn=20]     Compression algs: none,zlib
DEBUG:asyncssh:[conn=20] Beginning key exchange
DEBUG:asyncssh:[conn=20]   Key exchange alg: curve25519-sha256
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Received 16384 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052311552 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052327936 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052344320 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052360704 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052377088 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052393472 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052409856 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending read for 16384 bytes at offset 1052426240 in handle 313033
DEBUG:asyncssh:[conn=20, chan=0] Sending 28 data bytes
DEBUG:asyncssh:[conn=20] Sending keepalive request
DEBUG:asyncssh:[conn=20] Sending keepalive request
DEBUG:asyncssh:[conn=20] Sending keepalive request
INFO:asyncssh:[conn=20] Server not responding to keepalive
INFO:asyncssh:[conn=20, chan=0] Closing channel due to connection close
INFO:asyncssh:[conn=20, chan=0] Channel closed: Server not responding to keepalive
INFO:asyncssh.sftp:[conn=20, chan=0] SFTP client exited: Server not responding to keepalive
INFO:asyncssh:[conn=20, chan=0] Closing channel
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending close for handle 313033
INFO:asyncssh.sftp:[conn=20, chan=0] Starting SFTP get of XXXXXXXXXXXXXXXX to XXXXXXXXXXXXXXXX
DEBUG:asyncssh.sftp:[conn=20, chan=0] Sending lstat for XXXXXXXXXXXXXXXX
INFO:asyncssh:[conn=20] Closing connection
INFO:asyncssh:[conn=20] Sending disconnect: Disconnected by application (11)

So far I've tried modifying both kex_algs and rekey_bytes to try to resolve this, but the issue persists. In fact, rekey_bytes does not seem to have any impact as even lowing the value does not prompt the system to attempt to rekey sooner than the ~1GB mark.

Any advice on where to go from here?

[ronf]

So far, I'm unable to reproduce this here, using AsyncSSH as both client and server as well as an AsyncSSH client talking to an OpenSSH server. What client and server (and versions) are you using when you see the problem?

Changing rekey_bytes on the client won't do much here, as almost all of the data is being sent by the server and rekey_bytes kicks in based on the number of bytes sent. So, if your server has this set to 1 GB, that'll be when you see the rekey happening.

The fact that keepalive begins to fail at some point suggests that either something is hanging on the server side, or maybe that there's a problem with the network between the client and server after sending a large amount of data.

You might try running AsyncSSH as both client and server and see if you still see the problem. You could also setting a lower value for rekey_bytes on the server side in this case.

If you control the machine the SFTP server is running on, you could also change its config file to rekey sooner using OpenSSH as the server and see if the changes when the problem occurs.

[emeryalden]

Thank you again for your input on this. It's perhaps worth mentioning that this connection just uses username + password for authentication. Could that be tripping up the rekey process?

[ronf]

No - authentication isn't performed again during rekeying. Only the key exchange which happens prior to auth is repeated. Once complete, both sides switch over to new encryption keys, but the authentication information remains the same.