SSH Passwords in context #617

marcoooo · 2017-02-15T08:31:32Z

Hi All, here is a problem related to the 'good way to use password' for ssh saga connexions.

I store in database saga adaptors parameters, saga Service and Context are initialied in a class instantiated from params set up in DB, "let's call it myadaptor.full.path.ClusterOverSSHWIthUserPass.py".
This class store saga adaptors params as instances attributes such as:

username
password
host
protocol (let's say sge+ssh)

Now when I set up login / pass / host parameters in DB, and test connection with my connect function, everything goes well.

Later I come back to my backoffice, (even if I restart my web server), I change password to a erroneous one. Try to test connexion, and I still connect to saga.Service...

How could you explain this ? Saga Context logs the new erroneous password still....

I have no idea... does the saga context are store anywhere on my disk ? (I tried to erase .saga directory with no success either).
Is this related to the LIFE_TIME parameter automatically set in context to -1 ?

Is there any way clean context each time I call my connect function (which mainly create Service and Context) ?

andre-merzky · 2017-02-15T08:57:23Z

Hey Marc, thanks for moving that thread into a ticket! Let me explain parts of the magic happening here:

We try to optimize connection setup in two dimensions: performance, and number of connections. The former is not really crucial actually, but the second one is, as the number of connections we are allowed to establish is often limited both on client and server side. Coincidentally, we use the same approach for both: connection sharing.

That works like this: whenever a connection is requested by some SAGA component, we set up a master ssh channel towards the target host, and create a slave channel which reuses credentials from the master channel. All subsequent connection requests make use of the same master channel. The credential reuse makes channel setup faster, and different (usually less strict) limits apply to slave channels -- win/win :-)

Now, what I think happens in your case is that the master channel survives the session, and your next application run picks up the master channel with existing authentication and only creates slave channels. This is intentional to some extent, as its fast.

SSH internally keeps track of master channel processes via a socket in /tmp/, whose name we specify like this (simplified):

path = "/tmp/saga_ssh_%s_%%h_%%p.%s.ctrl" % (local_user, remote_user])

where %%h and %%p are filled by ssh with target host and port. If no remote_user is specified in the URL or otherwise now known, that part is omitted.

So my guess is that this socket is still around, your second attempt with a different password matches to the same socket name, finds the socket, and creates a slave channel from an existing, authorized master channel. This should be testable by removing /tmp/saga_ssh_* between the runs -- then the second run should fail with an authentication error because of the invalid password.

Can you please check if the above indeed applies? If so, lets then discuss what behavior you would actually expect, and how we can provide it.

Thanks, Andre.

marcoooo · 2017-02-15T09:09:31Z

Thanks Andre for this explanation, this explain why when I changed the user, connection attempt failed, but when I changed password connection was still ok (as the socket file name is related to user)

But, I just tried removing my tmp socket file /tmp/saga_ssh* but still I can connect with forced wrong credentials.

Regards,
Marc.

andre-merzky · 2017-02-15T09:13:39Z

But, I just tried removing my tmp socket file /tmp/saga_ssh* but still I can connect with forced wrong credentials.

Hmmm... Could you please check two things:

after deleting the /tmp/ files, can you please check if any ssh connections are still alive: ps -ef | grep ssh
when you see the second connection succeeding, can you please run on the command line:

ssh -vvv -l <user> -o ControlMaster=auto <host>

and (a) check if this succeeds, and (b) attach the (longish) output?

Thanks!

marcoooo · 2017-02-15T09:23:00Z

Andre, here is the output for ps -ef | grep ssh

marc      2085  2004  0 09:12 ?        00:00:00 /usr/bin/ssh-agent /usr/bin/ck-launch-session /usr/bin/dbus-launch --exit-with-session /usr/bin/im-launch /usr/bin/startkde
marc      6590  6557  0 10:17 pts/2    00:00:00 /usr/bin/ssh -t -o IdentityFile=/home/marc/.ssh/id_rsa -p 22 -o ControlMaster=auto -o ControlPath=/tmp/saga_ssh_marc_%h_%p.phylo.ctrl -o TCPKeepAlive=no -o ServerAliveInterval=10 -o ServerAliveCountMax=20 -o ConnectTimeout=10 phylo@wilkins
marc      6591  6557  0 10:17 pts/3    00:00:00 /usr/bin/ssh -t -o IdentityFile=/home/marc/.ssh/id_rsa -p 22 -o ControlMaster=auto -o ControlPath=/tmp/saga_ssh_marc_%h_%p.phylo.ctrl -o TCPKeepAlive=no -o ServerAliveInterval=10 -o ServerAliveCountMax=20 -o ConnectTimeout=10 phylo@wilkins

And the quite long output :-) (connection succeed)

OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "wilkins" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to wilkins [193.49.110.17] port 22.
debug1: Connection established.
debug1: identity file /home/marc/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/marc/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to wilkins:22 as 'phylo'
debug3: hostkeys_foreach: reading file "/home/marc/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/marc/.ssh/known_hosts:3
debug3: record_hostkey: found key type RSA in file /home/marc/.ssh/known_hosts:14
debug3: load_hostkeys: loaded 2 keys from wilkins
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Z3SvKfFUjRH6kKszzA65j6ccIuNUxLbUFKODH0EWRDQ
debug3: hostkeys_foreach: reading file "/home/marc/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/marc/.ssh/known_hosts:3
debug3: record_hostkey: found key type RSA in file /home/marc/.ssh/known_hosts:14
debug3: load_hostkeys: loaded 2 keys from wilkins
debug3: hostkeys_foreach: reading file "/home/marc/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/marc/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from 193.49.110.17
debug1: Host 'wilkins' is known and matches the ECDSA host key.
debug1: Found key in /home/marc/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug2: key: /home/marc/.ssh/id_rsa (0x557ebf3e4ef0)
debug2: key: /home/marc/.ssh/id_dsa ((nil))
debug2: key: /home/marc/.ssh/id_ecdsa ((nil))
debug2: key: /home/marc/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/marc/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/marc/.ssh/id_dsa
debug3: no such identity: /home/marc/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/marc/.ssh/id_ecdsa
debug3: no such identity: /home/marc/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/marc/.ssh/id_ed25519
debug3: no such identity: /home/marc/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
phylo@wilkins's password:

You see log related to my id_rsa, but my key is not registered on remote host.

Regards,
Marc

andre-merzky · 2017-02-15T09:34:10Z

wait - you say connection succeeded, but the output shows phylo@wilkins's password:: So I would assume that it won't succeed when you enter an incorrect password here, right?

Can you please try the following:

export RADICAL_SAGA_PTY_VERBOSE=DEBUG
export RADICAL_SAGA_VERBOSE=DEBUG
export RADICAL_SAGA_LOG_TGT=/tmp/saga.log

then run the second test again, and send the log file?

Thanks!

marcoooo · 2017-02-15T09:59:16Z

The log message issued from ssh -vvv -l <user> -o ControlMaster=auto <host> still ask for password, may be because I am not login in to remote with my machine user but with another one ?

I may have more information about the problem. Here is the process:

I started web django server
Setup right usr/pass
Test connection => OK
Delete the socket file from tmp
Setup and save wrong credentials
Test connection => OK
Kill and restart web server
Test connection => failed

Here is the log file:
saga.log.txt

Hope that help !
Thanxs

andre-merzky · 2017-02-15T10:11:44Z

Ah, that is useful, if that happens in the django instance, but a django restart resolves it, then it might e a garbage collection issue, in the sense that the original session is still around and gets reused, or that even the slave connection survives and gets reused (we also do that in a session, FWIW).

Do you have the option to explicitly delete the session instance between the tests, via del(session)? But I am not actually sure that is reliable: we have often seen things lingering around even after explicit deletion, because Python did not deem some object or the other viable for garbage collection, due to object dependencies... :(

marcoooo · 2017-02-15T10:15:42Z

Hmmm, I'll have a look for this option of killing the session, if you are talking about the Django Session.. But if I do so, I migth lost my connection to the Django BackOffice, nop ?

SO first I have to make my BO erasing the /tmp/saga_ssh... if the kill session works ?

Marc

andre-merzky · 2017-02-15T10:50:56Z

Ah, sorry - I mean the SAGA session! :)

marcoooo · 2017-02-15T10:56:11Z

I was wondering if this was yours or Django session I have to kill :-D. Is there a method to kill saga session ? (basiccally in my 'Django-Adaptor' class I create a new session Object like this:

class SshUserPassShellAdaptor(SshShellAdaptor):
...
    @property
    def session(self):
        """ Configuration of Saga-python session (with context) for adaptor """
        session = saga.Session()
        if self.context:
            session.add_context(self.context)
        return session
...

    @property
    def context(self):
        """ Setup saga context to connect over ssh with UserPass protocol """
        ctx = saga.Context('UserPass')
        ctx.user_id = self.user_id
        ctx.user_pass = self.crypt_user_pass
        return ctx

Perhaps, the 'default' = False Session constructor param may do the trick ? Or should I create a class member to keep session attached to my object ?

andre-merzky · 2017-02-15T11:27:40Z

Yes, setting default=False might help, as that will not use the default session singleton. But that has the drawback to also not get any default security contexts from the default session: so you need to manually add context for each ssh keypair you want to use (but, on the upside, ssh might pick most up on its own).

Otherwise I would think you can delete the session with:

my_session = SshUserPassShellAdaptor()
... # do stuff
del(my_session.session)

but as said, I am not sure this actually really deletes the session in fact, as I am not sure the garbage collector will kick in...

On an aside: I am not sure I understand your code above though: it looks like you create a new SAGA session instance everytime you access the session property on your class? So the following code would create two saga sessions, and also two ssh contexts:

my_session = SshUserPassShellAdaptor()
js_1 = saga.job.Service(url_1, session=my_session.session)
js_2 = saga.job.Service(url_2, session=my_session.session)

Is that the intended behavior?

marcoooo · 2017-02-15T12:24:20Z

I'll give a try to default=False, and delete session each time needed.

To explain my code, same SshUserPassShellAdaptor object can be used by multiple service, with different params. From Object model:

A service (ie a executable tool) is associated with a Runner (in charge of creating Adaptors instance from db init params - login / protocol / host etc...)
A same runner may be used by many Services with different configuration (different host / user / pass for example)
A job simply associate the submitted params to Service command line to be remotely executed on Runner host.
A daemon then treat current unfinished job to prepare / run / follow up execution on specified adaptor
according to current job state. The properties session / context are usually accessed once per state from the 'connect' method.

All keeping in mind that I do have implemented other adaptors to discuss with other platform such as Galaxy server (http://usegalaxy.org) or other remotes specific API (http://www.atgc-montpellier.fr/compphy/ for a service to create phylogenetics supertrees)
So I may say this is the intended behaviour, or am I missing a essential point about Saga ?
I migth associate session in a _session attribute, in order not to recreate one if Adaptor needs to access property more than once per step ?
Thanxs.

andre-merzky · 2017-02-15T12:38:16Z

I might associate session in a _session attribute, in order not to recreate one if Adaptor needs to access property more than once per step?

Yes, that is basically what I would suggest. I mean, I am in no position to give you advise on the code obviously, and the above points certainly make sense AFAICS -- but from a SAGA perspective it would be good to make sure that the number of sessions remains limited, and their lifetime managed.

marcoooo · 2017-02-15T12:45:45Z

ok I just did associated the _session attribute at least not to duplicate session object in same Adaptor class object during same process.

You don't know the code, but still you can have opinion :-)

As you said, from a SAGA point of view, what's the 'standard' LifeTime property to set (assuming -1 is forever).

marcoooo · 2017-02-15T13:34:49Z

Unfortunatlly deleting _session attribute upon 'disconnect' method do not help... still connect with wrong credentials... even with a life_time set to 0.
Forgive me for being annoying with that.

    def testWrongCredentials(self):
        import time
        self.adaptor.connect()
=> use standard good credentials
        self.assertTrue(self.adaptor.connected)
        self.adaptor.disconnect()
=> unset attribute _session (del)
        self.adaptor.crypt_user_pass = 'fake_pass'
        print "sleep"
        time.sleep(10)
=> to let me the time to delete tmp file :)
        print "awake"
        with self.assertRaises(AdaptorConnectException) as context:
            print self.adaptor.connect()
        self.assertTrue('AuthenticationFailed' in context.exception.message)

But still I am connected to remote.
M.

iparask · 2019-03-21T15:33:24Z

This is an interesting issue. I'm keeping it open for further discussion.

iparask · 2019-03-26T22:09:11Z

We discussed the issue internally and we saw that it is going to be addressed in version 2 (planned for 2020).
@marcoooo is this still an issue for you?

marcoooo · 2019-12-17T11:59:40Z

Hello, I haven't been working on my project for a while, and must say this is not an issue for now. Since I am planning to use the new python 3 version soon, I'll come back to you if I find the issue is still here.

andre-merzky assigned marcoooo and andre-merzky Feb 15, 2017

andre-merzky added the pty label Feb 15, 2017

andre-merzky added this to the Backburner milestone Feb 15, 2017

andre-merzky added topic:communication and removed pty labels Mar 4, 2018

andre-merzky removed their assignment Apr 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSH Passwords in context #617

SSH Passwords in context #617

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

marcoooo commented Feb 15, 2017

iparask commented Mar 21, 2019

iparask commented Mar 26, 2019

marcoooo commented Dec 17, 2019

SSH Passwords in context #617

SSH Passwords in context #617

Comments

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

andre-merzky commented Feb 15, 2017

marcoooo commented Feb 15, 2017

marcoooo commented Feb 15, 2017

iparask commented Mar 21, 2019

iparask commented Mar 26, 2019

marcoooo commented Dec 17, 2019