Auth failure on Centos9 Stream

[Lalufu]

I'm running the same pam_ssh_agent_auth configuration across a number of RH derived systems: Centos7. Centos8 Stream, Centos9 Stream, Fedora.
This configuration works on all of them, except for C9S, and I can't really figure out why.
Installation of the pam_ssh_agent_auth lib, configuration of /etc/pam.d/sudo and /etc/sudoers, creation of users and installation of .../authorized_keys is managed through ansible, and is the same on all of the above.
On C9S, attempting to sudo (as the user sun) results in the below logs:

Jul 17 16:33:33 centos-9-test sudo[56017]: Beginning pam_ssh_agent_auth for user sun
Jul 17 16:33:33 centos-9-test sudo[56017]: Attempting authentication: `sun' as `sun' using /home/sun/.ssh/authorized_keys
Jul 17 16:33:33 centos-9-test sudo[56017]: Contacted ssh-agent of user sun (1000)
Jul 17 16:33:33 centos-9-test sudo[56017]: trying public key file /home/sun/.ssh/authorized_keys
Jul 17 16:33:33 centos-9-test sudo[56017]: auth_secure_filename: checking for uid: 1000
Jul 17 16:33:33 centos-9-test sudo[56017]: secure_filename: checking '/home/sun/.ssh'
Jul 17 16:33:33 centos-9-test sudo[56017]: secure_filename: checking '/home/sun'
Jul 17 16:33:33 centos-9-test sudo[56017]: secure_filename: terminating check at '/home/sun'
Jul 17 16:33:33 centos-9-test sudo[56017]: matching key found: file/command /home/sun/.ssh/authorized_keys, line 1
Jul 17 16:33:33 centos-9-test sudo[56017]: Found matching RSA key: SHA256:pxEsjz3nkAhl8P0sFqpYigo0SL5jtdRcc+CXDzGHNiU
Jul 17 16:33:33 centos-9-test sudo[56017]: Failed Authentication: `sun' as `sun' using /home/sun/.ssh/authorized_keys

Setting a password for the user, and using that to sudo works, so the user is allowed to sudo in general.

In comparison, on all other systems (here a C8S) the same sequence results in these logs:

Jul 17 16:41:48 centos-8-test sudo[24707]: Beginning pam_ssh_agent_auth for user sun
Jul 17 16:41:48 centos-8-test sudo[24707]: Attempting authentication: `sun' as `sun' using /home/sun/.ssh/authorized_keys
Jul 17 16:41:48 centos-8-test sudo[24707]: Contacted ssh-agent of user sun (1000)
Jul 17 16:41:49 centos-8-test sudo[24707]: trying public key file /home/sun/.ssh/authorized_keys
Jul 17 16:41:49 centos-8-test sudo[24707]: auth_secure_filename: checking for uid: 1000
Jul 17 16:41:49 centos-8-test sudo[24707]: secure_filename: checking '/home/sun/.ssh'
Jul 17 16:41:49 centos-8-test sudo[24707]: secure_filename: checking '/home/sun'
Jul 17 16:41:49 centos-8-test sudo[24707]: secure_filename: terminating check at '/home/sun'
Jul 17 16:41:49 centos-8-test sudo[24707]: matching key found: file/command /home/sun/.ssh/authorized_keys, line 1
Jul 17 16:41:49 centos-8-test sudo[24707]: Found matching RSA key: SHA256:pxEsjz3nkAhl8P0sFqpYigo0SL5jtdRcc+CXDzGHNiU
Jul 17 16:41:49 centos-8-test sudo[24707]: Authenticated: `sun' as `sun' using /home/sun/.ssh/authorized_keys

As far as I can tell there's no difference in how pam_ssh_agent_auth is built between the different OSs. Any hint as to where in the code this might go wrong would be appreciated.

[shafer]

I don't know how to track down the problem, but I suspect it is related to el9 distros using OpenSSL 3.0 and something having changed there. I'm experiencing the same problem with Rocky Linux 9.

[beldmit]

To track down the problem, could you please enable LEGACY crypto policy?

[shafer]

I enabled the policy and things started to work again. Let me know if you need more information. Happy to help.

[beldmit]

Try to use the last build of OpenSSH (-19 in C9S)

[shafer]

I don't have C9S, but I found the SRPM for the -19 OpenSSH and built it on an updated Rocky 9. After installing the newer versions of packages for what I already have it works with the DEFAULT instead of LEGACY crypto profiles.

[Lalufu]

For what it's worth this looks like it could be a C9 specific issue somehow, there's a bug open at https://bugzilla.redhat.com/show_bug.cgi?id=2070113 Not sure this is strictly a OpenSSL3/LEGACY issue, as I'm using the same on F36 (which also has OpenSSL3) with the same crypto settings, and that works fine.

…

On Mon, Jul 18, 2022 at 03:05:34PM -0700, shafer wrote: I don't have C9S, but I found the SRPM for the -19 OpenSSH and built it on an updated Rocky 9. After installing the newer versions of packages for what I already have it works with the DEFAULT instead of LEGACY crypto profiles. -- Reply to this email directly or view it on GitHub: #39 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[beldmit]

Yes, and this bug was fixed recently.

I don't remember exactly to what extent SHA1-disablement patches are brought to F36.

[quanmqiu]

Run command update-crypto-policies --set DEFAULT:SHA1 in RHEL 9.0, then reboot. The problem disappeared.

But my pubkey format in .ssh/authorized_keys is SHA256:xxx. It has nothing to do with SHA1.

[beldmit]

Yes. The problem is that to proof the key ownership, sha1 signature is used internally.

[Lalufu]

Would you happen to know which particular bit in the crypto policies governs this specific use case? I'm running the same setup on C9S and Fedora, and the Fedora boxes mention a whole lot less SHA1 in /etc/crypto-policies/, yet things still work. Does RHEL9 carry more patches to SSH that extends the crypto-policy coverage?

[quanmqiu]

Another problem also disappeared after running update-crypto-policies --set DEFAULT:SHA1 in RHEL 9.0.

I'm trying to setup a SSH hostbased authentication cluster with RHEL 9.0. Passwordless login between nodes always failed:

debug3: authmethod_lookup hostbased
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled hostbased
debug1: Next authentication method: hostbased
debug3: userauth_hostbased: trying key type ssh-ed25519
debug1: userauth_hostbased: trying hostkey ssh-ed25519 SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=3040, exec /usr/libexec/openssh/ssh-keysign
debug3: send packet: type 50
debug2: we sent a hostbased packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: userauth_hostbased: trying key type ssh-ed25519
debug3: userauth_hostbased: trying key type ecdsa-sha2-nistp256
debug1: userauth_hostbased: trying hostkey ecdsa-sha2-nistp256 SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=3041, exec /usr/libexec/openssh/ssh-keysign
debug3: send packet: type 50
debug2: we sent a hostbased packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: userauth_hostbased: trying key type ssh-rsa
debug1: userauth_hostbased: trying hostkey ssh-rsa SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=3042, exec /usr/libexec/openssh/ssh-keysign
ssh-keysign: sshkey_sign failed: error in libcrypto
ssh_keysign: no reply
sign using hostkey ssh-rsa SHA256:xxxxxxx failed
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey

After running update-crypto-policies --set DEFAULT:SHA1, Passwordless login between nodes working too:

debug3: authmethod_lookup hostbased
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled hostbased
debug1: Next authentication method: hostbased
debug3: userauth_hostbased: trying key type ssh-ed25519
debug1: userauth_hostbased: trying hostkey ssh-ed25519 SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=1628, exec /usr/libexec/openssh/ssh-keysign
debug3: send packet: type 50
debug2: we sent a hostbased packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: userauth_hostbased: trying key type ssh-ed25519
debug3: userauth_hostbased: trying key type ecdsa-sha2-nistp256
debug1: userauth_hostbased: trying hostkey ecdsa-sha2-nistp256 SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=1629, exec /usr/libexec/openssh/ssh-keysign
debug3: send packet: type 50
debug2: we sent a hostbased packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,hostbased
debug3: userauth_hostbased: trying key type ssh-rsa
debug1: userauth_hostbased: trying hostkey ssh-rsa SHA256:xxxxxxx
debug2: userauth_hostbased: chost node01.
debug3: ssh_msg_send: type 2
debug3: ssh_msg_recv entering
debug3: ssh_keysign: [child] pid=1630, exec /usr/libexec/openssh/ssh-keysign
debug3: send packet: type 50
debug2: we sent a hostbased packet, wait for reply
debug3: receive packet: type 52
Authenticated to node02 ([192.168.1.2]:22) using "hostbased".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: Accepted for node01 [192.168.1.1] by /etc/ssh/shosts.equiv.
debug3: receive packet: type 4
debug1: Remote: Accepted for node01 [192.168.1.1] by /etc/ssh/shosts.equiv.
debug3: receive packet: type 4
debug1: Remote: Accepted for node01 [192.168.1.1] by /etc/ssh/shosts.equiv.
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

[beldmit]

OpenSSH in RHEL/Fedora uses OpenSSL 3.0 features to make FIPS certification easier. So the changes in CP are not located in the the OpenSSH file but in the OpenSSL one.

[Lalufu]

For those interested, the important setting is [evp_properties] rh-allow-sha1-signatures = yes in /etc/crypto-policies/back-ends/opensslcnf.config. This defaults to "yes" on fedora in the DEFAULT policy, but to "no" on RHEL9 with the DEFAULT policy.