Keychain cannot be unlocked using sufficient flag in macOS

[kun-zhou]

The first line in my pam.d/authorization file is
auth sufficient /usr/local/lib/security/pam_yubico.so mode=challenge-response

The issue is the macOS never let me enter my login password because I had once the first line in my pam is validated, the PAM finishes up and ignores the following required lines and therefore Keychain cannot get my login password and complains about it. Is there a workaround? My intention is to only use Yubikey for login and password only as a backup.