netconf_get fails with '''known_hosts_lookup' is not defined"

[jean-christophe-manciot]

SUMMARY

When using the netconf connection type, ansible tries to connect with SSH using the user's private key(s) even when ansible_private_key_file is set to another specific file.
No such issue when using the network_cli connection type alongside ansible_ssh_private_key_file.

ISSUE TYPE

• Bug Report

COMPONENT NAME

netconf

ANSIBLE VERSION

ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.2 (default, Mar 13 2020, 10:14:16) [GCC 9.3.0]

CONFIGURATION

ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
CACHE_PLUGIN(/etc/ansible/ansible.cfg) = redis
CACHE_PLUGIN_TIMEOUT(/etc/ansible/ansible.cfg) = 3600
DEFAULT_EXECUTABLE(/etc/ansible/ansible.cfg) = /bin/bash
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 1000
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = explicit
DEFAULT_GATHER_TIMEOUT(/etc/ansible/ansible.cfg) = 30
DEFAULT_HASH_BEHAVIOUR(/etc/ansible/ansible.cfg) = merge
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/hosts']
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_PRIVATE_ROLE_VARS(/etc/ansible/ansible.cfg) = False
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 180
DEFAULT_TRANSPORT(/etc/ansible/ansible.cfg) = ssh
ENABLE_TASK_DEBUGGER(/etc/ansible/ansible.cfg) = True
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = True
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = /usr/bin/python3
PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible.cfg) = 3599
PERSISTENT_CONNECT_RETRY_TIMEOUT(/etc/ansible/ansible.cfg) = 200
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 3600
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
SHOW_CUSTOM_STATS(/etc/ansible/ansible.cfg) = True

OS / ENVIRONMENT

controller host: Ubuntu focal 20.04
remote host: IOS-XEv 16.9.1

STEPS TO REPRODUCE

- name: Testing which private SSH key is used by network_cli connection
  hosts:
        - all
  vars:
        ansible_connection: network_cli
        ansible_network_os: ios
        ansible_ssh_private_key_file: "{{ private_key_file }}"
        ansible_user: "{{ username }}"
  tasks:
        - name: Getting all facts
          ios_facts:
                gather_subset: all
          
- name: Testing which private SSH key is used by netconf connection
  hosts:
        - all
  vars:
        ansible_connection: netconf
        ansible_network_os: ios
        ansible_private_key_file: "{{ private_key_file }}"
        ansible_user: "{{ username }}"
  tasks:
        - name: Getting running configuration and state data
          netconf_get:

EXPECTED RESULTS

Both plays should use the same SSH private key.

ACTUAL RESULTS

1. The first play uses the correct SSH key matching private_key_file.
2. The second play does not use the SSH key matching private_key_file and requests the password for another SSH private key (~/.ssh/id_rsa).

[jean-christophe-manciot]

It seems that this issue is linked with the fact that I use gpg-agent on the controller with multiple SSH keys.
I have already seen that strange behavior linked to an unsolved paramiko issue.
What is surprising is that I don't encounter that issue in the same environment with ansible_connection: network_cli which uses SSH and gpg-agent: doesn't the latter connection type rely on paramiko?

[jean-christophe-manciot]

If I leave in ~/.gnupg/sshcontrol only one reference to an SSH key matching {{ private_key_file }}, forcing netconf plugin to use the same key as network_cli plugin, I get this error:

The full traceback is:
Traceback (most recent call last):
  File ".ansible/tmp/ansible-local-247022i7ve4hnw/ansible-tmp-1586442456.2074456-127334655330005/AnsiballZ_netconf_rpc.py", line 102, in <module>
    _ansiballz_main()
  File ".ansible/tmp/ansible-local-247022i7ve4hnw/ansible-tmp-1586442456.2074456-127334655330005/AnsiballZ_netconf_rpc.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File ".ansible/tmp/ansible-local-247022i7ve4hnw/ansible-tmp-1586442456.2074456-127334655330005/AnsiballZ_netconf_rpc.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.network.netconf.netconf_rpc', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 206, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/modules/network/netconf/netconf_rpc.py", line 264, in <module>
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/modules/network/netconf/netconf_rpc.py", line 240, in main
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/module_utils/network/netconf/netconf.py", line 119, in dispatch
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/module_utils/network/netconf/netconf.py", line 41, in get_connection
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/module_utils/network/netconf/netconf.py", line 55, in get_capabilities
  File "/tmp/ansible_netconf_rpc_payload_srguph73/ansible_netconf_rpc_payload.zip/ansible/module_utils/connection.py", line 185, in __rpc__
ansible.module_utils.connection.ConnectionError: BadAuthenticationType('Bad authentication type', ['password'])

That is very strange because the password for that key has already been entered with the previous successful run of the network_cli plugin.

This seems to demonstrate that netconf plugin fails to connect with the SSH key set by ansible_private_key_file and tries the next one offered by gpg-agent SSH ring.

[jean-christophe-manciot]

The symptom has changed.
Now with:

• ansible 2.10.2
• ansible.netcommon: 1.3.0
• python3.8: 3.8.6-1

Running the following playbook over a CSR-1kv 16.11.1b:

- name: Getting running configuration and state data with NETCONF
  hosts:
        - all
  vars:
        ansible_connection: netconf
        ansible_network_os: ios
        ansible_private_key_file: "{{ private_key_file }}"
        ansible_user: "{{ username }}"
  tasks:
        - name: Getting running configuration and state data
          netconf_get:

leads to:

The full traceback is:
Traceback (most recent call last):
  File "/media/SAMSUNG5-Shared/home/admin/.ansible/tmp/ansible-local-649522gsrsgv1b/ansible-tmp-1603120778.204628-651873-146441039705922/AnsiballZ_netconf_get.py", line 102, in <module>
    _ansiballz_main()
  File "/media/SAMSUNG5-Shared/home/admin/.ansible/tmp/ansible-local-649522gsrsgv1b/ansible-tmp-1603120778.204628-651873-146441039705922/AnsiballZ_netconf_get.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/media/SAMSUNG5-Shared/home/admin/.ansible/tmp/ansible-local-649522gsrsgv1b/ansible-tmp-1603120778.204628-651873-146441039705922/AnsiballZ_netconf_get.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.ansible.netcommon.plugins.modules.netconf_get', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_netconf_get_payload_zy5esvn5/ansible_netconf_get_payload.zip/ansible_collections/ansible/netcommon/plugins/modules/netconf_get.py", line 292, in <module>
  File "/tmp/ansible_netconf_get_payload_zy5esvn5/ansible_netconf_get_payload.zip/ansible_collections/ansible/netcommon/plugins/modules/netconf_get.py", line 219, in main
  File "/tmp/ansible_netconf_get_payload_zy5esvn5/ansible_netconf_get_payload.zip/ansible_collections/ansible/netcommon/plugins/module_utils/network/netconf/netconf.py", line 60, in get_capabilities
  File "/tmp/ansible_netconf_get_payload_zy5esvn5/ansible_netconf_get_payload.zip/ansible/module_utils/connection.py", line 195, in __rpc__
ansible.module_utils.connection.ConnectionError: name 'known_hosts_lookup' is not defined

[jean-christophe-manciot]

The same issue has been opened with more details here.