dnf module errors with mitogen on Ansible 2.16.7+

[psvmcc]

I see errors on dnf tasks like:

PLAY [Common server configuration] *********************************************
Failed loading plugin "copr": module 'dnf' has no attribute 'cli'
Failed loading plugin "debuginfo-install": module 'dnf' has no attribute 'cli'
Failed loading plugin "copr": module 'dnf' has no attribute 'cli'
Failed loading plugin "debuginfo-install": module 'dnf' has no attribute 'cli'
Failed loading plugin "copr": module 'dnf' has no attribute 'cli'
Failed loading plugin "debuginfo-install": module 'dnf' has no attribute 'cli'
Failed loading plugin "copr": module 'dnf' has no attribute 'cli'

Task example:

- name: Setup rsync
  ansible.builtin.dnf:
    name:
      - rsync-daemon
    state: present

Output messages:

TASK [rsync : Setup rsync] ****************************************************
Failed loading plugin "debuginfo-install": module 'dnf' has no attribute 'cli'
ok: [hostname]

Without mitogen everything works well.

• Which version of Ansible are you running?
  2.16.6 works well, and starting from 2.16.7 I see issues, and with 2.17.4 also

• Is your version of Ansible patched in any way?
  No

• Are you running with any custom modules, or module_utils loaded?
  No

• Have you tried the latest master version from Git?
  I see issue with mitogen 0.3.9 and with just released 0.3.12

• Do you have some idea of what the underlying problem may be?
  I found this Failed loading plugin "debuginfo-install": module 'dnf' has no attribute 'cli' rpm-software-management/dnf-plugins-core#520
  And this https://github.com/ansible/ansible/pull/82725/files#diff-b1696caabd083d8176e1b14ad10716e8c88d882bf9e49e1a167f3a6cb702a21eL578

• Mention your host and target OS and versions
  My host - MacOS, but I use venv
  Target host - OracleLinux 9(latest)

• Mention your host and target Python versions
  My host - 3.12
  Target host - 3.9

[psvmcc]

@moreati maybe I can someone help with debug?

[moreati]

Looks screwy, probably of Mitogen's doing.

My first guess is that Mitogen is confusing one of the fully qualified Python module names ansible.modules.dnf, ansible.builtin.dnf ... with the FQN dnf. So on the target sys.modules['dnf'] ends up containing the wrong module object.

Notes to self

• Ansible module ansible.builtin.dnf
  • implemented in Python module ansible.modules.dnf
  • Ansible module can be referred in playbooks by unqualified name dnf.
  • Python implementation imports Python module with fully qualified name dnf. import is postponed https://github.com/ansible/ansible/blob/ce392dd86e35ba33bcdb76279f785f296008a1db/lib/ansible/modules/dnf.py#L475-L485
  • Python implementation no longer contains from __future__ import absolute import (reasonable and correct from Ansible project's POV)
• Ansible module community.general.copr
  • also imports the Python module with FQN dnf, not postponed https://github.com/ansible-collections/community.general/blob/10.3.0/plugins/modules/copr.py#L100-L109
  • Explicitly imports FQN dnf.cli, has done since creation
  • As of community.general 10.3.0 still contains from __future__ import absolute_import
• Python module dnf
  • typically installed from OS package, https://pypi.org/project/dnf/ is intentionally a dud.
  • makes little sense to attempt serving from Mitogen parent/master.

[psvmcc]

Yesterday, I delved into this issue by adding debugging to the DNF libraries. I have several hosts, but the problem doesn’t occur on all of them.

To investigate further, I picked a couple of hosts and began comparing them. I checked everything under /etc, /usr, and /var related to Python and DNF. In terms of the filesystem and versioning, everything was identical across the two hosts.

When I enabled debugging, I noticed that DNF loads its plugins from /usr/lib/python3.9/site-packages/dnf-plugins/ in a different order on the two hosts. This caught my attention, as it seemed to be the root of the issue. Specifically, if a plugin that already imports dnf.cli was loaded first, the problem didn’t occur.

I compared the ctime and inode of the files, and the order was identical. Additionally, I found that the find utility also returned the files in a different order, which is likely due to some quirks in the ext4 journaling system.

Bad Host

# find /usr/lib/python3.9/site-packages/dnf-plugins/ -type f | head -n 3
/usr/lib/python3.9/site-packages/dnf-plugins/debuginfo-install.py
/usr/lib/python3.9/site-packages/dnf-plugins/repoclosure.py
/usr/lib/python3.9/site-packages/dnf-plugins/config_manager.py

Good Host

# find /usr/lib/python3.9/site-packages/dnf-plugins/ -type f | head -n 3
/usr/lib/python3.9/site-packages/dnf-plugins/ulninfo.py
/usr/lib/python3.9/site-packages/dnf-plugins/config_manager.py
/usr/lib/python3.9/site-packages/dnf-plugins/copr.py

This behavior came as a surprise to me. However, I still don’t understand how Mitogen influences this or why vanilla Ansible doesn’t encounter the same issue. Even with maximum verbosity (-vvvvv), I don’t see any errors when using vanilla Ansible.

[moreati]

Reopening. I'm treating #1230 as a workaround, rather than a full fix.