[BUG] salt-call hangs after highstate #62550
Comments
OrangeDog
added
Bug
|
I probably should have checked first, but the rendering error has nothing to do with it. It does not hang after e.g. |
OrangeDog
changed the title
|
same here after updating to 3005, manually calling Summary for local
--------------
Succeeded: 170 (changed=11)
Failed: 0
--------------
Total states run: 170
Total run time: 69.927 s
^Cthe state finishes but the salt-call is kept running. Same behavior if executed from the master |
|
Able to reproduce. Upgraded minion to 3005, now hangs with "state.highstate" after printing the result. |
|
I'm affected, too. Ubuntu 20.04 with Salt minion 3005 (classic). Switching the salt-minion to the onedir installation fixes this problem. But since onedir has other issues (e.g. I cannot get the Here is an excerpt of Here is an excerpt of To my untrained eye, it looks like an event loop is not terminated properly. `salt-call --versions` for classic:Salt Version:
Salt: 3005
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: 1.4.4
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.10 (default, Jun 22 2022, 20:18:18)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-125-generic
system: Linux
version: Ubuntu 20.04 focal
`salt-call --versions` for onedir:Salt Version:
Salt: 3005
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.13 (main, Aug 23 2022, 18:31:04)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-125-generic
system: Linux
version: Ubuntu 20.04 focal
|
|
@d--j for the onedir issue did you try running And since you stated it works on onedir, it makes me think this is a dependency issue. It looks like there are difference in dependency versions. Here's the diff between the classic nad onedir packages: If anyone is willing to test out each dependency difference and install the newer versions on the classical packages and see if it works that would be great. If not whoever is assigned to this issue can try that, if that's discovered to be the issue. |
yes I tried => and => Packages without native extensions (e.g. IPy) can be installed this way but at least for the MySQL package (that I do need nearly on all minions) it does not work. Some modules also expect These issues are probably unrelated to this issue and I will open bug reports for them when I get the time.
Yes it might very well be. Or maybe usage of a specific state module might cause this. Since in my case the state run on onedir had some errors this state module might have not been executed by chance.
I will try to do that. |
|
Thanks, yes if you could open a separate issue I would appreciate it. |
|
still strange that after |
|
It looks like it is indeed a dependency problem (python3-zmq 18.1.1 from Ubuntu focal is the culprit). This works for me: resulting salt-call --versionsSalt Version:
Salt: 3005
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: 4.1.0
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: 1.4.4
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.10 (default, Jun 22 2022, 20:18:18)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-125-generic
system: Linux
version: Ubuntu 20.04 focal |
|
Wondering if something else is the problem here, since I have tried with the shipping pyzmq 18.1.1 and pip3 installed pyzmq 23.2.0 and both produce the same error: for state file However when I place the test.sls in /srv/salt on the salt-master it all works, even with pyzmq==18.1.1 |
|
@dmurphy18 the error message says that your pillar data failed to compile. What does your For me after the upgrading the master to 3005 this code in (could not find But after a reboot of the master the unchanged code started working again. Anyways, this looks like an unrelated issue and if it persists you should probably create a new issue for it. |
|
@OrangeDog I'm not able to reproduce this with Centos 7 salt-master running 3005 or 3004.2 |
|
@d--j My pillar top.sls is cleared out, the comment out stuff is due to old proxy testing |
dmurphy18
added
the
cannot-reproduce
|
@dmurphy18 the minion needs to run 3005. I do not think that the master's version matters to this issue. When the minion runs 3005 then execute |
|
@d--j The minion is 3005 classic state.apply without params results in highstate, but will call highstate separately :) |
|
Successful |
|
I think there must be something else going for the hang to occur. |
|
@dmurphy18 Hm, then it probably matters what states are in the highstate/apply, how many are in it (500+ in my case) or how long the |
|
@d--j So this might be a scale issue. If there is a logic problem, it will be seen with a single master and minion, usually. |
|
This is a minimal sls file that produces the issue on my test minion:
with With that executing |
|
I don't know whether everyone else here with the same problem is the same, but:
|
#!pydsl
__pydsl__.set(ordered=True)
state('/tmp/test').file.directory()
state().cmd.run('sleep 30')
for i in range(5000):
state('/tmp/test/file%s' % i).file.managed(contents='%s'%i)@OrangeDog I went thru my |
|
It looks like it's usage of |
Not all of them. I tried with a static |
|
Hmm, that seemed to be the pattern with my states. e.g. this hangs: client-pkgs:
pkg.installed:
- pkgs:
- krb5-user
- libpam-krb5
/etc/krb5.conf.d:
file.directory: []
/etc/krb5.conf:
file.managed:
- source: salt://{{ tpldir }}/krb5.conf
- require:
- file: /etc/krb5.conf.dBut this doesn't: vm.swappiness:
sysctl.present:
- value: 0
net.ipv4.tcp_timestamps:
sysctl.present:
- value: 0
net.ipv4.tcp_keepalive_time:
sysctl.present:
- value: 600
{% for pkg in ('htop', 'jq', 'unzip') %}
{{ pkg }}:
pkg.installed: []
{% endfor %}
en_GB.UTF-8:
locale.system: []
qemu-guest-agent:
{% if grains['virtual'] == 'kvm' %}
pkg.installed: []
{% else %}
pkg.purged: []
{% endif %}
{% if grains['os_family'] == 'Debian' %}
debian-goodies:
pkg.installed: []
{% endif %}
{% if salt['file.file_exists']('/etc/apt/trusted.gpg') and salt['file.stats']('/etc/apt/trusted.gpg')['size'] <= 32 %}
# an empty keyring causes invalid format warnings
/etc/apt/trusted.gpg:
file.absent: []
{% endif %} |
|
So in both our cases this fails: install-some-packages:
pkg.installed:
- pkgs:
- pkg1
- pkg2
/tmp/manage-a-file-with-source:
file.managed:
- source: salt://path/to/sourceSince your sls does not use |
|
@d--j @OrangeDog Thanks for that, used the last test case and got the hang |
dmurphy18
removed
the
cannot-reproduce
|
Yup using pyzmq 23.2.1 fixes the issue. Since 19.0.2 hangs and 20.0 doesn't. |
|
Using Ubuntu 20.04 python3-zmq v18.1.1-3 and pushing it differently and hitting that thread issue fixed in pyzmq v20.0. |
|
Don't see the problem hang on the following with classic packaging: Note: all onedir on Linux use pyzmq 23.2.0 so don't have hang issue. Looks like Ubuntu 20.04 is the only version affected, will fix packaging for Ubuntu 20.04, Debian 11 has pyzmq 20.0.0 which should work with Ubuntu 20.04 |
|
Any manual how i best install pyzmq 20.0.0 on Ubuntu 20.04? |
|
Fixed by saltstack/salt-pack-py3#255 |
|
Closing as fix is now available from https://repo.saltproject.io/py3/ubuntu/20.04/amd64/latest/ and https://repo.saltproject.io/py3/ubuntu/20.04/arm64/latest/ |
|
I still get this on a Centos 8.7 look-alike ( AlmaLinux 8.7 ) 3005.1 on it comes with PyZMQ 22.0.3 and not yet with PyZMQ 23.2.0. |
Description
salt-callfails to terminate after a rendering error during highstate. Must be manually killed.Setup
{{ salt["does.not.exist"]() }}Steps to Reproduce the behavior
Expected behavior
Process should terminate.
Versions Report
Master is 3004.2, minion is 3005 (classic package).
salt-call --versions
Additional context
There's no debug logging after printing the result either. Last messages (after all the rendering) are:
The text was updated successfully, but these errors were encountered: