Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

thermalctld crash on exit if connection to chassisdb is broken #21058

Open
patrickmacarthur opened this issue Dec 5, 2024 · 1 comment
Open

thermalctld crash on exit if connection to chassisdb is broken #21058

patrickmacarthur opened this issue Dec 5, 2024 · 1 comment
Assignees
@patrickmacarthur
Labels
Triaged this issue has been triaged

Comments

@patrickmacarthur
Copy link
Contributor

Description

When thermalctld exits, object finalizers run that attempt to connect to chassisdb to clean up table entries. This causes the process to crash if the database-chassis container isn't reachable, which can happen under certain circumstances during chassis reboot.

Steps to reproduce the issue:

  1. Boot up a chassis system.
  2. Kill the database-chassis container on the supervisor (may also need to disable the systemd service so it isn't automatically restarted)
  3. On a linecard, kill the thermalctld process.

Describe the results you received:

The thermalctld process exited with the following exception:

2024 Dec  5 21:01:54.131224 nfc420-7 INFO pmon#thermalctld: Stop thermal monitoring loop
2024 Dec  5 21:01:54.135852 nfc420-7 INFO pmon#supervisord: thermalctld Exception ignored in: <function TemperatureUpdater.__del__ at 0x7f33cceb0040>
2024 Dec  5 21:01:54.135927 nfc420-7 INFO pmon#supervisord: thermalctld Traceback (most recent call last):
2024 Dec  5 21:01:54.136005 nfc420-7 INFO pmon#supervisord: thermalctld   File "/usr/local/bin/thermalctld", line 561, in __del__
2024 Dec  5 21:01:54.136051 nfc420-7 INFO pmon#supervisord: thermalctld     self.chassis_table._del(tk)
2024 Dec  5 21:01:54.136105 nfc420-7 INFO pmon#supervisord: thermalctld   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 2970, in _del
2024 Dec  5 21:01:54.136634 nfc420-7 INFO pmon#supervisord: thermalctld     return self.delete(*args, **kwargs)
2024 Dec  5 21:01:54.136686 nfc420-7 INFO pmon#supervisord: thermalctld            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024 Dec  5 21:01:54.136724 nfc420-7 INFO pmon#supervisord: thermalctld   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 2964, in delete
2024 Dec  5 21:01:54.137163 nfc420-7 INFO pmon#supervisord: thermalctld     return _swsscommon.Table_delete(self, *args, **kwargs)
2024 Dec  5 21:01:54.137163 nfc420-7 INFO pmon#supervisord: thermalctld            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024 Dec  5 21:01:54.137182 nfc420-7 INFO pmon#supervisord: thermalctld RuntimeError: RedisError: Failed to redisGetReply in RedisPipeline::pop, err=3: errstr=Server closed the connection
2024 Dec  5 21:01:54.137182 nfc420-7 INFO pmon#supervisord: thermalctld terminate called after throwing an instance of 'swss::RedisError'
2024 Dec  5 21:01:55.188986 nfc420-7 INFO pmon#supervisord: thermalctld   what():  RedisError: Failed to redisGetReply in RedisPipeline::pop, err=3: errstr=Server closed the connection
2024 Dec  5 21:01:55.787212 nfc420-7 INFO pmon#supervisord 2024-12-05 21:01:55,786 WARN exited: thermalctld (terminated by SIGABRT (core dumped); not expected)

Describe the results you expected:

The thermalctld process exist cleanly.

Output of show version:

SONiC Software Version: SONiC.branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: 18f257b86
Build date: Thu Dec  5 09:32:19 UTC 2024
Built by: jenkins@jenkins-arsonic-k8s-6-j8lwz

Platform: x86_64-arista_7800r3_48cqm2_lc
HwSKU: Arista-7800R3-48CQM2-C48
ASIC: broadcom
ASIC Count: 1
Serial Number: SSN20370075
Model Number: 7800R3-48CQM2-LC
Hardware Revision: 2a.00
Uptime: 21:58:59 up 43 min,  1 user,  load average: 0.86, 1.05, 1.03
Date: Thu 05 Dec 2024 21:58:59

Docker images:
REPOSITORY                        TAG                                                                                IMAGE ID       SIZE
docker-macsec-dbg                 latest                                                                             c5538d12667d   602MB
docker-dhcp-relay-dbg             latest                                                                             7d258391a110   467MB
docker-teamd-dbg                  branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   d36212278788   595MB
docker-teamd-dbg                  latest                                                                             d36212278788   595MB
docker-teamd                      branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   d36212278788   595MB
docker-teamd                      latest                                                                             d36212278788   595MB
docker-sonic-mgmt-framework-dbg   branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   e5e04086a17a   502MB
docker-sonic-mgmt-framework-dbg   latest                                                                             e5e04086a17a   502MB
docker-sonic-mgmt-framework       branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   e5e04086a17a   502MB
docker-sonic-mgmt-framework       latest                                                                             e5e04086a17a   502MB
docker-snmp-dbg                   branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   8c10baf1f4a3   500MB
docker-snmp-dbg                   latest                                                                             8c10baf1f4a3   500MB
docker-snmp                       branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   8c10baf1f4a3   500MB
docker-snmp                       latest                                                                             8c10baf1f4a3   500MB
docker-sflow                      branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   4ee8d3968feb   596MB
docker-sflow                      latest                                                                             4ee8d3968feb   596MB
docker-sflow-dbg                  branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   4ee8d3968feb   596MB
docker-sflow-dbg                  latest                                                                             4ee8d3968feb   596MB
docker-router-advertiser-dbg      branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   fd7b9f10a0ba   452MB
docker-router-advertiser-dbg      latest                                                                             fd7b9f10a0ba   452MB
docker-router-advertiser          branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   fd7b9f10a0ba   452MB
docker-router-advertiser          latest                                                                             fd7b9f10a0ba   452MB
docker-platform-monitor-dbg       branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   6102150232a7   531MB
docker-platform-monitor-dbg       latest                                                                             6102150232a7   531MB
docker-platform-monitor           branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   6102150232a7   531MB
docker-platform-monitor           latest                                                                             6102150232a7   531MB
docker-orchagent                  branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   f262932f1b6f   613MB
docker-orchagent                  latest                                                                             f262932f1b6f   613MB
docker-orchagent-dbg              branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   f262932f1b6f   613MB
docker-orchagent-dbg              latest                                                                             f262932f1b6f   613MB
docker-nat-dbg                    branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   e862b821eceb   598MB
docker-nat-dbg                    latest                                                                             e862b821eceb   598MB
docker-nat                        branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   e862b821eceb   598MB
docker-nat                        latest                                                                             e862b821eceb   598MB
docker-mux-dbg                    branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   47fb4456afcc   473MB
docker-mux-dbg                    latest                                                                             47fb4456afcc   473MB
docker-mux                        branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   47fb4456afcc   473MB
docker-mux                        latest                                                                             47fb4456afcc   473MB
docker-lldp-dbg                   branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   0d6974019d8a   461MB
docker-lldp-dbg                   latest                                                                             0d6974019d8a   461MB
docker-lldp                       branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   0d6974019d8a   461MB
docker-lldp                       latest                                                                             0d6974019d8a   461MB
docker-sonic-gnmi-dbg             branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   35092e38b7c3   536MB
docker-sonic-gnmi-dbg             latest                                                                             35092e38b7c3   536MB
docker-sonic-gnmi                 branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   35092e38b7c3   536MB
docker-sonic-gnmi                 latest                                                                             35092e38b7c3   536MB
docker-fpm-frr-dbg                branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   7f99be82016f   650MB
docker-fpm-frr-dbg                latest                                                                             7f99be82016f   650MB
docker-fpm-frr                    branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   7f99be82016f   650MB
docker-fpm-frr                    latest                                                                             7f99be82016f   650MB
docker-eventd-dbg                 branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   5710224d0a7e   453MB
docker-eventd-dbg                 latest                                                                             5710224d0a7e   453MB
docker-eventd                     branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   5710224d0a7e   453MB
docker-eventd                     latest                                                                             5710224d0a7e   453MB
docker-database-dbg               branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   9c43dd9ec5a0   460MB
docker-database-dbg               latest                                                                             9c43dd9ec5a0   460MB
docker-database                   branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   9c43dd9ec5a0   460MB
docker-database                   latest                                                                             9c43dd9ec5a0   460MB
docker-syncd-brcm-dnx-dbg         branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   3a410553e76d   842MB
docker-syncd-brcm-dnx-dbg         latest                                                                             3a410553e76d   842MB
docker-syncd-brcm-dnx             branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   3a410553e76d   842MB
docker-syncd-brcm-dnx             latest                                                                             3a410553e76d   842MB
docker-gbsyncd-credo-dbg          branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   81dd3feb8458   425MB
docker-gbsyncd-credo-dbg          latest                                                                             81dd3feb8458   425MB
docker-gbsyncd-credo              branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   81dd3feb8458   425MB
docker-gbsyncd-credo              latest                                                                             81dd3feb8458   425MB
docker-gbsyncd-broncos-dbg        branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   3e419f6e4482   456MB
docker-gbsyncd-broncos-dbg        latest                                                                             3e419f6e4482   456MB
docker-gbsyncd-broncos            branch.202405-ars.1f8ff556-buildimage.origin.202405-nightly-dbg-2024.12.04.22.44   3e419f6e4482   456MB
docker-gbsyncd-broncos            latest                                                                             3e419f6e4482   456MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

This is occassionally triggered by sonic-mgmt reboot or config reload tests.
@arista-nwolfe
Copy link
Contributor

arista-nwolfe commented Dec 6, 2024 •
edited
Loading

@arlakshm @wenyiz2021

@rlhui rlhui added the Triaged this issue has been triaged label Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@patrickmacarthur patrickmacarthur

Labels
Triaged this issue has been triaged
Projects
SONiC Chassis
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@patrickmacarthur @rlhui @arista-nwolfe