Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: VOQ disrupts test log analyzer failures #15916

Open
arista-hpandya opened this issue Dec 5, 2024 · 0 comments · May be fixed by #15917
Open

[Bug]: VOQ disrupts test log analyzer failures #15916

arista-hpandya opened this issue Dec 5, 2024 · 0 comments · May be fixed by #15917
Labels

Comments

@arista-hpandya
Copy link
Contributor

arista-hpandya commented Dec 5, 2024

Issue Description

When an LC or a supervisor is rebooted it produces a lot of logs irrelevant to the individual testcases. This is one of the drawbacks faced by the voq disrupts tests that reboots the device and fails because of these unaccounted error logs. The errors can be reproduced by running test_reboot_supervisor in a T2 topology.

In testcases the involve a reboot the loganalyzer is usually disabled. This can be seen in testcases like:

Results you see

A number of log analyzer errors that are unrelated to VOQs

Results you expected to see

I expect the runs to catch/check voq related errors

Is it platform specific

generic

Relevant log output

	 E               Failed: Processes "['analyze_logs--<MultiAsicSonicHost cmp214>', 'analyze_logs--<MultiAsicSonicHost cmp214-6>', 'analyze_logs--<MultiAsicSonicHost cmp214-5>']" failed with exit code "1"
	 E               Exception:
	 E               match: 17
	 E               expected_match: 0
	 E               expected_missing_match: 0
	 E
	 E               Match Messages:
	 E               2024 Sep 23 07:13:24.492029 cmp214 ERR kernel: [18641.674628] pcieport 0000:68:02.0: Unable to change power state from D3hot to D0, device inaccessible
	 E
	 E               2024 Sep 23 07:14:49.234564 cmp214 ERR kernel: [    1.130366] tpm tpm0: [Firmware Bug]: TPM interrupt not working, polling instead
	 E
	 E               2024 Sep 23 07:14:49.235332 cmp214 WARNING kernel: [   18.724950] EXT4-fs (loop1): warning: mounting fs with errors, running e2fsck is recommended
	 E
	 E               2024 Sep 23 07:15:37.268005 cmp214 ERR CCmisApi: system_servicetoo many values to unpack (expected 2)
	 E
	 E               2024 Sep 23 07:15:39.982507 cmp214 ERR eventd#eventd: :- deserialize: deserialize Failed: input stream errorstr[0:64]:(#001) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&
	 E
	 E               2024 Sep 23 07:20:20.055070 cmp214 ERR syncd10#syncd: [3b:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.224719 cmp214 ERR syncd1#syncd: [10:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.394472 cmp214 ERR syncd7#syncd: [2e:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.402920 cmp214 ERR syncd9#syncd: [38:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.786212 cmp214 ERR syncd2#syncd: [13:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.856832 cmp214 ERR syncd8#syncd: [31:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.102896 cmp214 ERR syncd11#syncd: [42:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.433288 cmp214 ERR syncd3#syncd: [1a:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.607603 cmp214 ERR syncd0#syncd: [09:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.623071 cmp214 ERR syncd6#syncd: [27:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.780340 cmp214 ERR syncd5#syncd: [24:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.815145 cmp214 ERR syncd4#syncd: [1d:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               Traceback:
	 E               Traceback (most recent call last):
	 E                 File "/data/tests/common/helpers/parallel.py", line 35, in run
	 E                   Process.run(self)
	 E                 File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
	 E                   self._target(*self._args, **self._kwargs)
	 E                 File "/data/tests/common/helpers/parallel.py", line 245, in wrapper
	 E                   target(*args, **kwargs)
	 E                 File "/data/tests/common/plugins/loganalyzer/__init__.py", line 45, in analyze_logs
	 E                   dut_analyzer.analyze(markers[node.hostname], fail_test, store_la_logs=store_la_logs)
	 E                 File "/data/tests/common/plugins/loganalyzer/loganalyzer.py", line 409, in analyze
	 E                   self._verify_log(analyzer_summary)
	 E                 File "/data/tests/common/plugins/loganalyzer/loganalyzer.py", line 140, in _verify_log
	 E                   raise LogAnalyzerError(result_str)
	 E               tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzerError: match: 17
	 E               expected_match: 0
	 E               expected_missing_match: 0
	 E
	 E               Match Messages:
	 E               2024 Sep 23 07:13:24.492029 cmp214 ERR kernel: [18641.674628] pcieport 0000:68:02.0: Unable to change power state from D3hot to D0, device inaccessible
	 E
	 E               2024 Sep 23 07:14:49.234564 cmp214 ERR kernel: [    1.130366] tpm tpm0: [Firmware Bug]: TPM interrupt not working, polling instead
	 E
	 E               2024 Sep 23 07:14:49.235332 cmp214 WARNING kernel: [   18.724950] EXT4-fs (loop1): warning: mounting fs with errors, running e2fsck is recommended
	 E
	 E               2024 Sep 23 07:15:37.268005 cmp214 ERR CCmisApi: system_servicetoo many values to unpack (expected 2)
	 E
	 E               2024 Sep 23 07:15:39.982507 cmp214 ERR eventd#eventd: :- deserialize: deserialize Failed: input stream errorstr[0:64]:(#001) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&
	 E
	 E               2024 Sep 23 07:20:20.055070 cmp214 ERR syncd10#syncd: [3b:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.224719 cmp214 ERR syncd1#syncd: [10:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.394472 cmp214 ERR syncd7#syncd: [2e:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.402920 cmp214 ERR syncd9#syncd: [38:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.786212 cmp214 ERR syncd2#syncd: [13:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:20.856832 cmp214 ERR syncd8#syncd: [31:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.102896 cmp214 ERR syncd11#syncd: [42:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.433288 cmp214 ERR syncd3#syncd: [1a:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.607603 cmp214 ERR syncd0#syncd: [09:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.623071 cmp214 ERR syncd6#syncd: [27:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.780340 cmp214 ERR syncd5#syncd: [24:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).
	 E
	 E               2024 Sep 23 07:20:21.815145 cmp214 ERR syncd4#syncd: [1d:00.0] SAI_API_SWITCH:brcm_sai_get_switch_attribute:6654 Get switch attrib 37 failed with error Feature unavailable (0xfffffff0).

Output of show version

SONiC Software Version: SONiC.branch.202405-ars.bdc49266-buildimage.origin.202405-nightly-dbg-2024.11.28.22.44
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: 3c6372b8e
Build date: Fri Nov 29 08:41:16 UTC 2024

Platform: x86_64-arista_7800_sup
HwSKU: Arista-7804R3-FM
ASIC: broadcom
ASIC Count: 12
Serial Number: SSN20290119
Model Number: DCS-7800-SUP1A
Hardware Revision: 01.0a
Uptime: 18:55:31 up 2 days,  3:06,  1 user,  load average: 5.89, 6.11, 6.34
Date: Thu 05 Dec 2024 18:55:31

Attach files (if any)

No response

@arista-hpandya arista-hpandya linked a pull request Dec 5, 2024 that will close this issue
8 tasks
@yxieca yxieca added the Triaged label Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

Successfully merging a pull request may close this issue.

2 participants