Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky kselftest-rtc tests on Chromebooks #208

Closed
r-c-n opened this issue May 22, 2023 · 1 comment
Closed

Flaky kselftest-rtc tests on Chromebooks #208

r-c-n opened this issue May 22, 2023 · 1 comment

Comments

@r-c-n
Copy link

r-c-n commented May 22, 2023

kselftest-rtc.rtc_rtctest_rtc_alarm_alm_set_minute and other rtc alarm-related test cases are reported as a regression every now and then on multiple cros-ec-based systems, but the actual problem seems to be that the test is flaky, there doesn't seem to be a pattern for it and it's unpredictable when it'll pass or fail. Here are some results showing both scenarios on mainline/master:

kernel version commit config date status link
v5.18-rc1 3123109284176b1532874591f7c81f3837bbdc17 defconfig+arm64-chromebook+kselftest 2022-04-04 PASS https://linux.kernelci.org/test/case/id/624b5461993e6060dcae0689/
v5.18-rc3 b2d229d4ddb17db541098b83524d901257e93845 defconfig+arm64-chromebook+kselftest 2022-04-18 FAIL https://linux.kernelci.org/test/case/id/625d9a9da6ef3f87ffae0688/
v5.18-rc3-7-g559089e0a93d4 559089e0a93d44280ec3ab478830af319c56dbe3 defconfig+arm64-chromebook+kselftest 2022-04-19 PASS https://linux.kernelci.org/test/case/id/625f1ee53a98168a4eae06ac/
v5.18-rc3-16-gb253435746d9a b253435746d9a4a701b5f09211b9c14d3370d0da defconfig+arm64-chromebook+kselftest 2022-04-20 FAIL https://linux.kernelci.org/test/case/id/62608ecc739ab079d8ae06a2/
v5.18-rc3-80-gd569e86915b7f d569e86915b7f2f9795588591c8d5ea0b66481cb defconfig+arm64-chromebook+kselftest 2022-04-22 PASS https://linux.kernelci.org/test/case/id/62628e8306626c58f7ff95f0/
v5.18-rc3-95-g281b9d9a4b022 281b9d9a4b02229b602a14f7540206b0fbe4134f defconfig+arm64-chromebook+kselftest 2022-04-22 FAIL https://linux.kernelci.org/test/case/id/6262f84f2cf627aeb2ff9467/
v6.2-1847-gdb77b8502a40 db77b8502a4071a59c9424d95f87fe20bdb52c3a defconfig+arm64-chromebook 2023-02-21 FAIL https://linux.kernelci.org/test/case/id/63f41f64c31d2ebe8c8c8640/
v6.2-13534-gb01fe98d34f3 b01fe98d34f3bed944a93bd8119fed80c856fad8 defconfig+arm64-chromebook 2023-03-05 PASS https://linux.kernelci.org/test/case/id/6403e874d5dff7db4a8c8634/
v6.3-13390-g2e1e1337881b 2e1e1337881b0e9844d687982aa54b31b1269b11 defconfig+arm64-chromebook 2023-05-06 FAIL https://linux.kernelci.org/test/case/id/6455cc173769bba4af2e85f5/
v6.4-rc1-111-g47a2ee5d4a0bd 47a2ee5d4a0bda05decdda7be0a77e792cdb09a3 defconfig+arm64-chromebook 2023-05-12 PASS https://linux.kernelci.org/test/case/id/645e543c4dac4dedec2e85f0/

Also happening in other Chromebooks. EC-based RTC alarms look a bit flaky overall.

kevin

Test case failing: rtc.alarm_alm_set_minute
https://linux.kernelci.org/test/case/id/6465ac2dc75c8697d12e8605/
Log excerpt:

04:38:48.011436  # #  RUN           rtc.alarm_alm_set ...
04:38:48.018044  # # rtctest.c:221:alarm_alm_set:Alarm time now set to 04:38:51.
04:38:51.005389  # # rtctest.c:241:alarm_alm_set:data: 1a0
04:38:51.008927  # #            OK  rtc.alarm_alm_set
04:38:51.011512  # ok 5 rtc.alarm_alm_set
04:38:51.015889  # #  RUN           rtc.alarm_wkalm_set ...
04:38:51.023405  # # rtctest.c:281:alarm_wkalm_set:Alarm time now set to 18/05/2023 04:38:54.
04:38:54.004404  # #            OK  rtc.alarm_wkalm_set
04:38:54.007043  # ok 6 rtc.alarm_wkalm_set
04:38:54.012011  # #  RUN           rtc.alarm_alm_set_minute ...
04:38:54.018946  # # rtctest.c:331:alarm_alm_set_minute:Alarm time now set to 04:39:00.
04:39:56.076409  # # rtctest.c:343:alarm_alm_set_minute:Expected 0 (0) != rc (0)
04:39:56.081837  # # alarm_alm_set_minute: Test terminated by assertion
04:39:56.086355  # #          FAIL  rtc.alarm_alm_set_minute
04:39:56.089660  # not ok 7 rtc.alarm_alm_set_minute
04:39:56.095013  # #  RUN           rtc.alarm_wkalm_set_minute ...
04:39:56.102937  # # rtctest.c:391:alarm_wkalm_set_minute:Alarm time now set to 18/05/2023 04:40:00.
04:40:00.001231  # #            OK  rtc.alarm_wkalm_set_minute
04:40:00.004661  # ok 8 rtc.alarm_wkalm_set_minute
04:40:00.007888  # # FAILED: 7 / 8 tests passed

hatch

Test cases failing:

  • rtc.alarm_wkalm_set
  • rtc.alarm_wkalm_set_minute

https://linux.kernelci.org/test/case/id/6465b284cadf5813ba2e85ec/
Log excerpt:

05:06:19.019815  # #  RUN           rtc.alarm_alm_set ...
05:06:19.026920  # # rtctest.c:222:alarm_alm_set:Alarm time now set to 05:06:22.
05:06:22.012314  # # rtctest.c:241:alarm_alm_set:data: 1a0
05:06:22.015634  # #            OK  rtc.alarm_alm_set
05:06:22.018901  # ok 5 rtc.alarm_alm_set
05:06:22.022156  # #  RUN           rtc.alarm_wkalm_set ...
05:06:22.032493  # # rtctest.c:284:alarm_wkalm_set:Alarm time now set to 18/05/2023 05:06:25.
05:06:22.038740  # # rtctest.c:290:alarm_wkalm_set:Expected -1 (-1) != rc (-1)
05:06:22.042112  # # alarm_wkalm_set: Test terminated by assertion
05:06:22.049424  # #          FAIL  rtc.alarm_wkalm_set
05:06:22.053483  # not ok 6 rtc.alarm_wkalm_set
05:06:22.056793  # #  RUN           rtc.alarm_alm_set_minute ...
05:06:22.064001  # # rtctest.c:332:alarm_alm_set_minute:Alarm time now set to 05:07:00.
05:07:00.011831  # # rtctest.c:351:alarm_alm_set_minute:data: 1a0
05:07:00.015373  # #            OK  rtc.alarm_alm_set_minute
05:07:00.018960  # ok 7 rtc.alarm_alm_set_minute
05:07:00.025363  # #  RUN           rtc.alarm_wkalm_set_minute ...
05:07:00.031825  # # rtctest.c:394:alarm_wkalm_set_minute:Alarm time now set to 18/05/2023 05:08:00.
05:07:00.038342  # # rtctest.c:400:alarm_wkalm_set_minute:Expected -1 (-1) != rc (-1)
05:07:00.044824  # # alarm_wkalm_set_minute: Test terminated by assertion
05:07:00.048222  # #          FAIL  rtc.alarm_wkalm_set_minute
05:07:00.055002  # not ok 8 rtc.alarm_wkalm_set_minute
05:07:00.058276  # # FAILED: 6 / 8 tests passed.

hana

Tests failing:

  • rtc.alarm_alm_set
  • rtc.alarm_wkalm_set
  • rtc.alarm_alm_set_minute
  • rtc.alarm_wkalm_set_minute

https://linux.kernelci.org/test/case/id/6465b428b21d398af92e85f3/
Log excerpt:

05:14:05.011485  # #  RUN           rtc.alarm_alm_set ...
05:14:05.017370  # # rtctest.c:222:alarm_alm_set:Alarm time now set to 05:14:08.
05:14:05.022052  # # rtctest.c:226:alarm_alm_set:Expected -1 (-1) != rc (-1)
05:14:05.027008  # # alarm_alm_set: Test terminated by assertion
05:14:05.030632  # #          FAIL  rtc.alarm_alm_set
05:14:05.032487  # not ok 5 rtc.alarm_alm_set
05:14:05.035667  # #  RUN           rtc.alarm_wkalm_set ...
05:14:05.043100  # # rtctest.c:284:alarm_wkalm_set:Alarm time now set to 18/05/2023 05:14:08.
05:14:05.047820  # # rtctest.c:290:alarm_wkalm_set:Expected -1 (-1) != rc (-1)
05:14:05.052698  # # alarm_wkalm_set: Test terminated by assertion
05:14:05.056202  # #          FAIL  rtc.alarm_wkalm_set
05:14:05.059321  # not ok 6 rtc.alarm_wkalm_set
05:14:05.063735  # #  RUN           rtc.alarm_alm_set_minute ...
05:14:05.069712  # # rtctest.c:332:alarm_alm_set_minute:Alarm time now set to 05:15:00.
05:14:05.075079  # # rtctest.c:336:alarm_alm_set_minute:Expected -1 (-1) != rc (-1)
05:14:05.080361  # # alarm_alm_set_minute: Test terminated by assertion
05:14:05.084021  # #          FAIL  rtc.alarm_alm_set_minute
05:14:05.087776  # not ok 7 rtc.alarm_alm_set_minute
05:14:05.091594  # #  RUN           rtc.alarm_wkalm_set_minute ...
05:14:05.098967  # # rtctest.c:394:alarm_wkalm_set_minute:Alarm time now set to 18/05/2023 05:15:00.
05:14:05.105018  # # rtctest.c:400:alarm_wkalm_set_minute:Expected -1 (-1) != rc (-1)
05:14:05.110512  # # alarm_wkalm_set_minute: Test terminated by assertion
05:14:05.114746  # #          FAIL  rtc.alarm_wkalm_set_minute
05:14:05.118093  # not ok 8 rtc.alarm_wkalm_set_minute
05:14:05.120380  # # FAILED: 4 / 8 tests passed.

trogdor

Tests failing:

  • rtc.alarm_alm_set_minute
  • rtc.larm_wkalm_set
  • rtc.alarm_wkalm_set_minute

https://linux.kernelci.org/test/case/id/6465aaccb1c5af81fb2e8626/
https://linux.kernelci.org/test/case/id/6465b11a3bb3123b5d2e8611/
Log excerpts:

04:32:01.005975  # # rtctest.c:221:alarm_alm_set:Alarm time now set to 04:32:04.
04:32:03.994100  # # rtctest.c:241:alarm_alm_set:data: 1a0
04:32:03.997347  # #            OK  rtc.alarm_alm_set
04:32:03.997819  # ok 5 rtc.alarm_alm_set
04:32:04.003891  # #  RUN           rtc.alarm_wkalm_set ...
04:32:04.010776  # # rtctest.c:281:alarm_wkalm_set:Alarm time now set to 18/05/2023 04:32:07.
04:32:06.991128  # #            OK  rtc.alarm_wkalm_set
04:32:06.991628  # ok 6 rtc.alarm_wkal�-�et
04:32:06.998003  # #  RUN           rtc.alarm_alm_set_minute ...
04:32:07.004560  # # rtctest.c:331:alarm_alm_set_minute:Alarm time now set to 04:33:00.
04:33:09.064935  # # rtctest.c:343:alarm_alm_set_minute:Expected 0 (0) != rc (0)
04:33:09.071595  # # alarm_alm_set_minute: Test terminated by assertion
04:33:09.074638  # #          FAIL  rtc.alarm_alm_set_minute
04:33:09.077859  # not ok 7 rtc.alarm_alm_set_minute
04:33:09.080989  # #  RUN           rtc.alarm_wkalm_set_minute ...
04:33:09.091043  # # rtctest.c:391:alarm_wkalm_set_minute:Alarm time now set to 18/05/2023 04:34:00.
04:33:59.930842  # #            OK  rtc.alarm_wkalm_set_minute
04:33:59.933897  # ok 8 rtc.alarm_wkalm_set_minute
04:33:59.937003  # # FAILED: 7 / 8 tests passed.

05:00:35.995958  # #            OK  rtc.alarm_alm_set
05:00:35.996040  # ok 5 rtc.alarm_alm_set
05:00:36.002016  # #  RUN           rtc.alarm_wkalm_set ...
05:00:36.008694  # # rtctest.c:284:alarm_wkalm_set:Alarm time now set to 18/05/2023 05:00:39.
05:00:36.011798  # # rtctest.c:290:alarm_wkalm_set:Expected -1 (-1) != rc (-1)
05:00:36.018381  # # alarm_wkalm_set: Test terminated by assertion
05:00:36.022012  # #          FAIL  rtc.larm_wkalm_set
05:00:36.025175  # not ok 6 rtc.alarm_wkalm_set
05:00:36.028309  # #  RUN           rtc.alarm_alm_set_minute ...
05:00:36.035090  # # rtctest.c:332:alarm_alm_set_minute:Alarm time now set to 05:01:00.
05:00:59.978815  # # rtctest.c:351:alarm_alm_set_minute:data: 1a0
05:00:59.982333  # #            OK  rtc.alarm_alm_set_minute
05:00:59.985745  # ok 7 rtc.alarm_alm_set_minute
05:00:59.988553  # #  RUN           rtc.alarm_wkalm_set_minute ...
05:00:59.998754  # # rtctest.c:394:alarm_wkalm_set_minute:Alarm time now set to 18/05/2023 05:02:00.
05:01:00.005453  # # rtctest.c:400:alarm_wkalm_set_minute:Expected -1 (-1) != rc (-1)
05:01:00.008344  # # alarm_wkalm_set_minute: Test terminated by assertion
05:01:00.012096  # #          FAIL  rtc.alarm_wkalm_set_minute
05:01:00.015129  # not ok 8 rtc.alarm_wkalm_set_minute
05:01:00.018301  # # FAILED: 6 / 8 tests passed.
@padovan
Copy link
Contributor

padovan commented Jul 9, 2024

Old issue. (also we are not tracking kernel test failure/issues through GitHub anymore)

@padovan padovan closed this as completed Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@padovan @r-c-n