Semihosting not handled properly by QEMU

[awojasinski]

Describe the bug
I'm implementing backend driver for emulated flash that uses semihosting to utilize file stored on host machine as its flash storage. When I run tests on qemu_cortex_a53 with QEMU from Zephyr SDK 0.16.8 some tests are failing. It looks like next read/write operation is started before the previous one is completed or they are not flushed. Either way content of the file is not changed what causes unexpected values. When I run the same tests with upstream QEMU all tests are passing. The issue hasn't been discovered as semihosting wasn't used extensively in zephyr. This is not a regression. It looks like somewhere between 7.0 used in Zephyr SDK and newest 9.1 upstream version of QEMU the issue has been fixed.

To Reproduce

Checkout to the pull/76639/head:pr_76639 to get emulated flash tests with semihosting backend driver and execute following command:

west build -b qemu_cortex_a53 tests/drivers/flash_simulator/flash_sim_impl -T drivers.flash.flash_simulator.semihost -t run

If you want to run the test suite calling qemu-system-aarch64 directly, use following command:

qemu-system-aarch64 -cpu cortex-a53 -nographic -machine virt,secure=on,gic-version=3 -net none -pidfile qemu.pid -chardev stdio,id=con,mux=on -serial chardev:con -semihosting-config enable=on,target=auto,chardev=con -mon chardev=con,mode=readline -icount shift=4,align=off,sleep=on -rtc clock=vm -kernel build/zephyr/zephyr.elf

Expected behavior

All tests should pass

Impact

This bug stops me from being able to merge the #76639

Logs and console output

Zephyr SDK 0.16.8 (downstream QEMU 7.0)

*** Booting Zephyr OS build v3.7.0-2278-g9aa7f6c3d331 ***
Running TESTSUITE flash_sim_api
===================================================================
START - test_align
 PASS - test_align in 0.001 seconds
===================================================================
START - test_flash_fill

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/flash_simulator/flash_sim_impl/src/main.c:493: flash_sim_api_test_flash_fill: ((uint8_t)buf[i & (sizeof(buf) - 1)] not equal to 0x55)
Unexpected value at offset 0

 FAIL - test_flash_fill in 2.123 seconds
===================================================================
START - test_flash_flatten
 PASS - test_flash_flatten in 1.633 seconds
===================================================================
START - test_get_erase_value
 PASS - test_get_erase_value in 0.001 seconds
===================================================================
START - test_get_mock
 PASS - test_get_mock in 0.001 seconds
===================================================================
START - test_out_of_bounds
 PASS - test_out_of_bounds in 0.001 seconds
===================================================================
START - test_read
 PASS - test_read in 0.590 seconds
===================================================================
START - test_write_read_erase

    Assertion failed at WEST_TOPDIR/zephyr/tests/drivers/flash_simulator/flash_sim_impl/src/main.c:178: test_write_read: (val32 not equal to r_val32)
flash byte at offset 0x0 has value 0xffffffff, expected 0x00000000
 FAIL - test_write_read_erase in 2.082 seconds
===================================================================
TESTSUITE flash_sim_api failed.

------ TESTSUITE SUMMARY START ------

SUITE FAIL -  75.00% [flash_sim_api]: pass = 6, fail = 2, skip = 0, total = 8 duration = 6.432 seconds
 - PASS - [flash_sim_api.test_align] duration = 0.001 seconds
 - FAIL - [flash_sim_api.test_flash_fill] duration = 2.123 seconds
 - PASS - [flash_sim_api.test_flash_flatten] duration = 1.633 seconds
 - PASS - [flash_sim_api.test_get_erase_value] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_get_mock] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_out_of_bounds] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_read] duration = 0.590 seconds
 - FAIL - [flash_sim_api.test_write_read_erase] duration = 2.082 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION FAILED
QEMU: Terminated

upstream QEMU 9.1

*** Booting Zephyr OS build v3.7.0-2278-g9aa7f6c3d331 ***
Running TESTSUITE flash_sim_api
===================================================================
START - test_align
 PASS - test_align in 0.001 seconds
===================================================================
START - test_flash_fill
 PASS - test_flash_fill in 3.222 seconds
===================================================================
START - test_flash_flatten
 PASS - test_flash_flatten in 1.633 seconds
===================================================================
START - test_get_erase_value
 PASS - test_get_erase_value in 0.001 seconds
===================================================================
START - test_get_mock
 PASS - test_get_mock in 0.001 seconds
===================================================================
START - test_out_of_bounds
 PASS - test_out_of_bounds in 0.001 seconds
===================================================================
START - test_read
 PASS - test_read in 0.590 seconds
===================================================================
START - test_write_read_erase
Incremental pattern expected
Erased area expected
Incremental pattern expected
 PASS - test_write_read_erase in 2.697 seconds
===================================================================
TESTSUITE flash_sim_api succeeded

------ TESTSUITE SUMMARY START ------

SUITE PASS - 100.00% [flash_sim_api]: pass = 8, fail = 0, skip = 0, total = 8 duration = 8.146 seconds
 - PASS - [flash_sim_api.test_align] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_flash_fill] duration = 3.222 seconds
 - PASS - [flash_sim_api.test_flash_flatten] duration = 1.633 seconds
 - PASS - [flash_sim_api.test_get_erase_value] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_get_mock] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_out_of_bounds] duration = 0.001 seconds
 - PASS - [flash_sim_api.test_read] duration = 0.590 seconds
 - PASS - [flash_sim_api.test_write_read_erase] duration = 2.697 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION SUCCESSFUL
QEMU: Terminated

Environment (please complete the following information):

• OS: Linux (docker zephyrprojectrtos/zephyr:latest)
• Toolchain: Zephyr SDK 0.16.8
• drivers: flash_ flash_simulator: Add semihosting implementation #76639

[awojasinski]

I'm wondering whether it's better to update qemu fork to the newest version or try to apply patches from upstream on current version to fix this. Getting right patches is not trivial as there are many changes that depends on each other

[stephanosio]

I'm wondering whether it's better to update qemu fork to the newest version or try to apply patches from upstream on current version to fix this. Getting right patches is not trivial as there are many changes that depends on each other

Upgrade to QEMU 9.1 is planned for some time in 2025; so, I would say it would be better to identify and backport the QEMU patch that fixed this issue to the Zephyr QEMU fork.

[nashif]

@stephanosio

Upgrade to QEMU 9.1 is planned for some time in 2025;

why can't this be done before 2025? What is the holdup? Can we make this move faster? Let me know if anything is needed to make this go faster.

[stephanosio]

@stephanosio

Upgrade to QEMU 9.1 is planned for some time in 2025;

why can't this be done before 2025? What is the holdup? Can we make this move faster? Let me know if anything is needed to make this go faster.

Reworking all the existing patches in our fork (there are many) and making sure QEMU 9 is "compatible" with our CI (upgrading QEMU version has been quite painful in the past due to all sorts of breakages in the CI) takes a lot of work. I would prefer to tackle this alongside the rest of toolchain upgrade tasks.