You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running model tests cause Fatal Python Error on WH. After that, tt-smi reset was needed.
To reproduce
Checkout the latest main at the time of creating this issue
Run command:
pytest tests/models/unet_brain/test_unet_brain.py
Output:
platform linux -- Python 3.8.10, pytest-7.2.2, pluggy-1.5.0
rootdir: /home/ubuntu/pytorch2.0_ttnn/tests, configfile: pytest.ini
plugins: dash-2.15.0, anyio-4.5.2, split-0.8.2, timeout-2.2.0, xdist-3.6.1
collected 2 items
tests/models/unet_brain/test_unet_brain.py Fatal Python error: Aborted
Current thread 0x00007fa5f09dc740 (most recent call first):
File "/home/ubuntu/pytorch2.0_ttnn/tests/conftest.py", line 63 in device
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 901 in call_fixture_func
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 1129 in pytest_fixture_setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 1075 in execute
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 677 in _compute_fixture_value
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 591 in _get_active_fixturedef
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 569 in getfixturevalue
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 550 in _fillfixtures
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1792 in setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 492 in setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 155 in pytest_runtest_setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 125 in runtestprotocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/init.py", line 167 in main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/init.py", line 190 in console_main
File "/home/ubuntu/tt-metal/python_env/bin/pytest", line 8 in
Aborted (core dumped)
The text was updated successfully, but these errors were encountered:
Running model tests cause Fatal Python Error on WH. After that, tt-smi reset was needed.
To reproduce
Output:
The text was updated successfully, but these errors were encountered: