Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model tests cause Fatal Python Error #751

Open
amalbasaTT opened this issue Feb 11, 2025 · 0 comments
Open

Model tests cause Fatal Python Error #751

amalbasaTT opened this issue Feb 11, 2025 · 0 comments

Comments

@amalbasaTT
Copy link

amalbasaTT commented Feb 11, 2025

Running model tests cause Fatal Python Error on WH. After that, tt-smi reset was needed.

To reproduce

  1. Checkout the latest main at the time of creating this issue
  2. Run command:
pytest tests/models/unet_brain/test_unet_brain.py

Output:

platform linux -- Python 3.8.10, pytest-7.2.2, pluggy-1.5.0
rootdir: /home/ubuntu/pytorch2.0_ttnn/tests, configfile: pytest.ini
plugins: dash-2.15.0, anyio-4.5.2, split-0.8.2, timeout-2.2.0, xdist-3.6.1
collected 2 items
tests/models/unet_brain/test_unet_brain.py Fatal Python error: Aborted
Current thread 0x00007fa5f09dc740 (most recent call first):
File "/home/ubuntu/pytorch2.0_ttnn/tests/conftest.py", line 63 in device
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 901 in call_fixture_func
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 1129 in pytest_fixture_setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 1075 in execute
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 677 in _compute_fixture_value
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 591 in _get_active_fixturedef
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 569 in getfixturevalue
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/fixtures.py", line 550 in _fillfixtures
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1792 in setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 492 in setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 155 in pytest_runtest_setup
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 125 in runtestprotocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/init.py", line 167 in main
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/init.py", line 190 in console_main
File "/home/ubuntu/tt-metal/python_env/bin/pytest", line 8 in
Aborted (core dumped)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant