Agent's tests are failing with ScopeMismatch

[apryiomka]

System Info

WIndows/WSL Ubuntu 22.xx

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

Trying to run tests in v0.0.61:

pytest -s -m remote llama_stack/providers/tests/agents/test_agents.py --env REMOTE_ STACK_URL=$REMOTE_STACK_URL --env INFERENCE_MODEL=$INFERENCE_MODEL

Getting the error:

Error logs

================================================= test session starts ==================================================
platform linux -- Python 3.12.8, pytest-7.4.0, pluggy-1.5.0
rootdir: /meta/llama-stack
plugins: asyncio-0.21.0, anyio-4.7.0
asyncio: mode=Mode.STRICT
collected 25 items / 20 deselected / 5 selected

llama_stack/providers/tests/agents/test_agents.py EEEEE

======================================================== ERRORS ========================================================
_________________ ERROR at setup of TestAgents.test_agent_turns_with_safety[--remote][remote] _________________
ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object, involved factories:
llama_stack/providers/tests/agents/fixtures.py:61: def agents_stack(request, inference_model, safety_shield)
____________________ ERROR at setup of TestAgents.test_create_agent_turn[--remote][remote] ____________________
ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object, involved factories:
llama_stack/providers/tests/agents/fixtures.py:61: def agents_stack(request, inference_model, safety_shield)
________________ ERROR at setup of TestAgents.test_rag_agent_as_attachments[--remote][remote] _________________
ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object, involved factories:
llama_stack/providers/tests/agents/fixtures.py:61: def agents_stack(request, inference_model, safety_shield)
___________ ERROR at setup of TestAgents.test_create_agent_turn_with_brave_search[--remote][remote] ___________
ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object, involved factories:
llama_stack/providers/tests/agents/fixtures.py:61: def agents_stack(request, inference_model, safety_shield)
__________ ERROR at setup of TestAgents.test_create_agent_turn_with_tavily_search[--remote][remote] ___________
ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object, involved factories:
llama_stack/providers/tests/agents/fixtures.py:61: def agents_stack(request, inference_model, safety_shield)
=============================================== short test summary info ================================================
ERROR llama_stack/providers/tests/agents/test_agents.py::TestAgents::test_agent_turns_with_safety[--remote] - Failed: ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request obj...
ERROR llama_stack/providers/tests/agents/test_agents.py::TestAgents::test_create_agent_turn[--remote] - Failed: ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request obj...
ERROR llama_stack/providers/tests/agents/test_agents.py::TestAgents::test_rag_agent_as_attachments[--remote] - Failed: ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request obj...
ERROR llama_stack/providers/tests/agents/test_agents.py::TestAgents::test_create_agent_turn_with_brave_search[--remote] - Failed: ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request obj...
ERROR llama_stack/providers/tests/agents/test_agents.py::TestAgents::test_create_agent_turn_with_tavily_search[--remote] - Failed: ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request obj

Expected behavior

The tests run

[ashwinb]

I am somehow not able to reproduce this and might be a Windows issue. Llama stack's Windows support is quite sad right now.

[apryiomka]

this is not windows, this is linux/WSL. BTW, do you have a docker image just for the tests?

[apryiomka]

This is the Dockerfile, reproducible 100%:

FROM python:3.12

RUN apt update

WORKDIR /meta

RUN git clone https://github.com/meta-llama/llama-stack.git && cd llama-stack && git checkout tags/v0.0.61

WORKDIR /meta/llama-stack

RUN pip install -r requirements.txt

COPY test-requirements.txt .

RUN pip install -r test-requirements.txt

ENTRYPOINT ["pytest", "-s", "-m", "remote", "llama_stack/providers/tests/agents/test_agents.py"]

This is the test-requirements.txt

pytest==7.4.0
pytest-asyncio==0.21.0
llama-models==0.0.61