Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ollama ChatClientIntegrationTests.CompleteAsync_StructuredOutputEnum fails locally #5570

Closed
stephentoub opened this issue Oct 25, 2024 · 1 comment · Fixed by #5731
Closed

Comments

@stephentoub
Copy link
Member

 Microsoft.Extensions.AI.OllamaChatClientIntegrationTests.CompleteAsync_StructuredOutputEnum

  Message: 
Assert.Equal() Failure: Values differ
Expected: Arm64
Actual:   Arm

  Stack Trace: 
ChatClientIntegrationTests.CompleteAsync_StructuredOutputEnum() line 665
--- End of stack trace from previous location ---

cc: @SteveSandersonMS

@SteveSandersonMS
Copy link
Member

I ran this with "Run until failure" in VS and got 4 passes then a failure:

Image

To be honest I think this is a fair pass rate given the way the test is set up. M2 chips are ARM-based, even though ARM64 is a more precise statement. It's totally reasonable for this model to return ARM instead of ARM64 sometimes.

Given that what we want from our tests is a reliable signal about whether things are working or not, I'll update the test so that it's a less ambiguous test case by defining a different enum type.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants