Out of dataset answer and reference link provided for RAG example

[aiquick]

Is this a new bug?

• I believe this is a new bug
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm using the RAG example and feeding my own database of 1 football article.
The Pinecone DB is a brand new database and only contains vectors from the football article.

When I do qa_with_sources(query="Who is Sachin Tendulkar") it provides me an answer and a link as a reference. This is not the expected behavior.

I have not fed any article about Sachin Tendulkar to the database. How and why/where from is it getting the answer and the link?

Now, If I add more articles only about football, push the vector count in the database to around 90. And then I ask the same question, query="Who is Sachin Tendulkar", it is not able to give the answer, which is the expected behavior.

I wonder if the fullness of the vector db makes it more accurate? Has anyone else seen this?

Expected Behavior

Since the database does not contain any article or mention of Sachin Tendulkar, it should not provide any answer, and instead say "This is not mentioned in the database".

Steps To Reproduce

Create a new Vector DB on pinecone. Use this example to feed in a football article.

Run query="Who is Sachin Tendulkar". Note the result contains a reference and an answer. (Unexpected)

Now, create a more full db, with more articles and ask the same query. Note that the results is empty as expected.

Relevant log output

Answer is as above

Environment

No response

Additional Context

No response

[zackproser]

Hi @aiquick - thanks for filing an issue and for the detailed write-up.

Can I ask if you'd be willing to provide us with a minimal reproduction via a Python notebook on GitHub?

That might help us drill into the behavior you're seeing more efficiently.

Best,
Zack