index not found[collection=docs_embeddings]

[matsuo1234567]

I have created a program based on this page.
https://zilliz.com/learn/Exploring-BGE-M3-the-future-of-information-retrieval-with-milvus

zilliz.py

from FlagEmbedding import BGEM3FlagModel
import pymilvus
import numpy as np
from pymilvus import (
    connections,
    FieldSchema,
    CollectionSchema,
    DataType,
    Collection,
    utility,
)

connections.connect("default", host="127.0.0.1", port="19530")

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1024),
]

schema = CollectionSchema(fields, description="Document Embeddings Collection")
collection = Collection(name="docs_embeddings", schema=schema)


collection.load()

if utility.has_collection(collection_name):
    collection.release()


if collection.has_index():
    collection.drop_index()


index_params = {"index_type": "IVF_FLAT", "metric_type": "L2", "params": {"nlist": 128}}
collection.create_index("embedding", index_params)
print("New index created successfully.")


collection.load()


from pymilvus.model.hybrid import BGEM3EmbeddingFunction

bge_m3 = BGEM3EmbeddingFunction(model_name="BAAI/bge-m3", device="cpu", use_fp16=False)

documents = [
    "Climate change is a significant global issue.",
    "El cambio climático es un problema global significativo.",
    "气候变化是一个重大的全球问题。",
]


docs_embeddings = bge_m3.encode_documents(documents)
entities = [{"embedding": doc.tolist()} for doc in docs_embeddings["dense"]]
insert_result = collection.insert(entities)


collection.load()


search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
query_embeddings = docs_embeddings["dense"][2:3]
result = collection.search(
    query_embeddings, "embedding", search_params, limit=6, output_fields=["id"]
)

for hits in result:
    for hit in hits:
        print(f"hit: {hit}, id: {hit.id}")

When I run it, I get the following error.

% python zilliz.py
RPC error: [load_collection], <MilvusException: (code=700, message=index not found[collection=docs_embeddings])>, <Time:{'RPC start': '2024-09-11 16:15:35.363838', 'RPC error': '2024-09-11 16:15:35.368702'}>
Traceback (most recent call last):
  File "/Users/dev/milvus-lite/QAbot-lite/zilliz.py", line 32, in <module>
    collection.load()
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 425, in load
    conn.load_collection(
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 148, in handler
    raise e from e
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 144, in handler
    return func(*args, **kwargs)
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 183, in handler
    return func(self, *args, **kwargs)
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 123, in handler
    raise e from e
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 87, in handler
    return func(*args, **kwargs)
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 1160, in load_collection
    check_status(response)
  File "/Users/dev/milvus-lite/QAbot-lite/qa_venv/lib/python3.10/site-packages/pymilvus/client/utils.py", line 63, in check_status
    raise MilvusException(status.code, status.reason, status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=700, message=index not found[collection=docs_embeddings])>

[yhmo]

You call collection.load() twice, one is before collection.create_index(), the other is after collection.create_index(). The error is returned by the first call of collection.load(). Remove the first call and try again.

[matsuo1234567]

Resolved. Thank you.

But not output

print(f"hit: {hit}, id: {hit.id}")

[yhmo]

Set consistency_level="Strong" then you will see the results:

result = collection.search(
    query_embeddings, "embedding", search_params, limit=6, output_fields=["id"],
    consistency_level="Strong"
)

consistency_level is to control the data visibility for search/query. Read the doc for more info: https://milvus.io/docs/consistency.md#Consistency-levels

[matsuo1234567]

Thank you very much.

[cigdegoktas]

With collection.num_entities, you can check if your collection has any indexes. If it returns 0, it means there are no indexes in your collection. This code will return 0 in your case because in your collection indexes have not been saved.

The main cause of that problem is that the fields list and the entities list are not of the same size. You have created both the id and embedding fields, but you are only inserting the embedding field. You need to insert the id field as well. Once you do that, the index insertion problem will be resolved.