questions on indexing_process

[vihariazure]

I have the following understanding after reading the articles:

The inserted data is stored in segments. If a segment is sealed, an index is created for it. Each index is independent of the segment.
In the background, compaction takes place, which combines smaller segments into larger ones.
I inserted 1M records, and here is the output of the describe index command:

{
'm': '16',
'ef_construct': '100',
'index_type': 'HNSW',
'metric_type': 'L2',
'field_name': 'product_embeddings',
'index_name': 'milvus_index_uuid_trail_2',
'total_rows': 990000,
'indexed_rows': 297500,
'pending_index_rows': 897500,
'state': 'Finished'
}
The insertion of 1M records took 416 seconds with batch size 2500

I have the following questions:

1. When the state is marked as Finished, does it mean the index is built for all 1M records?
2. What do the pending_index_rows represent? Are they entities waiting for indexing?
3. Do the pending_index_rows come from non-sealed segments, segments undergoing compaction, or both?
4. If the state is Finished, can I proceed with a search? Will there be any drop in accuracy due to the presence of pending_index_rows (e.g., 100K rows)?
5. After waiting for 1 hour and 10 minutes, the pending_index_rows count became zero. If I perform a search now, will the results differ from a search performed earlier (when pending_index_rows was non-zero)?
6. Is there a drop in search accuracy when pending_index_rows > 0 (e.g., 500K) compared to when pending_index_rows = 0?
7. Does compaction eventually combine all segments into a single global index?
8. When the state is Finished, does it mean indices are built only for sealed segments? During compaction, when segments (e.g., Segment A and Segment B) are merged into a new segment (e.g., Segment C), are the older segments (A and B) deleted after Segment C is created? Does this process continue until there is only one global index?
9. In my case, is the total index-building time 416 seconds, or is it 416 seconds + 1 hour 10 minutes (including the time to index pending rows)?
10. If the state is Finished and I have pending_index_rows = 897500 and indexed_rows = 297,500, does this mean the search will only consider the indexed rows, ignoring the pending rows? In this case, does this result in a drop in search accuracy?
11. the ideal senario for search to get maximum accuracy is pending_index_rows=0
12. if total_rows and indexed_rows is same and status is finished. and there are some pending_index_rows count then what is the accuracy in this case

[yhmo]

1. When the state is marked as Finished, does it mean the index is built for all 1M records?
   At the time point you call the create_index(), if sealed segments contain 297500 rows, the index state will be "Finished" after the 297500 rows are indexed. But don't worry, the index tasks will continue untuil all the 1M rows are sealed and indexed.

To make sure the 1M rows are sealed, you can call flush() after all the 1M rows are inserted.

for i in range(100):
  collection.insert()
collection.flush()
collection.create_index()
print(utility.index_building_progress(collection_name=collection_name))

2. What do the pending_index_rows represent? Are they entities waiting for indexing?
   pending_index_rows is the row number that waiting for indexing.
   Note that compation task can generate new sealed segments, new sealed segments will increase the pending_index_rows
   For example, at the begining, we have 2 sealed segments A and B, A has 10000 rows, B ahs 2000 rows. The pending_index_rows is 10000+20000=30000. Let's say the index node is busy, A and B are waiting for indexing. A compaction task merges A and B to C, which has 30000 rows. The C also need to be indexed so the pending_index_rows become 30000+30000=60000. So, sometimes you will see pending_index_rows and indexed_rows larger than total_rows.

3. Do the pending_index_rows come from non-sealed segments, segments undergoing compaction, or both?
   pending_index_rows always come from sealed segments. Sealed segments come from flush or compaction.

4. If the state is Finished, can I proceed with a search? Will there be any drop in accuracy due to the presence of pending_index_rows (e.g., 100K rows)?
   Yes, if index state is finished, you can do search. Actually, even index state is not finished, you also can conduct search because milvus has temp index for growing segments(if temp index is disabled, growing segments will be brute-search). Since both sealed segments and growing segments can be searched, the accuracy can be ensure.

5. After waiting for 1 hour and 10 minutes, the pending_index_rows count became zero. If I perform a search now, will the results differ from a search performed earlier (when pending_index_rows was non-zero)?
   The recall might be a bit different between "full index search" and "brute-search on growing segment", in most cases, the difference can be ignored.

6. Is there a drop in search accuracy when pending_index_rows > 0 (e.g., 500K) compared to when pending_index_rows = 0?
   The same answer with 5. A bit difference can be ignored.

7. Does compaction eventually combine all segments into a single global index?
   Compaction eventually combine segemnts to be size of 1Gb. If your data exceed 1Gb, there should be one or two sealed segments, each segment has an independent index.
   If your data size is 100GB, there might be around 100 sealed segments, each segment has an independent index.

8. When the state is Finished, does it mean indices are built only for sealed segments? During compaction, when segments (e.g., Segment A and Segment B) are merged into a new segment (e.g., Segment C), are the older segments (A and B) deleted after Segment C is created? Does this process continue until there is only one global index?
   "Finished" means the sealed segments at the time point you call create_index() have been indexed. There are might be new sealed segments generated later, never mind, they will be indexed eventually.
   After A and B are merged to C, the A and B will be marked as "soft-delete", wait GC process to clean.
   No global index, always, each segment has an independent index.

9. In my case, is the total index-building time 416 seconds, or is it 416 seconds + 1 hour 10 minutes (including the time to index pending rows)?
   The total index-building time is 416 seconds + 1 hour 10 minutes.

10. If the state is Finished and I have pending_index_rows = 897500 and indexed_rows = 297,500, does this mean the search will only consider the indexed rows, ignoring the pending rows? In this case, does this result in a drop in search accuracy?
    Answer is in 4,5,6

11. the ideal senario for search to get maximum accuracy is pending_index_rows=0
    To get max accuracy, you can choose "FLAT" index, which is brute-search for the entire collection.

12. if total_rows and indexed_rows is same and status is finished. and there are some pending_index_rows count then what is the accuracy in this case
    Answer is in 4,5,6