Non Uniform Performance with GPU - Indexing running over days even with GPU.

[rohitreddy1698]

Hi,

I was trying to run some tests on GPU vs CPU Index performance.
I am trying to use the Zilliz Vector DB Bench Tool to run the benchmarking and capture performance metrics.

I have GPU Cluster setup on GKE :

Index Nodes : 3 n1-highmem-8 , nodes with 4 GPUs per node as index nodes :

4 NVIDIA T4 GPUs per node - 16* 4 GiB Memory = 64 GiB

index_node_replicas: 3
index_node_cpu_limit: 6
index_node_gpu_limit: 4
index_node_memory_limit: 48Gi
index_node_cpu_request: 3
index_node_gpu_request: 4
index_node_memory_request: 24Gi

1 ) When I run GPU IVF PQ for 100M Vectors 768 D with parameters :
nlist=10000, m=64 , n=16 - Is running over days. And ended in timeout.
On Index status is being build :

>>> 
>>> utility.load_state(COLLECTION_NAME)
<LoadState: NotLoad>
>>> utility.index_building_progress(COLLECTION_NAME)
{'total_rows': 100000000, 'indexed_rows': 0, 'pending_index_rows': 100000000, 'state': 'InProgress'}

>>> utility.wait_for_index_building_complete(COLLECTION_NAME) - BLOCKING , showing that index build is progress.

But on Grafana GPU Utilisation is show as 0. Attaching Image for reference.

2. Also is there a way to increase the Recall : with nlist=10000, m=32 , n=16 - recall is 31.42

For GPU CAGRA for same 100M Vectors 768 D dataset
Screenshot for parameters used.
The Indexing time is 5x more as compared to HNSW

Load Duration Output from Zilliz Vector DB Bench Tool :
GPU CAGRA : 195,700 s
HNSW : 37,292 s

The Same observed even for 10 M dataset :
GPU CAGRA : 32220 s
HNSW : 17430 s

But as per blogs from Milvus and in open source blogs I think CAGRA should be performing better.

Can you please help us on this.

Regards,
Rohit Mothe

[yhmo]

1. {'total_rows': 100000000, 'indexed_rows': 0, 'pending_index_rows': 100000000, 'state': 'InProgress'} ------------------------ I believe there must be errors in log of the index nodes, all the index tasks failed so the indexed_rows is zero.

2. nlist=10000, m=32 , n=16 - recall is 31.42 --------------------- I guess the ground-truth is wrong or something else. Wait @alwayslove2013 comment

3. Wait @Presburger comment

[Presburger]

@rohitreddy1698 Hi, please replace the IVF_PQ in the build algorithm with NN_DESCENT. IVF_PQ performs a brute-force search-like method to construct the adjacency list, which can be very time-consuming on cards like the T4.