Architecture

High level

Components

Query Path

Query will arrive at the Aggregator, which will then fan-out the request to all Index Server's, since data is doc-sharded.

Index Server will retrieve data from the Cold Storage. In the first version, we will use local disk as cold storage, but we will migrate that to S3.

Indexing Path

Index Writer will read raw data from Raw Cold Storage, build the index, and put it in the Cold Storage for Index Server to fetch.

Index

In the first version, we will only implement IVF-HNSW-PQ index. That means:

• Every request arriving at the IndexServer, we will find the sub-partition for that query using IVF.
• With that sub-partition, we will arrive at the HNSW graph.
• We will also transform the query with product quantization. Then we will traverse the HNSW graph to find the nearest neighbors.