Architecture

High level

Components

Query Path

Query will arrive at the Aggregator, which will then fan-out the request to all Index Server's, since data is doc-sharded.

Index Server will retrieve data from the Cold Storage. In the first version, we will use local disk as cold storage, but we will migrate that to S3.

Indexing Path

Index Writer will read raw data from Raw Cold Storage, build the index, and put it in the Cold Storage for Index Server to fetch.