GRPC Document API benchmark

[amberzsy]

Please describe the end goal of this project

based on GRPC document API implementation, perform benchmark on varies document apis with different scenarios.

Supporting References

#15190

Issues

• Bulk API benchmark.
• Index document benchmark.
• Update document benchmark
• Get document benchmark.

Related component

Indexing

[amberzsy]

assigned to @karenyrx.

[dblock]

[Catch All Triage - 1, 2, 3]

[karenyrx]

Results

Latency benchmark results from testing with a single data-node cluster and small GRPC payload:

Summary

1. GRPC latency of the first request is 1.2 - 1.25x higher than HTTP.
2. GRPC latency of the subsequent requests is 1.5 - 5x lower than HTTP.
3. For both GRPC and HTTP, CPU, memory, IO usage were similar, based on internal dashboard observations.

Next steps:

1. Confirm where the improvements are arising from: HTTP2, or Protobuf, or something else. To do so:
   i) Instrument codebase with detailed metrics to breakdown granular latency of GRPC vs. HTTP internal ops.
   ii) Confirm if results align with HTTP2 + JSON benchmarking results (to be performed by @reta )
2. Test with a larger payload (e.g. 5MB request) to gain more confidence
3. Submit PR for Protobuf and GRPC support for Bulk endpoint in Opensearch (in parallel with step 1 and 2)

[mgodwan]

@karenyrx Thanks for the benchmarks. The idea is indeed promising.
I would like to understand and discuss the proposal around how you're thinking to map document schema using protobuf, and how dynamic mappings would work for such scenarios.