No way to detect if native driver is ready

[dennwc]

The problem

Currently, the native protocol only allows Parse requests/responses. When the Go driver server starts the native driver, it has no way of knowing if the driver is ready to accept requests. Because of this, driver server cannot tell bblfshd if it's ready or not.

This lead to a situation where we cannot distinguish a real request timeouts from "driver not ready" kind of timeouts and bblfshd cannot dynamically add drivers to the pool when the driver is actually ready.

Proposal

1. Expose an health-check endpoint in Go driver server. bblfshd will be able to use it to check the status of the driver.

2. Change the native driver protocol to include a short "handshake". This way the driver server will know when native driver is ready.

Details

Health check

There is a standard way of defining a health-check for gRPC endpoints.

All we need is to enable it in the driver server. But to serve anything useful we will need additional information about the native driver's state.

Native protocol

To enable the handshake we will have to break the protocol compatibility. Luckily enough, we are prepared for this.

Each driver has a impl.go file that controls how exactly the driver interface is satisfied. For Java it runs the native driver assuming UTF8 encoding. For Go it serves results directly from the driver server, avoiding the protocol overhead.

Essentially, this file controls what native protocol the driver uses. And this file is not managed by SDK, so it can be changed separately in each driver. Thus, we can safely define native.NewDriverV2 in SDK and let drivers to switch to it when we decide it's time to.

I propose to leave the handshake to use the same JSON-lines protocol as we use right now. It may be as simple as:

native>  {"vers": 1, "status": "ok"}

It's not a real handshake since there is no response, but I think it's all we need.

The native driver advertises the maximal protocol version it supports, and the server will either continue, or kill the driver. Also, the native driver has a chance to send an error if something is not quite right.

This will allow us to detect when the native driver's runtime is up and running, as well as leaves some space for protocol improvements in the future.

[dennwc]

As discussed in Slack, the other option may be to use JSON-RPC v2 on the wire. The protocol is dead simple and can be served serially.

[mcuadros]

As I discussed with Michael, JSON-RPC sounds like a perfect solution for the communication between the native driver and the drivers. But we should do in a way that the adoption can be done incrementally

[creachadair]

[W]e should do in a way that the adoption can be done incrementally

Yes, agreed. I think if we start by making the daemon support both the existing protocol and the new one, then we can migrate each driver incrementally to use the new protocol as time allows, and that way things will continue working at all times. (Those drivers not yet migrated will still suffer from this issue, but we can focus on the higher-priority ones first)

[bzz]

JFYI, from the discussion elsewhere I noticed that this might start to be an impediment for C# clients soon.