Add guided decoding to TGIS gRPC API

[njhill]

Within the existing decoding request parameter section:

enum ResponseFormat {
  // Plain text, no constraints
  TEXT = 0;
  // Valid json
  JSON = 1;
}

message StringChoices {
  repeated string choices = 1;
}

// Mutually-exclusive guided decoding options
oneof guided {
  // Output will be in the specified format
  ResponseFormat format = 3;
  // Output will follow the provided JSON schema
  string json_schema = 4;
  // Output will follow the provided regex pattern
  string regex = 5;
  // Output will be exactly one of the specified choices
  StringChoices choice = 6;
  // Output will follow the provided context free grammar
  string grammar = 7;
}

[njhill]

Unfortunately you cannot have repeated fields directly within oneofs :(

protocolbuffers/protobuf#2592 (comment)

[joerunde]

I haven't looked much at logits processors, why does this require its own thread pool?

[maxdebayser]

It's the same code as here:

vllm/vllm/model_executor/guided_decoding/outlines_decoding.py

Line 71 in 4af59d3

global_thread_pool = concurrent.futures.ThreadPoolExecutor(

. If I'm not mistaken, only the construction of the logits processor happens in another thread. But if the logits processor is cached, I'm not sure what's the benefit of having another thread build the object.

[njhill]

Yes that's right. The code is just the same as that in the http API. It's dispatched to a threadpool to avoid blocking the asyncio event loop, but I think it could be made more efficient since we only care about this in the case that the LP is not already cached. In any case we can fix that as a follow-on since we need to fix that related concurrency bug anyhow.

[maxdebayser]

I've seen versions of the code where the get_tokenizer_group function exists and others where it doesn't. What's happening with this function?

[joerunde]

@maxdebayser that's from this upstream PR vllm-project/vllm#3512

It didn't get merged in a timely manner and is now buried in conflicts :(

[maxdebayser]

Since the bug reported in issue https://github.ibm.com/ai-foundation/fmaas-inference-server/issues/718 is not cause by the code in this PR, I think we can merge it and fix the problem in another PR.