Smithy GPTool

[Quafadas]

An idea. OpenAI recently built calling third party tools, right into their API.
https://openai.com/blog/function-calling-and-other-api-updates

For chatGPT to be really useful for me, I need to be able to tell it about my domain, and lots of my domain data is in private APIs… that I now have smithy definitions for :-). So I begin to wonder, if I can take advantage of all this somehow. What I realised that an awful lot of what I would need do, is already in Smithy in some form or another.

Here’s the notion - currently, I use two “targets“ provided by @simpleRestJson: X extends SmithyService[IO], and SimpleRestJsonBuilder.apply(SmithyService)

The idea, would be to add a third “target” let’s call it GPTool. Instantiated like val gptTool = GPTool.apply(List(anInstanceOfASmithyServce)) it exposes two methods:

gptTool.toolJson - generates a description of the JSON Schema to feed to the AI, telling it about the list of tools (endpoints) defined in the smithy specs. As smithy produces openAI specs already… in my head the heavy lifting is already done? It would be simply a question of extracting a subset?
https://platform.openai.com/docs/api-reference/chat/create#chat/create-functions

And gptTool.functionCall() which, given the JSON constructed by the LLM, calls our actual scala implementation of the service. I think this part would be a (relatively hides) simple problem. At heart JSON => JSON with a pattern match on the method name given by the LLM ...

So… my questions.

Is here an obvious “This isn’t going to work because X”, “this is a horrid idea” type problem with this?

The big one I see, is the “description” part of openAIs examples. Which I think would require something in the smithy specs themselves. On a Hail Mary: is there a “description” annotation already :-)? In my head, this might be a nasty “might need changes to smithy4s itself which are above my pay grade” kind of problem. So in a first pass… I might skip it (i.e leave them blank) and see if there's any value here first. I think this is the most awkward problem I saw.

I assume, that this is a bit niche for the “main” smithy repo itself due to maintenance concerns etc ? Would the published smithy artefact, expose enough functionality to be able to implement something like this on top of it downstream?

If the answer to all the above, is yes, has anyone followed enough to set out a more detailed solution sketch … i.e. put my feet on the path? I'll gladly answer more questions on the the thought process if anyone is interested enough to have them!

I’ve been lurking the discord … would it be worth waiting for 0.18, to attempt such a thing?

Is anyone interested enough to want to have a go, at shaving this yak with me :-)?

[Baccata]

in my head the heavy lifting is already done? It would be simply a question of extracting a subset?

More or less : the function that generates openapi is a build-time function that operates at the level of the software.amazon.smithy:smithy-model java library. That smithy-model library also has building blocks that do the heavy lifting to generate json-schema instead of full-blown openapi. Although there's a little bit of nuance (smithy-model doesn't know about alloy traits, like @discriminated or @untagged for unions), we've decided to use this instead of building our own run-time json-schema/openapi translation mechanism. We'd love to have this mechanism (ie : JsonSchema.fromSchema(smithy4sSchema)), but it's not a priority. We'd happily host it somebody wanted to contribute it ;)

Is here an obvious “This isn’t going to work because X”, “this is a horrid idea” type problem with this?

I actually think it's a good idea ! And also pretty straightforward to do, relatively to, say, implementing RESTful protocols like simpleRestJson.

The big one I see, is the “description” part of openAIs examples. Which I think would require something in the smithy specs themselves

There's the @documentation trait which may fit this role. Alternatively you can create your own traits in smithy land, and you'll be able to retrieve the corresponding values in Scala without any change to Smithy4s :) we've architected Smithy4s in order for users to be able to easily retrieve the values to their own traits.

I assume, that this is a bit niche for the “main” smithy repo itself due to maintenance concerns etc ? Would the published smithy artefact, expose enough functionality to be able to implement something like this on top of it downstream?

It absolutely does : we're very much making all of our decisions to ensure that downstreams are able to implement protocols on top of the constructs of smithy4s-core. Nothing in the generated code of smithy4s is protocol specific, nor is it biased towards any usecase. The generated code contains instances of the abstractions you need to implement the usecase you're describing :)

I’ve been lurking the discord … would it be worth waiting for 0.18, to attempt such a thing?

I don't think so : although we're breaking a few things, overall the stuff you need to build this functionality is not gonna be greatly impacted. Alternatively, we have snapshots of the 0.18 branch if you want to develop against the bleeding-edge. It may be easier to give you a hand if you do, as our heads are very much in 0.18 at the moment.

Latest snapshot we've released is 0.18.0-528-d32916a 

If the answer to all the above, is yes, has anyone followed enough to set out a more detailed solution sketch … i.e. put my feet on the path? I'll gladly answer more questions on the the thought process if anyone is interested enough to have them!

Yes, click here. This is a toy implementation of client/server interpreters for a Json => F[Json] protocol, where Json is smithy4s.Document. We use it for some tests.

The "server" essentially responds to requests that look like this :

{
   "OperationName" : JsonEncodedInput
} 

so if you have :

operation Foo {
    input: = {
       @required
       str: String
       @required 
       num: Integer
    } 
} 

then the following would be a valid request :

{
  "Foo" : { 
    "str": "hello", 
    "num": 1  
  }   
}

This should be sufficient to get you started :)

[Quafadas]

@Baccata I spent some time yesterday, sketching out the interactions with openAIs API - basically following their examples.
https://github.com/Quafadas/smithy-call-tool/blob/main/src/main/scala/callOpenAI.scala

I've realised, that there is some complexity in the control flow - but my view is that would clearly be "out of scope" of what I'm trying to achieve with smithy. I think it should be left to the caller. My goal here is only to enable the function calling.

Concretely, my goal would be to abstract this part with a call to newSmithyTool:
https://github.com/Quafadas/smithy-call-tool/blob/acbd15b0e6b84fd8802cdec45c7262ff93f533cc/src/main/scala/callOpenAI.scala#L37

And replace this "fakeFct" with the dispatcher you pointed me at above.
https://github.com/Quafadas/smithy-call-tool/blob/acbd15b0e6b84fd8802cdec45c7262ff93f533cc/src/main/scala/callOpenAI.scala#L99

Planned Path To Victory:

• write a tiny smithy weather service (nick it from Amazos examples) and publish local. Add the dependance on this artefact to this script and implement a "proper" weather service
• Add a dependance to the Smithy 0.18 snapshot
• Implement the two pieces of missing functionality above
• Once successful (!), extract into a new lib

Before I pour lots more time into this... I'd love to know if you thought that's a reasonable plan of attack / outcome?

[Baccata]

I'd love to know if you thought that's a reasonable plan of attack / outcome?

I do think it's very reasonable plan.

I'd greatly appreciate if the newSmithyTool was built on top of Smithy4s (at least the json-schema bit), as it'd be really helpful to our ecosystem, as opposed to being built on top of smithy-model. It'd be a more work for what you want to achieve but would be an interesting exercise for you to understand how smithy4s works. Also it'd have the advantage of working in scala-js and scala-native, which may be helpful in case you wanted to host this integration in AWS lambdas ... but I digress.

If you're interested in trying this approach out, the first step will be something like :

// Constant type 
import smithy4s._ 
import smithy4s.schema._ 

type JsonSchemaIR[A] = ??? 

object JsonSchemaVisitor extends SchemaVisitor[JsonSchema] {
   // implement all methods 
} 

The complexity is to find what JsonSchema[A] should be. We know that it's gonna be some Json eventually, but there's the additional complexity of recursion that needs to be taken into consideration, because if you have a recursive data model in your smithy spec, you'll need to create references in your JsonSchema. So you need to track the identifiers to the schemas you've already traversed to avoid looping infinitely. Roughly it should probably look like something like :

import smithy4s.ShapeId 

// nested keeps a map of nested definitions 
case class JsonSchemaRecord(nested: Map[ShapeId, Document], surface: Document.DObject)  

trait JsonSchema[A] {
    // `alreadySeen` should track the shapes you've already visited, to avoid looping. 
    def record(alreadySeen: Set[ShapeId]) : JsonSchemaRecord

   final def make: Document = {
     val jsonSchemaRecord = record(Set.empty) 
     // take a `JsonSchemaRecord` and create a Document.DObject with the `$defs` 
     // see https://json-schema.org/understanding-json-schema/structuring.html#defs
     ??? 
   } 
} 

Let us know if you want to have a go at it. It can totally occur in your codebase, but we'll be able to give pointers :)

[Quafadas]

I'd greatly appreciate if the newSmithyTool was built on top of Smithy4s (at least the json-schema bit), as it'd be really helpful to our ecosystem, as opposed to being built on top of smithy-model.

Well, there are two things warring here.

1. I very much want to contribute to the ecosystem
2. My original plan was somewhat predicated, on this part being largely solved. So this is an error in my assumptions, that perhaps leads to a revision of the time required.

So I am willing, but fearful, of the amount of time required. I'm aware of two pieces of prior art of generating JSON schema.

Quicktype has quite general inputs...
https://github.com/quicktype/quicktype/blob/master/packages/quicktype-core/src/language/JSONSchema.ts

Scala JSON Schema - which pretty much does this, but from the case classes themselves.
https://github.com/andyglow/scala-jsonschema/tree/master

I browsed through their source code, and neither project, is a trivial undertaking! I'll have a go. Let's see.

[Baccata]

I won't lie and say it's trivial but it's not too difficult either :) what makes it not too difficult is the fact that you can do it progressively : the SchemaVisitor gives you all the methods you need to implement in order to achieve the end goal, so the effort is very quantifiable, and very unit-test friendly.

Each generated data-type is accompanied with a Schema, you can set-up a smithy4s-project easily and play with schema visitors to verify your assumptions (and write unit tests) quite easily.

[Quafadas]

@Baccata @kubukoz So as you are by now aware, I experienced a complete and catastrophic loss of self control as my curiosity got the better of me.

https://github.com/Quafadas/smithy-call-tool is now a viable POC for the goal I outlined above.

scala-cli run . --main-class smithyOpenAI.SmithyModelled

It'll work for services and operations which can be described by the below subset of smithy.

[kubukoz]

complete and catastrophic loss of self control as my curiosity got the better of me

my career summarized in 1 sentence

[keynmol]

[Quafadas]

hehe - if you get to the printing stage, I'm good for a couple :-).

I haven't forgotten this discussion / idea either. I do want to come back to it but sadly, there's no way I can devote meaningful time before the new year without getting sacked (or divorced), so it'll stay dormant outside of the unlikely case of someone else wanting to pick it up.

Unless anyone wishes to offer me a job doing it (or their hand in marriage). Then I'm totally in.

[Quafadas]

@Baccata @kubukoz

Hi Guys,

I've had another look at this. As far as I can tell, my experiment produces valid JSON schema, under a wide range of conditions. It is incomplete in a few places (e.g. tagged unions), but this is my best starting point at a PR with the aim to introduce JSON schema generation.

I guess I'd welcome feedback on whether it's worth continuing / how to go about / roughly how far away you perceive a PR to smithy itself might be?

Here is my suggestion, on how you could investigate.
git clone https://github.com/Quafadas/smithy-call-tool.git

Step 1.

https://github.com/Quafadas/smithy-call-tool/blob/main/src/test/scala/json.schema.test.scala

The "example generation" test contains the finished schema gen, with both the shape and defs . Stick a breakpoint at the end, and skim the test debug log.

The assertions compare AWS generated schema, against this "homegrown" approach.

By playing with the smithy spec defined above and treating this as a black box, you should be able to fairly quickly gain a picture, of whether you think it's worth spending more of your time, looking into this, or not. To be clear if not : that would also be okay!

My experiments are that most of the time it produced valid schema - even if it often fails the "test" by not being the same as AWS. Usually, AWS has a different set of defs.

Step 2

Skim the other unit tests.

This should give you a feel, of whether it's possible to put on a "correctness collar" to keep it maintainable. My standard here will be lower than yours - as I don't maintain major projects.

I left a failing test in, to show that this is, sadly, not complete. In part that could provoke a discussion on how to break up a potential PR into smaller chunks?

Step 3

Skim the implementations themselves. I think everything you might be interested in, is in the visitors, and tests folder. Based on the discord discussion, all the other stuff is for me to play with and write up elsewhere - smithy4s has no interest in it and please ignore it for the sake of your time.

By my standards, the implementation is complex. It's probably messy in places. I am open to feedback. I am happy, to join a call to explain, as best I can, how / what / why, if that is helpful or you find anything unclear.