derive Circe codecs for smithy shapes (also, tapir integration? 🤔 )

[ShahOdin]

I'm in the funny situation where I have a smithy definition to implement but the company wants the services to be implemented in tapir. Now, smithy4s generates me the code for the service and I have an impl for it, but whereas normally you'd then feed this to SimpleRestJsonBuilder I now have to instead pass on the impl to tapir's Endpoints.

So i have:

def myEndpoint: Endpoint[
    Unit,
    (MyInput1, MyInput2),
    MyError,
    MyOutput,
    Any
  ] = endpoint.get
    .in("foo")
    .in(query[MyInput1]("foo_input_1"))
    .in(query[MyInput2]("foo_input_2"))
    .out(jsonBody[MyOutput])
    .errorOut(jsonBody[MyError])

Now in order to have this, I'd need:

  given Encoder[MyOutput] = ???
  given Decoder[MyOutput] = ???
  given sttp.tapir.Schema[MyOutput] = ???

  given Encoder[MyError] = ???
  given Decoder[MyError] = ???
  given sttp.tapir.Schema[MyError] = ???

  given sttp.tapir.Codec[List[String], MyInput1, TextPlain] = ???
  given sttp.tapir.Codec[List[String], MyInput2, TextPlain] = ???

given a smithy4s.Schema for A is there a way to derive circe Codecs for it? had a look at the SimpleRestJsonBuilder to get an idea of how it's done but there were so many nested layers of nesting and abstractions that I gave up, but I feel like this bit should be the easy bit.

How about sttp.tapir.{Codec, Schema}? 🤔

[kubukoz]

frankly, as much as I like both tapir and smithy4s, I don't think it's a good combination ;) they solve a similar problem in fundamentally opposite ways (code-first vs contract-first). (although worth mentioning, there's an experiment in progress by @Baccata to provide a code-first approach too).

Going through the effort of converting from Smithy4s's abstract representation (Schema) to another abstract representation (Endpoint) sounds counter-productive and I think it'd be much better to just pick one.

I don't know what the API of tapir's Codec/Schema looks like, so can't tell if these are possible to construct from smithy4s directly, but you can definitely construct a circe.Encoder/Decoder pair. We're not planning to support any of these in smithy4s, but you can give it a shot to implement a SchemaVisitor[io.circe.Codec], or more generally a Schema ~> io.circe.Codec.

Some things you can look at for inspiration:

• SchemaVisitor for the Schema ~> Document.Encoder conversion, which is close to what a circe encoder would look like
• Documentation for Schema

All in all, I think it would be in your (and the company's) best interest to revisit the decision to use both. If you already have a Smithy spec, use smithy4s. If you already have Tapir endpoints, use Tapir. Using both will land you in the position of having to implement not only your business logic and smithy definitions, but also the complex, costly and pretty low-level conversions - that can't possibly be a good use of time.

[ShahOdin]

All in all, I think it would be in your (and the company's) best interest to revisit the decision to use both.

fair point. I don't think having both is a stable or desirable strategy. I have a smithy definiton to implement and I'm trying to use that instead of going the open api route and I'm trying to get the team to appreciate smithy4s while not disrupting their existing setup.

That said, I have actually realised that, at least for my simple use cases, it's pretty easy to derive circe codecs for the smithy generated scala classes:

  given ofNewType[A](using
      instance: Newtype[A],
      encoder: Encoder[A],
      decoder: Decoder[A]
  ): Codec[instance.Type] = Codec.from(
    encodeA = encoder.imap(instance.apply)(instance.value),
    decodeA = decoder.imap(instance.apply)(instance.value)
  )

  given ofEnumeration[A <: smithy4s.Enumeration.Value](using A: smithy4s.Enumeration[A]): Codec[A] =
    Codec.from(
      encodeA = Encoder.encodeString.contramap(_.value),
      decodeA = Decoder.decodeString.emap: s =>
        A.fromString(s).toRight(s"Failed to parse $s as valid value for ${A.id.name}")
    )

The problem is that I have to manually pass the companion object as the companion objects aren't implicitly available. As a learning exercise tho, I'm trying to learn how to turn these into macros. the latter is easier, as both the type and the companion object itself share the A type. so it's just a matter of

    val companionSymbol = TypeRepr.of[T].typeSymbol.companionModule
    val companionTree   = Ref(companionSymbol).asExprOf[smithy4s.Enumeration[T]]
    val methodCallTree  = Select.unique(companionTree.asTerm, "fromString")

etc. With the ofNewType tho, it's more complicated and I'm still getting my head around it.

[Baccata]

You should really not use scala 3 metaprogramming to achieve your goal. You'll be easily violating the smithy semantics by doing that, for the simple reason that the in-memory representation is not necessarily a good reflection of the contract. A schema visitor, following the example linked by Jakub, is the right way to go for deriving Circe codecs for types generated by smithy4s.

[ShahOdin]

in-memory representation is not necessarily a good reflection of the contract.

Care to elaborate with an example?

[Baccata]

• smithy traits are not available from the case class type, so you won't be able to easily access things like jsonName
• The name of ADT members are capitalised and suffixed with Case, which doesn't make them a good reflection of the smithy union members they originate from
• case class fields can be prefixed with underscore when the name of the corresponding smithy member collides with a scala keyword, which has happens more often than you'd think
• enumerations can be open, in which case they have one non-singleton member so it's virtually impossible to distinguish them from union-reflecting ADTs

Those are just examples of things that will go wrong if you expect scala 3 metaprogramming to be a good solution to this problem. We have one construct that is designed to integrate smithy4s with third party libraries, and that is the schema. If you want to deviate from this approach, you're on your own.