Mill plugin

[Quafadas]

My situation is that I can't really use SBT at work (for dumb reasons, but that is out of my control).

I can however, use mill.

Now, I am not an expert on build tools, or build tool plugins, but from what I see here,
https://github.com/disneystreaming/smithy4s/blob/main/modules/codegen-plugin/src/smithy4s/codegen/CodegenPlugin.scala

The Smithy4s codegen plugin (which produces the key parts an application would wish to consume?), is

• a bunch of settings
• a single task

The task

• gets dependancies for smithy
• get paths from settings
• makes a call to the smithy library with CodeGen args.

I believe that smithy library is also published for scala 2.13.8?

First question : is all of the above, reasonably correct? Any material complexity I'm missing?

Because if that's all right, it looks like a tilt at a mill plugin, might be within my own (limited :-)!) skillz - finickity, but not "complex" per se... and with no knowledge needed beyond blindly translating the CLI / sbt plugin tasks...

Second question : if all of the above is right and I did have a go, are you interested in the outcome (if it were to work, that is, if I were to fail miserably, I likely wouldn't post that up :-)... ) ?

[lewisjkl]

Hey @Quafadas, thank you for your interest in smithy4s and your willingness to contribute! We actually have an example of a mill plugin for smithy4s that we can provide you. We may end up providing a mill plugin as part of the library down the road, but that has yet to be decided/mapped out. So in the meantime, you should be able to copy this one and modify it if need be to suit your needs. And of course, don't hesitate to reach out if you have questions along the way!

import $ivy.`com.disneystreaming.smithy4s::smithy4s-codegen:0.12.16`

import coursier.maven.MavenRepository
import mill._
import mill.api.PathRef
import mill.scalalib._
import os.Path
import smithy4s.codegen.{Codegen => Smithy4s}
import smithy4s.codegen.CodegenArgs

import scala.util._

trait Smithy4sModule extends ScalaModule {

  /** Input directory for .smithy files */
  protected def smithy4sInputDir: T[PathRef] = T.input {
    PathRef(millSourcePath / "smithy")
  }

  private def smithy4sCodegen: T[(PathRef, PathRef)] = T {
    val specFiles = if (os.exists(smithy4sInputDir().path)) {
      os.walk(smithy4sInputDir().path, skip = _.ext != "smithy")
    } else Seq.empty

    val scalaOutput   = T.ctx().dest / "scala"
    val openapiOutput = T.ctx().dest / "openapi"

    val args = CodegenArgs(
      specs         = specFiles.toList,
      output        = scalaOutput,
      openapiOutput = openapiOutput,
      skipScala     = false,
      skipOpenapi   = false,
      allowedNS     = None,
      repositories  = List.empty,
      dependencies  = List.empty,
      transformers  = List.empty
    )

    Smithy4s.processSpecs(args)
    (PathRef(scalaOutput), PathRef(openapiOutput))
  }

  override def generatedSources: T[Seq[PathRef]] = T {
    val (scalaOutput, _) = smithy4sCodegen()
    scalaOutput +: super.generatedSources()
  }

  // Considering the openapi output as localclasspath so that
  // its contents can be served by http servers
  override def localClasspath: T[Seq[PathRef]] = T {
    val (_, openapiOutput) = smithy4sCodegen()
    openapiOutput +: super.localClasspath()
  }

}

[keynmol]

I'd imagine providing a proper mill plugin is somewhat dependent on Mill establishing bincompat guarantees for plugins.

[Baccata]

Mill has already started bincompat guarantees as of late last year. MiMa is enabled since the 0.10 lineage. We have it in our backlog to add a proper mill module at some point in the future.

[Quafadas]

@lewisjkl Would you know by any chance, if something has changed in 0.14.0 that would impact the ability to server swagger specs with the example you've posted above?

I can't get the docs route to work :-/

[Baccata]

@Quafadas did change the import $ivy.... at the top of the plugin file above and fixed the compilation error ?

[Baccata]

Fix incoming #309

[Quafadas]

Er, wow. Thanks!

I shall ruthlessly copy paste this where it's needed :-). I'll link to this in the mill discussions forum as well, so the knowledge exists over there too...

Thanks again, I'm really looking forward to trying this.

[lefou]

May I suggest the use of T.source instead of T.input. Both represent an input task returning a single PathRef, but the former enables support for --watch, whereas the latter doesn't.

  /** Input directory for .smithy files */
  protected def smithy4sInputDir: Source = T.source(millSourcePath / "smithy")

[Baccata]

👍

[Quafadas]

I found I had to modify the module slightly, to include two extra params in the codgen args, (the bottom two) otherwise it grumbled.
For 0.14.0

    val args = CodegenArgs(
      specs         = specFiles.toList,
      output        = scalaOutput,
      openapiOutput = openapiOutput,
      skipScala     = false,
      skipOpenapi   = false,
      allowedNS     = None,
      repositories  = List.empty,
      dependencies  = List.empty,
      transformers  = List.empty,
      discoverModels = true,
      excludedNS = Some(Set.empty),
      localJars = List() // needed after 0.14.0
    )

But, I believe this works otherwise as intended. Which is really cool

Thankyou!

[kierendavies]

I had a crack at writing a Mill plugin (before I saw this discussion).

My first attempt was with a totally separate codebase, and I quickly ended up in class loader hell. The user should specify the versions of Scala and smithy4s, so the plugin has to load Codegen/CodegenArgs dynamically. The arguments in CodegenArgs are mostly Scala types, so those also need to be dynamically loaded... I gave up pretty quickly. One way to deal with this is to use a Java wrapper which only uses core Java types. See commentary here in the Mill Twirl plugin.

My second attempt was with a module in the smithy4s codebase. Seems OK, but I don't know how to write automated tests for this. See this branch.

[Quafadas]

If I understood this comment correctly (I may not have) Java wrapper types appears to be the canonical solution to the major version cli for 2.12 <-> mill 2.13 mismatch. Or at least that’s what @logab did for scalably typed…

[Baccata]

@kierendavies good effort. If you want to have publish a smithy4s mill plugin, the problem boils down to the following :

1. Either you need to have your plugin depend statically on smithy4s, which implies that the plugin needs to follow the versions of smithy4s, or the users need to add a line like

import $ivy.`com.disneystreaming.smithy4s::smithy4s-codegen:latest.release`

at the top of their build.sc to bump smithy4s (which is risky because we do not currently have strong guarantee for bincompat has the design of the core is still fluctuating and we need the flexibility to break some things)
2. Or you let the users control the version pulled by plugin via a def smithy4sVersion : T[String] task, which implies that you need to shell-out to smithy4s-cli (that you'd pull dynamicaly using coursier)

Either way, I'd advise against publishing a mill plugin as a third party library, it'll be a maintenance burden. The reason we have not published it ourselves is that we're waiting on the merging of this PR in mill, which will greatly facilitate the writing of tests, but @lefou has been busy and I think we can all agree that his endeavour to become a volunteer firefighter should have priority over any open-source work. So it'll happen when it happens 😄

In the meantime, the solution I'd advocate for would be the copy/pasting of the module that @lewisjkl shared above in the codebases that need it.

[daddykotex]

Happy to let you know that mill support is out and you're invited to test it out.

https://disneystreaming.github.io/smithy4s/docs/overview/installation#mill