Support for targeting multiple worlds

[karthik2804]

It would be great to support the ability to create a component that can target the union of multiple worlds in the cases where the user cannot edit the wit file to create a third world that includes the target worlds. An example of this would be an SDK built on top of a wit that bundles the wit in an npm package and the user application wanting to target their world where there would be 2 distinct wit folders.

This request is motivated by the ability to do that in componentize-py (the commit that added the support).

[guybedford]

@karthik2804 interesting use case. Is there a way to upstream the work done for componentize-py here in a way that can be shared? Alternatively I'd be open to a PR in these directions.

[dicej]

I can't think of a way to make the multi-world support reusable outside of componentize-py since it's pretty tightly coupled with how that tool represents worlds and binding generation internally, so I expect the equivalent for componentize-js would need to be written from scratch. I will note, however, that wit_parser::Resolve::merge does the heavy lifting of combining the contents of multiple WIT directories, so no need to re-invent that.

A couple of things to note based on the componentize-py implementation:

• During bindings generation, I had to emit a separate Python module hierarchy for each world being targeted, while also ensuring they share types wherever they share WIT interfaces. That means every time it needs to emit a type definition, it first checks to see if that type was already emitted for a different world and, if so, import it from the Python module which was already generated for that world.
• componentize-py supports pre-generating bindings which can be published as part of an SDK package alongside the WIT files those bindings were generated from. However, it internally replaces those bindings with its own version at component creation time, since the latter version includes hard-code offsets referencing Wasm code, and those offsets can only be calculated once all the worlds are known.

[guybedford]

We already use merge to merge our world with the StarlingMonkey WASI world (https://github.com/bytecodealliance/ComponentizeJS/blob/main/crates/spidermonkey-embedding-splicer/src/lib.rs#L168).

It sounds like this is just another set of merge calls, and everything should work out since merge will combined same-named worlds already. The bindgen for Jco works against a resolve object and a world, so if the final resolve object has all the merges, and the final world includes all the other worlds, it sounds like the bindgen should just work out to me as all the deduplication is otherwise already done at the other levels.

I don't mean to over-simplify, but am I missing anything here?

[dicej]

What componentize-py does is generate a separate Python module for each world separately rather than combine them into a single world and generate code for that. As described above, those modules can share code with each other, but from a application development perspective they can be used either independently or together.

For example, a developer might use the spin-sdk and target one of its worlds, e.g. spin-http, while also targeting some other world (call it foo). In that case, componentize-py will generate code for spin-http and foo separately (but sharing types behind the scenes as appropriate) rather than create a new world from the union of spin-http and foo. The distinction may seem subtle, but I think keeping the worlds separate matches the developer's expectation better than unioning them together (e.g. the spin-sdk API should not suddenly appear to have stuff from foo, nor should the Python module generated for foo contain anything from spin-sdk).

[guybedford]

@dicej thanks for explaining the use case better here, it sounds like they targeting different sets of exported interfaces in this case.

If I'm understanding right, we would then target multiple JS source files specifying n source.js files and n worlds to target respectively as part of a single compound component build.

That does not seem unviable to support within the scope of ComponentizeJS to me.

[karthik2804]

@guybedford revisiting this issue. I am a little confused by the following statement.

we would then target multiple JS source files specifying n source.js files and n worlds to target respectively as part of a single compound component build.

We could also have a single source file that has imports from multiple worlds correct? Something along the lines of

world a {
  include wasi:cli/[email protected];
}

world b {
  import wasi:http/[email protected];
  export wasi:http/[email protected];
}

The guest code could target both the worlds and look like something below

import { initialCwd } from "wasi:cli/[email protected]"

export const incomingHandler = {
  handle: () => {
    console.log(initialCwd())
  }
}

[guybedford]

@karthik2804 it would help to understand the use case for merging in this scenario. Of course it is possible, I was just fitting a direct source - world model which would effectively only gate the allowed imports per world. A merging semantic could also be designed of course, although there are likely other edge cases then too.

[karthik2804]

@guybedford A more concrete example would be the case of building an SDK for Spin that supports only the HTTP trigger. Since spin supports adding other types of triggers through plugins, it would be nice to be able to add support for them through other SDK packages.

In the ideal scenario, the base SDK package (call it spin-sdk) that targets a world spin-imports that just contains the imports and separate packages for the different triggers like spin-cron-sdk and spin-sqs-sdk that target worlds that only contain the exports. This would allow the user to grab a combination of packages as needed by the application.

[guybedford]

@karthik2804 yes we can just merge the worlds together then, and implement them all from one top-level JS file.

Something like:

export const cronTrigger = {
  // ... cron trigger world
}

export const sqsTrigger = {
  // ...sqs trigger world
}

Or alternatively via:

export * as cronTrigger from './cron.js';
export * as sqsTrigger from './sqs.js';

And we can even support multi version as per the export { cronTriggerA as 'spin:cron/[email protected]', cronTriggerB as 'spin:cron/[email protected]' } syntax support.

I was under the impression that the feature required here was the ability to author separate JS files targeting separate worlds like componentize-py, but the above would fit much more simply into the ComponentizeJS model anyway?

[dicej]

The important distinction about how componentize-py works is that the generated bindings can be split across multiple Python packages, but still reusing types and avoiding circular dependencies.

For example, I can say componentize-py --wit-path wit --world foo componentize --module-worlds spin_sdk=spin-http --module-worlds bar_sdk=some-other-world app -o app.wasm, which means:

• Generate bindings for the foo world as a top level module
• For the spin_sdk package (which is presumably a dependency of the app), look for a componentize-py.toml file which specifies where the WIT files (which should define a world named spin-http) are for that package and in which submodule of that package the generated bindings for the spin-http world should be placed.
• For the bar_sdk package, do the same, but use the some-other-world world for that package
• For each type shared by two or more of the foo, spin-http, and some-other-world worlds, define it only once in one of the generated modules and create aliases to that type in the other generated modules.

This allows packages like spin_sdk (which is real) and bar_sdk (which I made up for this example) to ship both WIT files and pre-generated bindings for the worlds they target. But note that those pre-generated bindings are only there for IDE and documentation purposes -- they won't be used at runtime. Instead, componentize-py componentize will replace them with bindings that provide the same API but deduplicate types and connect the imported and exported functions to generated Wasm code that handles the canonical ABI lifting and lowering.

I realize this is a bit complicated and subtle, so happy to elaborate if it's still not clear. The main takeaway is that componentize-py does what it does in order to support using one or more third-party packages which target possibly-overlapping worlds. There may be other ways to accomplish that, of course.

[guybedford]

If you're looking to publish spin_sdk to npm and have it "just work" when importing it in ComponentizeJS workflows, I think we should discuss this more generally from a JS ecosystem perspective as from what it sounds like you are describing here, this ties directly into Jco goals of transparent component usage on npm then.

If so, then perhaps we can arrange to discuss further around what has been considered here for Jco and how that could fit into a ComponentizeJS workflow here?

[dicej]

Would it make sense to discuss this at the next Jco meeting? @karthik2804 and @tschneidereit would you be able to make it on July 15 at 5pm UTC? I'm not sure myself what the plan is for a JS Spin SDK and how it will compare to the Python one.

[guybedford]

I'd be happy to discuss this at the next meeting.

[karthik2804]

I will be able to make the Jco meeting.