BP: Pluggable processors (WASM)

[maha-hajja]

Pluggable processors or bring-your-own-processor (BYOP) is the ability to provide a custom processor implementation, similarly to how we currently allow the user to provide standalone connector implementations. We already provide a similar feature using the JavaScript processor, although it has many drawbacks like poor performance and the fact that users are forced to write JavaScript. We should think about alternatives that would improve the experience of writing your own processor like go-plugin or WebAssembly.

• Implement a WASM processor plugin registry that can load WASM plugins
  • BP: WASM: Load available WASM processors #1322
• Plugins should be loaded at startup like connector plugins (hot reload can be added later)
• Implement an adapter that forwards calls from Conduit to the WASM plugin
  • BP: WASM: Load available WASM processors #1322
• Figure out the best way to pass records to the processor (we need to serialize them)
  • Research serialization options conduit-processor-sdk#5
• Figure out which host functions we should expose
• Figure out how we want logging to work (maybe also through host functions)
  • Implement logging conduit-processor-sdk#3

[ahmeroxa]

I spent a fair bit of time exploring WebAssembly and the WebAssembly ecosystem and unfortunately it has become quite fragmented with various runtimes and implementations/standards (e.g. base WASM, WASI and WASIX). What is promising though is the (relatively) mature support for Host Functions (i.e. a native function that can be exposed within the WASM runtime to provide additional functionality).

This feels like the right approach at least for now. Include a WASM runtime and ship a number of host functions that provide common functionality (such as HTTP Requests or maybe even SQL query access to popular databases). That would allow us to support multiple languages for transforms as well as see a significant performance boost (WASM vs JS on goja).

[lovromazgon]

For future reference: https://github.com/stealthrocket/net

This library provides net.Dial and net.Listen functions for GOOS=wasip1.

[hariso]

I've updated the description to include relevant issues and PRs

[hariso]

The WASM part itself is done, we're now hooking it into Conduit.