feat(new transform): Introduce WASM Plugins #2006

Hoverbear · 2020-03-09T15:03:47Z

Closes #2280, related to #2341

Platform support:

Linux (x86_64)
Linux (ARMv7) (Likely never)
Linux (Aarch64) (Probably coming, ARM contributing)
Mac (x86_64)
- Multiplatform signals bytecodealliance/lucet#437
Windows (x86_64)
FreeBSD (x86_64)
- Support FreeBSD bytecodealliance/lucet#419
- Multiplatform signals bytecodealliance/lucet#437

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear · 2020-03-31T00:13:08Z

@lukesteensen @LucioFranco Could you take an initial look at this (the RFC only?)

I suggest we merge this with just transforms specced and amend it with sinks and sources later. I'll work on getting the POC woven in and the RFC less draft-style.

Signed-off-by: Ana Hobden <[email protected]>

LucioFranco · 2020-03-31T16:21:05Z

Cargo.toml

@@ -177,11 +183,11 @@ dirs = "2.0.2"

 [features]
 # Default features for *-unknown-linux-gnu and *-apple-darwin
-default = ["sources", "transforms", "sinks", "vendored", "unix", "leveldb-plain", "rdkafka-plain"]
+default = ["sources", "transforms", "sinks", "vendored", "unix", "leveldb-plain", "rdkafka-plain", "engine"]


prob a better name than "engine"?

edit: I guess you do call it the vector engine but from reading this I wouldn't assume its the wasm engine.

Definitely open to new terms

LucioFranco

As sinks? For this we should consider how we handle batching, partitioning, etc, more.

I think we will want to expose premade host fn that basically just call out to and configure our current implementations.

What about the idea of codecs being separate?

I'm not totally following this question, could you explain more?

We should consider if we are satisfied with the idea of hostcalls being used for get and other API. We could also let
the host call the Guest allocate function and then pass it a C String pointer to let it work on. This, however, requires
serializing and deserializing the entire Event each time, which is a huge performance bottleneck.

Is this something where we could expose both? Would be nice to let the user choose the trade offs.

Do we one day want to support a cross platform cargo install vector option for installing a Vector binary?

To me this isn't that important since we should be having users go through their system package manager or download the binary from our hosted repo.

Should a user import vector::... to use the Vector guest API in their Rust module?

I would imagine we want to expose a separate smaller crate for rust users that want to compile to wasm? We probably also want to support rust as the best language to write in?

How can we let users see what's happening in WASM modules? Can we use tracing somehow? Lucet supports tracing, perhaps
we could hook in somehow?

Probably via the same method we have users interact with events via some host fn.

Overall this looks fantastic! Super excited to get WASM support! Great work :)

rfcs/2020-03-05-wasm-engine.md

LucioFranco · 2020-03-31T17:04:03Z

rfcs/2020-03-05-wasm-engine.md

+We noted that the existing lua runtime was able to accomplish these tasks quite elegantly, however it was an order of
+magnitude slower than a native transform.
+
+> TODO: Proof


proof is just "lua" :)

I'd like to do some real testing. :)

rfcs/2020-03-05-wasm-engine.md

Hoverbear · 2020-06-09T20:43:18Z

Ok lets leave it off by default and I'll remove the WATs and have them work through /target

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear · 2020-06-10T19:26:28Z

scripts/generate/templates/_partials/_component_header.md.erb

@@ -7,7 +7,7 @@ description: <%= "The Vector `#{component.name}` #{component.type} #{component_s
 event_types: <%= component.event_types.to_json %>
 function_category: <%= component.function_category.to_json %>
 issues_url: <%= metadata.links.fetch("urls.#{component.id}_issues") %>
-<%- if !component.transform? -%>
+<%- if !component.transform? || component.respond_to?(:operating_systems) -%>


@binarylogic I tried just removing this line but that broke, is this the correct way to introduce this value (operating_systems) to the transforms page?

src/event/mod.rs

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear · 2020-06-10T21:09:09Z

I asked @lukesteensen to take a final peak at this before merge to make sure I didn't miss anything. :)

lukesteensen

Looks good! Excited to start experimenting with this 🎉

lukesteensen · 2020-06-10T22:29:15Z

src/wasm/fingerprint.rs

+        let age = modified.duration_since(std::time::UNIX_EPOCH)?;
+        Ok(Self(age.as_secs()))


nit: Using the time itself instead of duration_since would be a more unique fingerprint. Not that I'd really expect a collision either way.

Heck, you're right! I wanted to improve the fingerprinter in general later, but it seems sufficient right now.

* Add wip Signed-off-by: Ana Hobden <[email protected]> * wip Signed-off-by: Ana Hobden <[email protected]> * wip Signed-off-by: Ana Hobden <[email protected]> * WIP Signed-off-by: Ana Hobden <[email protected]> * Fix bindings Signed-off-by: Ana Hobden <[email protected]> * Integrate Make task Signed-off-by: Ana Hobden <[email protected]> * Remove old cruft Signed-off-by: Ana Hobden <[email protected]> * More refinements to rfc Signed-off-by: Ana Hobden <[email protected]> * Get function passing working Signed-off-by: Ana Hobden <[email protected]> * Tracing and proper FFI strings Signed-off-by: Ana Hobden <[email protected]> * Wip Signed-off-by: Ana Hobden <[email protected]> * 🚀 Signed-off-by: Ana Hobden <[email protected]> * More RFC notes Signed-off-by: Ana Hobden <[email protected]> * Format wasm rfc Signed-off-by: Ana Hobden <[email protected]> * Add API notes Signed-off-by: Ana Hobden <[email protected]> * Write more of the RFC Signed-off-by: Ana Hobden <[email protected]> * Add plan of attack and platform support Signed-off-by: Ana Hobden <[email protected]> * Minor fixes. Signed-off-by: Ana Hobden <[email protected]> * Command line test passes Signed-off-by: Ana Hobden <[email protected]> * Extract foreign module APIs. Signed-off-by: Ana Hobden <[email protected]> * Do some documentation. Signed-off-by: Ana Hobden <[email protected]> * More RFC writing Signed-off-by: Ana Hobden <[email protected]> * Cleanup and RFC updating. Signed-off-by: Ana Hobden <[email protected]> * Checkpoint Signed-off-by: Ana Hobden <[email protected]> * Passing a Raw registration up works Signed-off-by: Ana Hobden <[email protected]> * Update RFC to reflect new simpler APIs. Signed-off-by: Ana Hobden <[email protected]> * Various cleaning of structure. Signed-off-by: Ana Hobden <[email protected]> * Some refining on docs and roles Signed-off-by: Ana Hobden <[email protected]> * Some new events Signed-off-by: Ana Hobden <[email protected]> * Remove RFC present in other PR Signed-off-by: Ana Hobden <[email protected]> * Fix release builds and add new metric Signed-off-by: Ana Hobden <[email protected]> * Rename somethings Signed-off-by: Ana Hobden <[email protected]> * Caching and fingerprinting works now. Signed-off-by: Ana Hobden <[email protected]> * Add benchmark for wasm protobuf Signed-off-by: Ana Hobden <[email protected]> * Add noop bench Signed-off-by: Ana Hobden <[email protected]> * Add wasm CI Signed-off-by: Ana Hobden <[email protected]> * Update lockfile Signed-off-by: Ana Hobden <[email protected]> * Clean up make jobs, benching Signed-off-by: Ana Hobden <[email protected]> * Fixup make jobs Signed-off-by: Ana Hobden <[email protected]> * Use WATS Signed-off-by: Ana Hobden <[email protected]> * Figure out less hazardous memory management (demo) Signed-off-by: Ana Hobden <[email protected]> * Responsible memory management Signed-off-by: Ana Hobden <[email protected]> * process takes pointer/len, emit works Signed-off-by: Ana Hobden <[email protected]> * Better benchmarks Signed-off-by: Ana Hobden <[email protected]> * Add docs * Fixup nits. Signed-off-by: Ana Hobden <[email protected]> * Add website files Signed-off-by: Ana Hobden <[email protected]> * Rework registration. Signed-off-by: Ana Hobden <[email protected]> * Some refining of the protobuf transform and tests. Signed-off-by: Ana Hobden <[email protected]> * Fixup benches. Signed-off-by: Ana Hobden <[email protected]> * Make emit more flexible. Signed-off-by: Ana Hobden <[email protected]> * Add tests for other mods Signed-off-by: Ana Hobden <[email protected]> * Add transform changes Signed-off-by: Ana Hobden <[email protected]> * Add error handling Signed-off-by: Ana Hobden <[email protected]> * Add panic handling Signed-off-by: Ana Hobden <[email protected]> * No more stale wats Signed-off-by: Ana Hobden <[email protected]> * wasm interop exposed and more protobuf guide Signed-off-by: Ana Hobden <[email protected]> * Clean up Signed-off-by: Ana Hobden <[email protected]> * Various small fixes Signed-off-by: Ana Hobden <[email protected]> * Add cached metric to compilation event Signed-off-by: Ana Hobden <[email protected]> * Clean up test spans * More robust protobuf Signed-off-by: Ana Hobden <[email protected]> * Add options calls to WASM modules. Signed-off-by: Ana Hobden <[email protected]> * checkpoint Signed-off-by: Ana Hobden <[email protected]> * Improve fingerprinter. Signed-off-by: Ana Hobden <[email protected]> * Add wasm-timings feature Signed-off-by: Ana Hobden <[email protected]> * Add assert config Signed-off-by: Ana Hobden <[email protected]> * Extract fingerprint and artifact cache. Signed-off-by: Ana Hobden <[email protected]> * Remove protobuf guide. Signed-off-by: Ana Hobden <[email protected]> * Make things much more safe. Signed-off-by: Ana Hobden <[email protected]> * Fix a whole swack of integer sizing issues. Signed-off-by: Ana Hobden <[email protected]> * Update to git master lucet Signed-off-by: Ana Hobden <[email protected]> * Add nix env Signed-off-by: Ana Hobden <[email protected]> * Update lucet/tracing Signed-off-by: Ana Hobden <[email protected]> * Note we support wat as well Signed-off-by: Ana Hobden <[email protected]> * Raise some debug to info Signed-off-by: Ana Hobden <[email protected]> * fmt Signed-off-by: Ana Hobden <[email protected]> * Add artifact_cache knob Signed-off-by: Ana Hobden <[email protected]> * Remove WATs Signed-off-by: Ana Hobden <[email protected]> * Clean up formatting and some clippy lints Signed-off-by: Ana Hobden <[email protected]> * Fixup features in benches Signed-off-by: Ana Hobden <[email protected]> * Checker now needs cmake Signed-off-by: Ana Hobden <[email protected]> * Fmt Signed-off-by: Ana Hobden <[email protected]> * Fixup bench features Signed-off-by: Ana Hobden <[email protected]> * Add OS support to transforms on website Signed-off-by: Ana Hobden <[email protected]> * Wasm doesn't need wabt anymore Signed-off-by: Ana Hobden <[email protected]> * Fixup CI Signed-off-by: Ana Hobden <[email protected]> * Remove bench features haha Signed-off-by: Ana Hobden <[email protected]> * Correct lib authors Signed-off-by: Ana Hobden <[email protected]> * Yank utf-8 handling change Signed-off-by: Ana Hobden <[email protected]> * generate Signed-off-by: Ana Hobden <[email protected]> * Decouple wasm modules from workspace Signed-off-by: Ana Hobden <[email protected]> Signed-off-by: Brian Menges <[email protected]>

Hoverbear added 4 commits March 2, 2020 15:16

Add wip

e1b5c2c

Signed-off-by: Ana Hobden <[email protected]>

wip

47f3231

Signed-off-by: Ana Hobden <[email protected]>

wip

13bac4d

Signed-off-by: Ana Hobden <[email protected]>

WIP

3aca385

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear self-assigned this Mar 9, 2020

Fix bindings

aaf2a4d

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear added needs: rfc Needs an RFC before work can begin. type: new feature labels Mar 10, 2020

Hoverbear mentioned this pull request Mar 10, 2020

Ability to call transform functions within a runtime #2015

Closed

Hoverbear added 7 commits March 11, 2020 11:43

Integrate Make task

764cd51

Signed-off-by: Ana Hobden <[email protected]>

Remove old cruft

d58b370

Signed-off-by: Ana Hobden <[email protected]>

More refinements to rfc

e6c368f

Signed-off-by: Ana Hobden <[email protected]>

Get function passing working

48ba013

Signed-off-by: Ana Hobden <[email protected]>

Tracing and proper FFI strings

5400b35

Signed-off-by: Ana Hobden <[email protected]>

Wip

6ef2bae

Signed-off-by: Ana Hobden <[email protected]>

🚀

28597be

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear linked an issue Mar 23, 2020 that may be closed by this pull request

New wasm transform #1802

Closed

Hoverbear added 4 commits March 24, 2020 08:12

More RFC notes

dd66604

Signed-off-by: Ana Hobden <[email protected]>

Format wasm rfc

a3b0ec3

Signed-off-by: Ana Hobden <[email protected]>

Add API notes

b8cca2b

Signed-off-by: Ana Hobden <[email protected]>

Write more of the RFC

4ec5cc7

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear changed the title ~~DRAFT: Wasm engine~~ feature: RFC - Wasm engine Mar 31, 2020

Hoverbear added domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources domain: transforms Anything related to Vector's transform components and removed needs: rfc Needs an RFC before work can begin. labels Mar 31, 2020

Add plan of attack and platform support

059cc68

Signed-off-by: Ana Hobden <[email protected]>

LucioFranco reviewed Mar 31, 2020

View reviewed changes

Hoverbear added 7 commits June 9, 2020 14:56

Remove WATs

c5be593

Signed-off-by: Ana Hobden <[email protected]>

Merge branch 'master' into wasm-engine

f7d938d

Clean up formatting and some clippy lints

07c4699

Signed-off-by: Ana Hobden <[email protected]>

Fixup features in benches

0627d7a

Signed-off-by: Ana Hobden <[email protected]>

Checker now needs cmake

b6d3064

Signed-off-by: Ana Hobden <[email protected]>

Fmt

c2764a0

Signed-off-by: Ana Hobden <[email protected]>

Fixup bench features

850671d

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear requested a review from MOZGIII as a code owner June 10, 2020 15:38

Hoverbear force-pushed the wasm-engine branch from f8e2422 to 850671d Compare June 10, 2020 15:40

Add OS support to transforms on website

03e921f

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear force-pushed the wasm-engine branch from f3e4b51 to 03e921f Compare June 10, 2020 17:40

Hoverbear added 4 commits June 10, 2020 11:42

Wasm doesn't need wabt anymore

9455eb5

Signed-off-by: Ana Hobden <[email protected]>

Fixup CI

8ac1555

Signed-off-by: Ana Hobden <[email protected]>

Remove bench features haha

84750b2

Signed-off-by: Ana Hobden <[email protected]>

Correct lib authors

f2d2483

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear commented Jun 10, 2020

View reviewed changes

src/event/mod.rs Outdated Show resolved Hide resolved

Hoverbear added 3 commits June 10, 2020 12:28

Yank utf-8 handling change

7d6f2b2

Signed-off-by: Ana Hobden <[email protected]>

generate

e12b6a2

Signed-off-by: Ana Hobden <[email protected]>

Decouple wasm modules from workspace

ba92982

Signed-off-by: Ana Hobden <[email protected]>

Hoverbear changed the title ~~feature(new transform): Introduce WASM Plugins~~ feat(new transform): Introduce WASM Plugins Jun 10, 2020

lukesteensen approved these changes Jun 10, 2020

View reviewed changes

Hoverbear merged commit 5706d2a into master Jun 11, 2020

binarylogic added type: feature A value-adding code addition that introduce new functionality. and removed type: new feature labels Jun 16, 2020

binarylogic deleted the wasm-engine branch July 23, 2020 17:32

JeanMertz added the transform: wasm label Aug 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new transform): Introduce WASM Plugins #2006

feat(new transform): Introduce WASM Plugins #2006

Hoverbear commented Mar 9, 2020 •

edited

Loading

Hoverbear commented Mar 31, 2020 •

edited

Loading

LucioFranco Mar 31, 2020 •

edited

Loading

Hoverbear Mar 31, 2020

LucioFranco left a comment

LucioFranco Mar 31, 2020

Hoverbear Apr 2, 2020

Hoverbear commented Jun 9, 2020

Hoverbear Jun 10, 2020 •

edited

Loading

Hoverbear commented Jun 10, 2020

lukesteensen left a comment

lukesteensen Jun 10, 2020

Hoverbear Jun 11, 2020

		let age = modified.duration_since(std::time::UNIX_EPOCH)?;
		Ok(Self(age.as_secs()))

feat(new transform): Introduce WASM Plugins #2006

feat(new transform): Introduce WASM Plugins #2006

Conversation

Hoverbear commented Mar 9, 2020 • edited Loading

Hoverbear commented Mar 31, 2020 • edited Loading

LucioFranco Mar 31, 2020 • edited Loading

Choose a reason for hiding this comment

Hoverbear Mar 31, 2020

Choose a reason for hiding this comment

LucioFranco left a comment

Choose a reason for hiding this comment

LucioFranco Mar 31, 2020

Choose a reason for hiding this comment

Hoverbear Apr 2, 2020

Choose a reason for hiding this comment

Hoverbear commented Jun 9, 2020

Hoverbear Jun 10, 2020 • edited Loading

Choose a reason for hiding this comment

Hoverbear commented Jun 10, 2020

lukesteensen left a comment

Choose a reason for hiding this comment

lukesteensen Jun 10, 2020

Choose a reason for hiding this comment

Hoverbear Jun 11, 2020

Choose a reason for hiding this comment

Hoverbear commented Mar 9, 2020 •

edited

Loading

Hoverbear commented Mar 31, 2020 •

edited

Loading

LucioFranco Mar 31, 2020 •

edited

Loading

Hoverbear Jun 10, 2020 •

edited

Loading