Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to call transform functions within a runtime #2015

Closed
binarylogic opened this issue Mar 10, 2020 · 3 comments
Closed

Ability to call transform functions within a runtime #2015

binarylogic opened this issue Mar 10, 2020 · 3 comments
Labels
domain: parsing Anything related to parsing within Vector domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) domain: transforms Anything related to Vector's transform components domain: vrl Anything related to the Vector Remap Language have: nice This feature is nice to have. It is low priority. needs: approval Needs review & approval before work can begin. needs: more demand Needs more demand before work can begin, +1 or comment to support. needs: requirements Needs a a list of requirements before work can be begin type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@binarylogic
Copy link
Contributor

binarylogic commented Mar 10, 2020

This has been discussed, and clarifies the intent behind #2012. It is becoming clear that runtimes will likely replace the entire transform pipeline, at least for operations that cross the fundamental boundary. We can further encourage this by allowing users to call other "function" transforms from within runtimes.

Note, this should be limited to transforms that do not aggregate, only simple input/output transforms. For example, we should not allow users to call the merge, dedupe, or transaction transforms within a runtime since this would complicate the runtime APIs. I'm open to solutions for these, but I'm not sure they are worth it.

Finally, I do not think we should address this in #2000. We should defer this and follow up with this addition, assuming it is possible.

@binarylogic binarylogic added type: enhancement A value-adding code change that enhances its existing functionality. needs: requirements Needs a a list of requirements before work can be begin domain: transforms Anything related to Vector's transform components labels Mar 10, 2020
@binarylogic binarylogic added this to the Improve data processing milestone Mar 10, 2020
@Hoverbear
Copy link
Contributor

This is interesting for #2006 as well since these would be hostcall candidates.

@binarylogic binarylogic added needs: approval Needs review & approval before work can begin. needs: more demand Needs more demand before work can begin, +1 or comment to support. labels Apr 17, 2020
@binarylogic
Copy link
Contributor Author

Removing this from the scripting project. It's not clear if this makes sense. I'm starting to think it does not since we are pushing users towards chaining transforms together instead of replacing the entire pipeline with a runtime.

@binarylogic binarylogic removed this from the Improve Data Processing milestone Apr 17, 2020
@binarylogic binarylogic added the have: nice This feature is nice to have. It is low priority. label Apr 21, 2020
@binarylogic binarylogic added domain: vrl Anything related to the Vector Remap Language domain: parsing Anything related to parsing within Vector domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) labels Aug 7, 2020
@binarylogic
Copy link
Contributor Author

Closing since this will be resolved in the new remap syntax. We'll add functions as it makes sense, and if possible share the code paths with any overlapping transforms. (Ex: JSON parsing).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: parsing Anything related to parsing within Vector domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) domain: transforms Anything related to Vector's transform components domain: vrl Anything related to the Vector Remap Language have: nice This feature is nice to have. It is low priority. needs: approval Needs review & approval before work can begin. needs: more demand Needs more demand before work can begin, +1 or comment to support. needs: requirements Needs a a list of requirements before work can be begin type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Development

No branches or pull requests

2 participants