From 36cd54472f12fe377268c41f180a2c48152a425e Mon Sep 17 00:00:00 2001 From: daidokoro Date: Mon, 18 Sep 2023 00:43:04 +0200 Subject: [PATCH 01/16] pytransform processor intial commit --- processor/pytransform/README.md | 479 ++++++++++++++++++ processor/pytransform/config.go | 5 + processor/pytransform/factory.go | 73 +++ processor/pytransform/factory_test.go | 16 + processor/pytransform/go.mod | 45 ++ processor/pytransform/go.sum | 124 +++++ processor/pytransform/helper_scripts.go | 32 ++ .../pytransform/internal/metadata/metadata.go | 12 + processor/pytransform/logprocessor.go | 67 +++ processor/pytransform/logprocessor_test.go | 51 ++ processor/pytransform/metricprocessor.go | 67 +++ processor/pytransform/metricprocessor_test.go | 53 ++ processor/pytransform/pytransform.go | 136 +++++ processor/pytransform/pytransform_test.go | 65 +++ processor/pytransform/testdata/builder.yaml | 19 + processor/pytransform/testdata/config.yaml | 124 +++++ .../testdata/log_event_example.json | 119 +++++ .../testdata/metric_event_example.json | 55 ++ processor/pytransform/testdata/test.log | 7 + .../testdata/trace_event_example.json | 135 +++++ processor/pytransform/traceprocessor.go | 67 +++ processor/pytransform/traceprocessor_test.go | 51 ++ 22 files changed, 1802 insertions(+) create mode 100644 processor/pytransform/README.md create mode 100644 processor/pytransform/config.go create mode 100644 processor/pytransform/factory.go create mode 100644 processor/pytransform/factory_test.go create mode 100644 processor/pytransform/go.mod create mode 100644 processor/pytransform/go.sum create mode 100644 processor/pytransform/helper_scripts.go create mode 100644 processor/pytransform/internal/metadata/metadata.go create mode 100644 processor/pytransform/logprocessor.go create mode 100644 processor/pytransform/logprocessor_test.go create mode 100644 processor/pytransform/metricprocessor.go create mode 100644 processor/pytransform/metricprocessor_test.go create mode 100644 processor/pytransform/pytransform.go create mode 100644 processor/pytransform/pytransform_test.go create mode 100644 processor/pytransform/testdata/builder.yaml create mode 100644 processor/pytransform/testdata/config.yaml create mode 100644 processor/pytransform/testdata/log_event_example.json create mode 100644 processor/pytransform/testdata/metric_event_example.json create mode 100644 processor/pytransform/testdata/test.log create mode 100644 processor/pytransform/testdata/trace_event_example.json create mode 100644 processor/pytransform/traceprocessor.go create mode 100644 processor/pytransform/traceprocessor_test.go diff --git a/processor/pytransform/README.md b/processor/pytransform/README.md new file mode 100644 index 000000000000..4c8bde5aa3e1 --- /dev/null +++ b/processor/pytransform/README.md @@ -0,0 +1,479 @@ +# pytransform + + +| Status | | +| ------------- |-----------| +| Stability | [alpha]: traces, metrics, logs | + + +[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta +[sumo]: https://github.com/SumoLogic/sumologic-otel-collector + + + +The pytransform processor modifies telemetry based on configuration using Python code. + +The processor leverages an embedded python version that has access to core python functionality and __does not__ require python be installed on the host. By leveraging Python code to modify telemetry, you have access to the full power of the Python language to modify data. + +## Config + +To configure pytransform, you add your code using the `code` option in the config. Each instance of pytransform can only have a single code section. + + +```yaml +processors: + pytransform: + code: | + {your python code} +``` + + +## How it works + +The python language is embedded to the Open Telemetry binary, this allows you to run this processor without having python installed on the host itself. + +When a python transform is executed, a controlled subprocess is called using the embedded python implementation that calls the user code. + +### Python Runtime and Helper Fucntions + +The pytransform processor sends the metric, trace or log events to the python runtime by setting the env variable `INPUT`. You do not need to read this environment variable, pytransform adds helper functions to handle this automatically. + +When a telemetry event is initialised in python, the event data will be accessible under the variable `event`. For example, the code below would print the event received from open telemetry. + +```yaml +processors: + pytransform: + code: | + print(event) +``` + +#### send() + +You are able to modify this event using python code, however you choose. This can be in the form of functions, classes, regex using the python [re](https://docs.python.org/3/library/re.html) library, etc. It is import however, to maintain the structure of the event as the modified event needs to be returned to the pytransform processor. Events can be returned using another helper function, `send`. + +For example, the config below adds a resource attribute to the telemetry event and returns it to the pytransform processor. + +```yaml +processors: + pytransform/logs: + code: | + # edit resource attributes + for data in event['resourceLogs']: + for attr in data['resource']['attributes']: + attr['value']['stringValue'] = 'source.log' + + send(event) +``` + +Note the use of the `send` function at the end of the code. If the event is not passed back to the processor using send, this will result in an error and the input event will be passed as unchanged. + +```log +pytransform/logprocessor.go:54 python script returned empty output, returning record unchanged {"kind": "processor", "name": "pytransform/test", "pipeline": "logs"} +``` + + +#### print() + +The `print` function in the pytransform Python runtime is overwritten with a special function that sends strings to the pytransform processor. These strings are then printed using the internal Open Telemetry logger. + +For example, take the following config: + +```yaml +processors: + pytransform: + code: | + print('hello world') + send(event) +``` + +This results in a log line appearing in the Open Telemetry process: + +```log +2023-09-16T18:00:21.551+0200 info pytransform/pytransform.go:45 hello world {"kind": "processor", "name": "pytransform/test", "pipeline": "logs", "src": "python.print()"} +``` + +The is useful for debugging python code. + +### Timeout + +There are a number of things that are possible running python within the Open Telemetry workflow, as such, to protect the runtime and host from long running zombie processes in python, each python invokation has a **maximum timeout of 1500ms** + +The goal is to encourage using this processor solely for complex telemetry transformations rather than long running tasks. + + +## Examples + +This section contains examples of the event payloads that are sent to the pytransform processor python runtime from each telemetry type. These examples can help you understand the structure of the telemetry events and how to modify them. + +##### Log Event Payload Example: + +```json +{ + "resourceLogs": [{ + "resource": { + "attributes": [{ + "key": "log.file.name", + "value": { + "stringValue": "test.log" + } + }] + }, + "scopeLogs": [{ + "scope": {}, + "logRecords": [{ + "observedTimeUnixNano": "1694127596456358000", + "body": { + "stringValue": "2023-09-06T01:09:24.045+0200 INFO starting app.", + "attributes": [{ + + "key": "app", + "value": { + "stringValue": "dev" + } + }], + "traceId": "", + "spanId": "" + } + }] + }] + }] +} +``` + +View the the log.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto) + +##### Metric Event Payload Example: + +```json +{ + "resourceMetrics": [{ + "resource": {}, + "scopeMetrics": [{ + "scope": { + "name": "otelcol/hostmetricsreceiver/memory", + "version": "0.84.0" + }, + "metrics": [{ + "name": "system.memory.usage", + "description": "Bytes of memory in use.", + "unit": "By", + "sum": { + "dataPoints": [{ + "attributes": [{ + "key": "state", + "value": { + "stringValue": "used" + } + }], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "1874247680" + }, + { + "attributes": [{ + "key": "state", + "value": { + "stringValue": "free" + } + }], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "29214199808" + } + ], + "aggregationTemporality": 2 + } + }] + }], + "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" + }] +} +``` +View the metric.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) + +##### Trace Event Payload Example: + +```json +{ + "resourceSpans": [{ + "resource": { + "attributes": [{ + "key": "telemetry.sdk.language", + "value": { + "stringValue": "python" + } + }, + { + "key": "telemetry.sdk.name", + "value": { + "stringValue": "opentelemetry" + } + }, + { + "key": "telemetry.sdk.version", + "value": { + "stringValue": "1.19.0" + } + }, + { + "key": "telemetry.auto.version", + "value": { + "stringValue": "0.40b0" + } + }, + { + "key": "service.name", + "value": { + "stringValue": "unknown_service" + } + } + ] + }, + "scopeSpans": [{ + "scope": { + "name": "opentelemetry.instrumentation.flask", + "version": "0.40b0" + }, + "spans": [{ + "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", + "spanId": "88079ad5c94b5b13", + "parentSpanId": "", + "name": "/roll", + "kind": 2, + "startTimeUnixNano": "1694388218052842000", + "endTimeUnixNano": "1694388218053415000", + "attributes": [{ + "key": "http.method", + "value": { + "stringValue": "GET" + } + }, + { + "key": "http.server_name", + "value": { + "stringValue": "0.0.0.0" + } + }, + { + "key": "http.scheme", + "value": { + "stringValue": "http" + } + }, + { + "key": "net.host.port", + "value": { + "intValue": "5001" + } + }, + { + "key": "http.host", + "value": { + "stringValue": "localhost:5001" + } + }, + { + "key": "http.target", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "net.peer.ip", + "value": { + "stringValue": "127.0.0.1" + } + }, + { + "key": "http.user_agent", + "value": { + "stringValue": "curl/7.87.0" + } + }, + { + "key": "net.peer.port", + "value": { + "intValue": "52365" + } + }, + { + "key": "http.flavor", + "value": { + "stringValue": "1.1" + } + }, + { + "key": "http.route", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "http.status_code", + "value": { + "intValue": "200" + } + } + ], + "status": {} + }] + }] + }] +} +``` + +View the trace.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto). + + +## Full Configuration Example + +For following configuration example demonstrates the pytransform processor telemetry events for logs, metrics and traces. + +```yaml +receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + grpc: + endpoint: "0.0.0.0:4317" + + filelog: + start_at: beginning + include_file_name: true + include: + - $LOGFILE + + operators: + - type: move + from: attributes["log.file.name"] + to: resource["log.file.name"] + + - type: add + field: attributes.app + value: dev + +processors: + + # - change resource attribute log.file.name to source.log + # - add resource attribute cluster: dev + # - filter out any logs that contain the word password + # - add an attribute to each log: language: golang + pytransform/logs: + code: | + # edit resource attributes + for data in event['resourceLogs']: + for attr in [ + attr for attr in data['resource']['attributes'] + if attr['key'] == 'log.file.name' + ]: + attr['value']['stringValue'] = 'source.log' + + # add resource attribute + data['resource']['attributes'].append({ + 'key': 'cluser', + 'value': { + 'stringValue': 'dev' + } + }) + + + # filter/delete logs + for data in event['resourceLogs']: + for slog in data['scopeLogs']: + slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'password' not in lr['body']['stringValue']] + + # add an attribute to each log + for lr in slog['logRecords']: + lr['attributes'].append({ + 'key': 'language', + 'value': { + 'stringValue': 'golang' + }}) + + send(event) + + # - print event received to otel runtime log + # - if there are no resources, add a resource attribute source pytransform + # - prefix each metric name with pytransform + pytransform/metrics: + code: | + print("received event") + for md in event['resourceMetrics']: + # if resources are empty + if not md['resource']: + md['resource'] = { + 'attributes': [ + { + "key": "source", + "value": { + "stringValue": "pytransform" + } + } + ] + } + + # prefix each metric name with pytransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'pytransform.' + m['name'] + + send(event) + + # - add resource attribute source pytransform + # - filter out any spans with http.target /roll attribute + pytransform/traces: + code: | + for td in event['resourceSpans']: + # add resource attribute + td['resource']['attributes'].append({ + 'key': 'source', + 'value': { + 'stringValue': 'pytransform' + } + }) + + # filter spans with http.target /roll attribute + has_roll_attr = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/roll'] + for sd in td['scopeSpans']: + sd['spans'] = [ + s for s in sd['spans'] + if not has_roll_attr(s['attributes']) + ] + + send(event) +exporters: + logging: + verbosity: detailed + + +service: + pipelines: + logs: + receivers: + - filelog + processors: + - pytransform/logs + exporters: + - logging + + metrics: + receivers: + - otlp + processors: + - pytransform/metrics + exporters: + - logging + + traces: + receivers: + - otlp + processors: + - pytransform/traces + exporters: + - logging +``` + +### Warnings + +The Pytransform processor allows you to modify all aspects of you telemetry data. This can result in valid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file diff --git a/processor/pytransform/config.go b/processor/pytransform/config.go new file mode 100644 index 000000000000..47e60ffd1354 --- /dev/null +++ b/processor/pytransform/config.go @@ -0,0 +1,5 @@ +package pytransform + +type Config struct { + Code string `mapstructure:"code"` +} diff --git a/processor/pytransform/factory.go b/processor/pytransform/factory.go new file mode 100644 index 000000000000..eb91cae513e4 --- /dev/null +++ b/processor/pytransform/factory.go @@ -0,0 +1,73 @@ +package pytransform + +import ( + "context" + + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor/internal/metadata" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/processor" +) + +// NewFactory creates a factory for the routing processor. +func NewFactory() processor.Factory { + return processor.NewFactory( + metadata.Type, + createDefaultConfig, + processor.WithLogs(createLogsProcessor, metadata.LogsStability), + processor.WithMetrics(createMetricsProcessor, metadata.MetricsStability), + processor.WithTraces(createTracesProcessor, metadata.TracesStability), + ) +} + +func createDefaultConfig() component.Config { + return &Config{ + // pass event back to otel without changes + Code: "send(event)", + } +} + +func createLogsProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Logs, +) (processor.Logs, error) { + + ep, err := getEmbeddedPython() + if err != nil { + return nil, err + } + + return newLogsProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil +} + +func createMetricsProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Metrics, +) (processor.Metrics, error) { + + ep, err := getEmbeddedPython() + if err != nil { + return nil, err + } + + return newMetricsProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil +} + +func createTracesProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Traces, +) (processor.Traces, error) { + + ep, err := getEmbeddedPython() + if err != nil { + return nil, err + } + + return newTraceProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil +} diff --git a/processor/pytransform/factory_test.go b/processor/pytransform/factory_test.go new file mode 100644 index 000000000000..09715ff2a1f4 --- /dev/null +++ b/processor/pytransform/factory_test.go @@ -0,0 +1,16 @@ +package pytransform + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "go.opentelemetry.io/collector/component/componenttest" +) + +func TestCreateDefaultConfig(t *testing.T) { + factory := NewFactory() + cfg := factory.CreateDefaultConfig() + assert.NoError(t, componenttest.CheckConfigStruct(cfg)) + assert.NotNil(t, cfg) + assert.Equal(t, "send(event)", cfg.(*Config).Code) +} diff --git a/processor/pytransform/go.mod b/processor/pytransform/go.mod new file mode 100644 index 000000000000..14f60d1120fd --- /dev/null +++ b/processor/pytransform/go.mod @@ -0,0 +1,45 @@ +module github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor + +go 1.20 + +require ( + github.com/daidokoro/go-embed-python v0.0.1 + github.com/stretchr/testify v1.8.4 + go.opentelemetry.io/collector/component v0.84.0 + go.opentelemetry.io/collector/consumer v0.84.0 + go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 + go.opentelemetry.io/collector/processor v0.84.0 + go.uber.org/zap v1.25.0 +) + +require ( + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang/protobuf v1.5.3 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/knadh/koanf/maps v0.1.1 // indirect + github.com/knadh/koanf/providers/confmap v0.1.0 // indirect + github.com/knadh/koanf/v2 v2.0.1 // indirect + github.com/mitchellh/copystructure v1.2.0 // indirect + github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect + github.com/mitchellh/reflectwalk v1.0.2 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect + github.com/rogpeppe/go-internal v1.10.0 // indirect + github.com/sirupsen/logrus v1.9.0 // indirect + go.opentelemetry.io/collector/config/configtelemetry v0.84.0 // indirect + go.opentelemetry.io/collector/confmap v0.84.0 // indirect + go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 // indirect + go.opentelemetry.io/otel v1.16.0 // indirect + go.opentelemetry.io/otel/metric v1.16.0 // indirect + go.opentelemetry.io/otel/trace v1.16.0 // indirect + go.uber.org/multierr v1.11.0 // indirect + golang.org/x/net v0.14.0 // indirect + golang.org/x/sys v0.11.0 // indirect + golang.org/x/text v0.12.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect + google.golang.org/grpc v1.57.0 // indirect + google.golang.org/protobuf v1.31.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) diff --git a/processor/pytransform/go.sum b/processor/pytransform/go.sum new file mode 100644 index 000000000000..94c7952877cd --- /dev/null +++ b/processor/pytransform/go.sum @@ -0,0 +1,124 @@ +github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A= +github.com/cyphar/filepath-securejoin v0.2.3 h1:YX6ebbZCZP7VkM3scTTokDgBL2TY741X51MTk3ycuNI= +github.com/daidokoro/go-embed-python v0.0.1 h1:4lPma0kaBwBfRMeEtk3KIf58ePzEffDRBl8sR32fvkw= +github.com/daidokoro/go-embed-python v0.0.1/go.mod h1:tsAnaTsq/UuEXbHXhzCBMYyz3TYudcsG9F+h2AhU4Xg= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs= +github.com/knadh/koanf/maps v0.1.1/go.mod h1:npD/QZY3V6ghQDdcQzl1W4ICNVTkohC8E73eI2xW4yI= +github.com/knadh/koanf/providers/confmap v0.1.0 h1:gOkxhHkemwG4LezxxN8DMOFopOPghxRVp7JbIvdvqzU= +github.com/knadh/koanf/providers/confmap v0.1.0/go.mod h1:2uLhxQzJnyHKfxG927awZC7+fyHFdQkd697K4MdLnIU= +github.com/knadh/koanf/v2 v2.0.1 h1:1dYGITt1I23x8cfx8ZnldtezdyaZtfAuRtIFOiRzK7g= +github.com/knadh/koanf/v2 v2.0.1/go.mod h1:ZeiIlIDXTE7w1lMT6UVcNiRAS2/rCeLn/GdLNvY1Dus= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= +github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= +github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 h1:BpfhmLKZf+SjVanKKhCgf3bg+511DmU9eDQTen7LLbY= +github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= +github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= +github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.opentelemetry.io/collector/component v0.84.0 h1:bh4Is5Z7TjuyF7Mab0rSNh2q3y15fImdNDRXqrqGlbA= +go.opentelemetry.io/collector/component v0.84.0/go.mod h1:uXteRaacYXXYhzeCJe5ClLz5gLzleXWz01IZ730w7FA= +go.opentelemetry.io/collector/config/configtelemetry v0.84.0 h1:pnZiYtppJN6SlzJNcrkA8R+Ur63e33qMO260m8JbK18= +go.opentelemetry.io/collector/config/configtelemetry v0.84.0/go.mod h1:+LAXM5WFMW/UbTlAuSs6L/W72WC+q8TBJt/6z39FPOU= +go.opentelemetry.io/collector/confmap v0.84.0 h1:fS62yIRrsTQwe1gyuEc8TQM0yUNfSAzPVi0A1665ZpQ= +go.opentelemetry.io/collector/confmap v0.84.0/go.mod h1:/SNHqYkLagF0TjBjQyCy2Gt3ZF6hTK8VKbaan/ZHuJs= +go.opentelemetry.io/collector/consumer v0.84.0 h1:sz8mXIdPACJArlRyFNXA1SScVoo954IU1qp9V78VUxc= +go.opentelemetry.io/collector/consumer v0.84.0/go.mod h1:Mu+KeuorwHHWd6iGxU7DMAhgsHZmmzmQgf3sSWkugmM= +go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 h1:C9o0mbP0MyygqFnKueVQK/v9jef6zvuttmTGlKaqhgw= +go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014/go.mod h1:0mE3mDLmUrOXVoNsuvj+7dV14h/9HFl/Fy9YTLoLObo= +go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 h1:iT5qH0NLmkGeIdDtnBogYDx7L58t6CaWGL378DEo2QY= +go.opentelemetry.io/collector/pdata v1.0.0-rcv0014/go.mod h1:BRvDrx43kiSoUx3mr7SoA7h9B8+OY99mUK+CZSQFWW4= +go.opentelemetry.io/collector/processor v0.84.0 h1:6VM5HLdkroeqNZ/jvjxA4nsgweJ87FDMsnNnzVBHpcY= +go.opentelemetry.io/collector/processor v0.84.0/go.mod h1:KWgBNQuA6wmmRqJwfvPaRybK2Di9X8nE2fraGuVLNJo= +go.opentelemetry.io/otel v1.16.0 h1:Z7GVAX/UkAXPKsy94IU+i6thsQS4nb7LviLpnaNeW8s= +go.opentelemetry.io/otel v1.16.0/go.mod h1:vl0h9NUa1D5s1nv3A5vZOYWn8av4K8Ml6JDeHrT/bx4= +go.opentelemetry.io/otel/metric v1.16.0 h1:RbrpwVG1Hfv85LgnZ7+txXioPDoh6EdbZHo26Q3hqOo= +go.opentelemetry.io/otel/metric v1.16.0/go.mod h1:QE47cpOmkwipPiefDwo2wDzwJrlfxxNYodqc4xnGCo4= +go.opentelemetry.io/otel/trace v1.16.0 h1:8JRpaObFoW0pxuVPapkgH8UhHQj+bJW8jJsCZEu5MQs= +go.opentelemetry.io/otel/trace v1.16.0/go.mod h1:Yt9vYq1SdNz3xdjZZK7wcXv1qv2pwLkqr2QVwea0ef0= +go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= +go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= +go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/zap v1.25.0 h1:4Hvk6GtkucQ790dqmj7l1eEnRdKm3k3ZUrUMS2d5+5c= +go.uber.org/zap v1.25.0/go.mod h1:JIAUzQIH94IC4fOJQm7gMmBJP5k7wQfdcnYdPoEXJYk= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= +golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= +golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc h1:XSJ8Vk1SWuNr8S18z1NZSziL0CPIXLCCMDOEFtHBOFc= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA= +google.golang.org/grpc v1.57.0 h1:kfzNeI/klCGD2YPMUlaGNT3pxvYfga7smW3Vth8Zsiw= +google.golang.org/grpc v1.57.0/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo= +google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= +google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/processor/pytransform/helper_scripts.go b/processor/pytransform/helper_scripts.go new file mode 100644 index 000000000000..e55af6949599 --- /dev/null +++ b/processor/pytransform/helper_scripts.go @@ -0,0 +1,32 @@ +package pytransform + +// prefixScript is a python script that is prepended to the user's script. +var prefixScript = ` +import json +import os +import urllib.request +import sys + +# override print function to send logs back to otel +def print(msg, endpoint=os.environ.get("LOG_URL"), pipeline=os.environ.get("PIPELINE")): + # Encode string to bytes + data_bytes = json.dumps({ + "message":str(msg), + "pipeline": pipeline, + }).encode('utf-8') + + req = urllib.request.Request(endpoint, data=data_bytes, method='POST') + + with urllib.request.urlopen(req) as resp: + return resp.read().decode() + + +def send(event): + sys.stdout.write(json.dumps(event)) + +# read input event from env var +try: + event = json.loads(os.environ.get("INPUT")) +except Exception as e: + raise Exception(f"error loading input event from env var: {e}") +` diff --git a/processor/pytransform/internal/metadata/metadata.go b/processor/pytransform/internal/metadata/metadata.go new file mode 100644 index 000000000000..dffae2856bd8 --- /dev/null +++ b/processor/pytransform/internal/metadata/metadata.go @@ -0,0 +1,12 @@ +package metadata + +import ( + "go.opentelemetry.io/collector/component" +) + +const ( + Type = "pytransform" + LogsStability = component.StabilityLevelAlpha + MetricsStability = component.StabilityLevelAlpha + TracesStability = component.StabilityLevelAlpha +) diff --git a/processor/pytransform/logprocessor.go b/processor/pytransform/logprocessor.go new file mode 100644 index 000000000000..de8ae7c10043 --- /dev/null +++ b/processor/pytransform/logprocessor.go @@ -0,0 +1,67 @@ +package pytransform + +import ( + "context" + "fmt" + + "github.com/daidokoro/go-embed-python/python" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/plog" + "go.uber.org/zap" +) + +func newLogsProcessor(ctx context.Context, logger *zap.Logger, + cfg *Config, ep *python.EmbeddedPython, consumer consumer.Logs) *logsProcessor { + return &logsProcessor{ + logger: logger, + cfg: cfg, + embeddedpython: ep, + next: consumer, + } +} + +type logsProcessor struct { + plog.JSONMarshaler + plog.JSONUnmarshaler + logger *zap.Logger + cfg *Config + embeddedpython *python.EmbeddedPython + next consumer.Logs +} + +func (lp *logsProcessor) Start(context.Context, component.Host) error { + startLogServer(lp.logger) + return nil +} + +func (lp *logsProcessor) Shutdown(context.Context) error { + return closeLogServer() +} + +func (lp *logsProcessor) ConsumeLogs(ctx context.Context, ld plog.Logs) error { + b, err := lp.MarshalLogs(ld) + if err != nil { + return err + } + + output, err := runPythonCode(lp.cfg.Code, string(b), "logs", lp.embeddedpython, lp.logger) + if err != nil { + return err + } + + if len(output) == 0 { + lp.logger.Error("python script returned empty output, returning record unchanged") + return lp.next.ConsumeLogs(ctx, ld) + } + + if ld, err = lp.UnmarshalLogs(output); err != nil { + return fmt.Errorf("error unmarshalling logs data from python: %w", err) + } + + return lp.next.ConsumeLogs(ctx, ld) +} + +func (lp *logsProcessor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/pytransform/logprocessor_test.go b/processor/pytransform/logprocessor_test.go new file mode 100644 index 000000000000..62e3badbcfb0 --- /dev/null +++ b/processor/pytransform/logprocessor_test.go @@ -0,0 +1,51 @@ +package pytransform + +import ( + "context" + "io" + "os" + "testing" + + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/plog" + "go.uber.org/zap" +) + +type fakeLogConsumer struct { + plog.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeLogConsumer) ConsumeLogs(ctx context.Context, ld plog.Logs) error { + b, err := f.MarshalLogs(ld) + require.NoError(f.t, err) + require.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeLogConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: false} +} + +func TestLogConsumer(t *testing.T) { + t.Parallel() + startLogServer(zap.NewNop()) + ep, err := getEmbeddedPython() + require.NoError(t, err) + next := &fakeLogConsumer{t: t, expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"test.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456543000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456572000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456576000","body":{"stringValue":"2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456611000","body":{"stringValue":"2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456668000","body":{"stringValue":"2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`} + lp := newLogsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) + + f, err := os.Open("testdata/log.json") + require.NoError(t, err) + + b, err := io.ReadAll(f) + require.NoError(t, err) + + ld, err := (&plog.JSONUnmarshaler{}).UnmarshalLogs(b) + require.NoError(t, err) + + err = lp.ConsumeLogs(context.Background(), ld) + require.NoError(t, err) +} diff --git a/processor/pytransform/metricprocessor.go b/processor/pytransform/metricprocessor.go new file mode 100644 index 000000000000..a689190a430a --- /dev/null +++ b/processor/pytransform/metricprocessor.go @@ -0,0 +1,67 @@ +package pytransform + +import ( + "context" + "fmt" + + "github.com/daidokoro/go-embed-python/python" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/pmetric" + "go.uber.org/zap" +) + +func newMetricsProcessor(ctx context.Context, logger *zap.Logger, + cfg *Config, ep *python.EmbeddedPython, consumer consumer.Metrics) *metricProcessor { + return &metricProcessor{ + logger: logger, + cfg: cfg, + embeddedpython: ep, + next: consumer, + } +} + +type metricProcessor struct { + pmetric.JSONMarshaler + pmetric.JSONUnmarshaler + logger *zap.Logger + cfg *Config + embeddedpython *python.EmbeddedPython + next consumer.Metrics +} + +func (mp *metricProcessor) Start(context.Context, component.Host) error { + startLogServer(mp.logger) + return nil +} + +func (mp *metricProcessor) Shutdown(context.Context) error { + return closeLogServer() +} + +func (mp *metricProcessor) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error { + b, err := mp.MarshalMetrics(md) + if err != nil { + return err + } + + output, err := runPythonCode(mp.cfg.Code, string(b), "metrics", mp.embeddedpython, mp.logger) + if err != nil { + return err + } + + if len(output) == 0 { + mp.logger.Error("python script returned empty output, returning record unchanged") + return mp.next.ConsumeMetrics(ctx, md) + } + + if md, err = mp.UnmarshalMetrics(output); err != nil { + return fmt.Errorf("error unmarshalling metric data from python: %v", err) + } + + return mp.next.ConsumeMetrics(ctx, md) +} + +func (mp *metricProcessor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/pytransform/metricprocessor_test.go b/processor/pytransform/metricprocessor_test.go new file mode 100644 index 000000000000..1851ecc04487 --- /dev/null +++ b/processor/pytransform/metricprocessor_test.go @@ -0,0 +1,53 @@ +package pytransform + +import ( + "context" + "io" + "os" + "testing" + + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/pmetric" + "go.uber.org/zap" +) + +type fakeMetricConsumer struct { + pmetric.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeMetricConsumer) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error { + b, err := f.MarshalMetrics(md) + require.NoError(f.t, err) + require.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeMetricConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} + +func TestMetricConsumer(t *testing.T) { + t.Parallel() + startLogServer(zap.NewNop()) + ep, err := getEmbeddedPython() + require.NoError(t, err) + next := &fakeMetricConsumer{t: t, expected: `{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{"name":"otelcol/hostmetricsreceiver/memory","version":"0.84.0"},"metrics":[{"name":"system.memory.usage","description":"Bytes of memory in use.","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"1874247680"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"29214199808"}],"aggregationTemporality":2}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.9.0"}]}`} + mp := newMetricsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) + + f, err := os.Open("testdata/metrics.json") + require.NoError(t, err) + + defer f.Close() + + b, err := io.ReadAll(f) + require.NoError(t, err) + + md, err := (&pmetric.JSONUnmarshaler{}).UnmarshalMetrics(b) + require.NoError(t, err) + + err = mp.ConsumeMetrics(context.Background(), md) + require.NoError(t, err) +} diff --git a/processor/pytransform/pytransform.go b/processor/pytransform/pytransform.go new file mode 100644 index 000000000000..3e4dd39d5e7a --- /dev/null +++ b/processor/pytransform/pytransform.go @@ -0,0 +1,136 @@ +package pytransform + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "math/rand" + "net/http" + "os" + "sync" + "time" + + "github.com/daidokoro/go-embed-python/python" + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor/internal/metadata" + "go.uber.org/zap" +) + +var ( + pylogserver *http.Server + embeddedPython *python.EmbeddedPython + initPythonEmbedOnce sync.Once + startOnce sync.Once + closeOnce sync.Once +) + +// The log server receives print statements from the python script +// and prints them using the zap logger to stdout + +func startLogServer(logger *zap.Logger) { + startOnce.Do(func() { + r := rand.New(rand.NewSource(time.Now().UnixNano())) + port := r.Intn(60000-10000) + 10000 + mux := http.NewServeMux() + + // handle log events from python script + mux.Handle("/log", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + defer r.Body.Close() + var m map[string]string + + if err := json.NewDecoder(r.Body).Decode(&m); err != nil { + logger.Error("failed reading python script log body", zap.Error(err)) + } + + logger.Info(m["message"], + zap.String("func", "python.print()"), + zap.String("source", fmt.Sprintf("pytransform/%s", m["pipeline"]))) + })) + + pylogserver = &http.Server{ + Addr: fmt.Sprintf(":%d", port), + Handler: mux, + } + + go func() { + logger.Info("starting python logging http server", zap.String("addr", pylogserver.Addr)) + if err := pylogserver.ListenAndServe(); err != http.ErrServerClosed { + logger.Error("error starting http server", zap.Error(err)) + return + } + + logger.Info("http server closed") + }() + }) + return +} + +func closeLogServer() error { + var err error + closeOnce.Do(func() { + if pylogserver != nil { + err = pylogserver.Close() + } + }) + return err +} + +func getEmbeddedPython() (*python.EmbeddedPython, error) { + var err error + initPythonEmbedOnce.Do(func() { + embeddedPython, err = python.NewEmbeddedPython(metadata.Type) + }) + return embeddedPython, err +} + +// runPythonCode - runs a python script and returns the stdout and stderr in bytes +func runPythonCode(code, pyinput, pipeline string, ep *python.EmbeddedPython, + logger *zap.Logger) ([]byte, error) { + + result := make(chan []byte) + errchan := make(chan error) + + code = fmt.Sprintf("%s\n%s", prefixScript, code) + cmd := ep.PythonCmd("-c", code) + + go func() { + var buf bytes.Buffer + + // add script prefix + cmd.Stdout = &buf + cmd.Stderr = &buf + cmd.Env = append( + os.Environ(), + fmt.Sprintf("INPUT=%s", pyinput), + fmt.Sprintf("PIPELINE=%s", pipeline), + fmt.Sprintf("LOG_URL=http://localhost%s/log", pylogserver.Addr)) + + if err := cmd.Run(); err != nil { + err = fmt.Errorf("error running python script: %s - %v", buf.String(), err) + logger.Error(err.Error()) + errchan <- err + } + + result <- buf.Bytes() + }() + + select { + case err := <-errchan: + return nil, err + case b := <-result: + return b, nil + case <-time.After(1500 * time.Millisecond): + logger.Error("timeout running python script") + logger.Debug("killing python process", zap.Int("pid", cmd.Process.Pid)) + process, err := os.FindProcess(cmd.Process.Pid) + if err != nil { + return nil, fmt.Errorf("error finding python process after timeout: %v", err) + } + + if err = process.Kill(); err != nil { + return nil, fmt.Errorf("error while cancelling python process after timeout: %v", err) + } + + return nil, errors.New("timeout running python script") + } +} diff --git a/processor/pytransform/pytransform_test.go b/processor/pytransform/pytransform_test.go new file mode 100644 index 000000000000..b1c4990e4a71 --- /dev/null +++ b/processor/pytransform/pytransform_test.go @@ -0,0 +1,65 @@ +package pytransform + +import ( + "errors" + "strings" + "testing" + + "github.com/stretchr/testify/require" + "go.uber.org/zap" +) + +func TestRunPythonCode(t *testing.T) { + startLogServer(zap.NewNop()) + ep, err := getEmbeddedPython() + require.NoError(t, err) + + testcases := []struct { + name string + event string + code string + expectError error + expectedOutput string + }{ + { + name: "update event input", + event: `{"name": "some_transform"}`, + code: `event['name'] = "pytransform"; send(event)`, + expectedOutput: `{"name": "pytransform"}`, + expectError: nil, + }, + { + name: "bad input", + event: `{"name": "some_transform"`, + code: `send(event)`, + expectedOutput: "", + expectError: errors.New("error loading input event from env var"), + }, + { + name: "timeout", + event: `{"name": "some_transform"}`, + code: `import time; time.sleep(2); send(event)`, + expectedOutput: "", + expectError: errors.New("timeout running python script"), + }, + { + name: "bad code", + event: `{"name": "some_transform"}`, + code: `send(event`, + expectedOutput: "", + expectError: errors.New("error running python script"), + }, + } + + for _, tt := range testcases { + t.Run(tt.name, func(t *testing.T) { + out, err := runPythonCode(tt.code, tt.event, "testing", ep, zap.NewNop()) + if tt.expectError != nil { + require.True(t, strings.Contains(err.Error(), tt.expectError.Error())) + } else { + require.NoError(t, err) + } + require.Equal(t, tt.expectedOutput, string(out)) + }) + } +} diff --git a/processor/pytransform/testdata/builder.yaml b/processor/pytransform/testdata/builder.yaml new file mode 100644 index 000000000000..b788b575ed2a --- /dev/null +++ b/processor/pytransform/testdata/builder.yaml @@ -0,0 +1,19 @@ +dist: + name: ddot + output_path: ./cmd/ddot + description: Daidokoro Open Telemetry Contrib binary. + version: 1.0.0 + +processors: + # add my processor + - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor v0.0.0" + path: "../" + +exporters: + - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.84.0 + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.84.0 + +receivers: + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.84.0 + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.84.0 + - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.84.0 diff --git a/processor/pytransform/testdata/config.yaml b/processor/pytransform/testdata/config.yaml new file mode 100644 index 000000000000..9fb906d42624 --- /dev/null +++ b/processor/pytransform/testdata/config.yaml @@ -0,0 +1,124 @@ +receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + grpc: + endpoint: "0.0.0.0:4317" + + filelog: + start_at: beginning + include_file_name: true + include: + - $LOGFILE + + operators: + - type: move + from: attributes["log.file.name"] + to: resource["log.file.name"] + + - type: add + field: attributes.app + value: dev + +processors: + pytransform/logs: + code: | + # edit resource attributes + for data in event['resourceLogs']: + for attr in data['resource']['attributes']: + attr['value']['stringValue'] = 'source.log' + + # filter/delete logs + for data in event['resourceLogs']: + for slog in data['scopeLogs']: + slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'internal' not in lr['body']['stringValue']] + + # add an attribute to each log + for lr in slog['logRecords']: + lr['attributes'].append({ + 'key': 'language', + 'value': { + 'stringValue': 'golang' + }}) + + send(event) + + pytransform/metrics: + code: | + for md in event['resourceMetrics']: + # if resources are empty + if not md['resource']: + md['resource'] = { + 'attributes': [ + { + "key": "source", + "value": { + "stringValue": "pytransform" + } + } + ] + } + + # prefix each metric name with pytransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'pytransform.' + m['name'] + + send(event) + + pytransform/traces: + code: | + for td in event['resourceSpans']: + # add resource attribute + td['resource']['attributes'].append({ + 'key': 'source', + 'value': { + 'stringValue': 'pytransform' + } + }) + + # filter spans with http.target /roll attribute + has_roll = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/roll'] + for sd in td['scopeSpans']: + sd['spans'] = [ + s for s in sd['spans'] + if not has_roll(s['attributes']) + ] + + send(event) + +exporters: + logging: + verbosity: detailed + + +service: + pipelines: + logs: + receivers: + - filelog + processors: + - pytransform/logs + exporters: + - logging + + metrics: + receivers: + - otlp + processors: + - pytransform/metrics + exporters: + - logging + + traces: + receivers: + - otlp + processors: + - pytransform/traces + exporters: + - logging + + # telemetry: + # logs: + # level: debug diff --git a/processor/pytransform/testdata/log_event_example.json b/processor/pytransform/testdata/log_event_example.json new file mode 100644 index 000000000000..598184dd0bd8 --- /dev/null +++ b/processor/pytransform/testdata/log_event_example.json @@ -0,0 +1,119 @@ +{ + "resourceLogs": [ + { + "resource": { + "attributes": [ + { + "key": "log.file.name", + "value": { + "stringValue": "test.log" + } + } + ] + }, + "scopeLogs": [ + { + "scope": {}, + "logRecords": [ + { + "observedTimeUnixNano": "1694127596456358000", + "body": { + "stringValue": "2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456543000", + "body": { + "stringValue": "2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456572000", + "body": { + "stringValue": "2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456576000", + "body": { + "stringValue": "2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456611000", + "body": { + "stringValue": "2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456668000", + "body": { + "stringValue": "2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + } + ] + } + ] + } + ] + } \ No newline at end of file diff --git a/processor/pytransform/testdata/metric_event_example.json b/processor/pytransform/testdata/metric_event_example.json new file mode 100644 index 000000000000..026f06779aa7 --- /dev/null +++ b/processor/pytransform/testdata/metric_event_example.json @@ -0,0 +1,55 @@ +{ + "resourceMetrics": [ + { + "resource": {}, + "scopeMetrics": [ + { + "scope": { + "name": "otelcol/hostmetricsreceiver/memory", + "version": "0.84.0" + }, + "metrics": [ + { + "name": "system.memory.usage", + "description": "Bytes of memory in use.", + "unit": "By", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "state", + "value": { + "stringValue": "used" + } + } + ], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "1874247680" + }, + { + "attributes": [ + { + "key": "state", + "value": { + "stringValue": "free" + } + } + ], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "29214199808" + } + ], + "aggregationTemporality": 2 + } + } + ] + } + ], + "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" + } + ] + } + \ No newline at end of file diff --git a/processor/pytransform/testdata/test.log b/processor/pytransform/testdata/test.log new file mode 100644 index 000000000000..81a4bf6c3a5c --- /dev/null +++ b/processor/pytransform/testdata/test.log @@ -0,0 +1,7 @@ +2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {"version": "0.84.0", "date": "2023-08-29T18:58:24Z"} +2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {"path": "builder.yaml"} +2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {"go-executable": "/usr/local/go/bin/go"} +2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {"path": "./cmd/ddot"} +2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules +2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling +2023-09-06T01:09:47.694+0200 INFO builder/main.go:102 Compiled {"binary": "./cmd/ddot/ddot"} \ No newline at end of file diff --git a/processor/pytransform/testdata/trace_event_example.json b/processor/pytransform/testdata/trace_event_example.json new file mode 100644 index 000000000000..ef165a57ceaa --- /dev/null +++ b/processor/pytransform/testdata/trace_event_example.json @@ -0,0 +1,135 @@ +{ + "resourceSpans": [ + { + "resource": { + "attributes": [ + { + "key": "telemetry.sdk.language", + "value": { + "stringValue": "python" + } + }, + { + "key": "telemetry.sdk.name", + "value": { + "stringValue": "opentelemetry" + } + }, + { + "key": "telemetry.sdk.version", + "value": { + "stringValue": "1.19.0" + } + }, + { + "key": "telemetry.auto.version", + "value": { + "stringValue": "0.40b0" + } + }, + { + "key": "service.name", + "value": { + "stringValue": "unknown_service" + } + } + ] + }, + "scopeSpans": [ + { + "scope": { + "name": "opentelemetry.instrumentation.flask", + "version": "0.40b0" + }, + "spans": [ + { + "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", + "spanId": "88079ad5c94b5b13", + "parentSpanId": "", + "name": "/roll", + "kind": 2, + "startTimeUnixNano": "1694388218052842000", + "endTimeUnixNano": "1694388218053415000", + "attributes": [ + { + "key": "http.method", + "value": { + "stringValue": "GET" + } + }, + { + "key": "http.server_name", + "value": { + "stringValue": "0.0.0.0" + } + }, + { + "key": "http.scheme", + "value": { + "stringValue": "http" + } + }, + { + "key": "net.host.port", + "value": { + "intValue": "5001" + } + }, + { + "key": "http.host", + "value": { + "stringValue": "localhost:5001" + } + }, + { + "key": "http.target", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "net.peer.ip", + "value": { + "stringValue": "127.0.0.1" + } + }, + { + "key": "http.user_agent", + "value": { + "stringValue": "curl/7.87.0" + } + }, + { + "key": "net.peer.port", + "value": { + "intValue": "52365" + } + }, + { + "key": "http.flavor", + "value": { + "stringValue": "1.1" + } + }, + { + "key": "http.route", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "http.status_code", + "value": { + "intValue": "200" + } + } + ], + "status": {} + } + ] + } + ] + } + ] + } + \ No newline at end of file diff --git a/processor/pytransform/traceprocessor.go b/processor/pytransform/traceprocessor.go new file mode 100644 index 000000000000..254a1e3e9967 --- /dev/null +++ b/processor/pytransform/traceprocessor.go @@ -0,0 +1,67 @@ +package pytransform + +import ( + "context" + "fmt" + + "github.com/daidokoro/go-embed-python/python" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/ptrace" + "go.uber.org/zap" +) + +func newTraceProcessor(ctx context.Context, logger *zap.Logger, + cfg *Config, ep *python.EmbeddedPython, consumer consumer.Traces) *traceProcessor { + return &traceProcessor{ + logger: logger, + cfg: cfg, + embeddedpython: ep, + next: consumer, + } +} + +type traceProcessor struct { + ptrace.JSONMarshaler + ptrace.JSONUnmarshaler + logger *zap.Logger + cfg *Config + embeddedpython *python.EmbeddedPython + next consumer.Traces +} + +func (tp *traceProcessor) Start(context.Context, component.Host) error { + startLogServer(tp.logger) + return nil +} + +func (tp *traceProcessor) Shutdown(context.Context) error { + return closeLogServer() +} + +func (tp *traceProcessor) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { + b, err := tp.MarshalTraces(td) + if err != nil { + return err + } + + output, err := runPythonCode(tp.cfg.Code, string(b), "traces", tp.embeddedpython, tp.logger) + if err != nil { + return err + } + + if len(output) == 0 { + tp.logger.Error("python script returned empty output, returning record unchanged") + return tp.next.ConsumeTraces(ctx, td) + } + + if td, err = tp.UnmarshalTraces(output); err != nil { + return fmt.Errorf("error unmarshalling metric data from python: %v", err) + } + + return tp.next.ConsumeTraces(ctx, td) +} + +func (tp *traceProcessor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/pytransform/traceprocessor_test.go b/processor/pytransform/traceprocessor_test.go new file mode 100644 index 000000000000..69e1c04c824d --- /dev/null +++ b/processor/pytransform/traceprocessor_test.go @@ -0,0 +1,51 @@ +package pytransform + +import ( + "context" + "io" + "os" + "testing" + + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/ptrace" + "go.uber.org/zap" +) + +type fakeTraceConsumer struct { + ptrace.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeTraceConsumer) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { + b, err := f.MarshalTraces(td) + require.NoError(f.t, err) + require.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeTraceConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: false} +} + +func TestTraceConsumer(t *testing.T) { + t.Parallel() + startLogServer(zap.NewNop()) + ep, err := getEmbeddedPython() + require.NoError(t, err) + next := &fakeTraceConsumer{t: t, expected: `{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"service.name","value":{"stringValue":"unknown_service"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9cb5bf738137b2248dc7b20445ec2e1c","spanId":"88079ad5c94b5b13","parentSpanId":"","name":"/roll","kind":2,"startTimeUnixNano":"1694388218052842000","endTimeUnixNano":"1694388218053415000","attributes":[{"key":"http.method","value":{"stringValue":"GET"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"5001"}},{"key":"http.host","value":{"stringValue":"localhost:5001"}},{"key":"http.target","value":{"stringValue":"/roll"}},{"key":"net.peer.ip","value":{"stringValue":"127.0.0.1"}},{"key":"http.user_agent","value":{"stringValue":"curl/7.87.0"}},{"key":"net.peer.port","value":{"intValue":"52365"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/roll"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}`} + tp := newTraceProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) + + f, err := os.Open("testdata/traces.json") + require.NoError(t, err) + + b, err := io.ReadAll(f) + require.NoError(t, err) + + td, err := (&ptrace.JSONUnmarshaler{}).UnmarshalTraces(b) + require.NoError(t, err) + + err = tp.ConsumeTraces(context.Background(), td) + require.NoError(t, err) +} From a028b18bbc45961dbc47c866112f2212ce4d5d0d Mon Sep 17 00:00:00 2001 From: daidokoro Date: Mon, 18 Sep 2023 01:03:59 +0200 Subject: [PATCH 02/16] fix test file paths --- processor/pytransform/logprocessor_test.go | 2 +- processor/pytransform/metricprocessor_test.go | 2 +- processor/pytransform/traceprocessor_test.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/processor/pytransform/logprocessor_test.go b/processor/pytransform/logprocessor_test.go index 62e3badbcfb0..636ce9871bd8 100644 --- a/processor/pytransform/logprocessor_test.go +++ b/processor/pytransform/logprocessor_test.go @@ -37,7 +37,7 @@ func TestLogConsumer(t *testing.T) { next := &fakeLogConsumer{t: t, expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"test.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456543000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456572000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456576000","body":{"stringValue":"2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456611000","body":{"stringValue":"2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456668000","body":{"stringValue":"2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`} lp := newLogsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - f, err := os.Open("testdata/log.json") + f, err := os.Open("testdata/log_event_example.json") require.NoError(t, err) b, err := io.ReadAll(f) diff --git a/processor/pytransform/metricprocessor_test.go b/processor/pytransform/metricprocessor_test.go index 1851ecc04487..c3925126cc6a 100644 --- a/processor/pytransform/metricprocessor_test.go +++ b/processor/pytransform/metricprocessor_test.go @@ -37,7 +37,7 @@ func TestMetricConsumer(t *testing.T) { next := &fakeMetricConsumer{t: t, expected: `{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{"name":"otelcol/hostmetricsreceiver/memory","version":"0.84.0"},"metrics":[{"name":"system.memory.usage","description":"Bytes of memory in use.","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"1874247680"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"29214199808"}],"aggregationTemporality":2}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.9.0"}]}`} mp := newMetricsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - f, err := os.Open("testdata/metrics.json") + f, err := os.Open("testdata/metric_event_example.json") require.NoError(t, err) defer f.Close() diff --git a/processor/pytransform/traceprocessor_test.go b/processor/pytransform/traceprocessor_test.go index 69e1c04c824d..ca15aaa81d34 100644 --- a/processor/pytransform/traceprocessor_test.go +++ b/processor/pytransform/traceprocessor_test.go @@ -37,7 +37,7 @@ func TestTraceConsumer(t *testing.T) { next := &fakeTraceConsumer{t: t, expected: `{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"service.name","value":{"stringValue":"unknown_service"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9cb5bf738137b2248dc7b20445ec2e1c","spanId":"88079ad5c94b5b13","parentSpanId":"","name":"/roll","kind":2,"startTimeUnixNano":"1694388218052842000","endTimeUnixNano":"1694388218053415000","attributes":[{"key":"http.method","value":{"stringValue":"GET"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"5001"}},{"key":"http.host","value":{"stringValue":"localhost:5001"}},{"key":"http.target","value":{"stringValue":"/roll"}},{"key":"net.peer.ip","value":{"stringValue":"127.0.0.1"}},{"key":"http.user_agent","value":{"stringValue":"curl/7.87.0"}},{"key":"net.peer.port","value":{"intValue":"52365"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/roll"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}`} tp := newTraceProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - f, err := os.Open("testdata/traces.json") + f, err := os.Open("testdata/trace_event_example.json") require.NoError(t, err) b, err := io.ReadAll(f) From c653ae980375b2da9fcbed62259346d61e693865 Mon Sep 17 00:00:00 2001 From: daidokoro Date: Mon, 18 Sep 2023 01:12:41 +0200 Subject: [PATCH 03/16] readme --- processor/pytransform/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/processor/pytransform/README.md b/processor/pytransform/README.md index 4c8bde5aa3e1..cbae6e224502 100644 --- a/processor/pytransform/README.md +++ b/processor/pytransform/README.md @@ -476,4 +476,4 @@ service: ### Warnings -The Pytransform processor allows you to modify all aspects of you telemetry data. This can result in valid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file +The Pytransform processor allows you to modify all aspects of you telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file From 0a976419e1afc03cc26179953db3ee096a66b3b2 Mon Sep 17 00:00:00 2001 From: daidokoro Date: Mon, 18 Sep 2023 01:20:27 +0200 Subject: [PATCH 04/16] modify readme --- processor/pytransform/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/processor/pytransform/README.md b/processor/pytransform/README.md index cbae6e224502..89530b3496c8 100644 --- a/processor/pytransform/README.md +++ b/processor/pytransform/README.md @@ -30,7 +30,9 @@ processors: ## How it works -The python language is embedded to the Open Telemetry binary, this allows you to run this processor without having python installed on the host itself. +The python language is embedded to the Open Telemetry binary, this allows you to run this processor without having python installed on the host itself. + +This design is based on Embedded Python for Go Project found [here](https://github.com/kluctl/go-embed-python). When a python transform is executed, a controlled subprocess is called using the embedded python implementation that calls the user code. From c0678fbbe605a42b2c0479c1098a494068f87549 Mon Sep 17 00:00:00 2001 From: daidokoro Date: Sun, 24 Sep 2023 15:39:44 +0200 Subject: [PATCH 05/16] removed pytransform processor code --- processor/pytransform/README.md | 481 ------------------ processor/pytransform/config.go | 5 - processor/pytransform/factory.go | 73 --- processor/pytransform/factory_test.go | 16 - processor/pytransform/go.mod | 45 -- processor/pytransform/go.sum | 124 ----- processor/pytransform/helper_scripts.go | 32 -- .../pytransform/internal/metadata/metadata.go | 12 - processor/pytransform/logprocessor.go | 67 --- processor/pytransform/logprocessor_test.go | 51 -- processor/pytransform/metricprocessor.go | 67 --- processor/pytransform/metricprocessor_test.go | 53 -- processor/pytransform/pytransform.go | 136 ----- processor/pytransform/pytransform_test.go | 65 --- processor/pytransform/testdata/builder.yaml | 19 - processor/pytransform/testdata/config.yaml | 124 ----- .../testdata/log_event_example.json | 119 ----- .../testdata/metric_event_example.json | 55 -- processor/pytransform/testdata/test.log | 7 - .../testdata/trace_event_example.json | 135 ----- processor/pytransform/traceprocessor.go | 67 --- processor/pytransform/traceprocessor_test.go | 51 -- 22 files changed, 1804 deletions(-) delete mode 100644 processor/pytransform/README.md delete mode 100644 processor/pytransform/config.go delete mode 100644 processor/pytransform/factory.go delete mode 100644 processor/pytransform/factory_test.go delete mode 100644 processor/pytransform/go.mod delete mode 100644 processor/pytransform/go.sum delete mode 100644 processor/pytransform/helper_scripts.go delete mode 100644 processor/pytransform/internal/metadata/metadata.go delete mode 100644 processor/pytransform/logprocessor.go delete mode 100644 processor/pytransform/logprocessor_test.go delete mode 100644 processor/pytransform/metricprocessor.go delete mode 100644 processor/pytransform/metricprocessor_test.go delete mode 100644 processor/pytransform/pytransform.go delete mode 100644 processor/pytransform/pytransform_test.go delete mode 100644 processor/pytransform/testdata/builder.yaml delete mode 100644 processor/pytransform/testdata/config.yaml delete mode 100644 processor/pytransform/testdata/log_event_example.json delete mode 100644 processor/pytransform/testdata/metric_event_example.json delete mode 100644 processor/pytransform/testdata/test.log delete mode 100644 processor/pytransform/testdata/trace_event_example.json delete mode 100644 processor/pytransform/traceprocessor.go delete mode 100644 processor/pytransform/traceprocessor_test.go diff --git a/processor/pytransform/README.md b/processor/pytransform/README.md deleted file mode 100644 index 89530b3496c8..000000000000 --- a/processor/pytransform/README.md +++ /dev/null @@ -1,481 +0,0 @@ -# pytransform - - -| Status | | -| ------------- |-----------| -| Stability | [alpha]: traces, metrics, logs | - - -[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta -[sumo]: https://github.com/SumoLogic/sumologic-otel-collector - - - -The pytransform processor modifies telemetry based on configuration using Python code. - -The processor leverages an embedded python version that has access to core python functionality and __does not__ require python be installed on the host. By leveraging Python code to modify telemetry, you have access to the full power of the Python language to modify data. - -## Config - -To configure pytransform, you add your code using the `code` option in the config. Each instance of pytransform can only have a single code section. - - -```yaml -processors: - pytransform: - code: | - {your python code} -``` - - -## How it works - -The python language is embedded to the Open Telemetry binary, this allows you to run this processor without having python installed on the host itself. - -This design is based on Embedded Python for Go Project found [here](https://github.com/kluctl/go-embed-python). - -When a python transform is executed, a controlled subprocess is called using the embedded python implementation that calls the user code. - -### Python Runtime and Helper Fucntions - -The pytransform processor sends the metric, trace or log events to the python runtime by setting the env variable `INPUT`. You do not need to read this environment variable, pytransform adds helper functions to handle this automatically. - -When a telemetry event is initialised in python, the event data will be accessible under the variable `event`. For example, the code below would print the event received from open telemetry. - -```yaml -processors: - pytransform: - code: | - print(event) -``` - -#### send() - -You are able to modify this event using python code, however you choose. This can be in the form of functions, classes, regex using the python [re](https://docs.python.org/3/library/re.html) library, etc. It is import however, to maintain the structure of the event as the modified event needs to be returned to the pytransform processor. Events can be returned using another helper function, `send`. - -For example, the config below adds a resource attribute to the telemetry event and returns it to the pytransform processor. - -```yaml -processors: - pytransform/logs: - code: | - # edit resource attributes - for data in event['resourceLogs']: - for attr in data['resource']['attributes']: - attr['value']['stringValue'] = 'source.log' - - send(event) -``` - -Note the use of the `send` function at the end of the code. If the event is not passed back to the processor using send, this will result in an error and the input event will be passed as unchanged. - -```log -pytransform/logprocessor.go:54 python script returned empty output, returning record unchanged {"kind": "processor", "name": "pytransform/test", "pipeline": "logs"} -``` - - -#### print() - -The `print` function in the pytransform Python runtime is overwritten with a special function that sends strings to the pytransform processor. These strings are then printed using the internal Open Telemetry logger. - -For example, take the following config: - -```yaml -processors: - pytransform: - code: | - print('hello world') - send(event) -``` - -This results in a log line appearing in the Open Telemetry process: - -```log -2023-09-16T18:00:21.551+0200 info pytransform/pytransform.go:45 hello world {"kind": "processor", "name": "pytransform/test", "pipeline": "logs", "src": "python.print()"} -``` - -The is useful for debugging python code. - -### Timeout - -There are a number of things that are possible running python within the Open Telemetry workflow, as such, to protect the runtime and host from long running zombie processes in python, each python invokation has a **maximum timeout of 1500ms** - -The goal is to encourage using this processor solely for complex telemetry transformations rather than long running tasks. - - -## Examples - -This section contains examples of the event payloads that are sent to the pytransform processor python runtime from each telemetry type. These examples can help you understand the structure of the telemetry events and how to modify them. - -##### Log Event Payload Example: - -```json -{ - "resourceLogs": [{ - "resource": { - "attributes": [{ - "key": "log.file.name", - "value": { - "stringValue": "test.log" - } - }] - }, - "scopeLogs": [{ - "scope": {}, - "logRecords": [{ - "observedTimeUnixNano": "1694127596456358000", - "body": { - "stringValue": "2023-09-06T01:09:24.045+0200 INFO starting app.", - "attributes": [{ - - "key": "app", - "value": { - "stringValue": "dev" - } - }], - "traceId": "", - "spanId": "" - } - }] - }] - }] -} -``` - -View the the log.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto) - -##### Metric Event Payload Example: - -```json -{ - "resourceMetrics": [{ - "resource": {}, - "scopeMetrics": [{ - "scope": { - "name": "otelcol/hostmetricsreceiver/memory", - "version": "0.84.0" - }, - "metrics": [{ - "name": "system.memory.usage", - "description": "Bytes of memory in use.", - "unit": "By", - "sum": { - "dataPoints": [{ - "attributes": [{ - "key": "state", - "value": { - "stringValue": "used" - } - }], - "startTimeUnixNano": "1694171569000000000", - "timeUnixNano": "1694189699786689531", - "asInt": "1874247680" - }, - { - "attributes": [{ - "key": "state", - "value": { - "stringValue": "free" - } - }], - "startTimeUnixNano": "1694171569000000000", - "timeUnixNano": "1694189699786689531", - "asInt": "29214199808" - } - ], - "aggregationTemporality": 2 - } - }] - }], - "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" - }] -} -``` -View the metric.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) - -##### Trace Event Payload Example: - -```json -{ - "resourceSpans": [{ - "resource": { - "attributes": [{ - "key": "telemetry.sdk.language", - "value": { - "stringValue": "python" - } - }, - { - "key": "telemetry.sdk.name", - "value": { - "stringValue": "opentelemetry" - } - }, - { - "key": "telemetry.sdk.version", - "value": { - "stringValue": "1.19.0" - } - }, - { - "key": "telemetry.auto.version", - "value": { - "stringValue": "0.40b0" - } - }, - { - "key": "service.name", - "value": { - "stringValue": "unknown_service" - } - } - ] - }, - "scopeSpans": [{ - "scope": { - "name": "opentelemetry.instrumentation.flask", - "version": "0.40b0" - }, - "spans": [{ - "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", - "spanId": "88079ad5c94b5b13", - "parentSpanId": "", - "name": "/roll", - "kind": 2, - "startTimeUnixNano": "1694388218052842000", - "endTimeUnixNano": "1694388218053415000", - "attributes": [{ - "key": "http.method", - "value": { - "stringValue": "GET" - } - }, - { - "key": "http.server_name", - "value": { - "stringValue": "0.0.0.0" - } - }, - { - "key": "http.scheme", - "value": { - "stringValue": "http" - } - }, - { - "key": "net.host.port", - "value": { - "intValue": "5001" - } - }, - { - "key": "http.host", - "value": { - "stringValue": "localhost:5001" - } - }, - { - "key": "http.target", - "value": { - "stringValue": "/roll" - } - }, - { - "key": "net.peer.ip", - "value": { - "stringValue": "127.0.0.1" - } - }, - { - "key": "http.user_agent", - "value": { - "stringValue": "curl/7.87.0" - } - }, - { - "key": "net.peer.port", - "value": { - "intValue": "52365" - } - }, - { - "key": "http.flavor", - "value": { - "stringValue": "1.1" - } - }, - { - "key": "http.route", - "value": { - "stringValue": "/roll" - } - }, - { - "key": "http.status_code", - "value": { - "intValue": "200" - } - } - ], - "status": {} - }] - }] - }] -} -``` - -View the trace.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto). - - -## Full Configuration Example - -For following configuration example demonstrates the pytransform processor telemetry events for logs, metrics and traces. - -```yaml -receivers: - otlp: - protocols: - http: - endpoint: "0.0.0.0:4318" - grpc: - endpoint: "0.0.0.0:4317" - - filelog: - start_at: beginning - include_file_name: true - include: - - $LOGFILE - - operators: - - type: move - from: attributes["log.file.name"] - to: resource["log.file.name"] - - - type: add - field: attributes.app - value: dev - -processors: - - # - change resource attribute log.file.name to source.log - # - add resource attribute cluster: dev - # - filter out any logs that contain the word password - # - add an attribute to each log: language: golang - pytransform/logs: - code: | - # edit resource attributes - for data in event['resourceLogs']: - for attr in [ - attr for attr in data['resource']['attributes'] - if attr['key'] == 'log.file.name' - ]: - attr['value']['stringValue'] = 'source.log' - - # add resource attribute - data['resource']['attributes'].append({ - 'key': 'cluser', - 'value': { - 'stringValue': 'dev' - } - }) - - - # filter/delete logs - for data in event['resourceLogs']: - for slog in data['scopeLogs']: - slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'password' not in lr['body']['stringValue']] - - # add an attribute to each log - for lr in slog['logRecords']: - lr['attributes'].append({ - 'key': 'language', - 'value': { - 'stringValue': 'golang' - }}) - - send(event) - - # - print event received to otel runtime log - # - if there are no resources, add a resource attribute source pytransform - # - prefix each metric name with pytransform - pytransform/metrics: - code: | - print("received event") - for md in event['resourceMetrics']: - # if resources are empty - if not md['resource']: - md['resource'] = { - 'attributes': [ - { - "key": "source", - "value": { - "stringValue": "pytransform" - } - } - ] - } - - # prefix each metric name with pytransform - for sm in md['scopeMetrics']: - for m in sm['metrics']: - m['name'] = 'pytransform.' + m['name'] - - send(event) - - # - add resource attribute source pytransform - # - filter out any spans with http.target /roll attribute - pytransform/traces: - code: | - for td in event['resourceSpans']: - # add resource attribute - td['resource']['attributes'].append({ - 'key': 'source', - 'value': { - 'stringValue': 'pytransform' - } - }) - - # filter spans with http.target /roll attribute - has_roll_attr = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/roll'] - for sd in td['scopeSpans']: - sd['spans'] = [ - s for s in sd['spans'] - if not has_roll_attr(s['attributes']) - ] - - send(event) -exporters: - logging: - verbosity: detailed - - -service: - pipelines: - logs: - receivers: - - filelog - processors: - - pytransform/logs - exporters: - - logging - - metrics: - receivers: - - otlp - processors: - - pytransform/metrics - exporters: - - logging - - traces: - receivers: - - otlp - processors: - - pytransform/traces - exporters: - - logging -``` - -### Warnings - -The Pytransform processor allows you to modify all aspects of you telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file diff --git a/processor/pytransform/config.go b/processor/pytransform/config.go deleted file mode 100644 index 47e60ffd1354..000000000000 --- a/processor/pytransform/config.go +++ /dev/null @@ -1,5 +0,0 @@ -package pytransform - -type Config struct { - Code string `mapstructure:"code"` -} diff --git a/processor/pytransform/factory.go b/processor/pytransform/factory.go deleted file mode 100644 index eb91cae513e4..000000000000 --- a/processor/pytransform/factory.go +++ /dev/null @@ -1,73 +0,0 @@ -package pytransform - -import ( - "context" - - "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor/internal/metadata" - "go.opentelemetry.io/collector/component" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/processor" -) - -// NewFactory creates a factory for the routing processor. -func NewFactory() processor.Factory { - return processor.NewFactory( - metadata.Type, - createDefaultConfig, - processor.WithLogs(createLogsProcessor, metadata.LogsStability), - processor.WithMetrics(createMetricsProcessor, metadata.MetricsStability), - processor.WithTraces(createTracesProcessor, metadata.TracesStability), - ) -} - -func createDefaultConfig() component.Config { - return &Config{ - // pass event back to otel without changes - Code: "send(event)", - } -} - -func createLogsProcessor( - ctx context.Context, - set processor.CreateSettings, - cfg component.Config, - nextConsumer consumer.Logs, -) (processor.Logs, error) { - - ep, err := getEmbeddedPython() - if err != nil { - return nil, err - } - - return newLogsProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil -} - -func createMetricsProcessor( - ctx context.Context, - set processor.CreateSettings, - cfg component.Config, - nextConsumer consumer.Metrics, -) (processor.Metrics, error) { - - ep, err := getEmbeddedPython() - if err != nil { - return nil, err - } - - return newMetricsProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil -} - -func createTracesProcessor( - ctx context.Context, - set processor.CreateSettings, - cfg component.Config, - nextConsumer consumer.Traces, -) (processor.Traces, error) { - - ep, err := getEmbeddedPython() - if err != nil { - return nil, err - } - - return newTraceProcessor(ctx, set.Logger, cfg.(*Config), ep, nextConsumer), nil -} diff --git a/processor/pytransform/factory_test.go b/processor/pytransform/factory_test.go deleted file mode 100644 index 09715ff2a1f4..000000000000 --- a/processor/pytransform/factory_test.go +++ /dev/null @@ -1,16 +0,0 @@ -package pytransform - -import ( - "testing" - - "github.com/stretchr/testify/assert" - "go.opentelemetry.io/collector/component/componenttest" -) - -func TestCreateDefaultConfig(t *testing.T) { - factory := NewFactory() - cfg := factory.CreateDefaultConfig() - assert.NoError(t, componenttest.CheckConfigStruct(cfg)) - assert.NotNil(t, cfg) - assert.Equal(t, "send(event)", cfg.(*Config).Code) -} diff --git a/processor/pytransform/go.mod b/processor/pytransform/go.mod deleted file mode 100644 index 14f60d1120fd..000000000000 --- a/processor/pytransform/go.mod +++ /dev/null @@ -1,45 +0,0 @@ -module github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor - -go 1.20 - -require ( - github.com/daidokoro/go-embed-python v0.0.1 - github.com/stretchr/testify v1.8.4 - go.opentelemetry.io/collector/component v0.84.0 - go.opentelemetry.io/collector/consumer v0.84.0 - go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 - go.opentelemetry.io/collector/processor v0.84.0 - go.uber.org/zap v1.25.0 -) - -require ( - github.com/davecgh/go-spew v1.1.1 // indirect - github.com/gogo/protobuf v1.3.2 // indirect - github.com/golang/protobuf v1.5.3 // indirect - github.com/json-iterator/go v1.1.12 // indirect - github.com/knadh/koanf/maps v0.1.1 // indirect - github.com/knadh/koanf/providers/confmap v0.1.0 // indirect - github.com/knadh/koanf/v2 v2.0.1 // indirect - github.com/mitchellh/copystructure v1.2.0 // indirect - github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect - github.com/mitchellh/reflectwalk v1.0.2 // indirect - github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect - github.com/modern-go/reflect2 v1.0.2 // indirect - github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/rogpeppe/go-internal v1.10.0 // indirect - github.com/sirupsen/logrus v1.9.0 // indirect - go.opentelemetry.io/collector/config/configtelemetry v0.84.0 // indirect - go.opentelemetry.io/collector/confmap v0.84.0 // indirect - go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 // indirect - go.opentelemetry.io/otel v1.16.0 // indirect - go.opentelemetry.io/otel/metric v1.16.0 // indirect - go.opentelemetry.io/otel/trace v1.16.0 // indirect - go.uber.org/multierr v1.11.0 // indirect - golang.org/x/net v0.14.0 // indirect - golang.org/x/sys v0.11.0 // indirect - golang.org/x/text v0.12.0 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect - google.golang.org/grpc v1.57.0 // indirect - google.golang.org/protobuf v1.31.0 // indirect - gopkg.in/yaml.v3 v3.0.1 // indirect -) diff --git a/processor/pytransform/go.sum b/processor/pytransform/go.sum deleted file mode 100644 index 94c7952877cd..000000000000 --- a/processor/pytransform/go.sum +++ /dev/null @@ -1,124 +0,0 @@ -github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A= -github.com/cyphar/filepath-securejoin v0.2.3 h1:YX6ebbZCZP7VkM3scTTokDgBL2TY741X51MTk3ycuNI= -github.com/daidokoro/go-embed-python v0.0.1 h1:4lPma0kaBwBfRMeEtk3KIf58ePzEffDRBl8sR32fvkw= -github.com/daidokoro/go-embed-python v0.0.1/go.mod h1:tsAnaTsq/UuEXbHXhzCBMYyz3TYudcsG9F+h2AhU4Xg= -github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= -github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= -github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y= -github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= -github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= -github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= -github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= -github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= -github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs= -github.com/knadh/koanf/maps v0.1.1/go.mod h1:npD/QZY3V6ghQDdcQzl1W4ICNVTkohC8E73eI2xW4yI= -github.com/knadh/koanf/providers/confmap v0.1.0 h1:gOkxhHkemwG4LezxxN8DMOFopOPghxRVp7JbIvdvqzU= -github.com/knadh/koanf/providers/confmap v0.1.0/go.mod h1:2uLhxQzJnyHKfxG927awZC7+fyHFdQkd697K4MdLnIU= -github.com/knadh/koanf/v2 v2.0.1 h1:1dYGITt1I23x8cfx8ZnldtezdyaZtfAuRtIFOiRzK7g= -github.com/knadh/koanf/v2 v2.0.1/go.mod h1:ZeiIlIDXTE7w1lMT6UVcNiRAS2/rCeLn/GdLNvY1Dus= -github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= -github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= -github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= -github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= -github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 h1:BpfhmLKZf+SjVanKKhCgf3bg+511DmU9eDQTen7LLbY= -github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= -github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= -github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= -github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= -github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= -github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= -github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= -github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= -github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= -github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= -github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= -github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -go.opentelemetry.io/collector/component v0.84.0 h1:bh4Is5Z7TjuyF7Mab0rSNh2q3y15fImdNDRXqrqGlbA= -go.opentelemetry.io/collector/component v0.84.0/go.mod h1:uXteRaacYXXYhzeCJe5ClLz5gLzleXWz01IZ730w7FA= -go.opentelemetry.io/collector/config/configtelemetry v0.84.0 h1:pnZiYtppJN6SlzJNcrkA8R+Ur63e33qMO260m8JbK18= -go.opentelemetry.io/collector/config/configtelemetry v0.84.0/go.mod h1:+LAXM5WFMW/UbTlAuSs6L/W72WC+q8TBJt/6z39FPOU= -go.opentelemetry.io/collector/confmap v0.84.0 h1:fS62yIRrsTQwe1gyuEc8TQM0yUNfSAzPVi0A1665ZpQ= -go.opentelemetry.io/collector/confmap v0.84.0/go.mod h1:/SNHqYkLagF0TjBjQyCy2Gt3ZF6hTK8VKbaan/ZHuJs= -go.opentelemetry.io/collector/consumer v0.84.0 h1:sz8mXIdPACJArlRyFNXA1SScVoo954IU1qp9V78VUxc= -go.opentelemetry.io/collector/consumer v0.84.0/go.mod h1:Mu+KeuorwHHWd6iGxU7DMAhgsHZmmzmQgf3sSWkugmM= -go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 h1:C9o0mbP0MyygqFnKueVQK/v9jef6zvuttmTGlKaqhgw= -go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014/go.mod h1:0mE3mDLmUrOXVoNsuvj+7dV14h/9HFl/Fy9YTLoLObo= -go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 h1:iT5qH0NLmkGeIdDtnBogYDx7L58t6CaWGL378DEo2QY= -go.opentelemetry.io/collector/pdata v1.0.0-rcv0014/go.mod h1:BRvDrx43kiSoUx3mr7SoA7h9B8+OY99mUK+CZSQFWW4= -go.opentelemetry.io/collector/processor v0.84.0 h1:6VM5HLdkroeqNZ/jvjxA4nsgweJ87FDMsnNnzVBHpcY= -go.opentelemetry.io/collector/processor v0.84.0/go.mod h1:KWgBNQuA6wmmRqJwfvPaRybK2Di9X8nE2fraGuVLNJo= -go.opentelemetry.io/otel v1.16.0 h1:Z7GVAX/UkAXPKsy94IU+i6thsQS4nb7LviLpnaNeW8s= -go.opentelemetry.io/otel v1.16.0/go.mod h1:vl0h9NUa1D5s1nv3A5vZOYWn8av4K8Ml6JDeHrT/bx4= -go.opentelemetry.io/otel/metric v1.16.0 h1:RbrpwVG1Hfv85LgnZ7+txXioPDoh6EdbZHo26Q3hqOo= -go.opentelemetry.io/otel/metric v1.16.0/go.mod h1:QE47cpOmkwipPiefDwo2wDzwJrlfxxNYodqc4xnGCo4= -go.opentelemetry.io/otel/trace v1.16.0 h1:8JRpaObFoW0pxuVPapkgH8UhHQj+bJW8jJsCZEu5MQs= -go.opentelemetry.io/otel/trace v1.16.0/go.mod h1:Yt9vYq1SdNz3xdjZZK7wcXv1qv2pwLkqr2QVwea0ef0= -go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= -go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= -go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= -go.uber.org/zap v1.25.0 h1:4Hvk6GtkucQ790dqmj7l1eEnRdKm3k3ZUrUMS2d5+5c= -go.uber.org/zap v1.25.0/go.mod h1:JIAUzQIH94IC4fOJQm7gMmBJP5k7wQfdcnYdPoEXJYk= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= -golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= -golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc h1:XSJ8Vk1SWuNr8S18z1NZSziL0CPIXLCCMDOEFtHBOFc= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA= -google.golang.org/grpc v1.57.0 h1:kfzNeI/klCGD2YPMUlaGNT3pxvYfga7smW3Vth8Zsiw= -google.golang.org/grpc v1.57.0/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo= -google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= -google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/processor/pytransform/helper_scripts.go b/processor/pytransform/helper_scripts.go deleted file mode 100644 index e55af6949599..000000000000 --- a/processor/pytransform/helper_scripts.go +++ /dev/null @@ -1,32 +0,0 @@ -package pytransform - -// prefixScript is a python script that is prepended to the user's script. -var prefixScript = ` -import json -import os -import urllib.request -import sys - -# override print function to send logs back to otel -def print(msg, endpoint=os.environ.get("LOG_URL"), pipeline=os.environ.get("PIPELINE")): - # Encode string to bytes - data_bytes = json.dumps({ - "message":str(msg), - "pipeline": pipeline, - }).encode('utf-8') - - req = urllib.request.Request(endpoint, data=data_bytes, method='POST') - - with urllib.request.urlopen(req) as resp: - return resp.read().decode() - - -def send(event): - sys.stdout.write(json.dumps(event)) - -# read input event from env var -try: - event = json.loads(os.environ.get("INPUT")) -except Exception as e: - raise Exception(f"error loading input event from env var: {e}") -` diff --git a/processor/pytransform/internal/metadata/metadata.go b/processor/pytransform/internal/metadata/metadata.go deleted file mode 100644 index dffae2856bd8..000000000000 --- a/processor/pytransform/internal/metadata/metadata.go +++ /dev/null @@ -1,12 +0,0 @@ -package metadata - -import ( - "go.opentelemetry.io/collector/component" -) - -const ( - Type = "pytransform" - LogsStability = component.StabilityLevelAlpha - MetricsStability = component.StabilityLevelAlpha - TracesStability = component.StabilityLevelAlpha -) diff --git a/processor/pytransform/logprocessor.go b/processor/pytransform/logprocessor.go deleted file mode 100644 index de8ae7c10043..000000000000 --- a/processor/pytransform/logprocessor.go +++ /dev/null @@ -1,67 +0,0 @@ -package pytransform - -import ( - "context" - "fmt" - - "github.com/daidokoro/go-embed-python/python" - "go.opentelemetry.io/collector/component" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/plog" - "go.uber.org/zap" -) - -func newLogsProcessor(ctx context.Context, logger *zap.Logger, - cfg *Config, ep *python.EmbeddedPython, consumer consumer.Logs) *logsProcessor { - return &logsProcessor{ - logger: logger, - cfg: cfg, - embeddedpython: ep, - next: consumer, - } -} - -type logsProcessor struct { - plog.JSONMarshaler - plog.JSONUnmarshaler - logger *zap.Logger - cfg *Config - embeddedpython *python.EmbeddedPython - next consumer.Logs -} - -func (lp *logsProcessor) Start(context.Context, component.Host) error { - startLogServer(lp.logger) - return nil -} - -func (lp *logsProcessor) Shutdown(context.Context) error { - return closeLogServer() -} - -func (lp *logsProcessor) ConsumeLogs(ctx context.Context, ld plog.Logs) error { - b, err := lp.MarshalLogs(ld) - if err != nil { - return err - } - - output, err := runPythonCode(lp.cfg.Code, string(b), "logs", lp.embeddedpython, lp.logger) - if err != nil { - return err - } - - if len(output) == 0 { - lp.logger.Error("python script returned empty output, returning record unchanged") - return lp.next.ConsumeLogs(ctx, ld) - } - - if ld, err = lp.UnmarshalLogs(output); err != nil { - return fmt.Errorf("error unmarshalling logs data from python: %w", err) - } - - return lp.next.ConsumeLogs(ctx, ld) -} - -func (lp *logsProcessor) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: true} -} diff --git a/processor/pytransform/logprocessor_test.go b/processor/pytransform/logprocessor_test.go deleted file mode 100644 index 636ce9871bd8..000000000000 --- a/processor/pytransform/logprocessor_test.go +++ /dev/null @@ -1,51 +0,0 @@ -package pytransform - -import ( - "context" - "io" - "os" - "testing" - - "github.com/stretchr/testify/require" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/plog" - "go.uber.org/zap" -) - -type fakeLogConsumer struct { - plog.JSONMarshaler - t *testing.T - expected string -} - -func (f *fakeLogConsumer) ConsumeLogs(ctx context.Context, ld plog.Logs) error { - b, err := f.MarshalLogs(ld) - require.NoError(f.t, err) - require.Equal(f.t, f.expected, string(b)) - return nil -} - -func (f *fakeLogConsumer) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: false} -} - -func TestLogConsumer(t *testing.T) { - t.Parallel() - startLogServer(zap.NewNop()) - ep, err := getEmbeddedPython() - require.NoError(t, err) - next := &fakeLogConsumer{t: t, expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"test.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456543000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456572000","body":{"stringValue":"2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456576000","body":{"stringValue":"2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456611000","body":{"stringValue":"2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""},{"observedTimeUnixNano":"1694127596456668000","body":{"stringValue":"2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`} - lp := newLogsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - - f, err := os.Open("testdata/log_event_example.json") - require.NoError(t, err) - - b, err := io.ReadAll(f) - require.NoError(t, err) - - ld, err := (&plog.JSONUnmarshaler{}).UnmarshalLogs(b) - require.NoError(t, err) - - err = lp.ConsumeLogs(context.Background(), ld) - require.NoError(t, err) -} diff --git a/processor/pytransform/metricprocessor.go b/processor/pytransform/metricprocessor.go deleted file mode 100644 index a689190a430a..000000000000 --- a/processor/pytransform/metricprocessor.go +++ /dev/null @@ -1,67 +0,0 @@ -package pytransform - -import ( - "context" - "fmt" - - "github.com/daidokoro/go-embed-python/python" - "go.opentelemetry.io/collector/component" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/pmetric" - "go.uber.org/zap" -) - -func newMetricsProcessor(ctx context.Context, logger *zap.Logger, - cfg *Config, ep *python.EmbeddedPython, consumer consumer.Metrics) *metricProcessor { - return &metricProcessor{ - logger: logger, - cfg: cfg, - embeddedpython: ep, - next: consumer, - } -} - -type metricProcessor struct { - pmetric.JSONMarshaler - pmetric.JSONUnmarshaler - logger *zap.Logger - cfg *Config - embeddedpython *python.EmbeddedPython - next consumer.Metrics -} - -func (mp *metricProcessor) Start(context.Context, component.Host) error { - startLogServer(mp.logger) - return nil -} - -func (mp *metricProcessor) Shutdown(context.Context) error { - return closeLogServer() -} - -func (mp *metricProcessor) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error { - b, err := mp.MarshalMetrics(md) - if err != nil { - return err - } - - output, err := runPythonCode(mp.cfg.Code, string(b), "metrics", mp.embeddedpython, mp.logger) - if err != nil { - return err - } - - if len(output) == 0 { - mp.logger.Error("python script returned empty output, returning record unchanged") - return mp.next.ConsumeMetrics(ctx, md) - } - - if md, err = mp.UnmarshalMetrics(output); err != nil { - return fmt.Errorf("error unmarshalling metric data from python: %v", err) - } - - return mp.next.ConsumeMetrics(ctx, md) -} - -func (mp *metricProcessor) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: true} -} diff --git a/processor/pytransform/metricprocessor_test.go b/processor/pytransform/metricprocessor_test.go deleted file mode 100644 index c3925126cc6a..000000000000 --- a/processor/pytransform/metricprocessor_test.go +++ /dev/null @@ -1,53 +0,0 @@ -package pytransform - -import ( - "context" - "io" - "os" - "testing" - - "github.com/stretchr/testify/require" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/pmetric" - "go.uber.org/zap" -) - -type fakeMetricConsumer struct { - pmetric.JSONMarshaler - t *testing.T - expected string -} - -func (f *fakeMetricConsumer) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error { - b, err := f.MarshalMetrics(md) - require.NoError(f.t, err) - require.Equal(f.t, f.expected, string(b)) - return nil -} - -func (f *fakeMetricConsumer) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: true} -} - -func TestMetricConsumer(t *testing.T) { - t.Parallel() - startLogServer(zap.NewNop()) - ep, err := getEmbeddedPython() - require.NoError(t, err) - next := &fakeMetricConsumer{t: t, expected: `{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{"name":"otelcol/hostmetricsreceiver/memory","version":"0.84.0"},"metrics":[{"name":"system.memory.usage","description":"Bytes of memory in use.","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"1874247680"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"29214199808"}],"aggregationTemporality":2}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.9.0"}]}`} - mp := newMetricsProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - - f, err := os.Open("testdata/metric_event_example.json") - require.NoError(t, err) - - defer f.Close() - - b, err := io.ReadAll(f) - require.NoError(t, err) - - md, err := (&pmetric.JSONUnmarshaler{}).UnmarshalMetrics(b) - require.NoError(t, err) - - err = mp.ConsumeMetrics(context.Background(), md) - require.NoError(t, err) -} diff --git a/processor/pytransform/pytransform.go b/processor/pytransform/pytransform.go deleted file mode 100644 index 3e4dd39d5e7a..000000000000 --- a/processor/pytransform/pytransform.go +++ /dev/null @@ -1,136 +0,0 @@ -package pytransform - -import ( - "bytes" - "encoding/json" - "errors" - "fmt" - "math/rand" - "net/http" - "os" - "sync" - "time" - - "github.com/daidokoro/go-embed-python/python" - "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor/internal/metadata" - "go.uber.org/zap" -) - -var ( - pylogserver *http.Server - embeddedPython *python.EmbeddedPython - initPythonEmbedOnce sync.Once - startOnce sync.Once - closeOnce sync.Once -) - -// The log server receives print statements from the python script -// and prints them using the zap logger to stdout - -func startLogServer(logger *zap.Logger) { - startOnce.Do(func() { - r := rand.New(rand.NewSource(time.Now().UnixNano())) - port := r.Intn(60000-10000) + 10000 - mux := http.NewServeMux() - - // handle log events from python script - mux.Handle("/log", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - defer r.Body.Close() - var m map[string]string - - if err := json.NewDecoder(r.Body).Decode(&m); err != nil { - logger.Error("failed reading python script log body", zap.Error(err)) - } - - logger.Info(m["message"], - zap.String("func", "python.print()"), - zap.String("source", fmt.Sprintf("pytransform/%s", m["pipeline"]))) - })) - - pylogserver = &http.Server{ - Addr: fmt.Sprintf(":%d", port), - Handler: mux, - } - - go func() { - logger.Info("starting python logging http server", zap.String("addr", pylogserver.Addr)) - if err := pylogserver.ListenAndServe(); err != http.ErrServerClosed { - logger.Error("error starting http server", zap.Error(err)) - return - } - - logger.Info("http server closed") - }() - }) - return -} - -func closeLogServer() error { - var err error - closeOnce.Do(func() { - if pylogserver != nil { - err = pylogserver.Close() - } - }) - return err -} - -func getEmbeddedPython() (*python.EmbeddedPython, error) { - var err error - initPythonEmbedOnce.Do(func() { - embeddedPython, err = python.NewEmbeddedPython(metadata.Type) - }) - return embeddedPython, err -} - -// runPythonCode - runs a python script and returns the stdout and stderr in bytes -func runPythonCode(code, pyinput, pipeline string, ep *python.EmbeddedPython, - logger *zap.Logger) ([]byte, error) { - - result := make(chan []byte) - errchan := make(chan error) - - code = fmt.Sprintf("%s\n%s", prefixScript, code) - cmd := ep.PythonCmd("-c", code) - - go func() { - var buf bytes.Buffer - - // add script prefix - cmd.Stdout = &buf - cmd.Stderr = &buf - cmd.Env = append( - os.Environ(), - fmt.Sprintf("INPUT=%s", pyinput), - fmt.Sprintf("PIPELINE=%s", pipeline), - fmt.Sprintf("LOG_URL=http://localhost%s/log", pylogserver.Addr)) - - if err := cmd.Run(); err != nil { - err = fmt.Errorf("error running python script: %s - %v", buf.String(), err) - logger.Error(err.Error()) - errchan <- err - } - - result <- buf.Bytes() - }() - - select { - case err := <-errchan: - return nil, err - case b := <-result: - return b, nil - case <-time.After(1500 * time.Millisecond): - logger.Error("timeout running python script") - logger.Debug("killing python process", zap.Int("pid", cmd.Process.Pid)) - process, err := os.FindProcess(cmd.Process.Pid) - if err != nil { - return nil, fmt.Errorf("error finding python process after timeout: %v", err) - } - - if err = process.Kill(); err != nil { - return nil, fmt.Errorf("error while cancelling python process after timeout: %v", err) - } - - return nil, errors.New("timeout running python script") - } -} diff --git a/processor/pytransform/pytransform_test.go b/processor/pytransform/pytransform_test.go deleted file mode 100644 index b1c4990e4a71..000000000000 --- a/processor/pytransform/pytransform_test.go +++ /dev/null @@ -1,65 +0,0 @@ -package pytransform - -import ( - "errors" - "strings" - "testing" - - "github.com/stretchr/testify/require" - "go.uber.org/zap" -) - -func TestRunPythonCode(t *testing.T) { - startLogServer(zap.NewNop()) - ep, err := getEmbeddedPython() - require.NoError(t, err) - - testcases := []struct { - name string - event string - code string - expectError error - expectedOutput string - }{ - { - name: "update event input", - event: `{"name": "some_transform"}`, - code: `event['name'] = "pytransform"; send(event)`, - expectedOutput: `{"name": "pytransform"}`, - expectError: nil, - }, - { - name: "bad input", - event: `{"name": "some_transform"`, - code: `send(event)`, - expectedOutput: "", - expectError: errors.New("error loading input event from env var"), - }, - { - name: "timeout", - event: `{"name": "some_transform"}`, - code: `import time; time.sleep(2); send(event)`, - expectedOutput: "", - expectError: errors.New("timeout running python script"), - }, - { - name: "bad code", - event: `{"name": "some_transform"}`, - code: `send(event`, - expectedOutput: "", - expectError: errors.New("error running python script"), - }, - } - - for _, tt := range testcases { - t.Run(tt.name, func(t *testing.T) { - out, err := runPythonCode(tt.code, tt.event, "testing", ep, zap.NewNop()) - if tt.expectError != nil { - require.True(t, strings.Contains(err.Error(), tt.expectError.Error())) - } else { - require.NoError(t, err) - } - require.Equal(t, tt.expectedOutput, string(out)) - }) - } -} diff --git a/processor/pytransform/testdata/builder.yaml b/processor/pytransform/testdata/builder.yaml deleted file mode 100644 index b788b575ed2a..000000000000 --- a/processor/pytransform/testdata/builder.yaml +++ /dev/null @@ -1,19 +0,0 @@ -dist: - name: ddot - output_path: ./cmd/ddot - description: Daidokoro Open Telemetry Contrib binary. - version: 1.0.0 - -processors: - # add my processor - - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/processor/pytransformprocessor v0.0.0" - path: "../" - -exporters: - - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.84.0 - - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.84.0 - -receivers: - - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.84.0 - - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.84.0 - - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.84.0 diff --git a/processor/pytransform/testdata/config.yaml b/processor/pytransform/testdata/config.yaml deleted file mode 100644 index 9fb906d42624..000000000000 --- a/processor/pytransform/testdata/config.yaml +++ /dev/null @@ -1,124 +0,0 @@ -receivers: - otlp: - protocols: - http: - endpoint: "0.0.0.0:4318" - grpc: - endpoint: "0.0.0.0:4317" - - filelog: - start_at: beginning - include_file_name: true - include: - - $LOGFILE - - operators: - - type: move - from: attributes["log.file.name"] - to: resource["log.file.name"] - - - type: add - field: attributes.app - value: dev - -processors: - pytransform/logs: - code: | - # edit resource attributes - for data in event['resourceLogs']: - for attr in data['resource']['attributes']: - attr['value']['stringValue'] = 'source.log' - - # filter/delete logs - for data in event['resourceLogs']: - for slog in data['scopeLogs']: - slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'internal' not in lr['body']['stringValue']] - - # add an attribute to each log - for lr in slog['logRecords']: - lr['attributes'].append({ - 'key': 'language', - 'value': { - 'stringValue': 'golang' - }}) - - send(event) - - pytransform/metrics: - code: | - for md in event['resourceMetrics']: - # if resources are empty - if not md['resource']: - md['resource'] = { - 'attributes': [ - { - "key": "source", - "value": { - "stringValue": "pytransform" - } - } - ] - } - - # prefix each metric name with pytransform - for sm in md['scopeMetrics']: - for m in sm['metrics']: - m['name'] = 'pytransform.' + m['name'] - - send(event) - - pytransform/traces: - code: | - for td in event['resourceSpans']: - # add resource attribute - td['resource']['attributes'].append({ - 'key': 'source', - 'value': { - 'stringValue': 'pytransform' - } - }) - - # filter spans with http.target /roll attribute - has_roll = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/roll'] - for sd in td['scopeSpans']: - sd['spans'] = [ - s for s in sd['spans'] - if not has_roll(s['attributes']) - ] - - send(event) - -exporters: - logging: - verbosity: detailed - - -service: - pipelines: - logs: - receivers: - - filelog - processors: - - pytransform/logs - exporters: - - logging - - metrics: - receivers: - - otlp - processors: - - pytransform/metrics - exporters: - - logging - - traces: - receivers: - - otlp - processors: - - pytransform/traces - exporters: - - logging - - # telemetry: - # logs: - # level: debug diff --git a/processor/pytransform/testdata/log_event_example.json b/processor/pytransform/testdata/log_event_example.json deleted file mode 100644 index 598184dd0bd8..000000000000 --- a/processor/pytransform/testdata/log_event_example.json +++ /dev/null @@ -1,119 +0,0 @@ -{ - "resourceLogs": [ - { - "resource": { - "attributes": [ - { - "key": "log.file.name", - "value": { - "stringValue": "test.log" - } - } - ] - }, - "scopeLogs": [ - { - "scope": {}, - "logRecords": [ - { - "observedTimeUnixNano": "1694127596456358000", - "body": { - "stringValue": "2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1694127596456543000", - "body": { - "stringValue": "2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1694127596456572000", - "body": { - "stringValue": "2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1694127596456576000", - "body": { - "stringValue": "2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1694127596456611000", - "body": { - "stringValue": "2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - }, - { - "observedTimeUnixNano": "1694127596456668000", - "body": { - "stringValue": "2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling" - }, - "attributes": [ - { - "key": "app", - "value": { - "stringValue": "dev" - } - } - ], - "traceId": "", - "spanId": "" - } - ] - } - ] - } - ] - } \ No newline at end of file diff --git a/processor/pytransform/testdata/metric_event_example.json b/processor/pytransform/testdata/metric_event_example.json deleted file mode 100644 index 026f06779aa7..000000000000 --- a/processor/pytransform/testdata/metric_event_example.json +++ /dev/null @@ -1,55 +0,0 @@ -{ - "resourceMetrics": [ - { - "resource": {}, - "scopeMetrics": [ - { - "scope": { - "name": "otelcol/hostmetricsreceiver/memory", - "version": "0.84.0" - }, - "metrics": [ - { - "name": "system.memory.usage", - "description": "Bytes of memory in use.", - "unit": "By", - "sum": { - "dataPoints": [ - { - "attributes": [ - { - "key": "state", - "value": { - "stringValue": "used" - } - } - ], - "startTimeUnixNano": "1694171569000000000", - "timeUnixNano": "1694189699786689531", - "asInt": "1874247680" - }, - { - "attributes": [ - { - "key": "state", - "value": { - "stringValue": "free" - } - } - ], - "startTimeUnixNano": "1694171569000000000", - "timeUnixNano": "1694189699786689531", - "asInt": "29214199808" - } - ], - "aggregationTemporality": 2 - } - } - ] - } - ], - "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" - } - ] - } - \ No newline at end of file diff --git a/processor/pytransform/testdata/test.log b/processor/pytransform/testdata/test.log deleted file mode 100644 index 81a4bf6c3a5c..000000000000 --- a/processor/pytransform/testdata/test.log +++ /dev/null @@ -1,7 +0,0 @@ -2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {"version": "0.84.0", "date": "2023-08-29T18:58:24Z"} -2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {"path": "builder.yaml"} -2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {"go-executable": "/usr/local/go/bin/go"} -2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {"path": "./cmd/ddot"} -2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules -2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling -2023-09-06T01:09:47.694+0200 INFO builder/main.go:102 Compiled {"binary": "./cmd/ddot/ddot"} \ No newline at end of file diff --git a/processor/pytransform/testdata/trace_event_example.json b/processor/pytransform/testdata/trace_event_example.json deleted file mode 100644 index ef165a57ceaa..000000000000 --- a/processor/pytransform/testdata/trace_event_example.json +++ /dev/null @@ -1,135 +0,0 @@ -{ - "resourceSpans": [ - { - "resource": { - "attributes": [ - { - "key": "telemetry.sdk.language", - "value": { - "stringValue": "python" - } - }, - { - "key": "telemetry.sdk.name", - "value": { - "stringValue": "opentelemetry" - } - }, - { - "key": "telemetry.sdk.version", - "value": { - "stringValue": "1.19.0" - } - }, - { - "key": "telemetry.auto.version", - "value": { - "stringValue": "0.40b0" - } - }, - { - "key": "service.name", - "value": { - "stringValue": "unknown_service" - } - } - ] - }, - "scopeSpans": [ - { - "scope": { - "name": "opentelemetry.instrumentation.flask", - "version": "0.40b0" - }, - "spans": [ - { - "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", - "spanId": "88079ad5c94b5b13", - "parentSpanId": "", - "name": "/roll", - "kind": 2, - "startTimeUnixNano": "1694388218052842000", - "endTimeUnixNano": "1694388218053415000", - "attributes": [ - { - "key": "http.method", - "value": { - "stringValue": "GET" - } - }, - { - "key": "http.server_name", - "value": { - "stringValue": "0.0.0.0" - } - }, - { - "key": "http.scheme", - "value": { - "stringValue": "http" - } - }, - { - "key": "net.host.port", - "value": { - "intValue": "5001" - } - }, - { - "key": "http.host", - "value": { - "stringValue": "localhost:5001" - } - }, - { - "key": "http.target", - "value": { - "stringValue": "/roll" - } - }, - { - "key": "net.peer.ip", - "value": { - "stringValue": "127.0.0.1" - } - }, - { - "key": "http.user_agent", - "value": { - "stringValue": "curl/7.87.0" - } - }, - { - "key": "net.peer.port", - "value": { - "intValue": "52365" - } - }, - { - "key": "http.flavor", - "value": { - "stringValue": "1.1" - } - }, - { - "key": "http.route", - "value": { - "stringValue": "/roll" - } - }, - { - "key": "http.status_code", - "value": { - "intValue": "200" - } - } - ], - "status": {} - } - ] - } - ] - } - ] - } - \ No newline at end of file diff --git a/processor/pytransform/traceprocessor.go b/processor/pytransform/traceprocessor.go deleted file mode 100644 index 254a1e3e9967..000000000000 --- a/processor/pytransform/traceprocessor.go +++ /dev/null @@ -1,67 +0,0 @@ -package pytransform - -import ( - "context" - "fmt" - - "github.com/daidokoro/go-embed-python/python" - "go.opentelemetry.io/collector/component" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/ptrace" - "go.uber.org/zap" -) - -func newTraceProcessor(ctx context.Context, logger *zap.Logger, - cfg *Config, ep *python.EmbeddedPython, consumer consumer.Traces) *traceProcessor { - return &traceProcessor{ - logger: logger, - cfg: cfg, - embeddedpython: ep, - next: consumer, - } -} - -type traceProcessor struct { - ptrace.JSONMarshaler - ptrace.JSONUnmarshaler - logger *zap.Logger - cfg *Config - embeddedpython *python.EmbeddedPython - next consumer.Traces -} - -func (tp *traceProcessor) Start(context.Context, component.Host) error { - startLogServer(tp.logger) - return nil -} - -func (tp *traceProcessor) Shutdown(context.Context) error { - return closeLogServer() -} - -func (tp *traceProcessor) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { - b, err := tp.MarshalTraces(td) - if err != nil { - return err - } - - output, err := runPythonCode(tp.cfg.Code, string(b), "traces", tp.embeddedpython, tp.logger) - if err != nil { - return err - } - - if len(output) == 0 { - tp.logger.Error("python script returned empty output, returning record unchanged") - return tp.next.ConsumeTraces(ctx, td) - } - - if td, err = tp.UnmarshalTraces(output); err != nil { - return fmt.Errorf("error unmarshalling metric data from python: %v", err) - } - - return tp.next.ConsumeTraces(ctx, td) -} - -func (tp *traceProcessor) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: true} -} diff --git a/processor/pytransform/traceprocessor_test.go b/processor/pytransform/traceprocessor_test.go deleted file mode 100644 index ca15aaa81d34..000000000000 --- a/processor/pytransform/traceprocessor_test.go +++ /dev/null @@ -1,51 +0,0 @@ -package pytransform - -import ( - "context" - "io" - "os" - "testing" - - "github.com/stretchr/testify/require" - "go.opentelemetry.io/collector/consumer" - "go.opentelemetry.io/collector/pdata/ptrace" - "go.uber.org/zap" -) - -type fakeTraceConsumer struct { - ptrace.JSONMarshaler - t *testing.T - expected string -} - -func (f *fakeTraceConsumer) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { - b, err := f.MarshalTraces(td) - require.NoError(f.t, err) - require.Equal(f.t, f.expected, string(b)) - return nil -} - -func (f *fakeTraceConsumer) Capabilities() consumer.Capabilities { - return consumer.Capabilities{MutatesData: false} -} - -func TestTraceConsumer(t *testing.T) { - t.Parallel() - startLogServer(zap.NewNop()) - ep, err := getEmbeddedPython() - require.NoError(t, err) - next := &fakeTraceConsumer{t: t, expected: `{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"service.name","value":{"stringValue":"unknown_service"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9cb5bf738137b2248dc7b20445ec2e1c","spanId":"88079ad5c94b5b13","parentSpanId":"","name":"/roll","kind":2,"startTimeUnixNano":"1694388218052842000","endTimeUnixNano":"1694388218053415000","attributes":[{"key":"http.method","value":{"stringValue":"GET"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"5001"}},{"key":"http.host","value":{"stringValue":"localhost:5001"}},{"key":"http.target","value":{"stringValue":"/roll"}},{"key":"net.peer.ip","value":{"stringValue":"127.0.0.1"}},{"key":"http.user_agent","value":{"stringValue":"curl/7.87.0"}},{"key":"net.peer.port","value":{"intValue":"52365"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/roll"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}`} - tp := newTraceProcessor(context.Background(), zap.NewNop(), createDefaultConfig().(*Config), ep, next) - - f, err := os.Open("testdata/trace_event_example.json") - require.NoError(t, err) - - b, err := io.ReadAll(f) - require.NoError(t, err) - - td, err := (&ptrace.JSONUnmarshaler{}).UnmarshalTraces(b) - require.NoError(t, err) - - err = tp.ConsumeTraces(context.Background(), td) - require.NoError(t, err) -} From 6fd58c9133d4ad8d1675a9dc14cd511f0fd0a962 Mon Sep 17 00:00:00 2001 From: daidokoro Date: Sun, 24 Sep 2023 15:40:00 +0200 Subject: [PATCH 06/16] added starlarktransform processor code --- processor/starlarktransform/LICENSE | 9 + processor/starlarktransform/README.md | 466 ++++++++++++++++++ processor/starlarktransform/config.go | 5 + processor/starlarktransform/factory.go | 74 +++ processor/starlarktransform/factory_test.go | 16 + processor/starlarktransform/go.mod | 44 ++ processor/starlarktransform/go.sum | 181 +++++++ .../starlarktransform/internal/config.go | 1 + .../internal/logs/processor.go | 94 ++++ .../internal/logs/processor_test.go | 226 +++++++++ .../internal/metadata/metadata.go | 12 + .../internal/metrics/processor.go | 93 ++++ .../internal/metrics/processor_test.go | 171 +++++++ .../internal/traces/processor.go | 93 ++++ .../internal/traces/processor_test.go | 174 +++++++ .../starlarktransform/testdata/builder.yaml | 19 + .../starlarktransform/testdata/config.yaml | 136 +++++ .../testdata/log_event_example.json | 119 +++++ .../testdata/metric_event_example.json | 55 +++ processor/starlarktransform/testdata/test.log | 7 + .../testdata/trace_event_example.json | 135 +++++ 21 files changed, 2130 insertions(+) create mode 100644 processor/starlarktransform/LICENSE create mode 100644 processor/starlarktransform/README.md create mode 100644 processor/starlarktransform/config.go create mode 100644 processor/starlarktransform/factory.go create mode 100644 processor/starlarktransform/factory_test.go create mode 100644 processor/starlarktransform/go.mod create mode 100644 processor/starlarktransform/go.sum create mode 100644 processor/starlarktransform/internal/config.go create mode 100644 processor/starlarktransform/internal/logs/processor.go create mode 100644 processor/starlarktransform/internal/logs/processor_test.go create mode 100644 processor/starlarktransform/internal/metadata/metadata.go create mode 100644 processor/starlarktransform/internal/metrics/processor.go create mode 100644 processor/starlarktransform/internal/metrics/processor_test.go create mode 100644 processor/starlarktransform/internal/traces/processor.go create mode 100644 processor/starlarktransform/internal/traces/processor_test.go create mode 100644 processor/starlarktransform/testdata/builder.yaml create mode 100644 processor/starlarktransform/testdata/config.yaml create mode 100644 processor/starlarktransform/testdata/log_event_example.json create mode 100644 processor/starlarktransform/testdata/metric_event_example.json create mode 100644 processor/starlarktransform/testdata/test.log create mode 100644 processor/starlarktransform/testdata/trace_event_example.json diff --git a/processor/starlarktransform/LICENSE b/processor/starlarktransform/LICENSE new file mode 100644 index 000000000000..9aaf79b4849c --- /dev/null +++ b/processor/starlarktransform/LICENSE @@ -0,0 +1,9 @@ +The MIT License (MIT) + +Copyright (c) 2023 Daidokoro, Ltd + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/processor/starlarktransform/README.md b/processor/starlarktransform/README.md new file mode 100644 index 000000000000..442bfeed1b25 --- /dev/null +++ b/processor/starlarktransform/README.md @@ -0,0 +1,466 @@ +# starlarktransform + + +| Status | | +| ------------- |-----------| +| Stability | [alpha]: traces, metrics, logs | + + +[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta +[sumo]: https://github.com/SumoLogic/sumologic-otel-collector + + + +The starlarktransform processor modifies telemetry based on configuration using Starlark code. + +Starlark is a scripting language used for configuration that is designed to be similar to Python. It is designed to be fast, deterministic, and easily embedded into other software projects. + +The processor leverages Starlark to modify telemetry data while using familiar, pythonic syntax. Modifying telemetry data is as a simple as modifying a `Dict`. + +## Config + +To configure starlarktransform, you add your code using the `code` option in the config. Each instance of starlarktransform can only have a single code section. + + +```yaml +processors: + starlarktransform: + code: | + def transform(event): + event = json.decode(event) + + + + return event +``` + +You must define a function called `transform` that accepts a single argument, `event`. This function is called by the processor and is passed the telemetry event. The function **must return** the modified, json decoded event. + +## How it works + +The processor uses the [Starlark-Go](https://github.com/google/starlark-go) interpreter, this allows you to run this processor without having to install a Starlark language interpreter on the host. + +## Features + +The starlarktransform processor gives you access to the full telemetry event payload. You are able to modify this payload using the Starklark code in any way you want. This allows you do various things such as: + +- Filtering +- Adding/Removing attributes +- Modifying attributes +- Modifying telemetry data directly +- Telemetry injection based on existing values +- And more + +## Libs, Functions and Functionality + +While similar in syntax to Python, Starlack does not have all the functionality associated with Python. This processor does not have access to Python standard libraries and the implementation found in this processor is limited further to only the following libraries and functions: + +- **json** + +> The JSON library allows you to encode and decode JSON strings. The use of this library is mandatory as the telemetry data is passed to the processor as a JSON string. You must decode the JSON string to a Dict before you can modify it. **You must also return a JSON decoded Dict to the processor.** + +```python +# encode dict string to json string +x = json.encode({"foo": ["bar", "baz"]}) +print(x) +# output: {"foo":["bar","baz"]} +``` + +```python +# decode json string to dict +x = json.decode('{"foo": ["bar", "baz"]}') +``` + +You can read more on the JSON library [here](https://qri.io/docs/reference/starlark-packages/encoding/json) + +- **print** +> You are able to use the print function to check outputs of your code. The output of the print function is sent to the Open Telemetry runtime log. Values printed by the Print function only show when running Open Telemetry in Debug mode. + +```python +def transform(event): + print("hello world") + return json.decode(event) +``` + +The print statement above would result in the following output in the Open Telemetry runtime log. Again, this output is only visible when running Open Telemetry in Debug mode. +```log +2023-09-23T16:50:17.328+0200 debug traces/processor.go:25 hello world {"kind": "processor", "name": "starlarktransform/traces", "pipeline": "traces", "thread": "trace.processor", "source": "starlark/code"} +``` + + +- **re** (regex) +> Support for Regular Expressions coming soon + + +Note that you can define your own functions within your Starlark code, however, there must be at least one function named `transform` that accepts a single argument `event` and returns a JSON decoded Dict, this function can call all your other functions as needed. + + +## Examples + +This section contains examples of the event payloads that are sent to the starlarktransform processor from each telemetry type. These examples can help you understand the structure of the telemetry events and how to modify them. + +##### Log Event Payload Example: + +```json +{ + "resourceLogs": [{ + "resource": { + "attributes": [{ + "key": "log.file.name", + "value": { + "stringValue": "test.log" + } + }] + }, + "scopeLogs": [{ + "scope": {}, + "logRecords": [{ + "observedTimeUnixNano": "1694127596456358000", + "body": { + "stringValue": "2023-09-06T01:09:24.045+0200 INFO starting app.", + "attributes": [{ + + "key": "app", + "value": { + "stringValue": "dev" + } + }], + "traceId": "", + "spanId": "" + } + }] + }] + }] +} +``` + +View the the log.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto) + +##### Metric Event Payload Example: + +```json +{ + "resourceMetrics": [{ + "resource": {}, + "scopeMetrics": [{ + "scope": { + "name": "otelcol/hostmetricsreceiver/memory", + "version": "0.84.0" + }, + "metrics": [{ + "name": "system.memory.usage", + "description": "Bytes of memory in use.", + "unit": "By", + "sum": { + "dataPoints": [{ + "attributes": [{ + "key": "state", + "value": { + "stringValue": "used" + } + }], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "1874247680" + }, + { + "attributes": [{ + "key": "state", + "value": { + "stringValue": "free" + } + }], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "29214199808" + } + ], + "aggregationTemporality": 2 + } + }] + }], + "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" + }] +} +``` +View the metric.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) + +##### Trace Event Payload Example: + +```json +{ + "resourceSpans": [{ + "resource": { + "attributes": [{ + "key": "telemetry.sdk.language", + "value": { + "stringValue": "python" + } + }, + { + "key": "telemetry.sdk.name", + "value": { + "stringValue": "opentelemetry" + } + }, + { + "key": "telemetry.sdk.version", + "value": { + "stringValue": "1.19.0" + } + }, + { + "key": "telemetry.auto.version", + "value": { + "stringValue": "0.40b0" + } + }, + { + "key": "service.name", + "value": { + "stringValue": "unknown_service" + } + } + ] + }, + "scopeSpans": [{ + "scope": { + "name": "opentelemetry.instrumentation.flask", + "version": "0.40b0" + }, + "spans": [{ + "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", + "spanId": "88079ad5c94b5b13", + "parentSpanId": "", + "name": "/roll", + "kind": 2, + "startTimeUnixNano": "1694388218052842000", + "endTimeUnixNano": "1694388218053415000", + "attributes": [{ + "key": "http.method", + "value": { + "stringValue": "GET" + } + }, + { + "key": "http.server_name", + "value": { + "stringValue": "0.0.0.0" + } + }, + { + "key": "http.scheme", + "value": { + "stringValue": "http" + } + }, + { + "key": "net.host.port", + "value": { + "intValue": "5001" + } + }, + { + "key": "http.host", + "value": { + "stringValue": "localhost:5001" + } + }, + { + "key": "http.target", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "net.peer.ip", + "value": { + "stringValue": "127.0.0.1" + } + }, + { + "key": "http.user_agent", + "value": { + "stringValue": "curl/7.87.0" + } + }, + { + "key": "net.peer.port", + "value": { + "intValue": "52365" + } + }, + { + "key": "http.flavor", + "value": { + "stringValue": "1.1" + } + }, + { + "key": "http.route", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "http.status_code", + "value": { + "intValue": "200" + } + } + ], + "status": {} + }] + }] + }] +} +``` + +View the trace.proto type definition [here](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto). + + +## Full Configuration Example + +For following configuration example demonstrates the starlarktransform processor telemetry events for logs, metrics and traces. + +```yaml +receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + grpc: + endpoint: "0.0.0.0:4317" + + filelog: + start_at: beginning + include_file_name: true + include: + - $LOGFILE + + operators: + - type: move + from: attributes["log.file.name"] + to: resource["log.file.name"] + + - type: add + field: attributes.app + value: dev + +processors: + + # - change resource attribute log.file.name to source.log + # - add resource attribute cluster: dev + # - filter out any logs that contain the word password + # - add an attribute to each log: language: golang + starlarktransform/logs: + code: | + def transform(event): + event = json.decode(event) + # edit resource attributes + for data in event['resourceLogs']: + for attr in data['resource']['attributes']: + attr['value']['stringValue'] = 'source.log' + + # filter/delete logs + for data in event['resourceLogs']: + for slog in data['scopeLogs']: + slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'internal' not in lr['body']['stringValue']] + + # add an attribute to each log + for lr in slog['logRecords']: + lr['attributes'].append({ + 'key': 'language', + 'value': { + 'stringValue': 'golang' + }}) + + return event + # - print event received to otel runtime log + # - if there are no resources, add a resource attribute source starlarktransform + # - prefix each metric name with starlarktransform + starlarktransform/metrics: + code: | + def transform(event): + print("received event", event) + event = json.decode(event) + for md in event['resourceMetrics']: + # if resources are empty + if not md['resource']: + md['resource'] = { + 'attributes': [ + { + "key": "source", + "value": { + "stringValue": "starlarktransform" + } + } + ] + } + + # prefix each metric name with starlarktransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'starlarktransform.' + m['name'] + + return event + + # - add resource attribute source starlarktransform + # - filter out any spans with http.target /roll attribute + starlarktransform/traces: + code: | + def transform(event): + event = json.decode(event) + for td in event['resourceSpans']: + # add resource attribute + td['resource']['attributes'].append({ + 'key': 'source', + 'value': { + 'stringValue': 'starlarktransform' + } + }) + + # filter spans with http.target /roll attribute + has_roll = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/cats'] + for sd in td['scopeSpans']: + sd['spans'] = [ + s for s in sd['spans'] + if not has_roll(s['attributes']) + ] + return event +exporters: + logging: + verbosity: detailed + + +service: + pipelines: + logs: + receivers: + - filelog + processors: + - starlarktransform/logs + exporters: + - logging + + metrics: + receivers: + - otlp + processors: + - starlarktransform/metrics + exporters: + - logging + + traces: + receivers: + - otlp + processors: + - starlarktransform/traces + exporters: + - logging +``` + + +### Warnings + +The starlarktransform processor allows you to modify all aspects of your telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file diff --git a/processor/starlarktransform/config.go b/processor/starlarktransform/config.go new file mode 100644 index 000000000000..07915fff8a75 --- /dev/null +++ b/processor/starlarktransform/config.go @@ -0,0 +1,5 @@ +package starlarktransform + +type Config struct { + Code string `mapstructure:"code"` +} diff --git a/processor/starlarktransform/factory.go b/processor/starlarktransform/factory.go new file mode 100644 index 000000000000..13e5d508518b --- /dev/null +++ b/processor/starlarktransform/factory.go @@ -0,0 +1,74 @@ +package starlarktransform + +import ( + "context" + "errors" + + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor/internal/logs" + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor/internal/metadata" + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor/internal/metrics" + "github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor/internal/traces" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/processor" +) + +// NewFactory creates a factory for the routing processor. +func NewFactory() processor.Factory { + return processor.NewFactory( + metadata.Type, + createDefaultConfig, + processor.WithLogs(createLogsProcessor, metadata.LogsStability), + processor.WithMetrics(createMetricsProcessor, metadata.MetricsStability), + processor.WithTraces(createTracesProcessor, metadata.TracesStability), + ) +} + +func createDefaultConfig() component.Config { + return &Config{ + // pass event back to otel without changes + Code: "def transform(e): return json.decode(e)", + } +} + +func createLogsProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Logs, +) (processor.Logs, error) { + config, ok := cfg.(*Config) + if !ok { + return nil, errors.New("invalid config type") + } + + return logs.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil +} + +func createMetricsProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Metrics, +) (processor.Metrics, error) { + config, ok := cfg.(*Config) + if !ok { + return nil, errors.New("invalid config type") + } + + return metrics.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil +} + +func createTracesProcessor( + ctx context.Context, + set processor.CreateSettings, + cfg component.Config, + nextConsumer consumer.Traces, +) (processor.Traces, error) { + config, ok := cfg.(*Config) + if !ok { + return nil, errors.New("invalid config type") + } + + return traces.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil +} diff --git a/processor/starlarktransform/factory_test.go b/processor/starlarktransform/factory_test.go new file mode 100644 index 000000000000..54b0bcea2d65 --- /dev/null +++ b/processor/starlarktransform/factory_test.go @@ -0,0 +1,16 @@ +package starlarktransform + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "go.opentelemetry.io/collector/component/componenttest" +) + +func TestCreateDefaultConfig(t *testing.T) { + factory := NewFactory() + cfg := factory.CreateDefaultConfig() + assert.NoError(t, componenttest.CheckConfigStruct(cfg)) + assert.NotNil(t, cfg) + assert.Equal(t, "def transform(event): return json.decode(event)", cfg.(*Config).Code) +} diff --git a/processor/starlarktransform/go.mod b/processor/starlarktransform/go.mod new file mode 100644 index 000000000000..5c220b12d2e1 --- /dev/null +++ b/processor/starlarktransform/go.mod @@ -0,0 +1,44 @@ +module github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor + +go 1.20 + +require ( + github.com/MakeNowJust/heredoc/v2 v2.0.1 + github.com/stretchr/testify v1.8.4 + go.opentelemetry.io/collector/component v0.84.0 + go.opentelemetry.io/collector/consumer v0.84.0 + go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 + go.opentelemetry.io/collector/processor v0.84.0 + go.starlark.net v0.0.0-20230912135651-745481cf39ed + go.uber.org/zap v1.25.0 +) + +require ( + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang/protobuf v1.5.3 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/knadh/koanf/maps v0.1.1 // indirect + github.com/knadh/koanf/providers/confmap v0.1.0 // indirect + github.com/knadh/koanf/v2 v2.0.1 // indirect + github.com/mitchellh/copystructure v1.2.0 // indirect + github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect + github.com/mitchellh/reflectwalk v1.0.2 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect + go.opentelemetry.io/collector/config/configtelemetry v0.84.0 // indirect + go.opentelemetry.io/collector/confmap v0.84.0 // indirect + go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 // indirect + go.opentelemetry.io/otel v1.16.0 // indirect + go.opentelemetry.io/otel/metric v1.16.0 // indirect + go.opentelemetry.io/otel/trace v1.16.0 // indirect + go.uber.org/multierr v1.11.0 // indirect + golang.org/x/net v0.14.0 // indirect + golang.org/x/sys v0.11.0 // indirect + golang.org/x/text v0.12.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect + google.golang.org/grpc v1.57.0 // indirect + google.golang.org/protobuf v1.31.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) diff --git a/processor/starlarktransform/go.sum b/processor/starlarktransform/go.sum new file mode 100644 index 000000000000..c6bb3590e60d --- /dev/null +++ b/processor/starlarktransform/go.sum @@ -0,0 +1,181 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/MakeNowJust/heredoc/v2 v2.0.1 h1:rlCHh70XXXv7toz95ajQWOWQnN4WNLt0TdpZYIR/J6A= +github.com/MakeNowJust/heredoc/v2 v2.0.1/go.mod h1:6/2Abh5s+hc3g9nbWLe9ObDIOhaRrqsyY9MWy+4JdRM= +github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= +github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= +github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs= +github.com/knadh/koanf/maps v0.1.1/go.mod h1:npD/QZY3V6ghQDdcQzl1W4ICNVTkohC8E73eI2xW4yI= +github.com/knadh/koanf/providers/confmap v0.1.0 h1:gOkxhHkemwG4LezxxN8DMOFopOPghxRVp7JbIvdvqzU= +github.com/knadh/koanf/providers/confmap v0.1.0/go.mod h1:2uLhxQzJnyHKfxG927awZC7+fyHFdQkd697K4MdLnIU= +github.com/knadh/koanf/v2 v2.0.1 h1:1dYGITt1I23x8cfx8ZnldtezdyaZtfAuRtIFOiRzK7g= +github.com/knadh/koanf/v2 v2.0.1/go.mod h1:ZeiIlIDXTE7w1lMT6UVcNiRAS2/rCeLn/GdLNvY1Dus= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= +github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= +github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 h1:BpfhmLKZf+SjVanKKhCgf3bg+511DmU9eDQTen7LLbY= +github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= +github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.opentelemetry.io/collector/component v0.84.0 h1:bh4Is5Z7TjuyF7Mab0rSNh2q3y15fImdNDRXqrqGlbA= +go.opentelemetry.io/collector/component v0.84.0/go.mod h1:uXteRaacYXXYhzeCJe5ClLz5gLzleXWz01IZ730w7FA= +go.opentelemetry.io/collector/config/configtelemetry v0.84.0 h1:pnZiYtppJN6SlzJNcrkA8R+Ur63e33qMO260m8JbK18= +go.opentelemetry.io/collector/config/configtelemetry v0.84.0/go.mod h1:+LAXM5WFMW/UbTlAuSs6L/W72WC+q8TBJt/6z39FPOU= +go.opentelemetry.io/collector/confmap v0.84.0 h1:fS62yIRrsTQwe1gyuEc8TQM0yUNfSAzPVi0A1665ZpQ= +go.opentelemetry.io/collector/confmap v0.84.0/go.mod h1:/SNHqYkLagF0TjBjQyCy2Gt3ZF6hTK8VKbaan/ZHuJs= +go.opentelemetry.io/collector/consumer v0.84.0 h1:sz8mXIdPACJArlRyFNXA1SScVoo954IU1qp9V78VUxc= +go.opentelemetry.io/collector/consumer v0.84.0/go.mod h1:Mu+KeuorwHHWd6iGxU7DMAhgsHZmmzmQgf3sSWkugmM= +go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014 h1:C9o0mbP0MyygqFnKueVQK/v9jef6zvuttmTGlKaqhgw= +go.opentelemetry.io/collector/featuregate v1.0.0-rcv0014/go.mod h1:0mE3mDLmUrOXVoNsuvj+7dV14h/9HFl/Fy9YTLoLObo= +go.opentelemetry.io/collector/pdata v1.0.0-rcv0014 h1:iT5qH0NLmkGeIdDtnBogYDx7L58t6CaWGL378DEo2QY= +go.opentelemetry.io/collector/pdata v1.0.0-rcv0014/go.mod h1:BRvDrx43kiSoUx3mr7SoA7h9B8+OY99mUK+CZSQFWW4= +go.opentelemetry.io/collector/processor v0.84.0 h1:6VM5HLdkroeqNZ/jvjxA4nsgweJ87FDMsnNnzVBHpcY= +go.opentelemetry.io/collector/processor v0.84.0/go.mod h1:KWgBNQuA6wmmRqJwfvPaRybK2Di9X8nE2fraGuVLNJo= +go.opentelemetry.io/otel v1.16.0 h1:Z7GVAX/UkAXPKsy94IU+i6thsQS4nb7LviLpnaNeW8s= +go.opentelemetry.io/otel v1.16.0/go.mod h1:vl0h9NUa1D5s1nv3A5vZOYWn8av4K8Ml6JDeHrT/bx4= +go.opentelemetry.io/otel/metric v1.16.0 h1:RbrpwVG1Hfv85LgnZ7+txXioPDoh6EdbZHo26Q3hqOo= +go.opentelemetry.io/otel/metric v1.16.0/go.mod h1:QE47cpOmkwipPiefDwo2wDzwJrlfxxNYodqc4xnGCo4= +go.opentelemetry.io/otel/trace v1.16.0 h1:8JRpaObFoW0pxuVPapkgH8UhHQj+bJW8jJsCZEu5MQs= +go.opentelemetry.io/otel/trace v1.16.0/go.mod h1:Yt9vYq1SdNz3xdjZZK7wcXv1qv2pwLkqr2QVwea0ef0= +go.starlark.net v0.0.0-20230912135651-745481cf39ed h1:kNt8RXSIU6IRBO9MP3m+6q3WpyBHQQXqSktcyVKDPOQ= +go.starlark.net v0.0.0-20230912135651-745481cf39ed/go.mod h1:jxU+3+j+71eXOW14274+SmmuW82qJzl6iZSeqEtTGds= +go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= +go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= +go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/zap v1.25.0 h1:4Hvk6GtkucQ790dqmj7l1eEnRdKm3k3ZUrUMS2d5+5c= +go.uber.org/zap v1.25.0/go.mod h1:JIAUzQIH94IC4fOJQm7gMmBJP5k7wQfdcnYdPoEXJYk= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= +golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= +golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc h1:XSJ8Vk1SWuNr8S18z1NZSziL0CPIXLCCMDOEFtHBOFc= +google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.57.0 h1:kfzNeI/klCGD2YPMUlaGNT3pxvYfga7smW3Vth8Zsiw= +google.golang.org/grpc v1.57.0/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= +google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= +google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/processor/starlarktransform/internal/config.go b/processor/starlarktransform/internal/config.go new file mode 100644 index 000000000000..d912156bec00 --- /dev/null +++ b/processor/starlarktransform/internal/config.go @@ -0,0 +1 @@ +package config diff --git a/processor/starlarktransform/internal/logs/processor.go b/processor/starlarktransform/internal/logs/processor.go new file mode 100644 index 000000000000..32f313949953 --- /dev/null +++ b/processor/starlarktransform/internal/logs/processor.go @@ -0,0 +1,94 @@ +package logs + +import ( + "context" + "errors" + "fmt" + + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/plog" + jsonlib "go.starlark.net/lib/json" + "go.starlark.net/starlark" + "go.uber.org/zap" +) + +func NewProcessor(ctx context.Context, logger *zap.Logger, + code string, consumer consumer.Logs) *Processor { + return &Processor{ + logger: logger, + code: code, + next: consumer, + thread: &starlark.Thread{ + Name: "log/processor", + Print: func(thread *starlark.Thread, msg string) { + logger.Debug(msg, zap.String("thread", thread.Name), zap.String("source", "starlark/code")) + }, + }, + } +} + +type Processor struct { + plog.JSONMarshaler + plog.JSONUnmarshaler + logger *zap.Logger + code string + thread *starlark.Thread + transformFn starlark.Value + next consumer.Logs +} + +func (p *Processor) Start(context.Context, component.Host) error { + + global := starlark.StringDict{ + "json": jsonlib.Module, + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, global) + if err != nil { + return err + } + + // Retrieve a module global. + var ok bool + if p.transformFn, ok = globals["transform"]; !ok { + return errors.New("starlark: no 'transform' function defined in script") + } + + return nil +} + +func (p *Processor) Shutdown(context.Context) error { return nil } + +func (p *Processor) ConsumeLogs(ctx context.Context, ld plog.Logs) error { + b, err := p.MarshalLogs(ld) + if err != nil { + return err + } + + // Call the function. + result, err := starlark.Call(p.thread, p.transformFn, starlark.Tuple{starlark.String(string(b))}, nil) + if err != nil { + return fmt.Errorf("error calling transform function: %w", err) + } + + if result.String() == "None" { + p.logger.Error("transform function returned an empty value, passing record with no changes", zap.String("result", result.String())) + return p.next.ConsumeLogs(ctx, ld) + } + + if ld, err = p.UnmarshalLogs([]byte(result.String())); err != nil { + return fmt.Errorf("error unmarshalling logs data from starlark: %w", err) + } + + // if there are no logs, return + if ld.LogRecordCount() == 0 { + return nil + } + + return p.next.ConsumeLogs(ctx, ld) +} + +func (p *Processor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/starlarktransform/internal/logs/processor_test.go b/processor/starlarktransform/internal/logs/processor_test.go new file mode 100644 index 000000000000..4a622cda79a7 --- /dev/null +++ b/processor/starlarktransform/internal/logs/processor_test.go @@ -0,0 +1,226 @@ +package logs + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/MakeNowJust/heredoc/v2" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/pcommon" + "go.opentelemetry.io/collector/pdata/plog" + "go.uber.org/zap" +) + +var testlogevent = `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"test.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}` + +type fakeLogConsumer struct { + plog.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeLogConsumer) ConsumeLogs(_ context.Context, ld plog.Logs) error { + if f.t == nil { + _ = ld // for benchmarking + return nil + } + + b, err := f.MarshalLogs(ld) + require.NoError(f.t, err) + assert.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeLogConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: false} +} + +func TestLogConsumer(t *testing.T) { + lp := NewProcessor(context.Background(), + zap.NewNop(), "", + &fakeLogConsumer{t: t, expected: testlogevent}) + + testcases := []struct { + name string + event string + code string + expectError error + expectStartError error + next consumer.Logs + }{ + { + name: "update event input", + event: testlogevent, + code: heredoc.Doc(` + def transform(event): + e = json.decode(event) + e["resourceLogs"][0]["resource"]["attributes"][0]["value"]["stringValue"] = "other.log" + return e`), + + next: &fakeLogConsumer{ + t: t, + expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"other.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`, + }, + }, + { + name: "nil or empty transform return", + event: testlogevent, + code: `def transform(event): return`, + next: &fakeLogConsumer{t: t, expected: testlogevent}, + }, + { + name: "bad transform syntax", + event: testlogevent, + code: `def transform(event): event["cats"]; return`, + next: &fakeLogConsumer{t: t, expected: testlogevent}, + expectError: errors.New("error calling transform function:"), + }, + { + name: "missing transform function", + event: testlogevent, + code: `def run(event): return event`, + next: &fakeLogConsumer{t: t, expected: testlogevent}, + expectError: nil, + expectStartError: errors.New("starlark: no 'transform' function defined in script"), + }, + } + + for _, tt := range testcases { + t.Run(tt.name, func(t *testing.T) { + lp.code = tt.code + err := lp.Start(context.Background(), nil) + if tt.expectStartError != nil { + require.ErrorContains(t, err, tt.expectStartError.Error()) + return + } + + require.NoError(t, err, nil) + + ld, err := (&plog.JSONUnmarshaler{}).UnmarshalLogs([]byte(tt.event)) + require.NoError(t, err) + + lp.next = tt.next + err = lp.ConsumeLogs(context.Background(), ld) + if tt.expectError != nil { + require.ErrorContains(t, err, tt.expectError.Error()) + return + } + require.Equal(t, tt.expectError, err) + }) + } +} + +var ( + TestLogTime = time.Date(2020, 2, 11, 20, 26, 12, 321, time.UTC) + TestLogTimestamp = pcommon.NewTimestampFromTime(TestLogTime) + + TestObservedTime = time.Date(2020, 2, 11, 20, 26, 13, 789, time.UTC) + TestObservedTimestamp = pcommon.NewTimestampFromTime(TestObservedTime) + + traceID = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} + spanID = [8]byte{1, 2, 3, 4, 5, 6, 7, 8} +) + +func constructLogs() plog.Logs { + td := plog.NewLogs() + rs0 := td.ResourceLogs().AppendEmpty() + rs0.Resource().Attributes().PutStr("host.name", "localhost") + rs0ils0 := rs0.ScopeLogs().AppendEmpty() + rs0ils0.Scope().SetName("scope") + fillLogOne(rs0ils0.LogRecords().AppendEmpty()) + fillLogTwo(rs0ils0.LogRecords().AppendEmpty()) + return td +} + +func fillLogOne(log plog.LogRecord) { + log.Body().SetStr("operationA") + log.SetTimestamp(TestLogTimestamp) + log.SetObservedTimestamp(TestObservedTimestamp) + log.SetDroppedAttributesCount(1) + log.SetFlags(plog.DefaultLogRecordFlags.WithIsSampled(true)) + log.SetSeverityNumber(1) + log.SetTraceID(traceID) + log.SetSpanID(spanID) + log.Attributes().PutStr("http.method", "get") + log.Attributes().PutStr("http.path", "/health") + log.Attributes().PutStr("http.url", "http://localhost/health") + log.Attributes().PutStr("flags", "A|B|C") + log.Attributes().PutStr("total.string", "123456789") + +} + +func fillLogTwo(log plog.LogRecord) { + log.Body().SetStr("operationB") + log.SetTimestamp(TestLogTimestamp) + log.SetObservedTimestamp(TestObservedTimestamp) + log.Attributes().PutStr("http.method", "get") + log.Attributes().PutStr("http.path", "/health") + log.Attributes().PutStr("http.url", "http://localhost/health") + log.Attributes().PutStr("flags", "C|D") + log.Attributes().PutStr("total.string", "345678") + +} + +func BenchmarkLogProcessor(b *testing.B) { + benchcases := []struct { + name string + code string + }{ + { + `set(attributes["test"], "pass") where body == "operationA"`, + heredoc.Doc(` + def transform(event): + e = json.decode(event) + for r in e["resourceLogs"]: + for sl in r["scopeLogs"]: + for lr in sl["logRecords"]: + if lr["body"]["stringValue"] == "operationA": + lr["attributes"].append({ + "key": "test", + "value": {"stringValue": "pass"} + }) + return e`), + }, + { + `set(attributes["test"], "pass") where resource.attributes["host.name"] == "localhost"`, + heredoc.Doc(` + def transform(event): + e = json.decode(event) + for rlogs in e["resourceLogs"]: + if [ + r for r in rlogs['resource']['attributes'] + if r['key'] == 'host.name' and r['value']['stringValue'] == 'localhost' + ]: + for sl in rlogs["scopeLogs"]: + for lr in sl["logRecords"]: + lr["attributes"].append({ + "key": "test", + "value": {"stringValue": "pass"} + }) + return e`), + }, + } + + for _, bc := range benchcases { + b.Run(bc.name, func(b *testing.B) { + for i := 0; i < b.N; i++ { + ld := constructLogs() + lp := NewProcessor(context.Background(), + zap.NewNop(), bc.code, + &fakeLogConsumer{}) + + if err := lp.Start(context.Background(), nil); err != nil { + b.Error(err) + } + + if err := lp.ConsumeLogs(context.Background(), ld); err != nil { + b.Error(err) + } + } + }) + } +} diff --git a/processor/starlarktransform/internal/metadata/metadata.go b/processor/starlarktransform/internal/metadata/metadata.go new file mode 100644 index 000000000000..770c6c6dcdb4 --- /dev/null +++ b/processor/starlarktransform/internal/metadata/metadata.go @@ -0,0 +1,12 @@ +package metadata + +import ( + "go.opentelemetry.io/collector/component" +) + +const ( + Type = "starlarktransform" + LogsStability = component.StabilityLevelAlpha + MetricsStability = component.StabilityLevelAlpha + TracesStability = component.StabilityLevelAlpha +) diff --git a/processor/starlarktransform/internal/metrics/processor.go b/processor/starlarktransform/internal/metrics/processor.go new file mode 100644 index 000000000000..b933dcdec923 --- /dev/null +++ b/processor/starlarktransform/internal/metrics/processor.go @@ -0,0 +1,93 @@ +package metrics + +import ( + "context" + "errors" + "fmt" + + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/pmetric" + jsonlib "go.starlark.net/lib/json" + "go.starlark.net/starlark" + "go.uber.org/zap" +) + +func NewProcessor(ctx context.Context, logger *zap.Logger, + code string, consumer consumer.Metrics) *Processor { + return &Processor{ + logger: logger, + code: code, + next: consumer, + thread: &starlark.Thread{ + Name: "metric/processor", + Print: func(thread *starlark.Thread, msg string) { + logger.Debug(msg, zap.String("thread", thread.Name), zap.String("source", "starlark/code")) + }, + }, + } +} + +type Processor struct { + pmetric.JSONMarshaler + pmetric.JSONUnmarshaler + logger *zap.Logger + code string + thread *starlark.Thread + transformFn starlark.Value + next consumer.Metrics +} + +func (p *Processor) Start(context.Context, component.Host) error { + + global := starlark.StringDict{ + "json": jsonlib.Module, + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, global) + if err != nil { + return err + } + + // Retrieve a module global. + var ok bool + if p.transformFn, ok = globals["transform"]; !ok { + return errors.New("starlark: no 'transform' function defined in script") + } + return nil +} + +func (p *Processor) Shutdown(context.Context) error { return nil } + +func (p *Processor) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error { + b, err := p.MarshalMetrics(md) + if err != nil { + return err + } + + // Call the function. + result, err := starlark.Call(p.thread, p.transformFn, starlark.Tuple{starlark.String(string(b))}, nil) + if err != nil { + return fmt.Errorf("error calling transform function: %w", err) + } + + if result.String() == "None" { + p.logger.Error("transform function returned an empty value, passing record with no changes", zap.String("result", result.String())) + return p.next.ConsumeMetrics(ctx, md) + } + + if md, err = p.UnmarshalMetrics([]byte(result.String())); err != nil { + return fmt.Errorf("error unmarshalling logs data from starlark: %w", err) + } + + // if there are no metrics, return + if md.ResourceMetrics().Len() == 0 { + return nil + } + + return p.next.ConsumeMetrics(ctx, md) +} + +func (p *Processor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/starlarktransform/internal/metrics/processor_test.go b/processor/starlarktransform/internal/metrics/processor_test.go new file mode 100644 index 000000000000..57f9dab7b60e --- /dev/null +++ b/processor/starlarktransform/internal/metrics/processor_test.go @@ -0,0 +1,171 @@ +package metrics + +import ( + "context" + "errors" + "testing" + + "github.com/MakeNowJust/heredoc/v2" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/pmetric" + "go.uber.org/zap" +) + +var testmetricevent = `{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{"name":"otelcol/hostmetricsreceiver/memory","version":"0.84.0"},"metrics":[{"name":"system.memory.usage","description":"Bytes of memory in use.","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"1874247680"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"29214199808"}],"aggregationTemporality":2}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.9.0"}]}` + +type fakeMetricConsumer struct { + pmetric.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeMetricConsumer) ConsumeMetrics(_ context.Context, md pmetric.Metrics) error { + if f.t == nil { + return nil + } + + b, err := f.MarshalMetrics(md) + require.NoError(f.t, err) + assert.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeMetricConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: false} +} + +func TestMetricConsumer(t *testing.T) { + mp := NewProcessor(context.Background(), + zap.NewNop(), "", + &fakeMetricConsumer{t: t, expected: testmetricevent}) + + testcases := []struct { + name string + event string + code string + expectError error + expectStartError error + next consumer.Metrics + }{ + { + name: "update event input", + event: testmetricevent, + code: heredoc.Doc(` + def transform(event): + event = json.decode(event) + for md in event['resourceMetrics']: + # prefix each metric name with starlarktransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'starlarktransform.' + m['name'] + + return event`), + + next: &fakeMetricConsumer{ + t: t, + expected: `{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{"name":"otelcol/hostmetricsreceiver/memory","version":"0.84.0"},"metrics":[{"name":"starlarktransform.system.memory.usage","description":"Bytes of memory in use.","unit":"By","sum":{"dataPoints":[{"attributes":[{"key":"state","value":{"stringValue":"used"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"1874247680"},{"attributes":[{"key":"state","value":{"stringValue":"free"}}],"startTimeUnixNano":"1694171569000000000","timeUnixNano":"1694189699786689531","asInt":"29214199808"}],"aggregationTemporality":2}}]}],"schemaUrl":"https://opentelemetry.io/schemas/1.9.0"}]}`, + }, + }, + { + name: "nil or empty transform return", + event: testmetricevent, + code: `def transform(event): return`, + next: &fakeMetricConsumer{t: t, expected: testmetricevent}, + }, + { + name: "bad transform syntax", + event: testmetricevent, + code: `def transform(event): event["cats"]; return`, + next: &fakeMetricConsumer{t: t, expected: testmetricevent}, + expectError: errors.New("error calling transform function:"), + }, + { + name: "missing transform function", + event: testmetricevent, + code: `def run(event): return event`, + next: &fakeMetricConsumer{t: t, expected: testmetricevent}, + expectError: nil, + expectStartError: errors.New("starlark: no 'transform' function defined in script"), + }, + } + + for _, tt := range testcases { + t.Run(tt.name, func(t *testing.T) { + mp.code = tt.code + err := mp.Start(context.Background(), nil) + if tt.expectStartError != nil { + require.ErrorContains(t, err, tt.expectStartError.Error()) + return + } + + require.NoError(t, err, nil) + + md, err := (&pmetric.JSONUnmarshaler{}).UnmarshalMetrics([]byte(tt.event)) + require.NoError(t, err) + + mp.next = tt.next + err = mp.ConsumeMetrics(context.Background(), md) + if tt.expectError != nil { + require.ErrorContains(t, err, tt.expectError.Error()) + return + } + require.Equal(t, tt.expectError, err) + }) + } +} + +func BenchmarkLogConsumer(b *testing.B) { + benchcases := []struct { + name string + event string + code string + expectError error + next consumer.Metrics + }{ + { + name: "update event input", + event: testmetricevent, + code: heredoc.Doc(` + def transform(event): + event = json.decode(event) + for md in event['resourceMetrics']: + # prefix each metric name with starlarktransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'starlarktransform.' + m['name'] + + return event`), + + next: &fakeMetricConsumer{ + t: nil, + expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"other.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`, + }, + expectError: nil, + }, + } + + for _, bc := range benchcases { + b.Run(bc.name, func(b *testing.B) { + for i := 0; i < b.N; i++ { + mp := NewProcessor(context.Background(), + zap.NewNop(), "", + bc.next) + + mp.code = bc.code + require.NoError(b, mp.Start(context.Background(), nil)) + + md, err := (&pmetric.JSONUnmarshaler{}).UnmarshalMetrics([]byte(bc.event)) + if err != nil { + b.Error(err) + } + + if err = mp.ConsumeMetrics(context.Background(), md); err != nil { + b.Error(err) + } + + } + }) + } +} diff --git a/processor/starlarktransform/internal/traces/processor.go b/processor/starlarktransform/internal/traces/processor.go new file mode 100644 index 000000000000..988931b55d4d --- /dev/null +++ b/processor/starlarktransform/internal/traces/processor.go @@ -0,0 +1,93 @@ +package traces + +import ( + "context" + "errors" + "fmt" + + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/ptrace" + jsonlib "go.starlark.net/lib/json" + "go.starlark.net/starlark" + "go.uber.org/zap" +) + +func NewProcessor(ctx context.Context, logger *zap.Logger, + code string, consumer consumer.Traces) *Processor { + return &Processor{ + logger: logger, + code: code, + next: consumer, + thread: &starlark.Thread{ + Name: "trace.processor", + Print: func(thread *starlark.Thread, msg string) { + logger.Debug(msg, zap.String("thread", thread.Name), zap.String("source", "starlark/code")) + }, + }, + } +} + +type Processor struct { + ptrace.JSONMarshaler + ptrace.JSONUnmarshaler + logger *zap.Logger + code string + thread *starlark.Thread + transformFn starlark.Value + next consumer.Traces +} + +func (p *Processor) Start(context.Context, component.Host) error { + + global := starlark.StringDict{ + "json": jsonlib.Module, + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, global) + if err != nil { + return err + } + + // Retrieve a module global. + var ok bool + if p.transformFn, ok = globals["transform"]; !ok { + return errors.New("starlark: no 'transform' function defined in script") + } + return nil +} + +func (p *Processor) Shutdown(context.Context) error { return nil } + +func (p *Processor) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { + b, err := p.MarshalTraces(td) + if err != nil { + return err + } + + // Call the function. + result, err := starlark.Call(p.thread, p.transformFn, starlark.Tuple{starlark.String(string(b))}, nil) + if err != nil { + return fmt.Errorf("error calling transform function: %w", err) + } + + if result.String() == "None" { + p.logger.Error("transform function returned an empty value, passing record with no changes", zap.String("result", result.String())) + return p.next.ConsumeTraces(ctx, td) + } + + if td, err = p.UnmarshalTraces([]byte(result.String())); err != nil { + return fmt.Errorf("error unmarshalling logs data from starlark: %w", err) + } + + // if there are no spans, return + if td.SpanCount() == 0 { + return nil + } + + return p.next.ConsumeTraces(ctx, td) +} + +func (p *Processor) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: true} +} diff --git a/processor/starlarktransform/internal/traces/processor_test.go b/processor/starlarktransform/internal/traces/processor_test.go new file mode 100644 index 000000000000..fffe3452bb18 --- /dev/null +++ b/processor/starlarktransform/internal/traces/processor_test.go @@ -0,0 +1,174 @@ +package traces + +import ( + "context" + "errors" + "testing" + + "github.com/MakeNowJust/heredoc/v2" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/consumer" + "go.opentelemetry.io/collector/pdata/ptrace" + "go.uber.org/zap" +) + +var testtraceevent = `{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"service.name","value":{"stringValue":"unknown_service"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9cb5bf738137b2248dc7b20445ec2e1c","spanId":"88079ad5c94b5b13","parentSpanId":"","name":"/roll","kind":2,"startTimeUnixNano":"1694388218052842000","endTimeUnixNano":"1694388218053415000","attributes":[{"key":"http.method","value":{"stringValue":"GET"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"5001"}},{"key":"http.host","value":{"stringValue":"localhost:5001"}},{"key":"http.target","value":{"stringValue":"/roll"}},{"key":"net.peer.ip","value":{"stringValue":"127.0.0.1"}},{"key":"http.user_agent","value":{"stringValue":"curl/7.87.0"}},{"key":"net.peer.port","value":{"intValue":"52365"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/roll"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}` + +type fakeTraceConsumer struct { + ptrace.JSONMarshaler + t *testing.T + expected string +} + +func (f *fakeTraceConsumer) ConsumeTraces(_ context.Context, md ptrace.Traces) error { + if f.t == nil { + return nil + } + + b, err := f.MarshalTraces(md) + require.NoError(f.t, err) + assert.Equal(f.t, f.expected, string(b)) + return nil +} + +func (f *fakeTraceConsumer) Capabilities() consumer.Capabilities { + return consumer.Capabilities{MutatesData: false} +} + +func TestTraceConsumer(t *testing.T) { + mp := NewProcessor(context.Background(), + zap.NewNop(), "", + &fakeTraceConsumer{t: t, expected: testtraceevent}) + + testcases := []struct { + name string + event string + code string + expectError error + expectStartError error + next consumer.Traces + }{ + { + name: "update event input", + event: testtraceevent, + code: heredoc.Doc(` + def transform(event): + event = json.decode(event) + for td in event['resourceSpans']: + # add resource attribute + td['resource']['attributes'].append({ + 'key': 'source', + 'value': { + 'stringValue': 'starlarktransform' + } + }) + + return event`), + + next: &fakeTraceConsumer{ + t: t, + expected: `{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"service.name","value":{"stringValue":"unknown_service"}},{"key":"source","value":{"stringValue":"starlarktransform"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9cb5bf738137b2248dc7b20445ec2e1c","spanId":"88079ad5c94b5b13","parentSpanId":"","name":"/roll","kind":2,"startTimeUnixNano":"1694388218052842000","endTimeUnixNano":"1694388218053415000","attributes":[{"key":"http.method","value":{"stringValue":"GET"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"5001"}},{"key":"http.host","value":{"stringValue":"localhost:5001"}},{"key":"http.target","value":{"stringValue":"/roll"}},{"key":"net.peer.ip","value":{"stringValue":"127.0.0.1"}},{"key":"http.user_agent","value":{"stringValue":"curl/7.87.0"}},{"key":"net.peer.port","value":{"intValue":"52365"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/roll"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}`, + }, + }, + { + name: "nil or empty transform return", + event: testtraceevent, + code: `def transform(event): return`, + next: &fakeTraceConsumer{t: t, expected: testtraceevent}, + }, + { + name: "bad transform syntax", + event: testtraceevent, + code: `def transform(event): event["cats"]; return`, + next: &fakeTraceConsumer{t: t, expected: testtraceevent}, + expectError: errors.New("error calling transform function:"), + }, + { + name: "missing transform function", + event: testtraceevent, + code: `def run(event): return event`, + next: &fakeTraceConsumer{t: t, expected: testtraceevent}, + expectError: nil, + expectStartError: errors.New("starlark: no 'transform' function defined in script"), + }, + } + + for _, tt := range testcases { + t.Run(tt.name, func(t *testing.T) { + mp.code = tt.code + err := mp.Start(context.Background(), nil) + if tt.expectStartError != nil { + require.ErrorContains(t, err, tt.expectStartError.Error()) + return + } + + require.NoError(t, err, nil) + + md, err := (&ptrace.JSONUnmarshaler{}).UnmarshalTraces([]byte(tt.event)) + require.NoError(t, err) + + mp.next = tt.next + err = mp.ConsumeTraces(context.Background(), md) + if tt.expectError != nil { + require.ErrorContains(t, err, tt.expectError.Error()) + return + } + require.Equal(t, tt.expectError, err) + }) + } +} + +func BenchmarkLogConsumer(b *testing.B) { + benchcases := []struct { + name string + event string + code string + expectError error + next consumer.Traces + }{ + { + name: "update event input", + event: testtraceevent, + code: heredoc.Doc(` + def transform(event): + event = json.decode(event) + for md in event['resourceMetrics']: + # prefix each metric name with starlarktransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'starlarktransform.' + m['name'] + + return event`), + + next: &fakeTraceConsumer{ + t: nil, + expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"other.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}"},"attributes":[{"key":"app","value":{"stringValue":"dev"}}],"traceId":"","spanId":""}]}]}]}`, + }, + expectError: nil, + }, + } + + for _, bc := range benchcases { + b.Run(bc.name, func(b *testing.B) { + for i := 0; i < b.N; i++ { + mp := NewProcessor(context.Background(), + zap.NewNop(), "", + bc.next) + + mp.code = bc.code + require.NoError(b, mp.Start(context.Background(), nil)) + + md, err := (&ptrace.JSONUnmarshaler{}).UnmarshalTraces([]byte(bc.event)) + if err != nil { + b.Error(err) + } + + if err = mp.ConsumeTraces(context.Background(), md); err != nil { + b.Error(err) + } + + } + }) + } +} diff --git a/processor/starlarktransform/testdata/builder.yaml b/processor/starlarktransform/testdata/builder.yaml new file mode 100644 index 000000000000..899b949b9ba1 --- /dev/null +++ b/processor/starlarktransform/testdata/builder.yaml @@ -0,0 +1,19 @@ +dist: + name: ddot + output_path: ./cmd/ddot + description: Daidokoro Open Telemetry Contrib binary. + version: 1.0.0 + +processors: + # add my processor + - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/processor/starlarktransformprocessor v0.0.0" + path: "../" + +exporters: + - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.84.0 + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.84.0 + +receivers: + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.84.0 + - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.84.0 + - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.84.0 diff --git a/processor/starlarktransform/testdata/config.yaml b/processor/starlarktransform/testdata/config.yaml new file mode 100644 index 000000000000..623f5a677547 --- /dev/null +++ b/processor/starlarktransform/testdata/config.yaml @@ -0,0 +1,136 @@ +receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + grpc: + endpoint: "0.0.0.0:4317" + + filelog: + start_at: beginning + include_file_name: true + include: + - $LOGFILE + + operators: + - type: move + from: attributes["log.file.name"] + to: resource["log.file.name"] + + - type: add + field: attributes.app + value: dev + +processors: + + # - change resource attribute log.file.name to source.log + # - add resource attribute cluster: dev + # - filter out any logs that contain the word password + # - add an attribute to each log: language: golang + starlarktransform/logs: + code: | + def transform(event): + event = json.decode(event) + # edit resource attributes + for data in event['resourceLogs']: + for attr in data['resource']['attributes']: + attr['value']['stringValue'] = 'source.log' + + # filter/delete logs + for data in event['resourceLogs']: + for slog in data['scopeLogs']: + slog['logRecords'] = [ lr for lr in slog['logRecords'] if 'internal' not in lr['body']['stringValue']] + + # add an attribute to each log + for lr in slog['logRecords']: + lr['attributes'].append({ + 'key': 'language', + 'value': { + 'stringValue': 'golang' + }}) + + return event + + # - print event received to otel runtime log + # - if there are no resources, add a resource attribute source starlarktransform + # - prefix each metric name with starlarktransform + starlarktransform/metrics: + code: | + def transform(event): + print("received event", event) + event = json.decode(event) + for md in event['resourceMetrics']: + # if resources are empty + if not md['resource']: + md['resource'] = { + 'attributes': [ + { + "key": "source", + "value": { + "stringValue": "starlarktransform" + } + } + ] + } + + # prefix each metric name with starlarktransform + for sm in md['scopeMetrics']: + for m in sm['metrics']: + m['name'] = 'starlarktransform.' + m['name'] + + return event + + # - add resource attribute source starlarktransform + # - filter out any spans with http.target /roll attribute + starlarktransform/traces: + code: | + def transform(event): + event = json.decode(event) + for td in event['resourceSpans']: + # add resource attribute + td['resource']['attributes'].append({ + 'key': 'source', + 'value': { + 'stringValue': 'starlarktransform' + } + }) + + # filter spans with http.target /roll attribute + has_roll = lambda attrs: [a for a in attrs if a['key'] == 'http.target' and a['value']['stringValue'] == '/cats'] + for sd in td['scopeSpans']: + sd['spans'] = [ + s for s in sd['spans'] + if not has_roll(s['attributes']) + ] + return event + +exporters: + logging: + verbosity: detailed + +service: + pipelines: + logs: + receivers: + - filelog + processors: + - starlarktransform/logs + exporters: + - logging + + metrics: + receivers: + - otlp + processors: + - starlarktransform/metrics + exporters: + - logging + + traces: + receivers: + - otlp + processors: + - starlarktransform/traces + exporters: + - logging + \ No newline at end of file diff --git a/processor/starlarktransform/testdata/log_event_example.json b/processor/starlarktransform/testdata/log_event_example.json new file mode 100644 index 000000000000..598184dd0bd8 --- /dev/null +++ b/processor/starlarktransform/testdata/log_event_example.json @@ -0,0 +1,119 @@ +{ + "resourceLogs": [ + { + "resource": { + "attributes": [ + { + "key": "log.file.name", + "value": { + "stringValue": "test.log" + } + } + ] + }, + "scopeLogs": [ + { + "scope": {}, + "logRecords": [ + { + "observedTimeUnixNano": "1694127596456358000", + "body": { + "stringValue": "2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {\"version\": \"0.84.0\", \"date\": \"2023-08-29T18:58:24Z\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456543000", + "body": { + "stringValue": "2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {\"path\": \"builder.yaml\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456572000", + "body": { + "stringValue": "2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {\"go-executable\": \"/usr/local/go/bin/go\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456576000", + "body": { + "stringValue": "2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {\"path\": \"./cmd/ddot\"}" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456611000", + "body": { + "stringValue": "2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + }, + { + "observedTimeUnixNano": "1694127596456668000", + "body": { + "stringValue": "2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling" + }, + "attributes": [ + { + "key": "app", + "value": { + "stringValue": "dev" + } + } + ], + "traceId": "", + "spanId": "" + } + ] + } + ] + } + ] + } \ No newline at end of file diff --git a/processor/starlarktransform/testdata/metric_event_example.json b/processor/starlarktransform/testdata/metric_event_example.json new file mode 100644 index 000000000000..026f06779aa7 --- /dev/null +++ b/processor/starlarktransform/testdata/metric_event_example.json @@ -0,0 +1,55 @@ +{ + "resourceMetrics": [ + { + "resource": {}, + "scopeMetrics": [ + { + "scope": { + "name": "otelcol/hostmetricsreceiver/memory", + "version": "0.84.0" + }, + "metrics": [ + { + "name": "system.memory.usage", + "description": "Bytes of memory in use.", + "unit": "By", + "sum": { + "dataPoints": [ + { + "attributes": [ + { + "key": "state", + "value": { + "stringValue": "used" + } + } + ], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "1874247680" + }, + { + "attributes": [ + { + "key": "state", + "value": { + "stringValue": "free" + } + } + ], + "startTimeUnixNano": "1694171569000000000", + "timeUnixNano": "1694189699786689531", + "asInt": "29214199808" + } + ], + "aggregationTemporality": 2 + } + } + ] + } + ], + "schemaUrl": "https://opentelemetry.io/schemas/1.9.0" + } + ] + } + \ No newline at end of file diff --git a/processor/starlarktransform/testdata/test.log b/processor/starlarktransform/testdata/test.log new file mode 100644 index 000000000000..81a4bf6c3a5c --- /dev/null +++ b/processor/starlarktransform/testdata/test.log @@ -0,0 +1,7 @@ +2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder {"version": "0.84.0", "date": "2023-08-29T18:58:24Z"} +2023-09-06T01:09:24.049+0200 INFO internal/command.go:150 Using config file {"path": "builder.yaml"} +2023-09-06T01:09:24.049+0200 INFO builder/config.go:106 Using go {"go-executable": "/usr/local/go/bin/go"} +2023-09-06T01:09:24.050+0200 INFO builder/main.go:69 Sources created {"path": "./cmd/ddot"} +2023-09-06T01:09:35.392+0200 INFO builder/main.go:121 Getting go modules +2023-09-06T01:09:38.554+0200 INFO builder/main.go:80 Compiling +2023-09-06T01:09:47.694+0200 INFO builder/main.go:102 Compiled {"binary": "./cmd/ddot/ddot"} \ No newline at end of file diff --git a/processor/starlarktransform/testdata/trace_event_example.json b/processor/starlarktransform/testdata/trace_event_example.json new file mode 100644 index 000000000000..ef165a57ceaa --- /dev/null +++ b/processor/starlarktransform/testdata/trace_event_example.json @@ -0,0 +1,135 @@ +{ + "resourceSpans": [ + { + "resource": { + "attributes": [ + { + "key": "telemetry.sdk.language", + "value": { + "stringValue": "python" + } + }, + { + "key": "telemetry.sdk.name", + "value": { + "stringValue": "opentelemetry" + } + }, + { + "key": "telemetry.sdk.version", + "value": { + "stringValue": "1.19.0" + } + }, + { + "key": "telemetry.auto.version", + "value": { + "stringValue": "0.40b0" + } + }, + { + "key": "service.name", + "value": { + "stringValue": "unknown_service" + } + } + ] + }, + "scopeSpans": [ + { + "scope": { + "name": "opentelemetry.instrumentation.flask", + "version": "0.40b0" + }, + "spans": [ + { + "traceId": "9cb5bf738137b2248dc7b20445ec2e1c", + "spanId": "88079ad5c94b5b13", + "parentSpanId": "", + "name": "/roll", + "kind": 2, + "startTimeUnixNano": "1694388218052842000", + "endTimeUnixNano": "1694388218053415000", + "attributes": [ + { + "key": "http.method", + "value": { + "stringValue": "GET" + } + }, + { + "key": "http.server_name", + "value": { + "stringValue": "0.0.0.0" + } + }, + { + "key": "http.scheme", + "value": { + "stringValue": "http" + } + }, + { + "key": "net.host.port", + "value": { + "intValue": "5001" + } + }, + { + "key": "http.host", + "value": { + "stringValue": "localhost:5001" + } + }, + { + "key": "http.target", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "net.peer.ip", + "value": { + "stringValue": "127.0.0.1" + } + }, + { + "key": "http.user_agent", + "value": { + "stringValue": "curl/7.87.0" + } + }, + { + "key": "net.peer.port", + "value": { + "intValue": "52365" + } + }, + { + "key": "http.flavor", + "value": { + "stringValue": "1.1" + } + }, + { + "key": "http.route", + "value": { + "stringValue": "/roll" + } + }, + { + "key": "http.status_code", + "value": { + "intValue": "200" + } + } + ], + "status": {} + } + ] + } + ] + } + ] + } + \ No newline at end of file From f147a460cacbf7241899af04e9b161358deadb04 Mon Sep 17 00:00:00 2001 From: daidokoro Date: Sun, 24 Sep 2023 17:33:56 +0200 Subject: [PATCH 07/16] reamme --- processor/starlarktransform/README.md | 40 +++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/processor/starlarktransform/README.md b/processor/starlarktransform/README.md index 442bfeed1b25..751f3013b8e3 100644 --- a/processor/starlarktransform/README.md +++ b/processor/starlarktransform/README.md @@ -26,16 +26,21 @@ To configure starlarktransform, you add your code using the `code` option in the processors: starlarktransform: code: | - def transform(event): + def transform(event): event = json.decode(event) - - - return event + return event ``` You must define a function called `transform` that accepts a single argument, `event`. This function is called by the processor and is passed the telemetry event. The function **must return** the modified, json decoded event. +## Why? + +While there are a number of transform processors, most notably, the main OTTL [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor), this processor aims to grant users more flexibility by allowing them to manipulate telemetry data using a familiar syntax. + +Python is a popular, well known language, even among non-developers. By allowing Starlark code to be used as an option to transform telemetry data, users can leverage their existing knowledge of Python. + + ## How it works The processor uses the [Starlark-Go](https://github.com/google/starlark-go) interpreter, this allows you to run this processor without having to install a Starlark language interpreter on the host. @@ -382,7 +387,7 @@ processors: starlarktransform/metrics: code: | def transform(event): - print("received event", event) + print("received event", event) event = json.decode(event) for md in event['resourceMetrics']: # if resources are empty @@ -463,4 +468,27 @@ service: ### Warnings -The starlarktransform processor allows you to modify all aspects of your telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. \ No newline at end of file +The starlarktransform processor allows you to modify all aspects of your telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. + + +```yaml + +```yaml +processors: + starlarktransform: + code: | + def transform(event): + e = json.decode(event) + for r in e["resourceLogs"]: + for sl in r["scopeLogs"]: + for lr in sl["logRecords"]: + if lr["body"]["stringValue"] == "operationA": + lr["attributes"].append({ + "key": "test", + "value": {"stringValue": "pass"} + }) + return e + +``` + +``` \ No newline at end of file From 465ee60dc5d23942d94d400becdfe5c3fcf714fd Mon Sep 17 00:00:00 2001 From: daidokoro Date: Sun, 24 Sep 2023 17:36:12 +0200 Subject: [PATCH 08/16] fix readme --- processor/starlarktransform/README.md | 22 ---------------------- 1 file changed, 22 deletions(-) diff --git a/processor/starlarktransform/README.md b/processor/starlarktransform/README.md index 751f3013b8e3..c5c18322e180 100644 --- a/processor/starlarktransform/README.md +++ b/processor/starlarktransform/README.md @@ -470,25 +470,3 @@ service: The starlarktransform processor allows you to modify all aspects of your telemetry data. This can result in invalid or bad data being propogated if you are not careful. It is your responsibility to inspect the data and ensure it is valid. - -```yaml - -```yaml -processors: - starlarktransform: - code: | - def transform(event): - e = json.decode(event) - for r in e["resourceLogs"]: - for sl in r["scopeLogs"]: - for lr in sl["logRecords"]: - if lr["body"]["stringValue"] == "operationA": - lr["attributes"].append({ - "key": "test", - "value": {"stringValue": "pass"} - }) - return e - -``` - -``` \ No newline at end of file From fef18f6255474e0337550d5aa9dab042de9f7f8a Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 18:32:33 +0100 Subject: [PATCH 09/16] dependencies --- processor/starlarktransform/go.mod | 2 ++ processor/starlarktransform/go.sum | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/processor/starlarktransform/go.mod b/processor/starlarktransform/go.mod index 5c220b12d2e1..1fec7c678cb7 100644 --- a/processor/starlarktransform/go.mod +++ b/processor/starlarktransform/go.mod @@ -4,6 +4,7 @@ go 1.20 require ( github.com/MakeNowJust/heredoc/v2 v2.0.1 + github.com/qri-io/starlib v0.5.0 github.com/stretchr/testify v1.8.4 go.opentelemetry.io/collector/component v0.84.0 go.opentelemetry.io/collector/consumer v0.84.0 @@ -26,6 +27,7 @@ require ( github.com/mitchellh/reflectwalk v1.0.2 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pkg/errors v0.8.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect go.opentelemetry.io/collector/config/configtelemetry v0.84.0 // indirect go.opentelemetry.io/collector/confmap v0.84.0 // indirect diff --git a/processor/starlarktransform/go.sum b/processor/starlarktransform/go.sum index c6bb3590e60d..d9b5050e6424 100644 --- a/processor/starlarktransform/go.sum +++ b/processor/starlarktransform/go.sum @@ -1,7 +1,10 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/360EntSecGroup-Skylar/excelize v1.4.1/go.mod h1:vnax29X2usfl7HHkBrX5EvSCJcmH3dT9luvxzu8iGAE= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/MakeNowJust/heredoc/v2 v2.0.1 h1:rlCHh70XXXv7toz95ajQWOWQnN4WNLt0TdpZYIR/J6A= github.com/MakeNowJust/heredoc/v2 v2.0.1/go.mod h1:6/2Abh5s+hc3g9nbWLe9ObDIOhaRrqsyY9MWy+4JdRM= +github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc= +github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y= github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= @@ -11,10 +14,12 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dustmop/soup v1.1.2-0.20190516214245-38228baa104e/go.mod h1:CgNC6SGbT+Xb8wGGvzilttZL1mc5sQ/5KkcxsZttMIk= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= @@ -41,6 +46,7 @@ github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/knadh/koanf/maps v0.1.1 h1:G5TjmUh2D7G2YWf5SQQqSiHRJEjaicvU0KpypqB3NIs= @@ -50,6 +56,8 @@ github.com/knadh/koanf/providers/confmap v0.1.0/go.mod h1:2uLhxQzJnyHKfxG927awZC github.com/knadh/koanf/v2 v2.0.1 h1:1dYGITt1I23x8cfx8ZnldtezdyaZtfAuRtIFOiRzK7g= github.com/knadh/koanf/v2 v2.0.1/go.mod h1:ZeiIlIDXTE7w1lMT6UVcNiRAS2/rCeLn/GdLNvY1Dus= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= @@ -62,12 +70,21 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8= +github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= +github.com/paulmach/orb v0.1.5/go.mod h1:pPwxxs3zoAyosNSbNKn1jiXV2+oovRDObDKfTvRegDI= +github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= +github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/qri-io/starlib v0.5.0 h1:NlveoBAhO6mNgM7+JpM9QlHh3/3pOtOiH6iXaqSdVK0= +github.com/qri-io/starlib v0.5.0/go.mod h1:FpVumyB2CMrKIrjf39fAi4uydYWVvnWEvXEOwfzZRHY= github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.2.3-0.20181224173747-660f15d67dbb/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -92,6 +109,7 @@ go.opentelemetry.io/otel/metric v1.16.0 h1:RbrpwVG1Hfv85LgnZ7+txXioPDoh6EdbZHo26 go.opentelemetry.io/otel/metric v1.16.0/go.mod h1:QE47cpOmkwipPiefDwo2wDzwJrlfxxNYodqc4xnGCo4= go.opentelemetry.io/otel/trace v1.16.0 h1:8JRpaObFoW0pxuVPapkgH8UhHQj+bJW8jJsCZEu5MQs= go.opentelemetry.io/otel/trace v1.16.0/go.mod h1:Yt9vYq1SdNz3xdjZZK7wcXv1qv2pwLkqr2QVwea0ef0= +go.starlark.net v0.0.0-20210406145628-7a1108eaa012/go.mod h1:t3mmBBPzAVvK0L0n1drDmrQsJ8FoIx4INCqVMTr/Zo0= go.starlark.net v0.0.0-20230912135651-745481cf39ed h1:kNt8RXSIU6IRBO9MP3m+6q3WpyBHQQXqSktcyVKDPOQ= go.starlark.net v0.0.0-20230912135651-745481cf39ed/go.mod h1:jxU+3+j+71eXOW14274+SmmuW82qJzl6iZSeqEtTGds= go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= @@ -108,13 +126,16 @@ golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvx golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= @@ -127,6 +148,7 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -138,6 +160,7 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= @@ -174,7 +197,10 @@ google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQ google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= From e427149d2edf43da0c7564ed70b5d457d36eac72 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 18:32:55 +0100 Subject: [PATCH 10/16] fix factory test --- processor/starlarktransform/factory_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/processor/starlarktransform/factory_test.go b/processor/starlarktransform/factory_test.go index 54b0bcea2d65..8d1e3f7e831b 100644 --- a/processor/starlarktransform/factory_test.go +++ b/processor/starlarktransform/factory_test.go @@ -12,5 +12,5 @@ func TestCreateDefaultConfig(t *testing.T) { cfg := factory.CreateDefaultConfig() assert.NoError(t, componenttest.CheckConfigStruct(cfg)) assert.NotNil(t, cfg) - assert.Equal(t, "def transform(event): return json.decode(event)", cfg.(*Config).Code) + assert.Equal(t, "def transform(e): return json.decode(e)", cfg.(*Config).Code) } From b3cc3d0cce919b06a10831e3740d536193fb1278 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 18:33:23 +0100 Subject: [PATCH 11/16] added regex module to starlark init modules --- .../internal/logs/processor.go | 14 ++++++++-- .../internal/logs/processor_test.go | 26 +++++++++++++++++++ .../internal/metrics/processor.go | 14 ++++++++-- .../internal/traces/processor.go | 14 ++++++++-- 4 files changed, 62 insertions(+), 6 deletions(-) diff --git a/processor/starlarktransform/internal/logs/processor.go b/processor/starlarktransform/internal/logs/processor.go index 32f313949953..28b47f36f5d4 100644 --- a/processor/starlarktransform/internal/logs/processor.go +++ b/processor/starlarktransform/internal/logs/processor.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" + "github.com/qri-io/starlib/re" "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/consumer" "go.opentelemetry.io/collector/pdata/plog" @@ -40,11 +41,20 @@ type Processor struct { func (p *Processor) Start(context.Context, component.Host) error { - global := starlark.StringDict{ + modules := starlark.StringDict{ "json": jsonlib.Module, } - globals, err := starlark.ExecFile(p.thread, "", p.code, global) + regexMod, err := re.LoadModule() + if err != nil { + return err + } + + for k, v := range regexMod { + modules[k] = v + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, modules) if err != nil { return err } diff --git a/processor/starlarktransform/internal/logs/processor_test.go b/processor/starlarktransform/internal/logs/processor_test.go index 4a622cda79a7..d7d5ecfeae46 100644 --- a/processor/starlarktransform/internal/logs/processor_test.go +++ b/processor/starlarktransform/internal/logs/processor_test.go @@ -87,6 +87,32 @@ func TestLogConsumer(t *testing.T) { expectError: nil, expectStartError: errors.New("starlark: no 'transform' function defined in script"), }, + { + name: "regex transform", + event: testlogevent, + code: heredoc.Doc(` + def transform(event): + event = json.decode(event) + val = event['resourceLogs'][0]['scopeLogs'][0]['logRecords'][0]['body']['stringValue'] + val = re.sub('{.*}', 'NONE', val) + start, end = re.search('[A-Z]{4}', val) + + # update log body + event['resourceLogs'][0]['scopeLogs'][0]['logRecords'][0]['body']['stringValue'] = val + + # add severity + event['resourceLogs'][0]['scopeLogs'][0]['logRecords'][0]['attributes'].append({ + "key": "severity", + "value": {"stringValue": val[start:end]} + }) + return event`), + + next: &fakeLogConsumer{ + t: t, + expected: `{"resourceLogs":[{"resource":{"attributes":[{"key":"log.file.name","value":{"stringValue":"test.log"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"observedTimeUnixNano":"1694127596456358000","body":{"stringValue":"2023-09-06T01:09:24.045+0200 INFO internal/command.go:117 OpenTelemetry Collector Builder NONE"},"attributes":[{"key":"app","value":{"stringValue":"dev"}},{"key":"severity","value":{"stringValue":"INFO"}}],"traceId":"","spanId":""}]}]}]}`}, + expectError: nil, + expectStartError: nil, + }, } for _, tt := range testcases { diff --git a/processor/starlarktransform/internal/metrics/processor.go b/processor/starlarktransform/internal/metrics/processor.go index b933dcdec923..313a1a6cec6e 100644 --- a/processor/starlarktransform/internal/metrics/processor.go +++ b/processor/starlarktransform/internal/metrics/processor.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" + "github.com/qri-io/starlib/re" "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/consumer" "go.opentelemetry.io/collector/pdata/pmetric" @@ -40,11 +41,20 @@ type Processor struct { func (p *Processor) Start(context.Context, component.Host) error { - global := starlark.StringDict{ + modules := starlark.StringDict{ "json": jsonlib.Module, } - globals, err := starlark.ExecFile(p.thread, "", p.code, global) + regexMod, err := re.LoadModule() + if err != nil { + return err + } + + for k, v := range regexMod { + modules[k] = v + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, modules) if err != nil { return err } diff --git a/processor/starlarktransform/internal/traces/processor.go b/processor/starlarktransform/internal/traces/processor.go index 988931b55d4d..698e92a7df96 100644 --- a/processor/starlarktransform/internal/traces/processor.go +++ b/processor/starlarktransform/internal/traces/processor.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" + "github.com/qri-io/starlib/re" "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/consumer" "go.opentelemetry.io/collector/pdata/ptrace" @@ -40,11 +41,20 @@ type Processor struct { func (p *Processor) Start(context.Context, component.Host) error { - global := starlark.StringDict{ + modules := starlark.StringDict{ "json": jsonlib.Module, } - globals, err := starlark.ExecFile(p.thread, "", p.code, global) + regexMod, err := re.LoadModule() + if err != nil { + return err + } + + for k, v := range regexMod { + modules[k] = v + } + + globals, err := starlark.ExecFile(p.thread, "", p.code, modules) if err != nil { return err } From 8575f6e36ff9c344207724425f0591114d5fdb51 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 22:51:33 +0100 Subject: [PATCH 12/16] added GetCode function and validation to config added scipr parameter --- processor/starlarktransform/config.go | 64 +++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/processor/starlarktransform/config.go b/processor/starlarktransform/config.go index 07915fff8a75..79badf588a8e 100644 --- a/processor/starlarktransform/config.go +++ b/processor/starlarktransform/config.go @@ -1,5 +1,69 @@ package starlarktransform +import ( + "errors" + "fmt" + "io" + "net/http" + "os" + "strings" +) + type Config struct { + + // in-line code Code string `mapstructure:"code"` + + // source of startlark script + // accepts both file path and url + Script string `mapstructure:"script"` +} + +func (c *Config) validate() error { + if c.Code == "" && c.Script == "" { + return errors.New("a value must be given for altest one, [code] or [script]") + } + + return nil +} + +func (c *Config) GetCode() (string, error) { + + if c.Code != "" { + return c.Code, nil + } + + var code string + switch { + case strings.HasPrefix(c.Script, "http"): + resp, err := http.Get(c.Script) + if err != nil { + return "", fmt.Errorf("failed to fetch script from %s -%w", c.Script, err) + } + + defer resp.Body.Close() + b, err := io.ReadAll(resp.Body) + if err != nil { + return code, fmt.Errorf("failed to read http response body after fetching script from: %w", err) + } + + code = string(b) + + default: + f, err := os.Open(c.Script) + if err != nil { + return code, fmt.Errorf("failed to read script from path %s - %w", c.Script, err) + } + + defer f.Close() + + b, err := io.ReadAll(f) + if err != nil { + return code, fmt.Errorf("failed while reading script file: %w", err) + } + + code = string(b) + } + + return code, nil } From 260b4c11338d74e48159ec2206729ec03a97c995 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 22:51:48 +0100 Subject: [PATCH 13/16] added config tests --- processor/starlarktransform/config_test.go | 66 ++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 processor/starlarktransform/config_test.go diff --git a/processor/starlarktransform/config_test.go b/processor/starlarktransform/config_test.go new file mode 100644 index 000000000000..067c0d2282e2 --- /dev/null +++ b/processor/starlarktransform/config_test.go @@ -0,0 +1,66 @@ +package starlarktransform + +import ( + "errors" + "fmt" + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestConfig(t *testing.T) { + testscript := "def transform(event):\n json.decode(event)\n return event" + mockHTTPServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + fmt.Fprint(w, testscript) + })) + + testcases := []struct { + name string + config Config + expected string + expectedErr error + validationErr error + }{ + { + name: "with file path", + config: Config{ + Script: "./testdata/test.star", + }, + expected: testscript, + }, + { + name: "with http url", + config: Config{ + Script: mockHTTPServer.URL, + }, + expected: testscript, + }, + { + name: "empty script and code", + config: Config{}, + validationErr: errors.New("a value must be given for altest one, [code] or [script]"), + }, + } + + for _, tc := range testcases { + t.Run(tc.name, func(t *testing.T) { + err := tc.config.validate() + require.Equal(t, tc.validationErr, err) + if err != nil { + return + } + + code, err := tc.config.GetCode() + require.Equal(t, tc.expectedErr, err) + if err != nil { + return + } + + assert.Equal(t, tc.expected, code) + }) + } + +} From 91d1f73e1a6171907c975aeb92b7cbc110ef8325 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 22:53:39 +0100 Subject: [PATCH 14/16] updated factory to use GetCode function --- processor/starlarktransform/factory.go | 33 +++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/processor/starlarktransform/factory.go b/processor/starlarktransform/factory.go index 13e5d508518b..67507e5a3128 100644 --- a/processor/starlarktransform/factory.go +++ b/processor/starlarktransform/factory.go @@ -42,7 +42,16 @@ func createLogsProcessor( return nil, errors.New("invalid config type") } - return logs.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil + if err := config.validate(); err != nil { + return nil, err + } + + code, err := config.GetCode() + if err != nil { + return nil, err + } + + return logs.NewProcessor(ctx, set.Logger, code, nextConsumer), nil } func createMetricsProcessor( @@ -56,7 +65,16 @@ func createMetricsProcessor( return nil, errors.New("invalid config type") } - return metrics.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil + if err := config.validate(); err != nil { + return nil, err + } + + code, err := config.GetCode() + if err != nil { + return nil, err + } + + return metrics.NewProcessor(ctx, set.Logger, code, nextConsumer), nil } func createTracesProcessor( @@ -70,5 +88,14 @@ func createTracesProcessor( return nil, errors.New("invalid config type") } - return traces.NewProcessor(ctx, set.Logger, config.Code, nextConsumer), nil + if err := config.validate(); err != nil { + return nil, err + } + + code, err := config.GetCode() + if err != nil { + return nil, err + } + + return traces.NewProcessor(ctx, set.Logger, code, nextConsumer), nil } From 4deb6b3a68defb98790d2de33e2da39875ff6e5d Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 22:53:56 +0100 Subject: [PATCH 15/16] added test.star script --- processor/starlarktransform/testdata/test.star | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 processor/starlarktransform/testdata/test.star diff --git a/processor/starlarktransform/testdata/test.star b/processor/starlarktransform/testdata/test.star new file mode 100644 index 000000000000..48b66fdfa0c7 --- /dev/null +++ b/processor/starlarktransform/testdata/test.star @@ -0,0 +1,3 @@ +def transform(event): + json.decode(event) + return event \ No newline at end of file From 23ec632021d8cc7a4df7c8c3ae3a3f706d301872 Mon Sep 17 00:00:00 2001 From: Shaun Remekie Date: Sat, 7 Oct 2023 23:07:42 +0100 Subject: [PATCH 16/16] updated readme --- processor/starlarktransform/README.md | 33 +++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/processor/starlarktransform/README.md b/processor/starlarktransform/README.md index c5c18322e180..bfc87182a8bd 100644 --- a/processor/starlarktransform/README.md +++ b/processor/starlarktransform/README.md @@ -17,9 +17,21 @@ Starlark is a scripting language used for configuration that is designed to be s The processor leverages Starlark to modify telemetry data while using familiar, pythonic syntax. Modifying telemetry data is as a simple as modifying a `Dict`. +## Why? + +While there are a number of transform processors, most notably, the main OTTL [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor), this processor aims to grant users more flexibility by allowing them to manipulate telemetry data using a familiar syntax. + +Python is a popular, well known language, even among non-developers. By allowing Starlark code to be used as an option to transform telemetry data, users can leverage their existing knowledge of Python. + + ## Config -To configure starlarktransform, you add your code using the `code` option in the config. Each instance of starlarktransform can only have a single code section. +| Parameter | Desc | +| ------- | ---- | +| code | Add in-line starlark code directly to your config | +| script | Allows you to set the script source. Supports __File path__ or __HTTP URL__ | + +To configure starlarktransform, you can add your code using the `code` option in the config. ```yaml @@ -32,13 +44,24 @@ processors: return event ``` -You must define a function called `transform` that accepts a single argument, `event`. This function is called by the processor and is passed the telemetry event. The function **must return** the modified, json decoded event. -## Why? +Alternatively, in cases where you prefer to not have starlark code present or visible in your Open Telemetry configuration, you can use the `script` parameter to pass your script from a file or http url. -While there are a number of transform processors, most notably, the main OTTL [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor), this processor aims to grant users more flexibility by allowing them to manipulate telemetry data using a familiar syntax. +```yaml +processors: + starlarktransform: + script: /path/to/script.star -Python is a popular, well known language, even among non-developers. By allowing Starlark code to be used as an option to transform telemetry data, users can leverage their existing knowledge of Python. +# or +processors: + starlarktransform: + script: https://some.url.com/script.star + +``` + + + +You must define a function called `transform` that accepts a single argument, `event`. This function is called by the processor and is passed the telemetry event. The function **must return** the modified, json decoded event. ## How it works