Dynamic log links #4774

eapolinario · 2024-01-24T20:43:42Z

Tracking issue

Why are the changes needed?

We need a way to specify dynamic log links from tasks as defined in the linked tracking issue. This PR implements this idea. Essentially operators are going to add a section to their config files for specific dynamic log links. For example, we have flyin as a plugin that would benefit from this feature.

What changes were proposed in this pull request?

We add a new subsection to the plugin section in the propeller config file that lets users specify dynamic plugins. The template uris for these dynamic plugins can be configured similarly to the usual plugins, the only difference that specific fields can be plucked from the task template to be used in the template uri. Take this example:

plugins:
  logs:
    dynamic-log-links-enabled: true
    dynamic-log-links:
      vscode: http://flyteinteractve.vs:{{ .taskConfig.port }}/{{ .podName }}

Notice how in the template uri for the vscode dynamic plugin we access the port (which is defined in a task template metadata) using the prefix .taskConfig, whereas the usual template variables are also available.

How was this patch tested?

Unit tests and tested locally in a sandbox.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

This reverts commit 3501675. Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>

codecov · 2024-01-24T20:52:13Z

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (73b1d40) 58.21% compared to head (d013b31) 58.56%.
Report is 16 commits behind head on master.

Files	Patch %	Lines
...ugins/go/tasks/pluginmachinery/tasklog/template.go	88.00%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4774      +/-   ##
==========================================
+ Coverage   58.21%   58.56%   +0.35%     
==========================================
  Files         626      624       -2     
  Lines       53855    53627     -228     
==========================================
+ Hits        31349    31404      +55     
+ Misses      19996    19705     -291     
- Partials     2510     2518       +8

Flag	Coverage Δ
unittests	`58.56% <94.23%> (+0.35%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

flyteplugins/go/tasks/logs/config.go

EngHabu · 2024-01-24T22:12:00Z

flyteplugins/go/tasks/pluginmachinery/tasklog/plugin.go

-	Scheme        TemplateScheme             `json:"scheme" pflag:",Templating scheme to use. Supported values are Pod and TaskExecution."`
+	DisplayName         string                     `json:"displayName" pflag:",Display name for the generated log when displayed in the console."`
+	TemplateURIs        []TemplateURI              `json:"templateUris" pflag:",URI Templates for generating task log links."`
+	DynamicTemplateURIs []DynamicTemplateURI       `json:"dynamicTemplateUris" pflag:",Dynamic URI Templates for generating task log links."`


Why do these need to be different?

A DynamicTemplateURI has a kind that we use to match with the values passed via task template. The use case is if we have multiple dynamic log links configured, but the task only enables a subset of them.

Can we just add a name field in the TemplateLogPlugin type?

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario · 2024-01-27T01:07:40Z

@EngHabu , as we talked about yesterday, this next batch of updates implements:

remove enabled flag
do not hardcode port
- In fact, we now let users match any value set in the task template config
Use vscode as the name of the log
Add name to TemplateLogPlugin to help distinguish which plugin is used
- Remove =DynamicTemplateURI=
Get rid of Schemes

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go

EngHabu · 2024-01-29T05:19:54Z

flyteplugins/go/tasks/logs/logging_utils.go

+			tasklog.TemplateLogPlugin{
+				Name:        logLinkType,
+				DisplayName: fmt.Sprintf("%s logs", logLinkType),
+				DynamicTemplateURIs: []tasklog.TemplateURI{


Is this field still needed? can't we use TemplateURIs...

This simplifies the distinction between task logs produced by dynamic log links vs regular log links.

For example, let's say we enable one dynamic log link in propeller. How do we make sure that dynamic log links only show up for tasks that have the right bits set in their task templates?

Signed-off-by: Eduardo Apolinario <[email protected]>

flyteplugins/go/tasks/logs/logging_utils.go

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go

Signed-off-by: Eduardo Apolinario <[email protected]>

EngHabu

Thank you ❤️ ❤️

EngHabu · 2024-01-29T21:52:35Z

flyteplugins/go/tasks/logs/logging_utils.go

+				Name:                logLinkType,
+				DisplayName:         dynamicLogLink.DisplayName,
+				DynamicTemplateURIs: dynamicLogLink.TemplateURIs,
+				MessageFormat:       core.TaskLog_JSON,


Does it not have a MessageFormat field?

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go

EngHabu · 2024-01-29T21:54:02Z

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go

+	for _, dynamicLogLinkType := range getDynamicLogLinkTypes(input.TaskTemplate) {
+		for _, dynamicTemplateURI := range p.DynamicTemplateURIs {
+			if p.Name == dynamicLogLinkType {
+				for _, match := range dynamicLogRegex.FindAllStringSubmatch(dynamicTemplateURI, -1) {


nit: Would be nice to support this for the "regular" logs as well...

It will be easy to rearrange this logic when we find a good use-case for regular logs.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Dynamic log links (#4774) * Bring Scheme back for backwards compatibility (#4789) * Bring Scheme back for backwards compatibility Signed-off-by: Eduardo Apolinario <[email protected]> * Rename deprecated field to `DeprecatedScheme` Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>

eapolinario added 7 commits December 29, 2023 10:00

Revert "Add Flyin propeller config (#4610)"

785599b

This reverts commit 3501675. Signed-off-by: Eduardo Apolinario <[email protected]>

wip

40bcac0

Signed-off-by: Eduardo Apolinario <[email protected]>

Add tests

930564e

Signed-off-by: Eduardo Apolinario <[email protected]>

Introduce Kind to DynamicTemplateURI and update tests

93f5845

Signed-off-by: Eduardo Apolinario <[email protected]>

Merge remote-tracking branch 'origin' into dynamic-log-links

33ff678

Signed-off-by: Eduardo Apolinario <[email protected]>

Use a map in the plugins config section

f168124

Signed-off-by: Eduardo Apolinario <[email protected]>

Lint

d66f1a0

Signed-off-by: Eduardo Apolinario <[email protected]>

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Jan 24, 2024

EngHabu reviewed Jan 24, 2024

View reviewed changes

eapolinario added 5 commits January 26, 2024 10:37

Remove boolean flag to enable dynamic log links

59dd968

Signed-off-by: Eduardo Apolinario <[email protected]>

Add name to TemplateLogPlugin and use that to identify dynamic log links

0a42ca1

Signed-off-by: Eduardo Apolinario <[email protected]>

Remove scheme switch

4b6dbc3

Signed-off-by: Eduardo Apolinario <[email protected]>

Remove scheme from log links

8049e31

Evaluate templatizable taskConfig values at runtime

6ca8fec

Signed-off-by: Eduardo Apolinario <[email protected]>

EngHabu reviewed Jan 29, 2024

View reviewed changes

eapolinario added 3 commits January 29, 2024 08:53

Use capture groups to create dynamic log regexes.

634c17c

Signed-off-by: Eduardo Apolinario <[email protected]>

Remove mentions to flyin

1058d03

Signed-off-by: Eduardo Apolinario <[email protected]>

Add unit test to demo the use of duplicated dynamic log links

ea9fdac

Signed-off-by: Eduardo Apolinario <[email protected]>

EngHabu reviewed Jan 29, 2024

View reviewed changes

flyteplugins/go/tasks/logs/logging_utils.go Outdated Show resolved Hide resolved

flyteplugins/go/tasks/pluginmachinery/tasklog/template.go Outdated Show resolved Hide resolved

eapolinario added 4 commits January 29, 2024 10:01

Simplify dynamic regex matching

b345864

Signed-off-by: Eduardo Apolinario <[email protected]>

One more dynamic test

4200022

Signed-off-by: Eduardo Apolinario <[email protected]>

Remove suffix from dynamic log links display name

5c3b813

Signed-off-by: Eduardo Apolinario <[email protected]>

Use a map of TemplateLogPlugin in log config

28aaa05

Signed-off-by: Eduardo Apolinario <[email protected]>

EngHabu previously approved these changes Jan 29, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jan 29, 2024

Rename taskConfig regex

d013b31

Signed-off-by: Eduardo Apolinario <[email protected]>

eapolinario dismissed EngHabu’s stale review via d013b31 January 29, 2024 22:16

EngHabu approved these changes Jan 29, 2024

View reviewed changes

eapolinario merged commit 460f0f1 into master Jan 29, 2024
45 checks passed

eapolinario deleted the dynamic-log-links branch January 29, 2024 22:55

eapolinario mentioned this pull request Jan 30, 2024

Bring Scheme back for backwards compatibility #4789

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic log links #4774

Dynamic log links #4774

eapolinario commented Jan 24, 2024 •

edited

Loading

codecov bot commented Jan 24, 2024 •

edited

Loading

EngHabu Jan 24, 2024

eapolinario Jan 24, 2024

EngHabu Jan 25, 2024

eapolinario commented Jan 27, 2024 •

edited

Loading

EngHabu Jan 29, 2024

eapolinario Jan 29, 2024

EngHabu left a comment

EngHabu Jan 29, 2024

EngHabu Jan 29, 2024

eapolinario Jan 29, 2024

Dynamic log links #4774

Dynamic log links #4774

Conversation

eapolinario commented Jan 24, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

codecov bot commented Jan 24, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eapolinario commented Jan 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EngHabu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eapolinario commented Jan 24, 2024 •

edited

Loading

codecov bot commented Jan 24, 2024 •

edited

Loading

eapolinario commented Jan 27, 2024 •

edited

Loading