Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support single_dep and multi_dep functionality #227

Closed
ahal opened this issue Apr 17, 2023 · 2 comments
Closed

Support single_dep and multi_dep functionality #227

ahal opened this issue Apr 17, 2023 · 2 comments
Assignees

Comments

@ahal
Copy link
Collaborator

ahal commented Apr 17, 2023

The multi_dep and single_dep loaders (and related transforms) are some of the most widely copied Taskgraph logic from project to project. We've mentioned upstreaming this into Taskgraph countless times! However, over the years I've come to realize that this loader is unnecessary and overly complicated (most recently while working on mozilla-releng/mozilla-taskgraph#7 in conjunction with the linked firefox-android PR).

Essentially it's doing too much at once, building upstream artifacts, copying attributes, resolving keys. The root issue is that it adds a lot of complexity right up front, which every later transform then needs to deal with. Instead we should add complexity little by little, and only as needed.

I'm still formulating thoughts here, but I think my rough plan is to:

  1. Add transforms for adding dependencies given a set of kinds.
  2. Add a utility file for deriving upstream-artifacts from a task's dependencies.
  3. Add transforms for copying attributes from a primary-dep.
  4. Push more logic into the scriptworker payload builders.

I believe with the above pieces, we'll be able to completely obsolete the single_dep and multi_dep loaders and replace it with a much simpler and easy to follow setup.

@ahal ahal self-assigned this Apr 17, 2023
@bhearsum
Copy link
Contributor

Maybe you omitted it because it's fairly trivial to deal with in transforms already, and we may not need or want a generic method, but just for the sake of completeness I'll note that the other major thing that multi_dep does is handles chunking and fanning out tasks (eg: by locale, partner repack, et. al). The new split_by_locale transform in Gecko is an example of how to do this in transforms.

@ahal
Copy link
Collaborator Author

ahal commented Apr 18, 2023

Ah yes, good call out!

ahal added a commit to ahal/taskgraph that referenced this issue May 2, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 2, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 25, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 25, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 25, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 25, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit to ahal/taskgraph that referenced this issue May 25, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227
ahal added a commit that referenced this issue May 26, 2023
These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: #227
@ahal ahal closed this as completed Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants