feat: create transforms for adding dependencies

These transforms will implement part of the work that the multi_dep
loader is doing in other projects.

Issue: taskcluster#227

docs/reference/index.rst

@@ -6,6 +6,7 @@ Reference
 
    cli
    parameters
+   transforms/index
    optimization-strategies
    source/modules
    migrations

docs/reference/source/taskgraph.transforms.rst

@@ -20,6 +20,14 @@ taskgraph.transforms.base module
    :undoc-members:
    :show-inheritance:
 
+taskgraph.transforms.after\_deps module
+-----------------------------------------
+
+.. automodule:: taskgraph.transforms.after_deps
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 taskgraph.transforms.cached\_tasks module
 -----------------------------------------
 

docs/reference/transforms/after_deps.rst

@@ -0,0 +1,148 @@
+.. _after_deps:
+
+After Dependencies
+==================
+
+The :mod:`taskgraph.transforms.after_deps` transforms can be used to create
+tasks based on the kind dependencies, filtering on common attributes like the
+``build-type``.
+
+These transforms are useful when you want to create follow-up tasks for some
+indeterminate subset of existing tasks. For example, maybe you want to run
+a signing task after each build task.
+
+
+Usage
+-----
+
+Add the transform to the ``transforms`` key in your ``kind.yml`` file:
+
+.. code-block:: yaml
+
+   transforms:
+     - taskgraph.transforms.add_deps
+     # ...
+
+Then create an ``after-deps`` section in your task definition, e.g:
+
+.. code-block:: yaml
+
+   kind-dependencies:
+     - build
+     - toolchain
+
+   tasks:
+     signing:
+       after-deps:
+         kinds: [build]
+
+This example will split the ``signing`` task into many, one for each build. If
+``kinds`` is unspecified, then it defaults to all kinds listed in the
+``kind-dependencies`` key. So the following is valid:
+
+.. code-block:: yaml
+
+   kind-dependencies:
+     - build
+     - toolchain
+
+   tasks:
+     signing:
+       after-deps: {}
+
+In this example, a task will be created for each build *and* each toolchain.
+
+Limiting Dependencies by Attribute
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It may not be desirable to create a new task for *all* tasks in the given kind.
+It's possible to limit the tasks given some attribute:
+
+.. code-block:: yaml
+
+   kind-dependencies:
+     - build
+
+   tasks:
+     signing:
+       after-deps:
+         with-attributes:
+           platform: linux
+
+In the above example, follow-up tasks will only be created for builds whose
+``platform`` attribute equals "linux". Multiple attributes can be specified,
+these are resolved using the :func:`~taskgraph.util.attributes.attrmatch`
+utility function.
+
+Grouping Dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes, it may be desirable to run a task after a group of related tasks
+rather than an individual task. One example of this, is a task to notify when a
+release is finished for each platform we're shipping. We want it to depend on
+every task from that particular release phase.
+
+To accomplish this, we specify the ``group-by`` key:
+
+.. code-block:: yaml
+
+   kind-dependencies:
+     - build
+     - signing
+     - publish
+
+   tasks:
+     notify:
+       after-deps:
+         group-by:
+           attribute: platform
+
+In this example, tasks across the ``build``, ``signing`` and ``publish`` kinds will
+be scanned for an attribute called "platform" and sorted into corresponding groups.
+Assuming we're shipping on Windows, Mac and Linux, it might create the following
+groups:
+
+.. code-block::
+
+   - build-windows, signing-windows, publish-windows
+   - build-mac, signing-mac, publish-mac
+   - build-linux, signing-linux, publish-linux
+
+Then the ``notify`` task will be duplicated into three, one for each group. The
+notify tasks will depend on each task in its associated group.
+
+Custom Grouping
+~~~~~~~~~~~~~~~
+
+Only the default ``single`` and the ``attribute`` group-by functions are
+built-in. But if more complex grouping is needed, custom functions can be
+implemented as well:
+
+.. code-block:: python
+
+   from typing import List
+
+   from taskgraph.task import Task
+   from taskgraph.transforms.after_deps import group_by
+   from taskgraph.transforms.base import TransformConfig
+
+   @group_by("custom-name")
+   def group_by(config: TransformConfig, tasks: List[Task]) -> List[List[Task]]:
+      pass
+
+It's also possible to specify a schema for your custom group-by function, which
+allows tasks to pass down additional context (such as with the built-in
+``attribute`` function):
+
+.. code-block:: python
+
+   from typing import List
+
+   from taskgraph.task import Task
+   from taskgraph.transforms.after_deps import group_by
+   from taskgraph.transforms.base import TransformConfig
+   from taskgraph.util.schema import Schema
+
+   @group_by("custom-name", schema=Schema(str))
+   def group_by(config: TransformConfig, tasks: List[Task], ctx: str) -> List[List[Task]]:
+      pass

docs/reference/transforms/index.rst

@@ -0,0 +1,11 @@
+Transforms
+==========
+
+Taskgraph includes several transforms out of the box. These can be used to
+accomplish common tasks such as adding dependencies programmatically or setting
+up notifications. This section contains a reference of the transforms Taskgraph
+provides, read below to learn how to use them.
+
+.. toctree::
+
+   add_deps

src/taskgraph/transforms/after_deps.py

@@ -0,0 +1,160 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""
+Transforms used to create tasks based on the kind dependencies, filtering on
+common attributes like the ``build-type``.
+
+These transforms are useful when follow-up tasks are needed for some
+indeterminate subset of existing tasks. For example, running a signing task
+after each build task, whatever builds may exist.
+"""
+from copy import deepcopy
+from textwrap import dedent
+
+from voluptuous import ALLOW_EXTRA, Any, Optional, Required
+
+from taskgraph.transforms.base import TransformSequence
+from taskgraph.util.attributes import attrmatch
+from taskgraph.util.schema import Schema
+
+# Define a collection of group_by functions
+GROUP_BY_MAP = {}
+
+
+def group_by(name, schema=None):
+    def wrapper(func):
+        GROUP_BY_MAP[name] = func
+        func.schema = schema
+        return func
+
+    return wrapper
+
+
+@group_by("single")
+def group_by_single(config, tasks):
+    for task in tasks:
+        yield [task]
+
+
+@group_by("attribute", schema=Schema(str))
+def group_by_attribute(config, tasks, attr):
+    groups = {}
+    for task in tasks:
+        val = task.attributes.get(attr)
+        if not val:
+            continue
+        groups.setdefault(val, []).append(task)
+
+    return groups.values()
+
+
+AFTER_DEPS_SCHEMA = Schema(
+    {
+        Required("after-deps"): {
+            Optional(
+                "kinds",
+                description=dedent(
+                    """
+                Limit dependencies to specified kinds (defaults to all kinds in
+                `kind-dependencies`).
+                """.lstrip()
+                ),
+            ): list,
+            Optional(
+                "with-attributes",
+                description=dedent(
+                    """
+                Limit dependencies to tasks whose attributes match
+                using :func:`~taskgraph.util.attributes.attrmatch`.
+                """.lstrip()
+                ),
+            ): {str: Any(list, str)},
+            Optional(
+                "group-by",
+                description=dedent(
+                    """
+                Group cross-kind dependencies using the given group-by
+                function. One task will be created for each group. If not
+                specified, the 'single' function will be used which creates
+                a new task for each individual dependency.
+                """.lstrip()
+                ),
+            ): Any(None, *GROUP_BY_MAP.keys()),
+        }
+    },
+    extra=ALLOW_EXTRA,
+)
+"""Schema for after_deps transforms."""
+
+transforms = TransformSequence()
+transforms.add_validate(AFTER_DEPS_SCHEMA)
+
+
+@transforms.add
+def after_deps(config, tasks):
+    for task in tasks:
+        # Setup and error handling.
+        after_deps = task.pop("after-deps")
+        kind_deps = set(config.config.get("kind-dependencies", []))
+        kinds = set(after_deps.get("kinds", kind_deps))
+
+        if kinds - kind_deps:
+            invalid = "\n".join(sorted(kinds - kind_deps))
+            raise Exception(
+                dedent(
+                    f"""
+                    The `after-deps.kinds` key contains the following kinds
+                    that are not defined in `kind-dependencies`:
+                    {invalid}
+                """.lstrip()
+                )
+            )
+
+        if not kinds:
+            raise Exception(
+                dedent(
+                    """
+                The `after_deps` transforms require at least one kind defined
+                in `kind-dependencies`!
+                """.lstrip()
+                )
+            )
+
+        # Resolve desired dependencies.
+        with_attributes = after_deps.get("with-attributes")
+        deps = [
+            task
+            for task in config.kind_dependencies_tasks.values()
+            if task.kind in kinds
+            if not with_attributes or attrmatch(task.attributes, **with_attributes)
+        ]
+
+        # Resolve groups.
+        group_by = after_deps.get("group-by", "single")
+        groups = set()
+
+        if isinstance(group_by, dict):
+            assert len(group_by) == 1
+            group_by, arg = group_by.popitem()
+            func = GROUP_BY_MAP[group_by]
+            if func.schema:
+                func.schema(arg)
+            groups = func(config, deps, arg)
+        else:
+            func = GROUP_BY_MAP[group_by]
+            groups = func(config, deps)
+
+        # Split the task, one per group.
+        for group in groups:
+            # Verify there is only one task per kind in each group.
+            kinds = {t.kind for t in group}
+            if len(kinds) < len(group):
+                raise Exception(
+                    "The after_deps transforms only allow a single task per kind in a group!"
+                )
+
+            new_task = deepcopy(task)
+            new_task["dependencies"] = {dep.kind: dep.label for dep in group}
+            yield new_task

src/taskgraph/util/attributes.py

@@ -7,18 +7,30 @@
 
 
 def attrmatch(attributes, **kwargs):
-    """Determine whether the given set of task attributes matches.  The
-    conditions are given as keyword arguments, where each keyword names an
-    attribute.  The keyword value can be a literal, a set, or a callable.  A
-    literal must match the attribute exactly.  Given a set, the attribute value
-    must be in the set.  A callable is called with the attribute value.  If an
-    attribute is specified as a keyword argument but not present in the
-    attributes, the result is False."""
+    """Determine whether the given set of task attributes matches.
+
+    The conditions are given as keyword arguments, where each keyword names an
+    attribute. The keyword value can be a literal, a set, or a callable:
+
+        * A literal must match the attribute exactly.
+        * Given a set or list, the attribute value must be contained within it.
+        * A callable is called with the attribute value and returns a boolean.
+
+    If an attribute is specified as a keyword argument but not present in the
+    task's attributes, the result is False.
+
+    Args:
+        attributes (dict): The task's attributes object.
+        kwargs (dict): The conditions the task's attributes must satisfy in
+                       order to match.
+    Returns:
+        bool: Whether the task's attributes match the conditions or not.
+    """
     for kwkey, kwval in kwargs.items():
         if kwkey not in attributes:
             return False
         attval = attributes[kwkey]
-        if isinstance(kwval, set):
+        if isinstance(kwval, (set, list)):
             if attval not in kwval:
                 return False
         elif callable(kwval):