WIP: Prototype to provide user-defined upsteam-to-downstream identity transformations/filters via Starlark or CEL functions

[cfryanr]

Problem Statement

During login, when an identity is established from an upstream OIDC or LDAP provider, then a Pinniped admin might like to perform arbitrary transformations like adding a prefix to usernames or group names, removing unwanted groups, down-casing usernames or group names, rejecting authentication based on group memberships, etc. This is logged as issue #558.

Hopefully this feature can pave the way towards supporting multiple simultaneous upstream identity providers for a single FederationDomain in the Supervisor. Once you are able to do things like prefix usernames and group names, then you can ensure that downstream usernames and group names from different simultaneous IDPs are unique and won't clash.

To support this feature, we need a way for users to craft their own arbitrary transformations in some kind of language.

Update February 2023 - Using CEL

In the 1.5 years since this prototype PR was first created, the Kubernetes project has adopted the CEL language as its preferred embedded language. The most recent release of Kubernetes contains at least two different usages of CEL, and there are more proposed as KEPs for the future. Also, the CEL language has progressed, and there are now extensions available to offer more functionality for string manipulation.

CEL offers many of the same benefits of Starlark in terms of being performant, thread-safe, sandboxed, international (supports UTF-8), simple, cancellable, and extendable.

CEL differs from Starlark in several ways. In CEL, the user cannot define functions, and cannot import libraries, although the system in which it is embedded can extend the CEL language using Go (similar to Starlark). CEL expressions can only have one return value, which impacts the design of the API, e.g. the username transformations would need to be separate CEL expressions from the groups transformations.

This PR has now been updated to also include an implementation of identity transformations using CEL, alongside the original Starlark proof of concept. The implementation using CEL is in the internal/idtransform and internal/celformer directories in this PR.

The intention is to choose either Starlark or CEL (not both), whichever is the best fit for our needs. Kubernetes broadly adopting CEL seems like a good argument to lean towards using CEL, assuming all other considerations were determined to be roughly equal. Kubernetes admins will need to learn CEL anyways as more CEL-based features are added to Kubernetes.

The rest of the description of this PR below is from 1.5 year ago when Starlark was investigated as a possible solution.

Old Starlark Investigation Notes

Using Starlark as Part of the Solution

Why consider using Starlark for custom user-defined upsteam-to-downstream identity transformations/filters?

• Starlark is implemented in Go by Google and they use it internally extensively for their build system, so we can expect it to be well supported. Google is also very conservative about allowing any language definition changes, so the language is stable.
• The language is very well documented. See https://github.com/google/starlark-go/blob/master/doc/spec.md
• Python-like syntax is understandable to any programmer, even to someone who has never used Python
• Scripts can define functions, use “if” statements and “for” loops, use built-in types like string, tuple, dict, list, set, etc.
• Starlark is simple and runs deterministically: no classes, no threads, no random numbers, no clock (can optionally expose the time lib), stable iteration order, etc.
• Designed to be hard to write scripts which do not terminate (e.g. no while loops)
• Effectively sandboxed, even though it runs in the same address space:
  • Scripts cannot do any IO
  • Can limit the number of abstract computation “steps” that are allowed (limiting CPU usage, roughly limiting time)
  • Can also cancel using a context (e.g. apply a timeout)
  • Cannot limit memory usage, unfortunately, but that seems okay for our use case where only administrators would write scripts
• Starts interpreter and runs functions in several microseconds, so performance would be excellent
• Safe to run many interpreters at the same time on the same loaded module/function (thread-safety is guaranteed)
• Functions can be called from golang and passed arguments, including passing complex types (with attributes defined in Go) as arguments, and can get back return values
• Libraries can be pre-defined and offered for use in the user-defined Starlark source, for example we could define common helper functions if we wanted
• Can run scripts using a golang program, e.g. a CLI, so we could easily provide a simple test framework for script authors to be able to TDD their scripts if we wanted
• Supports UTF-8 strings, with lots of built-in string manipulation and comparison functions/operators
• The Starlark Print() method can be attached to our logging system to allow scripts to debug log
• Neither the definition of these scripts nor the parameters passed to these scripts would come from end users, so that would make it even less likely that any mistake in the script implementation could be exploited in any way

Examples

Here is an example Starlark script to add a prefix to every upstream username and group name, and also downcase every upstream username and group name:

# During a login, given the upstream username and list of group names, return the downstream names.
def transform(username, groups):
    prefixedGroups = []
    for g in groups:
        prefixedGroups.append("group_prefix:" + g.lower())
    return "username_prefix:" + username.lower(), prefixedGroups

Here is an example Starlark script to reject logins from any user who does not belong to the upstream group "admins".

def transform(username, groups):
    if "admins" in set(groups):
        return username, groups
    else:
        fail("user", username, "does not belong to group admins")

Some possible example use cases for upstream-to-downstream identity transformations include:

• Disallow authn for any user who does not belong in a list of usernames (e.g. when the upstream IDP is GitHub)
• Disallow authn for any user who does not (or does) belong to a certain group or one of a list of groups
• Prefix or otherwise make unique the username or group names
• Downcase or otherwise normalize usernames and group names
• Even though a user belongs to lots of upstream groups, remove all expect for the interesting ones for downstream membership (also helps avoid client certs getting too large from having too many groups)

Some more “creative” (maybe less likely to be needed) use cases include:

• Change upstream group names into downstream names (e.g. change “my-admins” to “system:masters”)
• Change certain upstream usernames into downstream usernames (e.g. change “ryan” into “admin”)
• Look at other upstream ID token claims or LDAP record attributes as part of the business logic of doing these transformations (e.g. if the user is located in a California branch office of our company then add an artificial downstream group “ca-office”)
• Matt suggested that we might want to deprecate or make optional our existing settings for mapping upstream fields into downstream fields and just ask users to use the generic transformation capability instead

The Rest of the Solution

It would be easy to allow users to define such scripts as a string on a CRD, in a Secret, or wherever we like. For example, the FederationDomain CRD could have a field to define a custom transformation Starlark script which would be applied to each login via an IDP. These details are not yet designed.

If desired, we could allow a user-defined list of transform functions to be applied in order, since the input and output types are the similar enough that they are easy to chain together. This might be fancier than needed.

Broadly speaking, our Kube controllers would parse/load the user-defined Starlark scripts into memory using starformer.New(). Then our authentication code would execute the user-defined Starlark hooks during each login, which are thread-safe and performant, using starformer.Transform(). The starformer.Transformer instances, which are returned by starformer.New(), would probably live in our existing in-memory caches that are loaded by our controllers and used by our authentication code.

Limitations

Note that Starlark does not allow scripts to perform IO by default. This is a helpful feature. However, it also means that base Starlark will not allow configurations where an administrator wants to make HTTP calls to an external authorization engine to perform authentication policy checks. However, the Golang implementation of Starlark does allow language extensions to be authored in Golang, so it would be theoretically possible for us to create such an extension if desired.

Alternatives Considered

• Rego, which is based on Prolog. Unfortunately, the code is unreadable for an average end-user. See this example of Rego code from the OPA documentation: https://www.openpolicyagent.org/docs/latest/#putting-it-together
• CEL, https://github.com/google/cel-spec. CEL only allows defining one expression with one "return" value. It is a simple language built for fast expression evaluation for policy, with expressions usually evaluating to a boolean value. It lacks the features of a general purpose programming language, e.g. it does not have "if" statements (but does have a ternary operator), local variables, assignment operators, or the ability to add new functions written in the CEL language. CEL also has limited ability to work with strings compared to Starlark, and transformations are mainly operating on strings. CEL string operations are: size, concatenation, comparison, startsWIth, endsWith, and matches regexp. For these reasons, it would be harder for a user to write their own arbitrary transformation business logic in CEL.

Release Notes

Release note:

TBD

[enj]

@neolit123 mentioned that folks upstream had a hard time reading starlark when used with bazel.

CEL might be something to look into.

[codecov]

Codecov Report

Merging #694 (aa57a51) into main (60d12d8) will increase coverage by 0.11%.
The diff coverage is 94.28%.

❗ Current head aa57a51 differs from pull request most recent head 516abf6. Consider uploading reports for the commit 516abf6 to get more accurate results

@@            Coverage Diff             @@
##             main     #694      +/-   ##
==========================================
+ Coverage   75.39%   75.51%   +0.11%     
==========================================
  Files         165      166       +1     
  Lines       14905    15010     +105     
==========================================
+ Hits        11238    11335      +97     
- Misses       3377     3383       +6     
- Partials      290      292       +2     

Impacted Files	Coverage Δ
internal/starformer/starformer.go	94.28% <94.28%> (ø)
...l/localuserauthenticator/localuserauthenticator.go	52.17% <0.00%> (-0.73%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[cfryanr]

Closing because this implementation of CEL identity policies and transformations is included in PR #1419.