IAM <-> Workspace integration for scoped Workspace-API access

[achtsnits]

Builds on top of #34 introducing user<->group(=team) associaction!

The Workspace-API is a lightweight layer built on top of Kubernetes that exposes workspace-related information, such as credentials, via HTTP routes.

There is a single central deployment of the Workspace-API within the cluster, which is accessed by operators, (potentially) platform users , and services like ProcessingBB or MLOpsBB. These services use the API to retrieve credentials, for example, to stage data.

The API routes follow the pattern <workspace-api>/workspaces/<teamname>/..., ensuring that only authenticated users associated with a specific team can access the corresponding workspace resources.

Sign-Off Checklist:

• The Workspace-API is successfully deployed.
• Endpoints are secured using APX along with Keycloak or OPA policies, ensuring that platform users, such as userA associated with teamB, can access resources via HTTP GET requests like <workspace-api>/workspaces/teamB

[achtsnits]

As discussed in meeting 3 Dec 2024, a similar approach needs to be applied to the STAC API Admin component (see EOEPCA/resource-discovery#101).

In this case, coarse-grained authorization decisions will need to handle different HTTP methods rather than distinct routes - but should still be doable based on Keycloak policies.

Other important topics for future discussion:

• Delegated access for services: Services such as ProcessingBB and MLOpsBB will require delegated access, making token expiration a critical consideration.
• Fine-grained authorization: Example - Optimizing STAC queries to return results only for granted collections requires further exploration. This likely needs to be implemented directly in the code (e.g., through middleware) rather than via a separate proxy. But many other Examples exist where logic can be handled out of process, e.g. via gateway or OPA

[w-scho]

For the workspace API, what are the general rules for access protection?
Is it just an all-or-nothing protection like for the Workspace UIs, allowing users who are allowed to access workspace foo to also access path /workspaces/foo/* of the API by any HTTP method? If not: What else do we need to consider?
On whose behalf will external services (Processing BB, MLOps BB) access the workspace API, on their own or on behalf of some "real" user?

[achtsnits]

correct, if a user is associated to group foo he should be allowed to any route /workspaces/foo/* but not to /workspaces/bar/* or /workspaces/*

the HTTP method specific protection we need (at the moment) for the eoAPI STAC API Transaction tool

i.e. this coarse authorization checks (route-based here and HTTP method based there) are IMO the goal for this quarter, more granular (regardless if either external gateway resp. OPA or in app code) we need to discuss and plan for next quarter

hope @rconway can help and step in here regarding delegation as I was also not directly involved in past - my understanding is that the services (=ProcessingBB) themselves could access anything on the workspace-api and just used the passed JWT to extract the user=group=team and called the workspaces/<user> then?
so IMO we are "free" to define another approach for delegation here now, we have to consider that the user (with his token) calls the ProcessingBB but the process then is async, so the user token may be expired at the point when then ProcessingBB finished and would call the workspace-api for bucket crentials to subsequently upload results

I know that openEO (another new part of ProcessingBB) uses Device Token flow together with Refresh Tokens to mitigate that but personally I'm not sure about storing a Refresh Token on a service ...

[w-scho]

We are currently considering using offline tokens in the context of the Resource Health BB, for which they seem to be an appropriate solution. They would also work well for the processing and similar scenarios, but I am not yet sure if they are really the most appropriate choice there.
Offline tokens are long-living tokens (default: 30d) that can exist without an active session, but allow a user (or a service acting on the user's behalf) to request a session (i.e., access token) at a later time. The advantage as compared to a refresh token is that it is not necessary to keep a session open for a long time. Furthermore, Keycloak gets involved at the time when a session is actually requested and is able to intervene and reject the session, e.g. if the offline token has been revoked. So it is definitely better than using long-living access or refresh tokens.

[rconway]

The case of the Processing BB is that a processing job is invoked by a user (passing a JWT). During the execution of the job the Processing BB needs to call other BBs (acting as the originating user) - for example the Workspace BB for storage, or Resource Registration to publish outputs. This needs to take account of the fact, mentioned by @fabricebrito, that processing can take a (very) long time.

@w-scho Do you think the offline token approach would help with this case?

[w-scho]

I have no real-world experience with OIDC offline tokens and am therefore still a bit unsure about the details. I am quite sure, though, that they are an appropriate solution for the Resource Health case where jobs are run repeatedly on behalf of a user. This case is a bit simpler than the processing case, because it does not involve delegated authentication. I'd like to first evaluate the use of offline tokens in the Resource Health scenario to get a bit more familiar with them and then see if they really fit into the Processing scenario or would just be a partially disarmed variant of refresh tokens there.
Normally offline tokens are issued directly during the authentication flow if explicitly requested. Therefore I am not sure if it is possible to turn an existing JWT into an offline token. Also ideally a token that is intended for delegated access should be restricted as much as possible, and I am not sure to which extent this is possible with offline tokens. At least offline tokens do not seem to be explicitly designed for delegated access, and we will need to do some investigation to find out if they are still approriate for the processing scenario.

[rconway]

So what is the typical solution for delegated access - ignoring the 'token lifetime' problem?

[w-scho]

We initially considered UMA 2 for this case. However, UMA 2 does also not fit perfectly, because it focuses mainly on party-to-party (usually in the sense of person-to-person) authorization from the resource owner's perspective, is quite complex and does not address the token lifetime problem. A typical use case for UMA 2 would be granting another person (or maybe an app) access to one's bank account. Granting the Processing BB access to some workspace or data collection is similar to this, but does not exactly match the purpose of UMA 2.
So we actually have two candidate solutions that are completely different and do not exactly match our use case, but still may be viable approaches.
A nearly ideal solution for delegated access in a technical sense would be the proxy ticket feature of the CAS protocol. Unfortunately there is no exact equivalent for this in OIDC, and even CAS proxy tickets do not really solve the token lifetime problem.

[w-scho]

For evaluation and demonstration, I created two additional ingresses/ routes for the Workspace API. Both routes use a newly created client named "workspace-api".

A Keycloak-based one at https://ws-api-test.apx.develop.eoepca.org/
This route protects workspaces/ws-test4 through a resource that is configured in Keycloak and that requires the user to have role "ws_access" on client "ws-test4". Everything else is currently unprotected and only requires authentication.
The main advantage of this solution is that it is based on explicit configuration in Keycloak and does not make any assumptions. The downside is that this configuration has to be created individually for each workspace to protect.

An OPA-based one at https://ws-api-opa.apx.develop.eoepca.org/
This route uses the opa plugin of APISIX to connect directly to OPA. For authorization, it relies solely on OPA, i.e., it uses Keycloak only for authentication. The OPA policy behind this route currently protects /workspaces/{ws_name} generically by requiring the JWT to be valid and to claim that the user has role "ws_access" on the client with name {ws_name}. Everything else is currently blocked.
The main advantage of this solution is that it is generic and does not require any additional configuration in Keycloak in addition to what needs to be done for the Workspace UI anyway. However, it implicitly assumes that the workspace name matches the name of the corresponding client in Keycloak.

[rconway]

@w-scho
If the OPA approach is more generic, then it seems like a better solution.
For understanding - the OPA approach requires a dedicated client for each workspace, for attaching the ws_access role. Didn't we talk about using a Group for each workspace?

[w-scho]

@rconway
There is both, a group and a client. The group is required for associating users to a workspace. The client is required for the Workspace UIs, because each Workspace UI has its own DNS address. In principle it would also be possible to use a single client for multiple DNS addresses, but this would require adding and removing URIs to/from that client when a workspace is added/removed. I think it is cleaner to create a client when a workspace is created and delete it when the workspace is deleted.
For the Workspace API protection this implies that we can simply rely on that client and its associated role.
In principle it would also be possible to check directly for a group in an OPA policy, but it is cleaner to check for a role, because a role actually represents a set of permissions, whereas a group only represents a collection of users in the first place.

[rconway]

So each workspace has an associated client - each client has an associated ws_access role - and users are added to this role to obtain access?

[achtsnits]

we are mixing up topics here:

regarding #37 here:
there is just one workspace-api deployment in namespace workspace (-> just one DNS) but the individual routes must be protected via the team association of the user -> this could be handled via OPA without additional keycloak setup IMO

regarding #34:
the workspace-ui is deployed per team in a separate team namespace (-> multiple DNS) - there during the provisioning, the workspace-pipeline is creating keycloak group, client, ws_access role and so on and authorization is protected via keycloak

[achtsnits]

path based routes ./workspaces/{workspace} in Workspace-API are protected now

setup of OPA policy by IAM team: policy https://github.com/EOEPCA/iam-policies/blob/main/policies/eoepca/workspace/wsapi.rego

authorization enabled via OPA plugin here: https://github.com/EOEPCA/eoepca-plus/blob/deploy-develop/argocd/eoepca/workspace/parts/ingress-workspace-api.yaml