Enable e2e tests for real TEEs

[mkulke]

We would like to enable e2e tests on the repo for real TEE's. At moment we only use the dummy TEE to test the integration of attester + verifier components in general. We'd like to extend the coverage to the integration with real TEEs.

To do this we need to run the workflows on self-hosted runners or external CI systems. For Azure-based TEE's (az-{snp,tdx}-vtpm) we can use GARM, which we use in other kata + coco circumstances already. To make this work we need to provide a token scoped to the repository to CoCo's Garm instance and register a webhook secret on the kbs repo.

[Xynnn007]

To achieve this goal, two stages in my mind

1. generate the evidence inside the guest. attester: add evidence_getter binary guest-components#418 would help to provide such a tool
2. deliver the evidence to CoCo-AS. Added e2e test for CoCo-AS using SNP evidence #264 can be re-used.

[mkulke]

To achieve this goal, two stages in my mind

1. generate the evidence inside the guest. attester: add evidence_getter binary guest-components#418 would help to provide such a tool
2. deliver the evidence to CoCo-AS. Added e2e test for CoCo-AS using SNP evidence #264 can be re-used.

If we use a pre-generated fixture as evidence, wouldn't this just be a unit-test for a validator?

What I had in mind was simply running https://github.com/confidential-containers/kbs/blob/main/.github/workflows/kbs-e2e.yaml on a TEE-capable runner. If the test runs e.g. on a TDX worker it should work ootb (AA_SAMPLE_ATTESTER_TEST needs to be unset).

[Xynnn007]

If we use a pre-generated fixture as evidence, wouldn't this just be a unit-test for a validator?

Well, I misunderstood the idea. I thought there are two different e2e test

1. for CoCo-AS. including evidence generating on real TEE, API calling and get the attestation token
2. for KBS. including performing RCAR on real TEE, and get the resource

My last comment is for 1. Anyway, we both think that the action of obtaining evidence needs to be dynamically executed on the real TEE.

[mkulke]

Ah, I see. I was looking at #264. I saw fixtures in the tests/e2e folder and assumed this was the idea. But if we want to run both AS and KBS e2e tests on real TEEs then this issue is the right one, it's more about the repo configuration and the workers we need.

To be able to register a self hosted runner (ephemeral or a bare-metal box) we need the following token:

For ephemeral runners, we also need to configure a webhook + secret which listens on Workflow jobs

[Xynnn007]

Agreed. For the runner setting, I think @fitzthum would help.

For the self-host runner, we now have a CI machine in SGX (now working for enclave-cc and guest-components )but not configured in kbs repo. We might use that one?

[mkulke]

yes, it can be shared. for non-ephemeral runners (like bare-metal machines) you have to pay attention to not pollute the node in your workflows, since there is no isolation and a side-effect might break another workflow

[mkulke]

Finally it needs to be restricted who can run those jobs, you don't want people executing arbitrary code on your runner, because they opened a malicious PR.

Triggering an e2e test on real TEE on push: main might be a good compromise. In addition we can have a label "test-e2e" that people with privileges can set on PRs. This label triggers a PR with pull_request_target + additional GH logic to actually run the code from the PR.

@wainersm can you confirm? 👆

[fitzthum]

This all sounds good to me. If you need any help adding a machine to our runner pool or enabling it for this repo or whatever lmk.

There are also these security configs we can set

[mkulke]

This all sounds good to me. If you need any help adding a machine to our runner pool or enabling it for this repo or whatever lmk.

great, thx. I'll reach out.

[Xynnn007]

Is this error related？ https://github.com/confidential-containers/kbs/actions/runs/7523718849

[mkulke]

Is this error related？ https://github.com/confidential-containers/kbs/actions/runs/7523718849

yes, it is. There is a problem with referencing the reusable workflow.

[mkulke]

PTAL #291