How to skip downloading external dependencies

[osteffenrh]

I am working on using containers (docker) for the EDK2 CI on Linux.
The basic idea is to provide container images (see https://github.com/tianocore/containers) that include the required environment.

Now I am looking for a (good) way to tell the stuart system to skip downloading
some or all external dependencies (compilers, etc, described in the
ext_dep.yaml files) and use the tools provided by the image/host OS instead.

When I asked this question before, @spbrogan replied:

In short. All ext_deps and stuart files use a scope. The scope allows
platform and ci builds to opt-in or opt-out. We generally don't expect callers
to change scopes directly (and don't want that) but instead the build files
detect the environment and the caller's feature request and then set those
scope appropriately. For example in edk2 core CI if the caller wants to build
AARCH64 and is on Linux then it adds the scope to get a known version of the
compiler.

Stuart and ext_deps are intended to enable local and server based development
consistently and handle changes over time (meaning I can build a 2 year old
branch/release now just as i could 2 years ago). I think with the work you
are doing now around containers we can probably solve this problems more easily
using that.

To resolve your current question i think we would need to update CiSettings.py
([2]edk2/CISettings.py at master tianocore/edk2 (github.com)) to know if it
is containerized and then not add the scopes for compiler download and instead
leverage local compilers. If we want to drop support for non-containerized
linux builds from Edk2 CI I would be fine with that too but would be curious to
know what others think and find useful.

Before I start making changes, I would like to collect some comments on the matter.

Also the scope overrides would have to be done for each Platform-CI as well?

Thanks,
Oliver

[cfernald]

Hey Oliver,

@makubacki @spbrogan who may have ideas for this discussion.

Just to get this conversation kicked off, here are a few options I can think of.

1. A build argument to disable pulling in external tools. This is what I did in our original prototype but would be easy to forget to use.

2. Check for a global EDK2_USE_INSTALLED_TOOLS environment variable to indicate that this environment provides the needed tool chains.

3. Use contextual clues per tool. Specifically for the compilers, we already will prioritize the GCC_*_PREFIX shell variables over the extdep binaries in the compiler plugin, so if we added this knowledge to the platform and CI build scripts then we could avoid unnecessarily pulling in the compilers.

I personally find option 3 the most palatable but is pretty specific to the compiler use case. Option 2 would be simple and easy to understand/implement but due to its binary nature we might find it inflexible if we want to potentially have a mismatched of container provided tools and tools pulled at runtime. This breaks from the general idea of the containers honestly, so I'm not worried but would be the downside I would see.

[makubacki]

Controlling ext_deps with scope is the natural way to do this as mentioned. It seems the question here is what is the best way to set the scope in these scenarios?

Without knowing a lot about the container process currently implemented, I'd expect a generic scope could be added to the ext_deps (not specific to containers or anything else other than whether to install that ext_dep). This allows ext_deps to be controlled independently.

Then, the build scripts could be the binding point for controlling ext_deps in a given build environment. In this case, could the containers contain an environment variable (2) in Chris' suggestions that if missing does not set the tool scopes?

[makubacki]

To close this discussion, I saw https://edk2.groups.io/g/devel/message/96659, I think that's reasonable as well.

In the Tools & CI Meeting today that direction was considered fine, but we'd like to provide manual instructions too. I'm hoping that can be covered with some of the work already being done to improve documentation here. We'd make the container build instructions the default set of instructions but provide the manual instructions as a backup.