Request: Please make FIPS 140-2 compliant images available

[gesarki]

Describe the solution you'd like
At Acquia we’re currently using this component as part of our globally distributed Kubernetes infrastructure.

Like most providers, we expend a lot of effort ensuring that we remain compliant with varied regulatory regimes across different localities. Currently we're working towards FIPS compliance which is required by federal institutions like healthcare, banking, and defense organizations to set cryptographic standards for highly sensitive data.

The FIPS 140-2 encryption compliance requirement poses a common problem that is likely to be shared by any organizations looking to obtain a FEDRAMP certification and which use a myriad of OSS Kubernetes components under the hood.

FIPS compliance stands as a barrier to entry for smaller organizations. Accepting our contribution would take a small step towards lowering that barrier for the general public and other open source projects.

We, and any other cloud based software providers that are building atop Kubernetes, would benefit from the availability of off-the-shelf FIPS-compliant variants of this project’s images.

While many tools exist to tweak deployment manifests, it can be complex and fragile to rebuild upstream images with constraints placed on encryption libraries. This type of change would be far easier to make inside the originating repo.

Internally, we have explored several paths forward to achieve our compliance objectives. We are hoping to be able to contribute the required changes upstream in a way that benefits the community at large and minimizes toil.

Initially, we are requesting that the current maintainers and community consider supporting FIPS images. We have a notional approach to contribute and would like to collaborate.

Please let us know your thoughts in this Issue or by reaching out directly.

At this time, we have an internal approach that we are using on our internal Go-based controllers and other container images. There are some basic requirements that we’d ask to be built into a -fips variant including assertions that could live in the upstream build steps or in later CI steps.

What we do internally is:

• Image is built with a FIPS library such as boringcrypto for Go
• Image is based on Amazon Linux 2, Canonical Ubuntu 20.04 or later, or UBI8
• Image has appropriate group and user specified and final image executes in that context
• We assert that binaries call out to boringcrypto via dynamic linking (this is recommended during build but is not required, we will verify this ourselves in our CI/CD infra)

Thank you for your time considering this request as well as your efforts supporting this project!

Acquia KaaS Core Services team

Anything else you would like to add:
Similar to #5284.

The changes would be very minor, don't add toil, and wouldn't break things. We can't add this later because the change has to be at the step where the binary is built.

Environment:

• Velero version (use velero version):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[reasonerjt]

I'm more inclined to leave such enhancement downstream.

[kaovilai]

It's a widely followed government compliance requirement.

I would propose if we were to do this, we do it using boringcrypto flag during build time.

GOEXPERIMENT=boringcrypto go build .

Per their readme,

we are not making any statements or representations about
the suitability of this code in relation to the FIPS 140 standard.

This mean that velero project will not be officially certifying the project as FIPS 140 compliant. User will have to do their own verification.

[kaovilai]

We would use go tool nm $BINARY to validate FIPS mode.

[imuni4fun]

Exactly as mentioned above. The two minimum required changes would be to:

• build with boring crypto (in this repo)
• verify use of boring crypto (previous comment) (this can be downstream)

The portion that needs to be in this repo is very small and could be the defacto release or a different image tag. Either way, for any consumer that wishes to use velero in their clusters this step being in the upstream leads to MUCH less toil.

[ira-gordin-sap]

Hi, any update for this requirement?

[reasonerjt]

Hi, any update for this requirement?

ATM I don't think there's commitment to delivering the FIPS images. I know re-compiling it is relatively easy, but it would be a continuous effort to maintain and keep it compliant, IMO we need to discuss with other maintainers before making any public decision.

[kaovilai]

FYI added PR to #8412 for discussion etc.

[kaovilai]

We might want to use base images from https://github.com/microsoft/go/tree/microsoft/main/eng/doc/fips

They produces a bookworm 1.22 images which should slot right in for our Dockerfile.
https://github.com/microsoft/go-images/tree/microsoft/main/src/microsoft/1.22/bookworm

[kaovilai]

From initial experiments it seems to require more changes than just dropping in ENV when go building. Or perhaps it is because I am on GOOS=darwin which may not be able to (cross) compile this. Tried compiling with GOOS=linux and resulting is still standard crypto lib.

Either I need to install a new go CLI that is fips enabled or following quotes do not apply on macOS

Pass GOEXPERIMENT=boringcrypto to the go tool during build time. As simple as that.

❯ GOEXPERIMENT=boringcrypto GOOS=linux make local && go tool nm _output/bin/linux/arm64/velero > nm.txt && grep -E "sig.FIPSOnly|sig.BoringCrypto|sig.StandardCrypto" nm.txt
GOOS=linux \
        GOARCH=arm64 \
        GOBIN=$(pwd)/.go/bin \
        GOEXPERIMENT=boringcrypto \
        VERSION=main \
        REGISTRY=velero \
        PKG=github.com/vmware-tanzu/velero \
        BIN=velero \
        GIT_SHA=7882a5e6bb7e06121a4c63327dac5715e5e8e047 \
        GIT_TREE_STATE=dirty \
        OUTPUT_DIR=$(pwd)/_output/bin/linux/arm64 \
        ./hack/build.sh
...
  37d110 T crypto/internal/boring/sig.StandardCrypto.abi0

If cross compiling on darwin is the issue then all-builds will have to not produce darwin CLI when boringcrypto is used.

[kaovilai]

But at least IIUC darwin devs should still be able to build all-containers with the right base images and/or go binaries.

[kaovilai]

Probably doing something wrong here.

❯ docker run -it --rm -e GOEXPERIMENT=boringcrypto mcr.microsoft.com/oss/go/microsoft/golang:1.22-fips-bookworm sh -c 'go tool nm $(which go) | grep crypto/internal/boring && go env'
  1ec1f0 T crypto/internal/boring/sig.StandardCrypto.abi0

Details

❯ docker run -it --rm -e GOEXPERIMENT=boringcrypto mcr.microsoft.com/oss/go/microsoft/golang:1.22-fips-bookworm sh -c 'go tool nm $(which go) | grep crypto/internal/boring && go env'
  1ec1f0 T crypto/internal/boring/sig.StandardCrypto.abi0
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT='boringcrypto'
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.22.9'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build531265482=/tmp/go-build -gno-record-gcc-switches'

[kaovilai]

rh toolset works. opened issue with MSFT

❯ docker run --rm -it registry.access.redhat.com/ubi9/go-toolset sh -c 'go tool nm $(which go) | grep "crypto/internal/boring/sig."'
  5dd980 t crypto/internal/boring/sig.BoringCrypto.abi0

[kaovilai]

Closed PR for now until there's clarity on if there's a less complex way to drop-in changes.

[kaovilai]

Per microsoft/go#1412 (comment) it looks like microsoft bookworm images would be capable of producing fips complaint binaries.. but I'm not sure if that's enough for a complaint container image.

If folks really want it to be upstreamed here and are more aware of the needed changes, a PR is always welcome.

For now, the maintainers have agreement that it is out of scope of our priorities. If there's anything blocking your downstream FIPS build that need changes here, please let us know.

[reasonerjt]

Thanks @kaovilai I'm putting this in "icebox"

[kaovilai]

@reasonerjt thanks.

Another thing that would help simplify is if we can keep producing one kind of image.

If we can do FIPS image here, can it be the only image we publish?
ie do we need to keep producing non FIPS?
That would reduce the number of needed Docker files and/or simplifies build/release logic.