Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The FIPS enabled environment for developers #601

Closed
junaruga opened this issue Feb 21, 2023 · 12 comments
Closed

The FIPS enabled environment for developers #601

junaruga opened this issue Feb 21, 2023 · 12 comments

Comments

@junaruga
Copy link
Member

junaruga commented Feb 21, 2023

Dear Maintainers, we found a FIPS mode specific issue for ruby/openssl on RHEL 9. And I am thinking how to report it. Do you have FIPS mode enabled Linux machines to test? I can see the FIPS mode specific logic on the files below.

  • ext/openssl/ossl.c
  • test/openssl/test_fips.rb

As a reference, below are how to enable the FIPS mode for RHEL and Ubuntu.

  • RHEL 9 - Chapter 2. Installing the system in FIPS mode
  • Fedora (It's not official document. I just found it.) It seems the document is old.
  • Ubuntu
@junaruga junaruga changed the title Q. Do you have a FIPS mode enabled server? Q. Do you have a FIPS mode enabled machine to test? Feb 21, 2023
@hsbt
Copy link
Member

hsbt commented Feb 28, 2023

@junaruga Can we mix FIPS mode and normal(?) mode in one VM instance?

@junaruga
Copy link
Member Author

@hsbt This is a good point! We can save the infra resource if we can run both FIPS mode and non-FIPS mode on one VM instance. I am not sure about it. Let me confirm it to my colleagues who know more about it. I will let you know here. Thanks!

@junaruga
Copy link
Member Author

junaruga commented Mar 1, 2023

@hsbt I asked it my colleagues, collected the info from them, and I also experimented some things by myself. However, I couldn't find a way to mix the FIPS mode and non-FIPS mode (we can call it as "normal" mode) without rebooting OS in our case.

The reason is because below are the actual steps to switch the system to the FIPS mode written at the RHEL 9 document - 3.4. Switching the system to FIPS mode.

# fips-mode-setup --enable
# reboot

# fips-mode-setup --check

And if you see the fips-mode-setup (bash script) file, the script updates the kernel parameter as one of the processes in it.

$ cat /usr/bin/fips-mode-setup
...
fipsopts="fips=$enable_fips$boot_device_opt"
...
    grubby --update-kernel=ALL --args="$fipsopts"
...

In my testing RHEL 9.1 environment with FIPS mode enabled, I see the fips=1 in the kernel parameter. Note the command to see the kernel entries has 3 entries (index: 0, 1, 2). And the first 2 entries are for the latest installed kernel and the previously installed kernel, and the 3rd one is for the purpose of rescuing.

$ sudo grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.14.0-162.17.1.el9_1.x86_64"
args="... fips=1"
...
index=1
kernel="/boot/vmlinuz-5.14.0-162.6.1.el9_1.x86_64"
args="... fips=1"
...
index=2
...

Here the command to see the current running kernel and installed kernel packages.

$ uname -r
5.14.0-162.17.1.el9_1.x86_64

$ rpm -q kernel
kernel-5.14.0-162.6.1.el9_1.x86_64
kernel-5.14.0-162.17.1.el9_1.x86_64

Switching the system to FIPS or non-FIPS mode

When we cannot mix the FIPS and non-FIPS mode in one state in our case, we may be able to switch when we want to do it.

You can see the following options in the manual of the fips-mode-setup.

$ man fips-mode-setup
...
       •   --enable: Enables the system FIPS mode.

       •   --disable: Undo some of the FIPS-enablement steps (unsupported).
...

MAYBE the process to switch the system to non-FIPS mode again is below. But note the --disable option is unsupported. I tried to verify the commands below on my RHEL 9.1 VM instance with FIPS mode enabled. However, for some reason, I couldn't connect to the server any more. I suspect that it is our infra specific issue.

# fips-mode-setup --disable
Setting system policy to DEFAULT
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.
FIPS mode will be disabled.
Please reboot the system for the setting to take effect.

# reboot

According to the fips-mode-setup script, possibly the kernel parameters are set as fips=0. Note that it's not exactly the same with the normal status without fips=0 parameter.

$ sudo grubby --info=ALL | grep fips
args="... fips=0"
args="... fips=0"
args="... fips=0"

Or we may just be able to back up the RHEL 9 non-FIPS (normal) mode VM instance image.

Changing the VM instance to the smaller one

If we still want to keep both RHEL 9 FIPS mode and non-FIPS mode, but we want to save the infra cost, perhaps, we may be able to think about changing the VM instance size to smaller and cheaper one.

@junaruga
Copy link
Member Author

junaruga commented Mar 1, 2023

Oh wait! I just got a new information from one of my colleagues! Perhaps, this makes mix both FIPS and non-FIPS without rebooting OS in our case!

@junaruga
Copy link
Member Author

junaruga commented Mar 1, 2023

I don't find a way to mix the FIPS and non-FIPS mode for example on the process level. However, I was told that OpenSSL only depends on the content of the /proc/sys/crypto/fips_enabled. That means when running a bash script by root to change the system to a kind of the part of the user space FIPS mode, we may be able to create an environment to test ruby/openssl on FIPS. I will create the script and verify it.

@junaruga
Copy link
Member Author

junaruga commented Mar 1, 2023

I was able to enable/disable a part of the FIPS mode temporarily that is need to test with OpenSSL without rebooting OS!

Here is the repository.
https://github.com/junaruga/fips-mode-user-space

@junaruga
Copy link
Member Author

junaruga commented Mar 2, 2023

@rhenium, can you access Linux servers used in Ruby CI by SSH? If you don't know about it, you can ask on the maintainers Slack.

@junaruga
Copy link
Member Author

junaruga commented Mar 2, 2023

Note: I only see that my script above only works on RHEL 9 in my testing. I am testing it on Fedora 37. But I see one issue, and it doesn't work on Fedora 37. (junaruga/fips-mode-user-space#4)

I also checked it on Ubuntu 22.10. Ubuntu even doesn't have the /proc/sys/crypto/fips_enabled file. Today I heard the /proc/sys/crypto/fips_enabled is a downstream specific feature in Fedora and the downstream Linux distros (RHEL), while I want to make the script work on Ubuntu too. (junaruga/fips-mode-user-space#5)

@junaruga
Copy link
Member Author

junaruga commented Mar 3, 2023

OpenSSL maintainers,

I was able to disable/enable the kernel FIPS flag by using my script on the RHEL 9.1 server used in Ruby CI. and I was reproduce the error #603 on the on the environment with the kernel FIPS flag enabled.

Below are the steps to work on the kernel FIPS mode enabled on the RHEL 9 server.

Steps to work on the FIPS mode enabled.

Login to the RHEL 9 server by SSH.

Comment out a command running the test in a CI user's crontab.

Run the git clone .. for https://github.com/junaruga/fips-mode-user-space by your user account.

Check the current environment.

$ whoami
jaruga

$ sudo fips-mode-setup --check
Installation of FIPS modules is not completed.
FIPS mode is disabled.

$ echo $?
0

$ ./fips-mode-user-space-setup status
/proc/sys/crypto/fips_enabled: 0

Enable the kernel FIPS flag.

$ sudo ./fips-mode-user-space-setup enable

$ echo $?
0

$ ./fips-mode-user-space-setup status
/proc/sys/crypto/fips_enabled: 1

Work (debug, test and etc) on the environment.

Disable (undo) the kernel FIPS mode flag again.

$ sudo ./fips-mode-user-space-setup disable

$ ./fips-mode-user-space-setup status
/proc/sys/crypto/fips_enabled: 0

$ sudo fips-mode-setup --check
Installation of FIPS modules is not completed.
FIPS mode is disabled.

$ echo $?
0

Remove a command running the test in a CI user's crontab again.


That's all.

@junaruga
Copy link
Member Author

junaruga commented Mar 3, 2023

I would close this ticket, as we can provide the FIPS mode enabled environment in the Ruby project. Let me know when you have questions. Thanks.

@junaruga junaruga closed this as completed Mar 3, 2023
@junaruga
Copy link
Member Author

junaruga commented Mar 3, 2023

A very good news! You don't even run the script fips-mode-user-space-setup to test ruby/openssl with OpenSSL on the FIPS mode enabled in RHEL 9.

[jaruga@rhel9 work]$ rpm -q openssl
openssl-3.0.1-47.el9_1.x86_64

[jaruga@rhel9 work]$ openssl genrsa -out ./key.pem 4096

[jaruga@rhel9 work]$ ls -l /home/jaruga/work/key.pem
-rw-------. 1 jaruga wheel 3268 Mar  3 13:56 /home/jaruga/work/key.pem

[jaruga@rhel9 work]$ OPENSSL_FORCE_FIPS_MODE=1 /home/jaruga/.local/ruby-b49053a6be/bin/ruby -e "require 'openssl'; p OpenSSL::VERSION; OpenSSL::PKey.read(File.read('/home/jaruga/work/key.pem'))"
"3.1.0"
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from -e:1:in `<main>'

[jaruga@rhel9 work]$ OPENSSL_FORCE_FIPS_MODE=1 strace /home/jaruga/.local/ruby-b49053a6be/bin/ruby -e "require 'openssl'; p OpenSSL::VERSION; OpenSSL::PKey.read(File.read('/home/jaruga/work/key.pem'))" >& strace.log

[jaruga@rhel9 work]$ grep fips strace.log 
openat(AT_FDCWD, "/usr/lib64/ossl-modules/fips.so", O_RDONLY|O_CLOEXEC) = 5
openat(AT_FDCWD, "/usr/lib64/ossl-modules/fips.so", O_RDONLY) = 5
read(5, "providers/fips/fipsprov.c\0Pass\0F"..., 4096) = 4096

This comes from a Fedora stream line specific patch Fedora. But I cannot reproduce it on Fedora. I am not sure why.

@junaruga junaruga changed the title Q. Do you have a FIPS mode enabled machine to test? The FIPS enabled environment for developers Mar 8, 2023
@junaruga
Copy link
Member Author

I was able to create the FIPS mode enabled environment from the source-built OpenSSL. See #608.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants