Support OpenSSL 3 FIPS

[junaruga]

I would like to use this ticket to manage all the issues related to OpenSSL 3 FIPS. The goal of this ticket is to pass all the unit tests with the OpenSSL 3.2, 3.1 and 3.0 FIPS cases on the CI.

FIPS related issues/pull-requests

I update the list regularly. The issues are sorted by newest first.

• 11: Rakefile: Manage test files by excluding test files in the test_fips task. #820
• 10: Add passing test files in FIPS. #819
• 9: Fix test_provider.rb in FIPS. #794
• 8: Fix test_pkey_rsa.rb in FIPS. #790
• 7: Fix test_pkey_dsa.rb in FIPS. #729
• 6: Fix test_pkey_dh.rb in FIPS. #694
• 5: Fixed test/openssl/test_pkey_ec.rb. test_pkey_ec.rb test failures in OpenSSL FIPS #671 fixed by test/openssl/test_pkey_ec.rb: refactor tests for EC.builtin_curves #675, and Fix test_pkey_ec.rb on FIPS. #681.
• 4: Fixed the second issue in OpenSSL::PKey.read and test/openssl/test_pkey.rb. The issue was fixed by the the workaround ossl_pkey.c: Workaround: Decode with non-zero selections. #669.
  • ed25519: OpenSSL 3.0.8: ed25519 a decode from and then encode to a pem file corrupts the key if fips+base provider is used openssl/openssl#20758
  • x25519: OpenSSL 3: x25519 a decode from and then encode to a pem file corrupts the key if fips+base provider is used openssl/openssl#21493
  • Fixed PR in OpenSSL Decoder key export fixes openssl/openssl#21519
• 3: OpenSSL 3 FIPS mode - creating encrypted RSA key pair fails with PEM_write_bio_PrivateKey_traditional: initialization error (OpenSSL::PKey::PKeyError) #643
• 2: Fixed an issue in OpenSSL::PKey.read. Applied the workaround PR Fix OpenSSL::PKey.read that cannot parse PKey in the FIPS mode. #615 to avoid the OpenSSL issue OSSL_DECODER_CTX_set_selection doesn't apply the selection value properly openssl/openssl#20657.
• 1: Fixed the OpenSSL.fips_mode in OpenSSL 3.0+. The issue: OpenSSL 3: OpenSSL.fips_mode returns false in FIPS enabled environment #605, the PR: Implement FIPS functions, adding OpenSSL FIPS mode case on CI. #608

Remaining tasks to the goal

Fix other test files test/**/test_*.rb to pass in CI. The fixed test files 18/33.

$ date
Wed Oct  4 04:54:22 PM CEST 2023

$ find test/ -name "test_*.rb" | wc -l
33

Documents

• How to debug Ruby OpenSSL binding with OpenSSL 3 FIPS: Contribution is welcome!
• Check openssl-head fips, openssl-3.1.z fips or openssl-3.0 fips cases in this repository's GitHub Actions YAML file to know how to compile Ruby OpenSSL binding with OpenSSL FIPS.

Original comment

I deleted to simplify this issue ticket. I think we don't need the original comment's information any more.

[jackorp]

I'll add what we learned on our side in more detail.

Downstream bug report: https://bugzilla.redhat.com/show_bug.cgi?id=2170105

Some words and what they refer to throughout this report:
rubygem-openssl = this repo, the OpenSSL bindings gem shipped with Ruby
OpenSSL / C OpenSSL / libopenssl = the system OpenSSL

Problem description:
rubygem-openssl cannot load keys when the system is in FIPS mode. This was found on RHEL 9 and applies to RHEL 9.0 up to the package set available in C9S at the time of this report, which translates to RHEL 9.2.

Reproducer on RHEL 9:
0. have FIPS enabled system
1.a $ sudo dnf install ruby -- this will install Ruby 3.0 with backported patches to work with OpenSSL 3.0
1.b Optionally, you can use Ruby 3.1 which ships with rubygem-openssl 3.0 on RHEL 9.1 and later:

$ sudo dnf module enable ruby -y
$ sudo dnf install ruby -y

2. $ openssl genrsa -out key.pem 4096 -- create a key for reproducer purposes
3. ruby -ropenssl -e "OpenSSL::PKey.read(File.read('key.pem'))"

Actual results:

-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from -e:1:in `<main>'

Expected results:

No errors

This was observed with OpenSSL versions 3.0.1 through 3.0.7 and with rubygem-openssl 2.2.1, 3.0.0, 3.1.0 up to the more recent commits on the master branch of this repo: 9ea934a .

The problem is that ossl_pkey_read_generic fails to correctly parse the key.

I have distilled Ruby's code flow into a simple C example to illustrate better what is happening. See the comment on the downstream ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c7

I found that initializing OSSL_DECODER_CTX_new_for_pkey(...) with EVP_PKEY_KEYPAIR as the value for the selection argument of the function allows for the decoding to succeed. However, if I reuse the OSSL_DECODER_CTX structure, where selection is 0 and use OSSL_DECODER_CTX_set_selection before decoding with EVP_PKEY_KEYPAIR, decoding fails unless there is a proper call to OpenSSL_init_ssl() at the beginning of the main function of the C example. Currently these OpenSSL flags work for the example:

OPENSSL_INIT_NO_ADD_ALL_DIGESTS
OPENSSL_INIT_NO_LOAD_CONFIG
OPENSSL_INIT_ENGINE_RDRAND
PENSSL_INIT_ENGINE_CRYPTODEV
OPENSSL_INIT_ENGINE_PADLOCK
OPENSSL_INIT_ENGINE_ALL_BUILTIN

I'll emphasize that it is important that OSSL_DECODER_CTX_new_for_pkey with different selection arguments works when decoding a key in BIO struct multiple times.

I have also used system's python (python 3.9), where loading pem key simply works:

$ sudo dnf install -y python3-cryptography
$ cat script.py
from cryptography.hazmat.primitives import serialization

file = open("key.pem", "rb")
pem_bytes = file.read()
file.close()
print(serialization.load_pem_private_key(pem_bytes, password=None))
$ python script.py
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x7ff5ec2a7940>

They seem to initialize OpenSSL similarly to Ruby

From inspecting ltraces of both languages I noticed that python uses EVP_MD* function family from openssl unlike rubygem-openssl, where OSSL_DECODER* function family is used.

Traces for python and Ruby:
python was ran as follows: python script.py
python_crypto_ltrace.txt
python_crypto_strace.txt
Ruby was ran as follows: ruby -ropenssl -e "OpenSSL::PKey.read(File.read('key.pem'))"
rubygem_openssl_ltrace.txt
rubygem_openssl_ltrace.txt

To me it seems something isn't being loaded properly in Ruby when system is in FIPS mode as calling OSSL_PROVIDER_load(NULL, "fips") fixes the situation when inserted into rubygem-openssl code before decoding the key. However, from what I understand from Dmitry Belyavskiy on the downstream ticket, that call should not be needed, as correct providers should be loaded automatically, assuming OpenSSL was initialized: https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c29

[junaruga]

First, good news is the Ruby project has a RHEL 9.1 server that is one of the servers used in Ruby CI, where maintainers can access by SSH. And I chat with hsbt, asking him if we can have the RHEL 9 server with FIPS mode enabled. After that, maintainers can access to the server by SSH.

Note that now the maintainers can try the FIPS mode enabled environment on the RHEL 9.1 server in the Ruby project. For the steps, see #601 (comment) . I was able to reproduce this issue ticket' error on the environment.

[junaruga]

$ sudo dnf install -y python3-cryptography
...

I noted the the mentioned RPM package "python3-cryptography" is not available on Fedora: https://src.fedoraproject.org/rpms/python3-cryptography .

So, just installing the PyPI package "cryptography" directly is convenient. The GitHub is here.

$ pip3 install --user cryptography

$ pip3 list | grep cryptography
cryptography         37.0.2

[junaruga]

We can reproduce this issue on GitHub Actions after this PR #608 is merged!

[jackorp]

I noted the the mentioned RPM package "python3-cryptography" is not available on

I am not sure you are right in this. I was able to find this via both the dnf info and dnf install on containers of Fedora 36 up to Rawhide.

it is actually coming from this repo: https://src.fedoraproject.org/rpms/python-cryptography

[junaruga]

@jackorp You are right. Thanks for the info. I also can see the installed package on my environment. What was I seeing at that time... Sorry.

$ cat /etc/fedora-release 
Fedora release 37 (Thirty Seven)

$ rpm -q python3-cryptography
python3-cryptography-37.0.2-4.fc37.x86_64

It seems that the python-cryptography repository's python-cryptography.spec (recipe file) manages the entry of the python3-cryptography package.

[junaruga]

I am debugging this issue on Fedora 37. Let me share my current status. I think my small program: fips_mode command is helpful for you to check if you are running the correct FIPS mode on your environment. You can run like this.

$ cat /home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf
config_diagnostics = 1
openssl_conf = openssl_init

.include /home/jaruga/.local/openssl-3.0.8-fips/ssl/fipsmodule.cnf
#.include ./fipsmodule.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

$ LD_LIBRARY_PATH=/home/jaruga/.local/openssl-3.0.8-fips/lib/ \
  OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf \
  ~/git/openssl-test/fips_mode
FIPS mode provider available: 1
FIPS mode enabled: 1

I am debugging this issue with GDB on the commits in the #608 on Fedora. There is an issue gdb with LD_LIBRARY_PATH=/path/to/openssl_lib (this issue). So, I am setting the LD_LIBRARY_PATH in the GDB prompt.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf \
  gdb --args ruby -I lib -e "require 'openssl'; OpenSSL::PKey.read(File.read('key.pem'))"
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.0.8-fips/lib/

Some of the content might repeat what jarkorp said.
The error is happening on the code below, because the pkey is NULL.

openssl/ext/openssl/ossl_pkey.c

Lines 215 to 218 in a14055a

	pkey = ossl_pkey_read_generic(bio, ossl_pem_passwd_value(pass));
	BIO_free(bio);
	if (!pkey)
	ossl_raise(ePKeyError, "Could not parse PKey");

Here is the backtrace.

(gdb) f
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, 
    self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
218		ossl_raise(ePKeyError, "Could not parse PKey");
(gdb) bt
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, 
    self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
#1  0x00007ffff7b309f7 in vm_call_cfunc_with_frame (ec=0x40a0c0, reg_cfp=0x7ffff7542f90, 
    calling=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_insnhelper.c:3268
#2  0x00007ffff7b35d44 in vm_sendish (method_explorer=<optimized out>, 
    block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, 
    ec=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_callinfo.h:367
#3  vm_exec_core (ec=0x7, initial=4214800, initial@entry=0)
    at /home/jaruga/src/ruby-3.2.1/insns.def:820
#4  0x00007ffff7b3bdf9 in rb_vm_exec (ec=0x40a0c0, jit_enable_p=jit_enable_p@entry=true)
    at vm.c:2383
#5  0x00007ffff7b3cde8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#6  0x00007ffff7951755 in rb_ec_exec_node (ec=ec@entry=0x40a0c0, n=n@entry=0x7ffff7e7bab8)
    at eval.c:289
#7  0x00007ffff7957c7b in ruby_run_node (n=0x7ffff7e7bab8) at eval.c:330
#8  0x0000000000401102 in rb_main (argv=0x7fffffffda58, argc=5) at ./main.c:38
#9  main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57
(gdb) bt
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
#1  0x00007ffff7b309f7 in vm_call_cfunc_with_frame (ec=0x40a0c0, reg_cfp=0x7ffff7542f90, calling=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_insnhelper.c:3268
#2  0x00007ffff7b35d44 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
    at /home/jaruga/src/ruby-3.2.1/vm_callinfo.h:367
#3  vm_exec_core (ec=0x7, initial=4214800, initial@entry=0) at /home/jaruga/src/ruby-3.2.1/insns.def:820
#4  0x00007ffff7b3bdf9 in rb_vm_exec (ec=0x40a0c0, jit_enable_p=jit_enable_p@entry=true) at vm.c:2383
#5  0x00007ffff7b3cde8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#6  0x00007ffff7951755 in rb_ec_exec_node (ec=ec@entry=0x40a0c0, n=n@entry=0x7ffff7e7bab8) at eval.c:289
#7  0x00007ffff7957c7b in ruby_run_node (n=0x7ffff7e7bab8) at eval.c:330
#8  0x0000000000401102 in rb_main (argv=0x7fffffffda58, argc=5) at ./main.c:38
#9  main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57

In the process, the ossl_pkey_read_generic is called from the ext/openssl/ossl_pkey_dh.c ossl_dh_initialize, and also called from ext/openssl/ossl_pkey.c ossl_pkey_new_from_data.

When the ossl_pkey_read_generic called from the ossl_dh_initialize, the OSSL_DECODER_from_bio(dctx, bio) returns 1, and set the value to the poiner pkey in this timing. But the ossl_pkey_read_generic called from the ossl_pkey_new_from_data, OSSL_DECODER_from_bio(dctx, bio) doesn't return 1, and the variable pkey is still NULL.

openssl/ext/openssl/ossl_pkey.c

Lines 144 to 145 in a14055a

	if (OSSL_DECODER_from_bio(dctx, bio) == 1)
	goto out;

[junaruga]

I opened the issue ticket openssl/openssl#20657 on the OpenSSL project to ask a queston.

[junaruga]

In the issue ticket on the openssl/openssl above, the root cause was found. In short summary, the OSSL_DECODER_CTX_set_selection has a bug in the FIPS mode case. And I confirmed this issue was fixed with my patch on openssl/openssl#20657 (comment). As a note, I am working for that on my forked repository's branch https://github.com/junaruga/openssl/tree/wip/fips-read.

By applying this patch, the unit test result in the FIPS mode changes in a better way from

https://github.com/junaruga/openssl/actions/runs/4681990779/jobs/8295272065

513 tests, 1963 assertions, 14 failures, 331 errors, 0 pendings, 1 omissions, 0 notifications
32.6172% passed

to

https://github.com/junaruga/openssl/actions/runs/4690199157/jobs/8312977212

513 tests, 3082 assertions, 17 failures, 135 errors, 0 pendings, 1 omissions, 0 notifications
70.3125% passed

.

I am still working to fix other issues on the FIPS mode. Seeing the code below in the test, it seems that the FIPS mode case was not tested before. And I would like to remove this logic to run the unit tests in the FIPS mode.

openssl/test/openssl/utils.rb

Lines 5 to 8 in 6182ac0

	# Disable FIPS mode for tests for installations
	# where FIPS mode would be enabled by default.
	# Has no effect on all other installations.
	OpenSSL.fips_mode=false

[junaruga]

I changed this issue ticket's title to the "Support OpenSSL 3 FIPS mode" to manage all the issues on this one ticket. I think it's easy for people to track. This ticket's goal is to pass all the unit tests on the OpenSSL 3.0 FIPS mode on GitHub Actions CI.

My current focus is to fix the 2 test failures test_ed25519 and test_x25519 in the test/openssl/test_pkey.rb on the OpenSSL 3 FIPS mode. And I opened an issue ticket to ask people with my reproducing program written by C at the OpenSSL project on openssl/openssl#20758.

$ LD_LIBRARY_PATH=/home/jaruga/.local/openssl-3.0.8-fips-debug/lib/ \
  OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips-debug/ssl/openssl_fips.cnf \
  ruby -I lib test/openssl/test_pkey.rb -n test_ed25519
Loaded suite test/openssl/test_pkey
Started
F
==================================================================================================
Failure: test_ed25519(OpenSSL::TestPKey)
test/openssl/test_pkey.rb:107:in `test_ed25519'
     104:     assert_instance_of OpenSSL::PKey::PKey, pub
     105:     assert_equal priv_pem, priv.private_to_pem
     106:     assert_equal pub_pem, priv.public_to_pem
  => 107:     assert_equal pub_pem, pub.public_to_pem
     108: 
     109:     sig = [<<~EOF.gsub(/[^0-9a-f]/, "")].pack("H*")
     110:     92a009a9f0d4cab8720e820b5f642540
<"-----BEGIN PUBLIC KEY-----\n" +
"MCowBQYDK2VwAyEAPUAXw+hDiVqStwqnTRt+vJyYLM8uxJaMwM1V8Sr0Zgw=\n" +
"-----END PUBLIC KEY-----\n">(UTF-8) expected but was
<"-----BEGIN PUBLIC KEY-----\n" +
"MCowBQYDK2VwAyEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n" +
"-----END PUBLIC KEY-----\n">(ASCII-8BIT)

diff:
  -----BEGIN PUBLIC KEY-----
? MCowBQYDK2VwAyEAPUAXw+hDiVqStwqnTRt+vJyYLM8uxJaMwM1V8Sr0Zgw  =
?                    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 
?                 -- ???????????????????????????????????????? 
  -----END PUBLIC KEY-----
  
? Encoding: UTF  -8   
?           ASCII  BIT
?           ???  +++
==================================================================================================

Finished in 0.08343667 seconds.
--------------------------------------------------------------------------------------------------
1 tests, 6 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
--------------------------------------------------------------------------------------------------
11.99 tests/s, 71.91 assertions/s

$ LD_LIBRARY_PATH=/home/jaruga/.local/openssl-3.0.8-fips-debug/lib/ \
  OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips-debug/ssl/openssl_fips.cnf \
  ruby -I lib test/openssl/test_pkey.rb -n test_x25519
Loaded suite test/openssl/test_pkey
Started
F
==================================================================================================
Failure: test_x25519(OpenSSL::TestPKey)
test/openssl/test_pkey.rb:151:in `test_x25519'
     148:     end
     149:     assert_instance_of OpenSSL::PKey::PKey, alice
     150:     assert_equal alice_pem, alice.private_to_pem
  => 151:     assert_equal bob_pem, bob.public_to_pem
     152:     assert_equal [shared_secret].pack("H*"), alice.derive(bob)
     153:   end
     154: 
<"-----BEGIN PUBLIC KEY-----\n" +
"MCowBQYDK2VuAyEA3p7bfXt9wbTTW2HC7OQ1Nz+DQ8hbeGdNrfx+FG+IK08=\n" +
"-----END PUBLIC KEY-----\n">(UTF-8) expected but was
<"-----BEGIN PUBLIC KEY-----\n" +
"MCowBQYDK2VuAyEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n" +
"-----END PUBLIC KEY-----\n">(ASCII-8BIT)

diff:
  -----BEGIN PUBLIC KEY-----
? MCowBQYDK2VuAyEA3p7bfXt9wbTTW2HC7OQ1Nz+DQ8hbeGdNrfx+FG+IK08=
?                 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 
?                 ??????????????????????????????????????????? 
  -----END PUBLIC KEY-----
  
? Encoding: UTF  -8   
?           ASCII  BIT
?           ???  +++
==================================================================================================

Finished in 0.013378889 seconds.
--------------------------------------------------------------------------------------------------
1 tests, 4 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
--------------------------------------------------------------------------------------------------
74.74 tests/s, 298.98 assertions/s

[cwjenkins]

@junaruga you still working on this? Need help?

[junaruga]

@cwjenkins Yes, I am still working on this. And yes! Thank you for your help.

For the original issue on this ticket, I plan to rebase the PR #615 as a workaround of this issue and merge it to the master branch. If you want to see it soon, I can work on it as a higher priority. The root cause was found on the OpenSSL side. Here is the issue ticket openssl/openssl#20657 Folks are working on it.

$ ruby -e "require 'openssl'; p OpenSSL::VERSION; OpenSSL::PKey.read(File.read('/home/jaruga/work/key.pem'))"
"3.1.0"
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
  from -e:1:in `<main>'

In the other issues, I want to fix the issue that I wrote on the comment above, #603 (comment). The ticket is openssl/openssl#20758. They are working on it. I don't have a time to debug it by myself (yet).

Right now my focus is to add checks for the compiler warnings on GitHub Actions. Here is the ticket, #626. Because I made a bug #620 related to compiler warning. I think checking compiler warnings by -Werror at least on the currently passing cases on the CI is useful and effective. But the #628 is a blocker.

So, I need your help for any things related to this issue. Please let me know if you take a task. Thanks!

[junaruga]

By the way, I created a document of how to debug Ruby OpenSSL bindings with upstream OpenSSL FIPS mode and, OpenSSL FIPS mode itself (without Ruby). I wish that that I add some of the contents to the CONTRIBUTING.md. And here is Fedora OpenSSL RPM specific document as I am also working in Fedora project.

[junaruga]

Note I merged the #615, and updated this ticket's first comment.

[cwjenkins]

Thanks @junaruga. I'll take a look at #628 if you'd like.
Also, you have a way to communicate (e.g. discord/email/etc.) in case any questions arise ?

[junaruga]

@cwjenkins Yes I like it! Thank you!

As a note, now I am rethinking about how to check the compiler warnings on CI. Because in the way with the -Werror the compiler (gcc, clang and etc) exits as an error immediately. This may not be convenient when people want to see all the warnings in several .c files on CI, that is like the "macos-latest 2.6" case on the #615 (comment). What I am thinking is if we can check the output from the rake compile, then grep by '^something: warning: ' to check if there are warnings. And if they find the warnings, exit as an error (CI failure). Considering Windows cases on CI, it may be good to implement in Ruby rather than Bash or Sh.

However, the way with -Werror is still more convenient than the current situation (no check for the compiler warnings). So, your investigation for this issue is very helpful. Perhaps, it may be a Ruby's bug.

Also, you have a way to communicate (e.g. discord/email/etc.) in case any questions arise ?

My email is "jaruga AT ruby-lang.org" in this case. I normally would like open communication as a way to scale. But you can comment on #628 or email me if you have questions.

I have to do other work other than ruby/openssl for a few days or 1 week from now.

[tarnowsc]

since this is supposed to track the FIPS mode related issues, I've added one with issue on creation of RSA key pair: #643

[junaruga]

@tarnowsc thanks! I updated the first comment on this ticket with your issue link.

[junaruga]

However, the way with -Werror is still more convenient than the current situation (no check for the compiler warnings). So, your investigation for this issue is very helpful. Perhaps, it may be a Ruby's bug.

As you may know the compiler warnings checks are added to the master branch. So, my focus is to debug the issue openssl/openssl#20758 to fix. @cwjenkins you can help it for that too when you have spare time.

I have to do other work other than ruby/openssl for a few days or 1 week from now.

I finished the other work above, came back slightly..

[cwjenkins]

Thanks @junaruga. I'll try finding some time this week.

[cwjenkins]

@junaruga took a look this weekend and noticed with fips/base providers on 3.1.0 the key is NULL.
Created a dockerfile to allow building different versions quickly to see if anything has changed in subsequent openssl releases here

Looks like the provider lookup has been refactored so can compare on the changes against 3.0.x to see what exactly produced a mangled key.

[junaruga]

@cwjenkins That's a nice finding! Thanks! I plan to check the issue with the openssl/openssl master branch. By the way, I will take a vacation from tomorrow 5th to 12th July. Then I will come back.

[junaruga]

Created a dockerfile to allow building different versions quickly to see if anything has changed in subsequent openssl releases here

@cwjenkins I executed your Dockerfile with OpenSSL 3.0.8 and 3.1.0 with x86_64 image as my local machine is x86_64. And I was able to see the difference of both results as you reported.

I plan to check the issue with the openssl/openssl master branch.

I tested this on OpenSSL latest master branch, and reported it at openssl/openssl#20758 (comment).

Note I updated the README for the instructions of my reproducing program by adding the gcc -I ... -L ... and LD_LIBRARY_PATH to compile and run the program with a proper target OpenSSL.

[cwjenkins]

@junaruga thanks.

The NULL pkey was from OSSL_DECODER_CTX_new_for_pkey and according to the upstream report it sounds like that should be the behavior? (ed25519 not supported in NIST).

The segfault was in ossl_pkey_export_spki which I believe is part of the test program, therefore upstream ruby should work.

I did notice the fips_mode= func isn't setup correctly, but that is a separate issue.

irb(main):004:0> OpenSSL.fips_mode=true
Traceback (most recent call last):
        5: from ./bin/irb:23:in `<main>'
        4: from ./bin/irb:23:in `load'
        3: from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
        2: from (irb):4
        1: from (irb):4:in `fips_mode='
OpenSSL::OpenSSLError (This version of OpenSSL does not support FIPS mode)

[junaruga]

The NULL pkey was from OSSL_DECODER_CTX_new_for_pkey and according to the upstream report it sounds like that should be the behavior? (ed25519 not supported in NIST).

Right now the OSSL_DECODER_CTX_new_for_pkey can be called 3 times at maximum in the ossl_pkey_read_generic at ext/openssl/ossl_pkey.c. So, I suppose that if the pkey is NULL after calling the ossl_pkey_read_generic, we can say the encryption is not supported in OpenSSL 3.1 FIPS and later versions.

The logic is PROV_NAMES_ED25519, FIPS_UNAPPROVED_PROPERTIES, ... in the providers/fips/fipsprov.c by the commit openssl/openssl@9fa5532. However, problem is OpenSSL 3.0 FIPS case. Right now the OSSL_DECODER_CTX_new_for_pkey can return the non-NULL pkey, and decode a wrong text without error in OpenSSL 3.0 FIPS case.

And one person said "OpenSSL 3.0 has banned EdDSA by policy, so a technical fix isn't required." in a openssl ticket. So perhaps this issue in OpenSSL 3.0 FIPS case may be not fixed. And perhaps we may need to implement the error check by ourselves.

I did notice the fips_mode= func isn't setup correctly, but that is a separate issue.

What's the value of OpenSSL::OPENSSL_LIBRARY_VERSION? Are you compiling the Ruby OpenSSL binding for the OpenSSL 1.1 or 1.0?

The following command only works when the library directory of the openssl is "lib" such as "/path/to/openssl_dir/lib"

$ MAKEFLAGS="V=1" \
  bundle exec rake compile -- \
  --with-openssl-dir=/path/to/openssl_dir

If your openssl's library directory name is lib64 (default library name), maybe like this.

$ MAKEFLAGS="V=1" \
  bundle exec rake compile -- \
  --with-openssl-dir=/path/to/openssl_dir \
  --with-openssl-lib=/path/to/openssl_dir/lib64

Then if you are compiling in Linux environment, you can check the link like this.

$ ldd lib/openssl.so 
	linux-vdso.so.1 (0x00007ffd375bb000)
	libruby.so.3.3 => /home/jaruga/.local/ruby-30b960ba34-debug/lib/libruby.so.3.3 (0x00007ff679a00000)
	libssl.so.3 => /home/jaruga/.local/openssl-3.0.9-fips-debug/lib/libssl.so.3 (0x00007ff67a08f000)
	libcrypto.so.3 => /home/jaruga/.local/openssl-3.0.9-fips-debug/lib/libcrypto.so.3 (0x00007ff679400000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ff67991f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff679222000)
	libz.so.1 => /lib64/libz.so.1 (0x00007ff67a05b000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007ff679fb6000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007ff6791e9000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff6798fb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff67a1d8000)

[junaruga]

Note I opened another issue ticket openssl/openssl#21493 with x25519 pem file, the case is convenient to debug and fix the issue as we can reproducer the issue on OpenSSL master branch.

[junaruga]

Note after openssl/openssl#21519 will be merged, I think we can work to add the unit tests related to the issues to the FIPS cases in this repository.

[junaruga]

Note after openssl/openssl#21519 will be merged, I think we can work to add the unit tests related to the issues to the FIPS cases in this repository.

First, I added OpenSSL head non-FIPS and FIPS cases to the CI to run the unit tests on the fixed version of the OpenSSL by the #658.

I managed to pass the pending tests in the fixed version of the OpenSSL (openssl-head) cases. Here is my work on my forked repository. The last 4 commits are for this topic.

[junaruga]

I sent the PR #664, to pass the pending tests in FIPS in the fixed versions of the OpenSSL.

[junaruga]

I sent the PR #669 that is to implement a workaround to avoid the issue openssl/openssl#21519, and it's a rework of the #664. I believe the workaround is better than pending the related tests.

[junaruga]

I am debugging the test failures in FIPS in the test/openssl/test_pkey_ec.rb now, and opened the issue #671.

[junaruga]

As an experiment, I would share the status update about this ticket here once in around 3 months, or when I achieve something.

During the term from July 2023 to September, 2023, we finished the task 4 and 5 written in the first comment.

• 4: OpenSSL::PKey.read (test/openssl/test_pkey.rb)
• 5: test/openssl/test_pkey_ec.rb

[junaruga]

I would share the status update about this ticket.

During the term from October 2023 to December 2023, we finished the task 6 below.

• 6: Fix test_pkey_dh.rb in FIPS. #694

[junaruga]

I would share the status update about this ticket.

During the term from January 2024 to March 2024, we finished the task 7 below.

• 7: Fix test_pkey_dsa.rb in FIPS. #729

[junaruga]

I would share the status update about this ticket.

During the term from April 2024 to June 2024, there is no progress for this ticket. I couldn't have time at all for this FIPS task, because I was busy for another task in my company.

[junaruga]

I would share the status update about this ticket.

During the term from July 2024 to September 2024, we finished the task 8 and 9 below.

• 8: Fix test_pkey_rsa.rb in FIPS. #790
• 9: Fix test_provider.rb in FIPS. #794

[junaruga]

Note below is a list of failing or error test files in FIPS. The fist column means a number of the failing or error tests in the test file. Feel free to fix the files!

      1 test/openssl/test_asn1.rb
      5 test/openssl/test_cipher.rb
     13 test/openssl/test_digest.rb
      5 test/openssl/test_hmac.rb
      3 test/openssl/test_kdf.rb
      1 test/openssl/test_ns_spki.rb
     20 test/openssl/test_ocsp.rb
     10 test/openssl/test_pkcs12.rb
      5 test/openssl/test_pkcs7.rb
      4 test/openssl/test_ssl.rb
     25 test/openssl/test_ts.rb
      4 test/openssl/test_x509cert.rb
      3 test/openssl/test_x509crl.rb
      1 test/openssl/test_x509name.rb
     10 test/openssl/test_x509req.rb