Checking private key with an X509 cert fails with 0.14.2

[jamis]

I'm trying to build a (signed) gem, but with JOpenSSL 0.14.2 the cert check fails during build. This works on JOpenSSL 0.12.2. I've narrowed it down to a consistently reproducible and minimal script, but I'm also fairly weak in OpenSSL and may be doing something wrong.

To reproduce the issue

1. Generate a private key.

openssl genrsa -out gem-private_key.pem 4096

2. Generate an X509 certificate. I'm using gem cert for this, for convenience, but the issue occurs even if I use the openssl CLI to generate this.

gem cert --build [email protected] --private-key gem-private_key.pem

(Have to use MRI ruby for this; trying to run this command with JRuby results in an error.)

3. Attempt to verify the private key for the certificate.

# frozen_string_literal: true

require 'openssl'

key = OpenSSL::PKey.read(File.read('gem-private_key.pem'), nil)
cert = OpenSSL::X509::Certificate.new(File.read('gem-public_cert.pem'))

puts "ruby: #{RUBY_DESCRIPTION}"
puts "openssl: #{OpenSSL::VERSION}"
puts "jopenssl: #{JOpenSSL::VERSION}" if defined?(JOpenSSL)

puts "check private key: #{cert.check_private_key(key)}"

Running with MRI 3.1.4:

ruby: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [arm64-darwin23]
openssl: 3.0.1
check private key: true

Running with JRuby 9.2.21:

ruby: jruby 9.2.21.0 (2.5.8) 2022-06-27 49e5080a7c OpenJDK 64-Bit Server VM 21 on 21 +jit [darwin-aarch64]
openssl: 2.2.1
jopenssl: 0.12.2
check private key: true

Running with JRuby 9.3.13:

ruby: jruby 9.3.13.0 (2.6.8) 2023-11-02 09b6f2263a OpenJDK 64-Bit Server VM 21 on 21 +jit [arm64-darwin]
openssl: 2.2.1
jopenssl: 0.14.2
check private key: false

Running with JRuby 9.4.5:

ruby: jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 21 on 21 +jit [arm64-darwin]
openssl: 2.2.1
jopenssl: 0.14.2
check private key: false

As seen in the output, JRuby 9.3.13 and 9.4.5 fail to check to the private key (returning false). MRI and JRuby 9.2.21 succeed (returning true).

Possibly relevant: if I get the public key from the private key and convert it to PEM, the resulting key is empty on JRuby 9.3.13 and 9.4.5:

p key.public_key.to_pem
# -> "-----BEGIN PUBLIC KEY-----\nMAA=\n-----END PUBLIC KEY-----\n"

[segiddins]

I ran into the same issue, I have a repro script that succeeds on mri and truffleruby and fails on jruby 9.4:

#!/usr/bin/env ruby

require 'openssl'
ENV["MT_NO_PLUGINS"] = "1" # Work around autoloading of plugins
require 'minitest/autorun'

class T < Minitest::Test
  make_my_diffs_pretty!
  def test_round_trip
    pem = "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEEXsz3SZXFb8jMV42j6pJlyjbjR8K\nN3Bwocexq6LMIb5qsWKOQvLN16NUefLc4HswOoumRsVVaajSpQS6fobkRw==\n-----END PUBLIC KEY-----\n"
    key = OpenSSL::PKey.read(pem)
    
    assert_equal pem, key.to_pem
  end
end

[kares]

thank you both - much appreciated. I dag in and found the cause to have happened in 0.14.1
namely the changes from #267 are causing the issue, specifically: a7994b3#diff-c007f003c44fb9a713d8e8e9c78b80196499ad88a01e43787165a7dde8664367R355