Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Dockerfile #4068

Merged
merged 2 commits into from
Jan 13, 2025
Merged

Fix Dockerfile #4068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
294668b
Upgrade module streams for nodejs and Ruby in the Dockerfile
wpoely86 Jan 10, 2025
5b4205b
add missing dependency (xz) for nokogiri gem
wpoely86 Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ RUN dnf -y install https://yum.osc.edu/ondemand/latest/ondemand-release-web-late
RUN dnf -y update && \
dnf install -y dnf-utils && \
dnf config-manager --set-enabled powertools && \
dnf -y module enable nodejs:18 ruby:3.1 && \
dnf -y module enable nodejs:20 ruby:3.3 && \
dnf install -y \
file \
lsof \
Expand All @@ -23,6 +23,7 @@ RUN dnf -y update && \
patch \
lua-posix \
rsync \
xz \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this for sure? Isn't this the one that just had that vulnerability?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the one. I'm also not a fan of it.

But it seems a requirement for building nokogiri: https://nokogiri.org/tutorials/installing_nokogiri.html#fedora-red-hat-and-centos

Or can we avoid building nokogiri?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As reference:

--------- IMPORTANT NOTICE ----------
Building Nokogiri with a packaged version of libxml2-2.11.7.
Configuration options: --host\=x86_64-redhat-linux --enable-static
--disable-shared
--libdir\=/opt/ood/ondemand/root/usr/share/gems/3.3/gems/nokogiri-1.15.6/ports/x86_64-linux/libxml2/2.11.7/lib
--with-iconv\=yes --disable-dependency-tracking --without-python
--without-readline --with-c14n --with-debug --with-threads --disable-shared
--enable-static CPPFLAGS\= CFLAGS\=-O2\ -U_FORTIFY_SOURCE\ -g\ -fPIC
The following patches are being applied:
  - 0001-Remove-script-macro-support.patch
  - 0002-Update-entities-to-remove-handling-of-ssi.patch
  - 0003-libxml2.la-is-in-top_builddir.patch
  - 0009-allow-wildcard-namespaces.patch
  - 0010-update-config.guess-and-config.sub-for-libxml2.patch
  - 0011-rip-out-libxml2-s-libc_single_threaded-support.patch

The Nokogiri maintainers intend to provide timely security updates, but if
this is a concern for you and want to use your OS/distro system library
instead, then abort this installation process and install nokogiri as
instructed at:

https://nokogiri.org/tutorials/installing_nokogiri.html#installing-using-standard-system-libraries

Note, however, that nokogiri cannot guarantee compatibility with every
version of libxml2 that may be provided by OS/package vendors.

Extracting libxml2-2.11.7.tar.xz into
tmp/x86_64-redhat-linux/ports/libxml2/2.11.7... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include=${opt-dir}/include
	--without-opt-include
	--with-opt-lib=${opt-dir}/lib64
	--without-opt-lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/bin/$(RUBY_BASE_NAME)
	--help
	--clean
	--prevent-strip
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-static
	--disable-static
	--enable-cross-build
	--disable-cross-build
	--enable-cross-build
	--disable-cross-build
	--with-zlib-dir
	--without-zlib-dir
	--with-zlib-include=${zlib-dir}/include
	--without-zlib-include
	--with-zlib-lib=${zlib-dir}/lib64
	--without-zlib-lib
	--with-xml2-source-dir
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:557:in
`xzcat_exe': xzcat not found (RuntimeError)
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:568:in
`tar_command'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:579:in
`extract_file'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:140:in
`block in extract'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:138:in
`each'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:138:in
`extract'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:233:in
`cook'
	from extconf.rb:540:in `block (2 levels) in process_recipe'
	from extconf.rb:319:in `chdir'
	from extconf.rb:319:in `chdir_for_build'
	from extconf.rb:540:in `block in process_recipe'
	from <internal:kernel>:90:in `tap'
	from extconf.rb:438:in `process_recipe'
	from extconf.rb:872:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/opt/ood/ondemand/root/usr/share/gems/3.3/extensions/x86_64-linux/3.3.0/nokogiri-1.15.6/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/opt/ood/ondemand/root/usr/share/gems/3.3/gems/nokogiri-1.15.6 for inspection.
Results logged to
/opt/ood/ondemand/root/usr/share/gems/3.3/extensions/x86_64-linux/3.3.0/nokogiri-1.15.6/gem_make.out

  /usr/share/rubygems/rubygems/ext/builder.rb:125:in `run'
  /usr/share/rubygems/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /usr/share/rubygems/rubygems/ext/builder.rb:193:in `build_extension'
  /usr/share/rubygems/rubygems/ext/builder.rb:227:in `block in build_extensions'
  /usr/share/rubygems/rubygems/ext/builder.rb:224:in `each'
  /usr/share/rubygems/rubygems/ext/builder.rb:224:in `build_extensions'
  /usr/share/rubygems/rubygems/installer.rb:855:in `build_extensions'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/rubygems_gem_installer.rb:71:in
`build_extensions'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/source/rubygems.rb:204:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/gem_installer.rb:54:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/parallel_installer.rb:186:in
`do_install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/parallel_installer.rb:177:in
`block in worker_pool'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:62:in
`apply_func'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:57:in
`block in process_queue'
  <internal:kernel>:187:in `loop'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:54:in
`process_queue'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:91:in
`block (2 levels) in create_threads'

An error occurred while installing nokogiri (1.15.6), and Bundler cannot
continue.

In Gemfile:
  ood_appkit was resolved to 2.1.4, which depends on
    rails was resolved to 7.0.8.5, which depends on
      actioncable was resolved to 7.0.8.5, which depends on
        actionpack was resolved to 7.0.8.5, which depends on
          actionview was resolved to 7.0.8.5, which depends on
            rails-dom-testing was resolved to 2.2.0, which depends on
              nokogiri
rake aborted!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it issue seems to be that nokogiri ship it's own version libxml2 in a tar.xz file...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try with zlib-devel instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should likely just pull a package off of yum.osc.edu. Or expect a local package to exist somewhere.

Building is sort of the issue - but really it's installing. Installing from source is very flaky and error prone. We have a lot of steps in the packages (rpms and debs) that we can just utilize instead of doing it all over again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but that means creating packages for all these which is not that trivial and has a large maintenance overhead.

It's the reason we run a container for ondemand in the first place ;)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but that means creating packages for all these which is not that trivial and has a large maintenance overhead.

Not sure I follow what you mean by all these. We already build lots of packages. In fact those are the actual artifacts we're supporting. Container installations aren't really supported by us.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, now I'm a bit lost. What's the reason than for the bundle install and rake install in this Dockerfile?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, now I'm a bit lost. What's the reason than for the bundle install and rake install in this Dockerfile?

I think I see, we're getting tripped up on the word packages maybe? When I mean we build and support packages I mean .deb and .rpm files that we host on yum.osc.edu.

bundle and rake install install our dependencies (ruby gems) along with some nodejs packages.

Or .rpms for example pull in all our dependencies,

ondemand/packaging/rpm/ondemand.spec

Line 147 in 359e2aa

rake --trace -mj%{ncpus} build

But also do all sorts of other stuff like moving stuff around and creating directories and so on.

A Dockerfile that has to replicate all of these is bound to be error prone.

ondemand-gems \
ondemand-runtime \
ondemand-build \
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.dev
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN cp /etc/yum.repos.d/ondemand-web.repo /etc/yum.repos.d/ondemand-nightly-web.
RUN dnf -y update && \
dnf install -y dnf-utils && \
dnf config-manager --set-enabled powertools && \
dnf -y module enable nodejs:18 ruby:3.1 && \
dnf -y module enable nodejs:20 ruby:3.3 && \
dnf install -y epel-release && \
dnf install -y ondemand ondemand-dex && \
dnf install -y gcc gcc-c++ libyaml-devel nc && \
Expand Down
Loading