metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script #151

CAMOBAP · 2022-02-15T16:31:58Z

metanorma/eccma-iso-scor-vocab#1
OCRA bundler issues #100

ronaldtse · 2022-02-15T16:43:00Z

Issue encountered on macOS:

Run build/bin/metanorma-darwin-x86_64 help
[17](https://github.com/metanorma/packed-mn/runs/5203614747?check_suite_focus=true#step:15:17)
/__tebako_memfs__/lib/ruby/2.7.0/bundler/process_lock.rb:9:in `initialize': Read-only file system @ rb_sysopen - /__tebako_memfs__/lib/ruby/gems/2.7.0/bundler.lock (Errno::EROFS)
[18](https://github.com/metanorma/packed-mn/runs/5203614747?check_suite_focus=true#step:15:18)
	from /__tebako_memfs__/lib/ruby/2.7.0/bundler/process_lock.rb:9:in `open'

CAMOBAP · 2022-02-16T08:03:38Z

@maxirmx do we have any restrictions/expectations about the version of Apache Thrift?

I'm asking because when I build temtebako locally I get this error

In file included from /Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs/src/dwarfs/entry.cpp:39:
/Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs-build/thrift/dwarfs/gen-cpp2/metadata_types.h:1138:12: error: no matching constructor for initialization of '::apache::thrift::optional_field_ref<unsigned int &>'
    return {this->time_resolution_sec, __isset.time_resolution_sec};
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs-build/thrift/dwarfs/gen-cpp2/metadata_types.h:1220:12: note: in instantiation of function template specialization 'dwarfs::thrift::metadata::fs_options::time_resolution_sec_ref<, unsigned int>' requested here
    return time_resolution_sec_ref() ? std::addressof(time_resolution_sec) : nullptr;
           ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:394:15: note: candidate constructor not viable: no known conversion from 'bool' to 'apache::thrift::detail::is_set_t<value_type> &' (aka 'unsigned char &') for 2nd argument
  FOLLY_ERASE optional_field_ref(
              ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:408:30: note: candidate constructor template not viable: requires single argument 'other', but 2 arguments were provided
  FOLLY_ERASE /* implicit */ optional_field_ref(
                             ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:417:24: note: candidate constructor template not viable: requires single argument 'other', but 2 arguments were provided
  FOLLY_ERASE explicit optional_field_ref(
                       ^
/usr/local/include/thrift/lib/cpp2/FieldRefTraits.h:27:7: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 2 were provided
class optional_field_ref;
      ^
/usr/local/include/thrift/lib/cpp2/FieldRefTraits.h:27:7: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 2 were provided
In file included from /Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs/src/dwarfs/entry.cpp:39:

maxirmx · 2022-02-16T11:06:54Z

@maxirmx do we have any restrictions/expectations about the version of Apache Thrift?

I'm asking because when I build temtebako locally I get this error

In file included from /Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs/src/dwarfs/entry.cpp:39:
/Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs-build/thrift/dwarfs/gen-cpp2/metadata_types.h:1138:12: error: no matching constructor for initialization of '::apache::thrift::optional_field_ref<unsigned int &>'
    return {this->time_resolution_sec, __isset.time_resolution_sec};
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs-build/thrift/dwarfs/gen-cpp2/metadata_types.h:1220:12: note: in instantiation of function template specialization 'dwarfs::thrift::metadata::fs_options::time_resolution_sec_ref<, unsigned int>' requested here
    return time_resolution_sec_ref() ? std::addressof(time_resolution_sec) : nullptr;
           ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:394:15: note: candidate constructor not viable: no known conversion from 'bool' to 'apache::thrift::detail::is_set_t<value_type> &' (aka 'unsigned char &') for 2nd argument
  FOLLY_ERASE optional_field_ref(
              ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:408:30: note: candidate constructor template not viable: requires single argument 'other', but 2 arguments were provided
  FOLLY_ERASE /* implicit */ optional_field_ref(
                             ^
/usr/local/include/thrift/lib/cpp2/FieldRef.h:417:24: note: candidate constructor template not viable: requires single argument 'other', but 2 arguments were provided
  FOLLY_ERASE explicit optional_field_ref(
                       ^
/usr/local/include/thrift/lib/cpp2/FieldRefTraits.h:27:7: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 2 were provided
class optional_field_ref;
      ^
/usr/local/include/thrift/lib/cpp2/FieldRefTraits.h:27:7: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 2 were provided
In file included from /Users/camobap/Developers/Projects/riboseinc/packed-mn/.archive/tebako/deps/src/_dwarfs_wr/deps/src/_dwarfs/src/dwarfs/entry.cpp:39:

We use dwarfs that uses specific verion of fbthrift.
I did my best to create an isolated environment that loads and sets up all required versions and I can see that it takes files from correct locations.

I suspect issues with incompatible C++ standart applied. The author of dwarfs wrote that he developed it to try all newest C++ feature so it is fragile.
In the version that I am testing now I enforce C++ standart but not in the version that is published

What compiler are you running ?
Could you please force the standard with export CXXFLAGS="$CXXFLAGS -std=c++17

ronaldtse · 2022-03-02T09:02:04Z

@maxirmx would you have time to help resolve this error? This causes packed-mn not to be able to run at https://github.com/metanorma/eccma-iso-scor-vocab/issues/1 . Thanks!

CAMOBAP · 2022-03-02T09:15:18Z

@maxirmx I feel like I need your help here. I have checked flags and std=c++17 passed to the compiler, but this doesn't help, so it's hard to develop it locally

Let me describe the original problem: binary from Tebako breaks if Gemfile exists in the working directory https://github.com/metanorma/eccma-iso-scor-vocab/issues/1

We need somehow embed Gemfile into binary, probably with the definition ENV["BUNDLE_GEMFILE"] or with bundler/inline

ronaldtse · 2022-03-02T13:06:57Z

@CAMOBAP but can we just remove the Gemfile, and have Tebako Ruby ignore any local Gemfile?

maxirmx · 2022-03-02T14:46:21Z

@CAMOBAP @ronaldtse
I will be happy to look at but I need your help first.

It looks like there is some local ("extra") Gemfile that is processed by bundler when

require 'eccma-iso-scor-vocab'

Is it correct understanding ?

If so is it really necessary ? I mean, do you expect some extra dependencies to be really installed ?

maxirmx · 2022-03-02T16:56:50Z

It cannot open libsass.bundle and after that goes build it from Gemfile

Traceback (most recent call last):
	17: from /__tebako_memfs__/local/metanorma:18:in `<main>'
	16: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/inline.rb:54:in `gemfile'
	15: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/settings.rb:115:in `temporary'
	14: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/inline.rb:70:in `block in gemfile'
	13: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:50:in `require'
	12: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:50:in `each'
	11: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:61:in `block in require'
	10: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:61:in `each'
	 9: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:66:in `block (2 levels) in require'
	 8: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:66:in `require'
	 7: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc.rb:31:in `<top (required)>'
	 6: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc.rb:31:in `require_relative'
	 5: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:5:in `<top (required)>'
	 4: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:6:in `<module:SassC>'
	 3: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:11:in `<module:Native>'
	 2: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
	 1: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/libsass.bundle': dlopen(/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/libsass.bundle, 0x0005): tried: '/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/libsass.bundle' (no such file), '/usr/local/lib/libsass.bundle' (no such file), '/usr/lib/libsass.bundle' (no such file) (LoadError)
	18: from /__tebako_memfs__/local/metanorma:18:in `<main>'
	17: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/inline.rb:54:in `gemfile'
	16: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/settings.rb:115:in `temporary'
	15: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/inline.rb:70:in `block in gemfile'
	14: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:50:in `require'
	13: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:50:in `each'
	12: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:61:in `block in require'
	11: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:61:in `each'
	10: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:66:in `block (2 levels) in require'
	 9: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/runtime.rb:66:in `require'
	 8: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc.rb:31:in `<top (required)>'
	 7: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc.rb:31:in `require_relative'
	 6: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:5:in `<top (required)>'
	 5: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:6:in `<module:SassC>'
	 4: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:10:in `<module:Native>'
	 3: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/lib/sassc/native.rb:13:in `rescue in <module:Native>'
	 2: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
	 1: from /__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/ext/libsass.bundle': dlopen(/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/ext/libsass.bundle, 0x0005): tried: '/__tebako_memfs__/lib/ruby/gems/2.7.0/gems/sassc-2.4.0/ext/libsass.bundle' (no such file), '/usr/local/lib/libsass.bundle' (no such file), '/usr/lib/libsass.bundle' (no such file) (LoadError)```

maxirmx · 2022-03-02T19:30:32Z

If I understand it correctly, when 'bundler/inline' is used it automatically installs any missing gems and then requires them.
It creates two issues related to tebako/metanorma packaging

FFI. The call sequence should be

require 'ffi'
apply FFI hack (redefinition/chaining of map_library_name)
require any gems that rely on ffi
In the code

gemfile do
  source 'https://rubygems.org'

  if Gem.win_platform?
    gem "fontist"
    gem "net-ssh"
    gem "zlib"
  end

  gem "ffi"
  gem "rake"
  gem "sassc"
  gem "seven_zip_ruby"

  gem "metanorma-cli", "= 1.5.9"
end

the sequence is different. Probably we shall try something like

gemfile do
  source 'https://rubygems.org'
  gem "ffi"
end

-- ffi hack---

gemfile do
  source 'https://rubygems.org'

  if Gem.win_platform?
    gem "fontist"
    gem "net-ssh"
    gem "zlib"
  end

  gem "rake"
  gem "sassc"
  gem "seven_zip_ruby"

  gem "metanorma-cli", "= 1.5.9"
end

Not sure if it can be splitted this way at all

Tebako starts the package in a jail. GEM_HOME and GEM_PATH point to memfs. I do not think that bundler will be able to install anything

This is the patch that I apply

# ....................................................
# WE DO NOT ACCEPT OUTSIDE GEM PATHS
# ruby/lib/rubygems/path_support.rb
restore_and_save "$1/lib/rubygems/path_support.rb"

re="  @home = env\[\"GEM_HOME\"\] || Gem.default_dir"
# shellcheck disable=SC2251
! IFS= read -r -d '' sbst << EOM
    @home = env\["GEM_HOME"\] || Gem.default_dir
# -- Start of tebako patch --
    unless env\['TEBAKO_PASS_THROUGH'\]
      @home = Gem.default_dir unless @home.index('\/__tebako_memfs__') == 0
    end
# -- End of tebako patch --
EOM

"$gSed" -i "s/$re/${sbst//$'\n'/"\\n"}/g" "$1/lib/rubygems/path_support.rb"

re="@path = split_gem_path env\[\"GEM_PATH\"\], @home"
# shellcheck disable=SC2251
! IFS= read -r -d '' sbst << EOM
    @path = split_gem_path env\["GEM_PATH"\], @home
# -- Start of tebako patch --
    unless env\['TEBAKO_PASS_THROUGH'\]
      @path.keep_if do |xpath|
        xpath.index('\/__tebako_memfs__') == 0
      end
    end
# -- End of tebako patch --
EOM

"$gSed" -i "s/$re/${sbst//$'\n'/"\\n"}/g" "$1/lib/rubygems/path_support.rb"

It can be fixes but it is against initial idea to maintain 'Ruby jail'

ronaldtse · 2022-03-03T06:19:48Z

@maxirmx I believe the packed Ruby should not support external gems that are not packaged. This goes against the original logic of jailing Ruby.

maxirmx · 2022-03-03T10:10:46Z

I agree. Support of external gems will deflate the whole idea

Can you please explain me the original issue with gemfile in cwd ?
I looked at this one: https://github.com/metanorma/eccma-iso-scor-vocab/issues/1. but it looks like an issue with sassc. I think it was fixed though - #146

What shall I do to reproduce it locally ?

CAMOBAP · 2022-03-03T11:19:28Z

I agree. Support of external gems will deflate the whole idea

+1, this is against the packed-mn idea

Can you please explain me the original issue with gemfile in cwd ?

I don't have many technical details, but if Gemfile in cwd, bundler behaves really strange way, looks like it trying lookup gems based on this local Gemfile, also isn't clear why exactly saasc fails, but we have the same issue for ocra (windows packer) and it fails on different gems

What shall I do to reproduce it locally ?

Just clone repo https://github.com/metanorma/eccma-iso-scor-vocab/ locally and try execute metanorma site generate . --agree-to-terms from the root of the repo

maxirmx · 2022-03-03T15:52:53Z

This is the beginning of metanorma script

#!/usr/bin/env ruby
# frozen_string_literal: true

require 'rubygems'
require 'bundler/setup'                <---------  issue
require 'openssl'
require 'open-uri'
require 'net/https'
require 'tempfile'

require 'bundler/setup' discovers Gemfile and puts all of the gems on the load path
simple solution would be remove require 'bundler/setup' from bin/metanorma script

ronaldtse · 2022-03-04T00:00:19Z

@maxirmx @CAMOBAP with #153 done can we close this or is this still pending? Thanks!

maxirmx · 2022-03-04T07:12:24Z

@maxirmx @CAMOBAP with #153 done can we close this or is this still pending? Thanks!

As far as I understand this is not required with #153

CAMOBAP self-assigned this Feb 15, 2022

CAMOBAP force-pushed the feature/embed-gemfile-into-script branch 3 times, most recently from 2b04b0f to 7515995 Compare February 15, 2022 18:47

metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script

3647b57

CAMOBAP force-pushed the feature/embed-gemfile-into-script branch from 7515995 to 3647b57 Compare February 16, 2022 08:09

CAMOBAP closed this Mar 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script #151

metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script #151

CAMOBAP commented Feb 15, 2022

ronaldtse commented Feb 15, 2022

CAMOBAP commented Feb 16, 2022

maxirmx commented Feb 16, 2022

ronaldtse commented Mar 2, 2022

CAMOBAP commented Mar 2, 2022 •

edited by ronaldtse

Loading

ronaldtse commented Mar 2, 2022

maxirmx commented Mar 2, 2022

maxirmx commented Mar 2, 2022

maxirmx commented Mar 2, 2022 •

edited

Loading

ronaldtse commented Mar 3, 2022

maxirmx commented Mar 3, 2022

CAMOBAP commented Mar 3, 2022

maxirmx commented Mar 3, 2022

ronaldtse commented Mar 4, 2022

maxirmx commented Mar 4, 2022

metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script #151

metanorma/eccma-iso-scor-vocab#1 embed Gemfile into metanroma script #151

Conversation

CAMOBAP commented Feb 15, 2022

ronaldtse commented Feb 15, 2022

CAMOBAP commented Feb 16, 2022

maxirmx commented Feb 16, 2022

ronaldtse commented Mar 2, 2022

CAMOBAP commented Mar 2, 2022 • edited by ronaldtse Loading

ronaldtse commented Mar 2, 2022

maxirmx commented Mar 2, 2022

maxirmx commented Mar 2, 2022

maxirmx commented Mar 2, 2022 • edited Loading

ronaldtse commented Mar 3, 2022

maxirmx commented Mar 3, 2022

CAMOBAP commented Mar 3, 2022

maxirmx commented Mar 3, 2022

ronaldtse commented Mar 4, 2022

maxirmx commented Mar 4, 2022

CAMOBAP commented Mar 2, 2022 •

edited by ronaldtse

Loading

maxirmx commented Mar 2, 2022 •

edited

Loading