Various errors using register_pyroot_binding()

[lcorcodilos]

Hello,

I've tried what I think is the correct way to establish pyroot bindings according to this resource [1]. However, I get a bunch of C++ related errors (below at [2]). Based on [2] and some Googling, I think at least on issue is that my ROOT version is compiled for C++14 and, from what I can tell, std::variant was introduced in C++17. Does this sound correct and do I need to recompile ROOT for C++17? Is there anything else I might be missing?

Here is a minimal reproducer of the issue:

python3 -m virtualenv test-env # Python 3.8.5
source test-env/bin/activate
python -m pip install correctionlib
python -c "import correctionlib; correctionlib.register_pyroot_binding()"
python -c "import ROOT" # confirms that pyroot is working

ROOT version is 6.24.0 and is compiled for python 2.7 and python 3.8.5. I've confirmed these are linked correctly with the final line of the reproducer.

Thanks,
Lucas

[1] - https://gist.github.com/pieterdavid/a560e65658386d70a1720cb5afe4d3e9
[2] -

In file included from input_line_22:1:
In file included from /home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:9:
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correctionlib_version.h:7:30: error: constexpr variable 'correctionlib_version' must be initialized by a constant expression
  constexpr std::string_view correctionlib_version{"2.0.0"};
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/9/experimental/string_view:109:39: note: non-constexpr function 'length' cannot be used in a constant expression
      : _M_len{__str == nullptr ? 0 : traits_type::length(__str)},
                                      ^
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correctionlib_version.h:7:30: note: in call to 'basic_string_view(&"2.0.0"[0])'
  constexpr std::string_view correctionlib_version{"2.0.0"};
                             ^
/usr/include/c++/9/bits/char_traits.h:329:7: note: declared here
      length(const char_type* __s)
      ^
In file included from input_line_22:1:
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:20:18: error: no template named 'variant' in namespace 'std'
    typedef std::variant<int, double, std::string> Type;
            ~~~~~^
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:41:14: error: no template named 'variant' in namespace 'std'
typedef std::variant<double, Formula, FormulaRef, Transform, Binning, MultiBinning, Category> Content;
        ~~~~~^
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:73:18: error: no template named 'variant' in namespace 'std'
    typedef std::variant<
            ~~~~~^
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:74:12: error: no member named 'monostate' in namespace 'std'
      std::monostate,
      ~~~~~^
/home/lucas/CMS/temp/correctionlib_test/test-env/lib/python3.8/site-packages/correctionlib/include/correction.h:176:10: error: no template named 'variant' in namespace 'std'
    std::variant<IntMap, StrMap> map_;
    ~~~~~^

[pieterdavid]

I agree that that's likely the problem. You can check the C++ version your ROOT was compiled for with root-config --has-cxx17 or root-config --features, I think it's automatically deduced from the compiler used to build ROOT, and as far as I know the compiler can only use this one version.
IIRC it worked fine for me with ROOT 6.24/00 set up using source /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc10-opt/setup.sh

[lcorcodilos]

I was actually too quick to mark this as resolved...

Now I can do python -c "import correctionlib; correctionlib.register_pyroot_binding()" with no issue.

However, when I try the example linked above (df.py) where I've plugged in a local raw NanoAOD file, I get

IncrementalExecutor::executeFunction: symbol '_ZN10correction13CorrectionSet9from_fileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE' unresolved while linking function '_GLOBAL__sub_I_cling_module_339'!
You are probably missing the definition of correction::CorrectionSet::from_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
Maybe you need to load the corresponding shared library?
 *** Break *** segmentation violation
  

and then the stack trace. Does the library need to be recompiled or something?

[pieterdavid]

I always used a local build from source, but indeed, this is because the wheels are built with the pre-C++11 ABI for std::string, they have this one instead of correction::CorrectionSet::from_file(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), which ROOT expects:

% nm -D test_venv/lib/python3.9/site-packages/correctionlib/lib/libcorrectionlib.so | grep _ZN10correction13CorrectionSet9from_file | c++filt
0000000000045ed0 T correction::CorrectionSet::from_file(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)

I think it should be fine to use the new one here... actually I'm a bit surprised that's not the default yet, it may be due to the gcc and cmake version in the environment that builds the wheels. It could be that installing from source fixes it (otherwise you can try passing -DCMAKE_CXX_STANDARD=17 explicitly from setup.py).

[lcorcodilos]

Okay, I was able to get this working with some wrangling. On top of making sure I had a ROOT version compiled for python3 and cxx17, I did the following steps

python3 -m virtualenv test-env
source test-env/bin/activate
git clone --recursive [email protected]:cms-nanoAOD/correctionlib.git
pip install scikit-build setuptools-scm
pip install -e correctionlib
python -c "import correctionlib; correctionlib.register_pyroot_binding()" # works
python df.py # works but I had to modify

I had to modify df.py slightly to the following

import ROOT as gbl
import correctionlib
correctionlib.register_pyroot_binding()

f = gbl.TFile.Open("file.root")
df = gbl.ROOT.RDataFrame(f.Get("Events"))

gbl.gInterpreter.Declare('auto csetEl = correction::CorrectionSet::from_file("electron.json");')
gbl.gInterpreter.Declare('auto csetEl_2016preID = csetEl->at("UL-Electron-ID-SF");')
df = df.Filter("nElectron >= 1 && Electron_pt[0] > 10.").Define( # added index to Electron_pt
        "leadElSF", # just used multiline str so I could read it easier
        '''csetEl_2016preID->evaluate(
            {
                "2016preVFP", "sf", "wp90iso",
                std::min((float)2.49,std::abs(Electron_eta[0])), // needed mins on these lines or I get
                std::min((float)499.0,Electron_pt[0])            // a Multibinning error for going outside
            }                                                    // the range of the SF measurement
        )''')
h = df.Histo1D("leadElSF")
h.Draw()

So I think my base issue is solved but I'm curious about my need for std::min. The error I got without this was:

RDataFrame::Run: event loop was interrupted
Traceback (most recent call last):
  File "test_df.py", line 23, in <module>
    h.Draw()
cppyy.gbl.std.runtime_error: TH1D& ROOT::RDF::RResultPtr<TH1D>::operator*() =>
    runtime_error: Index above bounds in MultiBinning input argument4 val: 798.477295

and one similar for eta. Is there a way to have correctionlib take care of the cases of overflow (like here where the highest pt bin is just used for electrions with pt > 500 GeV)? Or is there a good way to know what the cutoffs are without opening the huge json?

[pieterdavid]

Thanks for the feedback, the Electron_pt[0] is fixed in the example. I think I had something with out-of-range arguments at some point, but I don't remember the details, I'll check again (it may have changed since).
Aside: to install 2.0 from source (non-editable) this should take care of the (build and install) dependencies:

pip install 'git+https://github.com/cms-nanoAOD/[email protected]#egg=correctionlib'

[nsmith-]

Sorry I didn't see this until now but

it may be due to the gcc and cmake version in the environment that builds the wheels

Indeed the python manylinux wheels have _GLIBCXX_USE_CXX11_ABI set to 0 since they use RHEL7 stock gcc. See #86 (reply in thread) for a bit more info. In short, you can add --no-binary=correctionlib to your pip install command to build it from source and avoid this issue.

Is there a way to have correctionlib take care of the cases of overflow (like here where the highest pt bin is just used for electrions with pt > 500 GeV)?

It is up to the correction json author how to handle out-of-range values. The options include clamping to nearest bin edge, returning a default value, or throwing an error. I guess here the author decided to use the error option.

Or is there a good way to know what the cutoffs are without opening the huge json?

Try correction summary electron.json from the command line.