Core dependencies for running AGC implementation

[alexander-held]

We had some discussions in a few places (#140, #141, #144) about how to provide the relevant information for environment setup. This is meant to gather the core pieces that would allow running the latest non-notebook version (otherwise Jupyter is needed) with Python futures (see also #143, otherwise Dask is required as well).

The list currently includes:

coffea cabinetry servicex-clients scikit-learn xgboost

This will pick up a lot of core dependencies as well automatically.

Please report any issues and missing libraries encountered.

cc @eguiraud

[alexander-held]

Possible future pitfalls to be aware of: coffea versions beyond 0.7 ("coffea 2023") will only be compatible with future versions of the AGC implementation here (#116) and otherwise not work with the current notebook. The same holds for awkward>=2.

[eguiraud]

Hi, I tried just putting those 5 packages in a requirements.txt and run pip install over it in an empty virtualenv, and pip takes forever to find a set of versions that are compatible with each other.

I still find the conda environment.yml is the simplest way to provide an environment that works out of the box (as it also fixes the python version, see #144 ).

[matthewfeickert]

@eguiraud Can I have you look at https://iris-hep.org/analysis-systems-env-nightlies/ and check to see what additional packages would be needed to test a nightly AGC lock file build?

[alexander-held]

Based on some discussion today I believe we should revisit this topic of specifying environments with pixi @matthewfeickert.

[matthewfeickert]

@alexander-held Is there an existing requirements file somewhere?

[alexander-held]

Yes, there is https://github.com/iris-hep/analysis-grand-challenge/blob/main/analyses/cms-open-data-ttbar/requirements.txt which then we realized isn't ideal (see e.g. #144). The list of core dependencies in #199 (comment) could perhaps also be a basis for such a file, but would require at least a coffea<1 restriction (and perhaps more, I have not tried this from scratch recently).