Skip to content

DMWM Packaging with PyPi

Jump to bottom
Erik Gough edited this page Jun 23, 2020 · 19 revisions

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

General Information

DMWM PyPi account: cms-oc-dmwm

Erik Gough and Andrew Melo have the password if needed.

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

Packaging and uploading WMCore/WMAgent to PyPI

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Make a ~/.pypirc file with required contents for uploading to PyPI. In previous testing, it would prompt for a password. Now it seems it is required to put the password in this file, which is a bad idea. Need to look into alternatives to uploading.

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: cms-oc-dmwm
password: <password>

Clone the WMCore repo and fetch the set of tags

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

Checkout what tag you want

$ git checkout <tag>

Run the PyPI build script to build and upload the version you checked out

$ sh tools/build_pypi_packages.sh <wmagent|wmcore|all>

(pypibuild) [username@lxplus789 WMCore]$ sh tools/build_pypi_packages.sh wmcore
Building wmcore package
/afs/cern.ch/user/u/username/pypibuild/lib/python2.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '1.3.6.pre3' to '1.3.6rc3'
  normalized_version,
running sdist
<snip>
Writing wmcore-1.3.6rc3/setup.cfg
Creating tar archive
removing 'wmcore-1.3.6rc3' (and everything under it)
running upload
Submitting dist/wmcore-1.3.6rc3.tar.gz to https://upload.pypi.org/legacy/
Server response (200): OK

Check the cms-oc-dmwm PyPI page to find the new uploaded version: https://pypi.org/project/wmcore/#history

Install WMCore via PyPI

[username@lxplus789 WMCore]$ virtualenv wmcore-pypi
New python executable in /afs/cern.ch/user/u/username/WMCore/wmcore-pypi/bin/python
Please make sure you remove any previous custom paths from your /afs/cern.ch/user/u/username/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
[username@lxplus789 WMCore]$ source wmcore-pypi/bin/activate
(wmcore-pypi) [username@lxplus789 WMCore]$ pip install wmcore
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wmcore
  Downloading wmcore-1.3.5.tar.gz (2.2 MB)
<snip>
Successfully built wmcore
Installing collected packages: wmcore
Successfully installed wmcore-1.3.5

Dependencies (non-python) needed for WMAgent

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

  • MariaDB
  • CouchDB
  • Oracle Client
  • Condor
  • libcurl-devel
  • jemalloc
  • libuuid
  • yui (might not be necessary)
  • pystack (might not be necessary)
  • gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

  • Add mariadb bin and gcc bin directory to PATH
  • Add gcc lib directory to LD_LIBRARY_PATH
  • Set ORACLE_HOME to Oracle client location

Download and install WMAgent

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Packaging DBS Applications

Instructions on how to build and upload DBS Client and Pycurl Client to PyPi.

One time setup for users wanting to upload new releases to PyPi

Create a .pypirc file in your home directory with the following contents:

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: cms-oc-dmwm

Changes to be made at release time

The following changes should be made when a new version of DBS is released. Some version numbers will have to be updated manually for now. This will (hopefully) be automated in the future.

  1. dbs-client version in setup_dbs_client.py, L10
  2. pycurl-client version in setup_pycurl_client.py, L10
  3. pycurl-client version in requirements.dbs_client.py, L1

Build and upload to PyPi

Clone the DBS repo and checkout the latest tag:

$ git clone https://github.com/dmwm/DBS.git
$ git fetch && git fetch --tags
$ git checkout <tag>

Run the build_pypi_packages.sh script build and upload the packages. The script takes the following options, but you will usually want to build "all".

  all              Build all DBS packages
  dbs-client       Build dbs-client package
  pycurl-client    Build pycurl-client package

$ sh Tools/build_pypi_packages.sh all

Enter the cms-oc-dmwm account password when prompted. This can also be set in the .pypirc file if you don't mind storing passwords in clear text.

Check the details of the new build here: https://pypi.org/user/cms-oc-dmwm/

Installing DBS client and Pycurl client via PyPI on lxplus7

Source a CMSSW environment.

[user@lxplusXXX ~]$ export SCRAM_ARCH=<arch>
[user@lxplusXXX ~]$ cmsrel CMSSW_X_Y_Z
[user@lxplusXXX ~]$ cd CMSSW_X_Y_Z/src
[user@lxplusXXX src]$ cmsenv

To use DBS client, one needs to install it via pip and then activate a CMSSW envrionment. Using DBS client with user certificates and proxies requires a curl shared library built with special configuration options. This curl library is pulled in when setting the CMSSW environment. Using virtual python environments (virtualenv) in conjunction with the CMSSW environment gets messy real quick, so it is best to just install DBS client into your home directory with the --user option.

[user@lxplusXXX ~]$ pip install --user dbs-client
Collecting dbs-client
<snip>
Successfully installed dbs-client-3.7.8 pycurl-7.19.0.3 pycurl-client-3.7.8 python-cjson-1.2.1

Create a proxy

[user@lxplusXXX ~]$ voms-proxy-init -rfc -voms cms

Test access with a simple python script

[user@lxplusXXX ~]$ cat dbs-test.py
from dbs.apis.dbsClient import DbsApi
dbs = DbsApi('https://cmsweb.cern.ch/dbs/prod/global/DBSReader')
ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
print(ds)

[user@lxplusXXX ~]$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]
Clone this wiki locally