Skip to content

Understanding Jenkins

Jump to bottom
Alan Malta Rodrigues edited this page Mar 24, 2020 · 26 revisions

Introduction

The homepage for our Jenkins tests is https://cmssdt.cern.ch/dmwm-jenkins/ which is behind the CERN SSO service.

There are two tests that ultimately decide what Jenkins thinks about your pull request: DMWM-WMCore-PR-test and DMWM-WMAgent-TestAll. Each of these tests or "builds" starts other builds that test specific aspects.

DMWM-WMCore-PR-test starts DMWM-PR-unittests, DMWM-WMCore-PR-pylint, and DMWM-WMCore-PR-27. DMWM-PR-unittests actually starts 12 jobs, each of which tests a "slice" of the unit tests since running them all in one process would take too long. Each of these slices feeds up when done to DMWM-PR-unittests. DMWM-PR-unittests, DMWM-WMCore-PR-pylint, and DMWM-WMCore-PR-27 all feed their information up to DMWM-WMCore-PR-test which does a final analysis of the results and decides if your PR passes or fails.

How to interpret Jenkins's comments on your pull request

Jenkins will post a summary message of how our tests and checks went into the GitHub conversation. It also produces a detailed report which is linked. The summary and report are broken down into three different sections:

Unit tests

Jenkins runs every unit test we have in the code base (currently about 1300) and compares them with the last time the baseline unit tests were run against the master branch (this should happen twice per day). It reports on the differences it finds. Your PR is failed if you add a failing unit test or cause any existing unit tests to fail that previously succeeded. Unit tests that are known to be unstable (test/etc/UnstableTests.txt) are reported on, but won't trigger a failure of your pull request.

If you do have a failure and want to see the unit test output

  • The detailed report will tell you which tests failed, but not the details of how
  • If you click on "details" under Unit Tests in the merge status box
  • then click on Test Result
  • and finally find the failing test you are interested in and click on it. This will give you all the info that nose captured while it was running your test.

False positives: Because of how the snapshot of the master branch is taken, it happens occasionally that code added to master since then will show up in your report. These should only be successful unit tests because no one would push a broken test to master, right?

pylint code quality checks

Jenkins runs a pylint check on changed files on both the master (or other) branch you are pulling towards and your proposed branch. It fails your PR under some circumstances. Check the full report. Things in bold are causing your pull request fail testing. Everything listed under the report is something you should investigate fixing, even it is in a part of the code you did not change. Things that are making the check fail must be fixed unless it is not possible.

python future code quality checks

The last set of checks run is to verify that our code did not regress when it comes to our python 3 compatibility. This is the "27" (python 2.7) check.

Three checks are done:

  1. The entire code base is checked for the "futurize stage 1" changes that were already made. If any regressions are found, your PR is failed.
  2. Any new files you add are checked to make sure you are using python3 versions of print and division. If new files do not contain from __future__ import (print_function, division) your PR is failed
  3. Any files you change are checked for non-breaking but suggested future changes. This is mostly for things like type(var) == int instead of the preferred isintance(var, int). Your PR is not failed for these infractions, but you should fix it if you can to be more pythonic

How to debug Jenkins when it doesn't produce results

In principle, none of these sub-jobs is supposed to fail. It doesn't work out that way, though

  • Any of them can fail because they can't install the WMAgent RPMs from cmsweb
  • Any of them can fail because they can't contact GitHub
  • Unit test slices can fail because they get stuck or take too long, in which case Jenkins kills them
  • PR-27 can fail if the proposed code introduces a python syntax error. That never happens because everyone runs their code, right?

Any time a failure occurs, the job that started it also fails. If this is the case, DMWM-WMCore-PR-test fails and is restarted (up to four times). It is not restarted if it determines that your code was bad, only if there is a problem with the infrastructure.

Ok, so let's look back at that top level page: https://cmssdt.cern.ch/jenkins/view/DMWM/ You may see DMWM-WMCore-PR-test as red and with stormy skies (previous builds failed). You shouldn't see the sub-tests mentioned with anything but green and sunny skies.

Now, you can click on DMWM-WMCore=PR-test and see the history of tests on our pull requests. On the left you will see a list of pull requests and Jenkins build numbers. If you see a little green arrow in that column, that means Jenkins had to automatically restart the pull request for one of the reasons above. If not, that build was started because of a new commit or someone saying "test this please" on the pull request.

Now lets say you want to see why the unit tests were unable to complete.

  • Click on the build # next to the red dot
  • Click on console log (these two steps can be combined with a dropdown)
  • Scroll down to the bottom and you will see something like
Waiting for the completion of DMWM-PR-unittests
DMWM-PR-unittests #642 completed. Result was FAILURE
  • This tells you the build # of the sub-build (642). Click on that link, not the generic one for all DMWM-PR-unittests
  • Scroll to the bottom. You will see the status of each slice. Yellow is OK (tests failing). Red is not. Find the red one and click on that
  • Click on console output (again these can be combined using the drop down)
  • Usually at the end you will see the problem. It should be one of the causes listed above.

The steps to diagnose pylint or "27" (the python 2.7 future checker) are a little easier since they don't have slices. And in the "27" case at the end you might find something like this:

RefactoringTool: There was 1 error:
RefactoringTool: Can't parse src/python/CRABInterface/HTCondorDataWorkflow.py: ParseError: bad input: type=8, value=u')', context=('', (643, 107))

This means that line 643 of the proposed version of HTCondorDataWorkflow.py had a syntax error.

Tasks executed by the CI Jenkins setup

This is a short summary and break down of every single job/project executed in the DMWM Jenkins setup. I also try to highlight jobs using a docker container and checks for python3 compatibility.

HEAD/master tasks

DMWM-WMCore-TagBaseline This one is simply creating and pushing a tag to the repository (and deleting old tags), such that we know what the baseline is (which is compared against each PR results), this is done via docker container. This also triggers the "DMWM-WMAgent-TestAll" task.

DMWM-WMAgent-TestAll It installs WMAgent and test the master branch using this script: https://github.com/cms-sw/cms-bot/blob/master/DMWM/test-wmcore.sh . It also triggers the "DMWM-WMCore-TagBaseline" task. It's also supposed to run a coverage check, which is apparently disabled at the moment. Comand line is

# coverage xml -i --include=$PWD/install* || true

DMWM-WMCore-UnitTests It aggregates the results of "DMWM-WMAgent-TestAll" and archive the results for posterior comparison with PR tests.

DMWM-WMCore-pylint Run pylint checks on the master branch using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/wmcore_tests/TestScripts/pylintWMCore.sh . It keeps a history of the WMCore pylint/pep8 analysis. Besides the pylint check, it also runs a pep8 scan like:

# Fix pep8 which has the wrong python executable
echo "#! /usr/bin/env python" > ../pep8
cat `which pep8` >> ../pep8
chmod +x ../pep8
# Run PEP-8 checker but not in pylint format
../pep8 --format=default --exclude=test/data,.svn,CVS,.bzr,.hg,.git,__pycache__,.tox.

DMWM-WMCore-pylint3 Analyze code base for python3 issues reported by pylint using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/wmcore_tests/TestScripts/pylint3WMCore.sh Which basically runs this check against the master/HEAD branch:

pylint --py3k -f parseable -d W1618 src/python/* test/python/*

DMWM-WMCore-TestOracle This project is disabled at the moment, but it's meant to have the same executing flow as "DMWM-WMAgent-TestAll", however, executing tests in a single worker node and using Oracle as a database backend.

Pull request triggered checks

DMWM-WMCore-PR-test Responsible for testing Pull requests; including mark the PR as pending status using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/jenkins_python/scripts/PullRequestTestBegin.py

then triggering all these other tests: DMWM-PR-unittests, DMWM-WMCore-PR-27, DMWM-WMCore-PR-pylint

and finally aggregating all those results/artifacts and reporting back to the PR using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/jenkins_python/scripts/PullRequestReport.py

DMWM-PR-unittests It installs WMAgent and rebase it to the PR branch, then it executes the unit tests wit this script: https://github.com/cms-sw/cms-bot/blob/master/DMWM/test-wmcore.sh

DMWM-WMCore-PR-27 Check pull requests for inadvertent reintroduction of pre-python 2.7 idioms using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/wmcore_base/ContainerScripts/pyfutureTest.sh Which basically checks all python files changed in the PR against futurize, e.g.:

  futurize -1 $name >> test.patch
  futurize -f execfile -f filter -f raw_input $name >> test.patch || true
  futurize -f idioms $name  >> idioms.patch || true

In addition to that, it also checks whether new python files have from __future__ import division, from this script: https://github.com/dmwm/Docker/blob/master/wmcore_base/ContainerScripts/AnalyzePyFuture.py

DMWM-WMCore-PR-pylint Compares pylint report between master and the feature branch using a docker container and running this script: https://github.com/dmwm/Docker/blob/master/wmcore_base/ContainerScripts/pylintTest.sh thus a test like the following is executed:

  pylint --evaluation='10.0 - ((float(5 * error + warning) / statement) * 10)'  --rcfile standards/.pylintrc --msg-template='{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}'  $name > pylint.out || true
 ${HOME}/ContainerScripts/AggregatePylint.py base

In addition to that, it also creates a PEP8 report for all python files that were changed in the PR, e.g.:

./pep8 NOTHING `< changedFiles.txt` > pep8.txt
Clone this wiki locally