Jenkins is a Java application and as such it requires that Java is installed on the workstation that will be configured as the Jenkins server. It is recommended that a recent version of the Java JDK is installed ( This Jenkins-LSCI implementation has been successfully tested with Java version 1.8.x)
Groovy (http://groovy-lang.org/) is the native scripting language in Jenkins. Most of the provided scripts are implemented in Groovy. The latest version of the Groovy SDK (2.4.x as of this writing) is recommended. Jenkins can directly install tools and one of them is Groovy. If you don’t already have Groovy installed on your server you can configure Jenkins to install Groovy as needed.
Git (https://git-scm.com/ ) is a mature, actively maintained open source project for source code version control. You will need to have git installed on the computer where Jenkins-LSCI will be installed. This will allow you to install and update the Jenkins-LSCI code directly from its git repository.
Alternatively you can download a zip archive from the git repository and expand it in your JENKINS_HOME folder
H2 is a Java SQL database. In Jenkins-LSCI it is used to manipulate large CSV files as database tables. H2 is included as a jar file in this distribution.
CellProfiler is free open-source software designed to enable biologists without training in computer vision or programming to quantitatively measure phenotypes from thousands of images automatically. It can be downloaded from http://cellprofiler.org/releases/ .
Download and install the latest stable version (all of our code has been developed and tested with CellProfiler v2.1.2)
For HP image processing on a Linux cluster, CellProfiler and its library dependencies must be installed and correctly working on the destination Linux cluster. Detailed instructions are provided on the CellProfiler wiki page https://github.com/CellProfiler/CellProfiler/wiki/Source-installation-%28Linux%29
The provided installation instructions are for a Windows workstation or server, but can be easily adapted to other platforms. Jenkins-LSCI and all the software libraries used are Java based, and so they should be cross platform compatible.
Jenkins will look for a JENKINS_HOME environment variable on first run and will create the required installation file structure in that folder. This is preferred than installing Jenkins in its default location and then needing to clone the Jenkins-LSCI project into it.
-
Create a folder in a suitable location on your file system where you will install Jenkins. For example create a C:\DEVTOOLS\Jenkins folder
-
Create a JENKINS_HOME environment variable for your operating system. This variable should point to the folder you just created. For example JENKINS_HOME=C:\DEVTOOLS\Jenkins
Before installing or running Jenkins for the first time clone the Jenkins-LSCI git repository into the JENKINS_HOME folder.
-
From the command line, change the working directory to JENKINS_HOME
-
Then execute the command:
git clone https://github.com/Novartis/Jenkins-LSCI.git .
This should clone the required Jenkins-LSCI jobs, plugins, scripts, and other resource folders into the JENKINS_HOME folder.
After cloning the Jenkins-LSCI repository you should have the following JENKINS_HOME subfolders.
-
jobs
-
plugins
-
scriptler
-
userContent
This minimal installation will allow you to run the Jenkins-LSCI job examples once Jenkins is installed and configured.
You can either configure the global site variables now or wait till later. The system comes preconfigured with defaults that use examples in the ‘userContent’ folder for testing. To use data from your own site requires that you customize the site.properties file for your installation environment.
Now you should be ready to install Jenkins and take the system for a spin!
Download a recent version of the Jenkins war file and place it in the JENKINS_HOME folder.
-
Open a command line window and change your working directory to JENKINS_HOME.
-
Launch Jenkins by executing java -jar jenkins.war
-
Once it launches, visit http://localhost:8080/ in your browser to get to the dashboard.
-
Review the console for any error messages (there should not be any if all goes well!)
On Windows, you can even choose to install Jenkins as a service afterwards.
At this point you should have a Jenkins-LSCI server running on you workstation and accessible from your browser at http://localhost:8080/
The dashboard should look similar to the one below:
Note: You will navigate to the various configuration options described below by clicking links on the left side menu or by entering a specific URL in your browser.
-
Go to: http://localhost:8080/configure
- Or click on the ‘Manage Jenkins’->’Configure System’ links
-
In the ‘Jenkins Location’ section enter a Jenkins URL that includes the name of your system instead of ‘localhost’
-
Click the <Save> button to save your changes. This is important as it addresses a bug in Jenkins that can prevent some of the Jenkins-LSCI jobs from working correctly with a minimally configured Jenkins!
Note: This initial server is not configured yet for Security and Groovy. As a result it will not function correctly until the next two steps are completed.
We will next configure a minimal security server that will allow us to test-run the Jenkins-LSCI jobs, but will not be appropriate for a production environment.
Review the Jenkins wiki on how to secure Jenkins for a production use.
-
Go to http://localhost:8080/configureSecurity
- Or click on ‘Manage Jenkins’->’Configure Global Security’ links
-
In the ‘Markup Formatter’ drop down box select ‘Safe HTML’
This allows HTML to be used in the description of projects and parameters and is required for correct display of Jenkins-LSCI job forms.
Access the scriptlet scripts and allow execution on those that are used as build steps in Jenkins-LSCI jobs as follows:
-
Go to: http://localhost:8080/scriptler
- Or click on the ‘Scriptler’ link
-
Click on each of the following scripts and check the ‘Permission’ checkbox as shown in the example below to ‘allow execution by user with RunScript permission’
-
RenameBuildArtifact
-
cpSgeTaskArray
-
trackListMetadata
-
unZipArchive
-
writeXMLProperties_scriptlet
-
HTTP proxy configuration is required if your Jenkins server will be accessing the internet via a proxy server. Proxy configuration is required for installing Groovy via Jenkins and for updating required plugins from the Jenkins update center.
-
To access the HTTP proxy configuration go to: http://localhost:8080/pluginManager/advanced
-
Provide a Server name and port for your HTTP proxy
-
Click the <Advanced> button and provide a test URL to validate your proxy settings
The proxy settings might not take effect until the Jenkins server is restarted.
So exit and restart the Jenkins server.
-
To stop the server type: http://localhost:8080/exit in your browser and click on the ‘Try Posting’ button
-
Or go to the command line console and hit <Ctrl>-C to exit the batch script
-
Restart Jenkins by executing java -jar jenkins.war
Before executing Jenkins-LSCI jobs that utilize Groovy scripts, we need to install Groovy on the server. Jenkins has the ability to auto-install certain tools such as Groovy.
-
Go to: http://localhost:8080/configureTools
- Or click on ‘Manage Jenkins’->’Global Tool Configuration’ links
-
In the ‘Groovy’ section click <Add Installer> and select ‘Install from Groovy Site’
-
Select the most recent version of Groovy
-
Name this Groovy installation: Default
-
Click on the <Save> or <Apply> buttons to save your changes
This will insure that before a Groovy script is executed Jenkins will download and install the requested version. Note the Groovy installation directory:
JENKINS_HOME/tools/hudson.plugins.groovy.GroovyInstallation/Default
To demonstrate the use of Jenkins as an integration and management platform for life-sciences research applications, we provide a set of Jenkins-LSCI jobs that in combination with CellProfiler can be used to build an effective infrastructure for image analysis as used in high content screening applications.
These related Jenkins-LSCI jobs will enable users to:
-
Create a library of re-usable CellProfiler pipelines.
- Specifically users will be able to annotate, upload, review and re-use CellProfiler image processing pipelines
-
Create a library of re-usable CellProfiler image lists
- Specifically users will be able to upload, create and re-use CellProfiler compatible image lists that can be processed through a CellProfiler pipeline
-
Prepare images for CellProfiler high performance image analysis (CellProfiler batch mode) on a Linux cluster
The ‘OS_CONTRIBUTE_PIPELINE’ job allows users to annotate and upload a CellProfiler pipeline to the Jenkins server. The build generates a pipeline summary report combining user and internal pipeline annotations. The pipeline is stored on the Jenkins server and can be used by CellProfiler running on the Jenkins or the user’s desktop.
The following image analysis pipelines are made available with the Jenkins-LSCI system to provide example starter pipelines. Read the associated notes. All pipelines used in Jenkins-LSCI must use the LoadData module to load and associate image files with metadata.
| Pipeline Name | Utility |
|---|---|
| Cytoplasmic Intensity | Determine the cytoplasmic intensity of a fluorescent cell marker |
| Translocation protocol | Determine the nuclear to cytoplasmic ratio of a marker that can translocate |
A few example CellProfiler pipelines can be found under JENKINS_HOME\userContent\cp_pipelines
Consult the readme.txt file included in the cp_pipelines folder for some useful information.
You can test the OS_CONTRIBUTE_PIPELINE job by choosing a ‘user_pipeline’ from the example pipelines. The example pipelines are already configured with the required LoadData import module and source image lists served from a Jenkins URL. As result, you don’t have to supply an ‘example_image_list’ file.
Click <Build> and review the corresponding build summary report.
The ‘OS_CONTRIBUTE_IMAGELIST’ job allows users to create correctly formatted image lists that can be used for processing large number of images through a CellProfiler pipeline. The creation of an image list depends on the correct parsing of imager-specific metadata generated during the image acquisition phase.
- We provide parsers for InCell and Yokogawa imagers.
In the absence of instrument metadata, image lists can be created using the CellProfiler desktop client and uploaded to the Jenkins server for adaptation to formats compatible with high performance cluster analysis.
We provide example InCell and Yokogawa acquisition metadata useful for image list creation in the following folders:
JENKINS_HOME\userContent\cp_pimageLists\InCell
JENKINS_HOME\userContent\cp_pimageLists\Yokogawa
Consult the readme.txt file included in each corresponding (or example) folder for some useful information.
You can test the OS_CONTRIBUTE_IMAGELIST job by generating image lists from the provided example InCell and Yokogawa acquisition metadata files and reviewing the corresponding build summary reports.
To create an image list adapted for another operating system (such as Linux) or path you need to provide a replacement path for either a part of the current image path or for the entire image path.
As an example, let’s assume that the current image path is:
C:\CWorkspace\jenkins_lsci\userContent\InCell\example_01_4_channel
And that the images can also be found on a Linux path at:
/labdata/InCell/ example_01_4_channel
We could assign a PATH_MAPPING by replacing the entire path
Alternatively, we could remove the common path of the mapping (InCell\example_01_4_channel) and replace just the divergent part:
OS_CONTRIBUTE_IMAGELIST is aware of operating system conventions and will replace backward slashes in Windows paths with forward slashes in Linux paths.
The ‘OS_CELLPROFILER_BATCH’ job allows users to generate correctly formatted CellProfiler task array scripts that can be submitted to a Linux cluster for parallel processing. The job builds take as input a reference to an ‘OS_CONTRIBUTE_PIPELINE’ build and a reference to an ‘OS_CONTRIBUTE_IMAGELIST’ build. Theses builds define the two inputs for a typical CellProfiler run, the image processing pipeline and the image list.
Due to the unpredictable, local, Linux cluster environment dependencies of a clustered job, the submission process of the task array is left up to the user. In addition, CellProfiler and its library dependencies must be installed and correctly working on the destination Linux cluster.
The task array is currently formatted for the UNIVA grid engine scheduler and it should require little effort to customize for other environments.
‘OS_CELLPROFILER_BATCH’ can define how images to be processed on the same compute node are grouped.
-
If the CellProfiler pipeline does not define a grouping then images are grouped by simply dividing the image list is ‘chunks’ of 12 images per node (can be configures in site.properties)
-
If the CellProfiler pipeline defines metadata-based grouping of the images (required for correction illumination, time-series, z-stacks and other advanced pipelines) then image list metadata are automatically used to create the specified groups
In addition, ‘OS_CELLPROFILER_BATCH’ allows users to select for processing a subset of images from the image list using image list metadata as filtering criteria.
The input to OS_CELLPROFILER_BATCH jobs are pipelines and image lists created by the corresponding Jenkins-LSCI jobs :
-
OS_CONTRIBUTE_PIPELINE and
-
OS_CONTRIBUTE_IMAGE_LIST
So, you should have at least one of each of these jobs executed before you attempt to build a OS_CELL_PROFILER_JOB.
In addition, a Linux formatted list is required as input. Insure that you create Linux adapted image lists when using the OS_CONTRIBUTE_IMAGELIST jobs.
On the OS_CELLPROFILER_BATCH build form select from the available options:
-
A PIPELINE reference to previous OS_CONTRIBUTE_PIPELINE builds
-
A GENERATED_IMAGE_LIST reference to previous OS_CONTRIBUTE_IMAGELIST builds
-
(Optionally) in section 4 select a subset of the image list by constructing and adding selection criteria from the available metadata of the list
-
Annotate you build
Click the <Build> button and review the resulting build summary report.
If you need additional information on administering and securing your Jenkins server, review the Jenkins wiki:
-
https://wiki.jenkins-ci.org/display/JENKINS/Administering+Jenkins
-
https://wiki.jenkins-ci.org/display/JENKINS/Securing+Jenkins
A Jenkins server installation depends on the core Jenkins (http://jenkins-ci.org) software that is freely available on the web, and a number of required plugins that are also freely available from the Jenkins plugin repository (https://wiki.jenkins-ci.org/display/JENKINS/Plugins).
| Software | Link | Category | Comment |
|---|---|---|---|
| Jenkins (Windows) | http://mirrors.jenkins-ci.org/windows/latest | Installation | Download Jenkins for Windows |
| Jenkins (Linux) | http://jenkins-ci.org/ | Installation | Download Jenkins package for appropriate version of Linux-OS |
Follow the installation directions relevant to the OS on which Jenkins will be running found on the Jenkins wiki
We will refer to the Jenkins installation directory as the JENKINS_HOME. For a typical Windows Jenkins installation JENKINS_HOME may look like: D:\DEVTOOLS\Jenkins
This folder will contain additional standardized folders but may vary slightly depending on the Jenkins plugins used and other configuration options:
JENKINS_HOME
| | config.xml (jenkins root configuration)
| | *.xml (other sidte-wide configuration files)
|
|__jobs
| |
| |__ [JOBNAME] (subdirectory for each job)
|
|__log
|
|__plugins (installed plugins)
|
|__scriptler (reusable groovy scripts)
|
|__userContent (static content that can be served from a URL)
|
|__users (local security and personalization)
|
|__war (server files)
|
|__workspace (temporary workspace for job builds)
|
|__ [JOBNAME] (subdirectory for each job)
Jenkins-LSCI jobs are installed in the JENKINS_HOME/jobs folder.
The configuration file for each job, and the associated scripts and properties are in a corresponding folder with the same name. The typical folder structure is shown in the example below:
JENKINS_HOME
|
|__ jobs
|
|__OS_CONTRIBUTE_PIPELINE
| |config.xml (job configuration file)
|
|__buildProps (build report configuration files)
| | action.csv
| | report.properties
|
|__buildScripts(build scripts not in scriptler)
| | adaptImageList.groovy
| | xdceSlurperMapRecurese.groovy
| | ...(other scripts)
|
|__ builds (build archived results)
|
|__ 1
|__ 2
|__ n...
Note that while the Jenkins server is running you can add new job definitions to the JENKINS_HOME\jobs folder.
However, for these jobs to appear on your server, you will need to reload the job definitions
You can enter the following command http://[yourserver]:[port]/reload in your browser to reload job configurations.