diff --git a/README.md b/README.md index 0277cb1..3f5d2fc 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,15 @@ Continuous Integration Scripts for Satellite 6 ============================================== +- - - + +* Author: Patrick C. F. Ernzer +* Email: +* Revision: 0.4 +* Add puppet-only workflow + +- - - + * Domain Architect: Eric Lavarde * Email: * Consultant: Patrick C. F. Ernzer @@ -35,14 +44,14 @@ The CII system consists of the following components: * Red Hat Satellite 6. This acts as the repository for Red Hat-provided and 3rd party packages, kickstarts and puppet modules. The Foreman module is also used to deploy test clients. * A virtualisation infrastructure to run test clients. I have used KVM/Libvirt, VMware and RHEV in different engagements. -The architecture is shown in [this YeD diagram](https://github.com/RedHatEMEA/soe-ci/blob/master/Engineering%20Platform.graphml). +The architecture is shown in [this YeD diagram](https://github.com/RedHatEMEA/soe-ci/blob/master/Engineering%20Platform.graphml). View it with [yEd](http://www.yworks.com/downloads#yEdLive). ## Setup The following steps should help you get started with CII. ### Jenkins Server -NB I have SELinux disabled on the Jenkins server as I ran into too many problems with it enabled and didn't have the time to fix them. +NB SELinux in _Enforcing_ mode with the RHEL7 _Targeted_ policy works just fine, #### Installation @@ -87,8 +96,10 @@ NB I have SELinux disabled on the Jenkins server as I ran into too many problems * Restart Jenkins * Add the `jenkins` user to the `mock` group (`usermod -a -G mock jenkins`). This will allow Jenkins to build RPMs. * Create `/var/www/html/pub/soe-repo` and `/var/www/html/pub/soe-puppet` and assign their ownership to the `jenkins` user. These will be used as the upstream repositories to publish artefacts to the satellite. + * Create `/var/www/html/pub/soe-puppet-only` for the puppet only workflow and assign their ownership to the `jenkins` user. These serve for the puppet only workflow. * `su` to the `jenkins` user (`su jenkins -s /bin/bash`) and use `ssh-keygen` to create an ssh keypair. These will be used for authentication to both the git repository, and to the satellite server. * Create one build plan per release you want to build for in Jenkins by creating the directory `/var/lib/jenkins/jobs/SOE- (e.g. SOE-el7)` and copying in the [config.xml] file + * create a similar build plan for the puppet only workflow, omitting the RPM stuff. * Check that the build plan is visible and correct via the Jenkins UI, you will surely need to adapt the parameter values to your environment. * you might need to reload the configuration from disk using 'Manage Jenkins -> Reload Configuration from Disk'. diff --git a/common.sh b/common.sh index 9ab8139..0a3b8cc 100755 --- a/common.sh +++ b/common.sh @@ -12,7 +12,7 @@ SRPMBUILD_ERR=6 MODBUILD_ERR=7 function tell() { - echo "${@} [$(date)]" | fold -s + echo "${@} [$(date)]" | fold --spaces --width=240 } function info() { diff --git a/config-puppet-only.xml b/config-puppet-only.xml new file mode 100644 index 0000000..6efca00 --- /dev/null +++ b/config-puppet-only.xml @@ -0,0 +1,259 @@ + + + + Generic plan to create puppet module environments + false + + + + 7 + 100 + -1 + -1 + + + + + + PUSH_USER + The username to use when pushing artefacts to the satellite. The build job will need to be able to ssh to the satellite as this user so you need to have keys set up. + jenkins + + + RSA_ID + RSA ID of the Jenkins user. + /var/lib/jenkins/.ssh/id_rsa + + + KNOWN_HOSTS + Known hosts file of the Jenkins user. + /var/lib/jenkins/.ssh/known_hosts + + + SATELLITE + The FQDN of the Red Hat Satellite. + satellite.example.com + + + ORG + This is the name of the organisation you are using on the Red Hat Satellite. + Default_Organization + + + PUPPET_REPO_ID + The numeric ID of the puppet module repository on the Red Hat Satellite that you want to use for your custom puppet modules. + + + + TESTVM_HOSTCOLLECTION + The name of the Satellite 6 host collection containing your test machines. + Test Servers puppet only + + + TESTVM_ENV + The numeric ID of the environment you want to promote CI builds to. This will typically be a 'Crash' or 'EngTest' environment. + +You can find the IDs of your environments using: +hammer lifecycle-environment list --organization "Default_Organization" + + + + TEST_ROOT + This is the root password of your test machines, as set in your hostgroup definition. + redhat + + + MODAUTHOR + This is the authorname for your puppet modules. + acme + + + PUPPET_REPO + This is the location of the puppet repository on the Jenkins server used to export puppet modules to the satellite server. + /var/www/html/pub/soe-puppet-only + + + CV + This is the name of the Content View on the satellite that you want to publish your build into. + cv-puppet-only + + + CV_PASSIVE_LIST + A comma separated list of content view names to publish and optionally promote as part of the build. +Those CVs won't get new Puppet Modules assigned to them, meaning you must (and are free to) assign (and unassign) them yourself directly in Satellite. + + + + CCV_NAME_PATTERN + Enter here a name pattern if you want Composite Content Views using the given CV(s) to be updated with the latest version and as well published and promoted. +CCVs respecting the pattern but not containing any CV will not be touched. +Example of pattern: "ccv-test-*" + + + + CONDITIONAL_VM_BUILD + If this parameter is true/set, only the hosts existing in Satellite with a comment matching +the modified Puppet module, RPM package or kickstart file, each surrounded by hashes (#), will be built and tested. +<br/> +Examples: <code>#ssh#bats#kickstart.erb#</code> +<br/> +<b>CAUTION:</b> be aware that in this case, no host will be tested if only the scripts have been modified. +<br/> +If unset/false, all existing hosts, within the given Host Collection, will be rebuilt independent of changes done in Git. + true + + + MOCK_CONFIG + This will be passed to mock --root +e.g. mock --root rhel-6-x86_64 +needed to ensure we can build RPMs against more then just the default mock chroot. + rhel-7-x86_64 + + + PUPPET_DONE_SLEEP + Number of seconds to wait in puppet-done-test.sh +if you set up things with puppet that take a while to settle (e.g. NTP), you may want to wait a bit (e.g. 75 seconds is good for ntp to settle) + 0 + + + + + + + + 2 + + + git@localhost:soe-ci.git + + + + + */master + + + false + + + + scripts + + + + + 2 + + + git@localhost:acme-soe.git + + + + + */master + + + false + + + + soe + + + + + + true + false + false + false + + + * * * * * + false + + + false + + + echo "#####################################################" +echo "# BUILDING PUPPET MODULES #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/puppetbuild.sh ${WORKSPACE}/soe/puppet + + + + echo "#####################################################" +echo "# PUSH PUPPET MODULES TO SATELLITE #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/puppetpush.sh ${WORKSPACE}/artefacts +echo "#####################################################" +echo "# REPUBLISH CONTENT VIEW #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/publishcv.sh + + + echo "#####################################################" +echo "# MAKING SURE TEST VMs ARE ON #" +echo "#####################################################" + +/bin/bash ${WORKSPACE}/scripts/starttestvms.sh + + + + echo "#####################################################" +echo "# PUSHING TESTS to TEST VMS #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/pushtests.sh + + + + + + SUCCESS + 0 + BLUE + true + + + SUCCESS + 0 + BLUE + true + + + + echo "#####################################################" +echo "# POWER OFF TEST VMs #" +echo "#####################################################" +/bin/bash ${WORKSPACE}/scripts/powerofftestvms.sh + +echo "#####################################################" +echo "# CLEAN-UP AFTER SUCCESS BUILD #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/cleanup.sh + + + + + + + + test_results/*.tap + false + false + false + false + true + false + false + false + true + true + false + false + false + false + + + + diff --git a/config.xml b/config.xml index 64a623b..39595a1 100644 --- a/config.xml +++ b/config.xml @@ -4,6 +4,14 @@ Generic plan to create Red Hat Enterprise Linux builds false + + + 60 + 30 + -1 + -1 + + @@ -107,9 +115,15 @@ If unset/false, all existing hosts, within the given Host Collection, will be re MOCK_CONFIG This will be passed to mock --root -e.g. mock --root rhel-6-x86_64 +e.g. mock --root rhel-7-x86_64 needed to ensure we can build RPMs against more then just the default mock chroot. - rhel-6-x86_64 + rhel-7-x86_64 + + + PUPPET_DONE_SLEEP + Number of seconds to wait in puppet-done-test.sh +if you set up things with puppet that take a while to settle (e.g. NTP), you may want to wait a bit (e.g. 75 seconds is good for ntp to settle) + 75 @@ -163,10 +177,9 @@ needed to ensure we can build RPMs against more then just the default mock chroo false false - - * * * * * - false - + + H H(3-5) * * * + false @@ -258,6 +271,32 @@ echo "#####################################################" true true + + + + echo "#####################################################" +echo "# POWER OFF TEST VMs #" +echo "#####################################################" +/bin/bash ${WORKSPACE}/scripts/powerofftestvms.sh + + + echo "#####################################################" +echo "# CLEAN-UP AFTER SUCCESS BUILD #" +echo "#####################################################" +/bin/bash -x ${WORKSPACE}/scripts/cleanup.sh + +echo "#####################################################" +echo "# run cvmanager #" +echo "# on Satellite or Jenkins box? #" +echo "#####################################################" + + + + true + false + false + + diff --git a/publishcv.sh b/publishcv.sh index 90ba484..7e26ce0 100755 --- a/publishcv.sh +++ b/publishcv.sh @@ -14,7 +14,7 @@ # has anything changed? If yes, then MODIFIED_CONTENT_FILE is not 0 bytes if [[ ! -s "${MODIFIED_CONTENT_FILE}" ]] then - echo "No entries in ${MODIFIED_CONTENT_FILE} no need to continue with $0" + info "No entries in ${MODIFIED_CONTENT_FILE} no need to continue with $0" exit 0 fi diff --git a/puppet-done-test.sh b/puppet-done-test.sh index 3c175a0..4310912 100755 --- a/puppet-done-test.sh +++ b/puppet-done-test.sh @@ -1,5 +1,19 @@ #!/bin/bash +NO_ARGS=0 +OPTERROR=65 + +usage() +{ + echo "Usage: $0 -s " + echo " s = number of seconds to sleep" + exit 1 +} +if [ $# -eq "$NO_ARGS" ] +then + usage +fi + look_for_state() { while [[ ! -d /var/lib/puppet/state ]] @@ -43,10 +57,23 @@ look_for_finished() date -# waiting just over one minute for puppet to start -# this script gets run on first boot after a system was installed -echo "waiting 75 seconds (one minute and a bit) before checking on puppet" -sleep 75 +while getopts "c:n:s:h" Option +do + case $Option in + s) SLEEP=${OPTARG} + ;; + h) + usage + ;; + *) echo "Non valid switch" + ;; + esac +done + +# if you run puppet on boot, sleep 75 seconds (just over a minute) +# if this is a run in the puppet only workflow, no real need to sleep +echo "waiting ${SLEEP} seconds before checking on puppet" +sleep ${SLEEP} # /var/lib/puppet/state gets created on first run? look_for_state diff --git a/pushtests.sh b/pushtests.sh index efe4340..fc1f94f 100755 --- a/pushtests.sh +++ b/pushtests.sh @@ -17,9 +17,6 @@ for I in "${TEST_VM_LIST[@]}"; do vmcopy[$I]=$I; done WAIT=0 while [[ ${#vmcopy[@]} -gt 0 ]] do - sleep 10 - ((WAIT+=10)) - info "Waiting 10 seconds" for I in "${vmcopy[@]}" do info "Checking if test server $I has rebooted into OS so that tests can be run" @@ -33,7 +30,9 @@ do tell "Success!" unset vmcopy[$I] else - tell "Not yet." + tell "Not yet. Sleeping 30 seconds." + sleep 30 + ((WAIT+=30)) fi done if [[ ${WAIT} -gt 6000 ]] @@ -53,9 +52,9 @@ export TEST_ROOT for I in ${TEST_VM_LIST[@]} do # Check the host's entitlements - info "Checking entitlements for test server $I" + inform "Displaying entitlements for test server $I" ssh -q -l ${PUSH_USER} -i ${RSA_ID} ${SATELLITE} \ - "hammer content-host info --name ${I} --organization \"${ORG}\"" + "hammer host info --name ${I}" info "Setting up ssh keys for test server $I" sed -i.bak "/^$I[, ]/d" ${KNOWN_HOSTS} # remove test server from the file @@ -80,9 +79,13 @@ do scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${RSA_ID} \ ${WORKSPACE}/scripts/puppet-done-test.sh root@$I: + # run puppet once, this will skip if puppet already running, so no need for if clause + ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${RSA_ID} root@$I \ + "puppet agent -t" + # wait for puppet to finish ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${RSA_ID} root@$I \ - "/root/puppet-done-test.sh" + "/root/puppet-done-test.sh -s ${PUPPET_DONE_SLEEP}" info "Installing bats and rsync on test server $I" if ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${RSA_ID} root@$I \ diff --git a/starttestvms.sh b/starttestvms.sh new file mode 100755 index 0000000..2d7de5d --- /dev/null +++ b/starttestvms.sh @@ -0,0 +1,79 @@ +#!/bin/bash + +# Instruct Foreman to start the test VMs (just in case they are off) +# +# e.g ${WORKSPACE}/scripts/starttestvms.sh 'test' +# +# this will tell Foreman to rebuild all machines in hostgroup TESTVM_HOSTGROUP + +# Load common parameter variables +. $(dirname "${0}")/common.sh + +if [[ -z ${PUSH_USER} ]] || [[ -z ${SATELLITE} ]] || [[ -z ${RSA_ID} ]] \ + || [[ -z ${ORG} ]] || [[ -z ${TESTVM_HOSTCOLLECTION} ]] +then + err "Environment variable PUSH_USER, SATELLITE, RSA_ID, ORG " \ + "or TESTVM_HOSTCOLLECTION not set or not found." + exit ${WORKSPACE_ERR} +fi + +get_test_vm_list # populate TEST_VM_LIST + +# TODO: Error out if no test VM's are available. +if [ $(echo ${#TEST_VM_LIST[@]}) -eq 0 ]; then + err "No test VMs configured in Satellite" +fi + +# rebuild test VMs +for I in "${TEST_VM_LIST[@]}" +do + info "Making sure VM ID $I is on" + + _PROBED_STATUS=$(ssh -q -l ${PUSH_USER} -i ${RSA_ID} ${SATELLITE} "hammer host status --id $I" | grep Power | cut -f2 -d: | tr -d ' ') + + # different hypervisors report power status with different words. parse and get a single word per status + # KVM uses running / shutoff + # VMware uses poweredOn / poweredOff + # add other hypervisors as you come across them and please submit to https://github.com/RedHatEMEA/soe-ci + + case "${_PROBED_STATUS}" in + running) + _STATUS=On + ;; + poweredOn) + _STATUS=On + ;; + up) + _STATUS=On + ;; + shutoff) + _STATUS=Off + ;; + poweredOff) + _STATUS=Off + ;; + down) + _STATUS=Off + ;; + off) + _STATUS=Off + ;; + *) + echo "can not parse power status, please review $0" + esac + + if [[ ${_STATUS} == 'On' ]] + then + info "Host $I is already on." + elif [[ ${_STATUS} == 'Off' ]] + then + info "Host $I is off, switching it on." + ssh -q -l ${PUSH_USER} -i ${RSA_ID} ${SATELLITE} \ + "hammer host start --id $I" + else + err "Host $I is neither running nor shutoff. No action possible!" + # exit 0 while testingi for issue #50, + # allows for manual rebooting of the test VM(s) + exit 0 + fi +done