Skip to content

Latest commit

 

History

History
232 lines (188 loc) · 20.3 KB

Builds.md

File metadata and controls

232 lines (188 loc) · 20.3 KB

The Build Infrastructure of Xtext

Xtext uses Gradle to build the Java projects that are independent of Eclipse, Maven to build the Maven plug-ins, and Tycho to build the Eclipse plug-ins. The builds are executed on a Jenkins server.

An overview of the project dependencies and their build systems is shown here:

xtext-lib
(Gradle*)
    |
    |
xtext-core
(Gradle*)
    |
    |
xtext-extras________ ___________
 (Gradle*)          |           |
    |               |           |
    |               |           |
xtext-eclipse   xtext-web   xtext-maven
 (Tycho)        (Gradle)      (Maven)
    |
    |
xtext-xtend
(Gradle + Maven + Tycho)
    |
    |
xtext-umbrella
 (Tycho)

The Gradle projects marked with * include generated Tycho builds for creating p2 repositories as described below. For xtext-xtend, only the Tycho build depends on xtext-eclipse, while the Gradle and Maven builds just require xtext-extras.

The Build Systems

Gradle

The projects that include Gradle builds are xtext-lib, xtext-core, xtext-extras, xtext-web, and xtext-xtend. These repositories share some common concepts:

  • settings.gradle in the project root lists the subprojects to be included in the build.
  • build.gradle in the project root applies commonly used plugin-ins such as java and applies configuration scripts from the gradle directory to all subprojects. The intent of each configuration script is explained in a file header comment.
  • Each subproject has its own build.gradle that declares a title, description, and dependencies.
  • The project version as well the versions of common dependencies are declared in versions.gradle.
  • The remote repositories from which dependencies are drawn are declared in upstream-repositories.gradle. Depending on the useJenkinsSnapshots property, dependencies to upstream Xtext projects are resolved either against the Maven repositories created by Jenkins build jobs or against public snapshots.
  • The Xtend code is compiled with the xtext-gradle-plugin. Since this plug-in has dependencies to projects that we want to compile, we have a bootstrap problem, which is solved using a shared dependency configuration declared in the root project (see bootstrap-setup.gradle). For xtext-web the Java code generated by Xtend is not tracked in the git repository. For the other repositories the generated code is tracked in order to verify changes to the compiler. Here the compiler is activated with the compileXtend property.
  • All build artifacts are installed into a local Maven repository build/maven-repository (see maven-deployment.gradle). The pom.xml files are generated with the maven Gradle plug-in.
  • The Gradle projects xtext-lib, xtext-core, xtext-extras, and xtext-xtend are published both as Maven artifacts and as Eclipse plug-ins. Here the MANIFEST.MF files are maintained manually, which means they need to be kept consistent with the build.gradle configuration. The build timestamp is inserted into the version qualifier of these manifests automatically (see manifest-gen.gradle). For the generated source bundles, the manifest is completely generated from the project metadata. The xtext-web project is not published for Eclipse, but here we generate OSGi-compatible manifests using the osgi Gradle plug-in.
  • The builds of xtext-lib, xtext-core, and xtext-extras include a Gradle plug-in that can generate a second Tycho-based build into the releng directory by running ./gradlew generateP2Build (see p2-deployment.gradle). This Tycho build creates a local p2 repository in build/p2-repository. This approach eliminates the need to manually keep these Tycho builds consistent with the Gradle builds, e.g. regarding version numbers.
  • The Xtext generator can be invoked from Gradle for the languages contained in the Xtext projects (see mwe2-workflows.gradle). For example, test languages can be generated with ./gradlew generateTestLanguages.

Maven Plug-ins

Maven plug-ins are built by the xtext-maven project and by the org.eclipse.xtend.maven.* subprojects in xtext-xtend.

  • The root pom.xml (in xtext-xtend it's named maven-pom.xml) lists the subprojects to be included in the build.
  • The parent modules org.eclipse.xtext.maven.parent and org.eclipse.xtend.maven.parent define the common plug-in configuration and metadata.
  • Depending on the chosen profile, dependencies to upstream Xtext projects are resolved either against the Maven repositories created by Jenkins build jobs (useJenkinsSnapshots profile) or against public snapshots (useSonatypeSnapshots profile).
  • The deploy goal installs the artifacts into a local Maven repository build/maven-repository.

Tycho

The Eclipse plug-ins and features of xtext-eclipse and xtext-xtend are built with Tycho.

  • The root pom.xml (in xtext-xtend it's named tycho-pom.xml) lists the subprojects to be included in the build.
  • The parent modules org.eclipse.xtext.tycho.parent and org.eclipse.xtend.tycho.parent define the common plug-in configuration.
  • The target platform modules org.eclipse.xtext.target and org.eclipse.xtend.target define from where to fetch the dependencies. This includes external dependencies such as EMF as well as upstream Xtext projects, which are referenced via the p2 repositories created by Jenkins build jobs.
  • The modules org.eclipse.xtext.p2-repository and org.eclipse.xtend.p2-repository list the plug-ins and features to be exported. The resulting p2 repositories are copied to build/p2-repository.
  • The Xtend code is compiled with the xtend-maven-plugin.

Another Tycho build is found in xtext-umbrella, with a very similar structure as described above. The difference is that the umbrella does not build its own plug-ins, but fetches all plug-ins from the other projects, builds some cross-project features (Xtext and Xtend SDKs), and exports them to a common p2 repository. The resulting p2 repository is used to create the Xtext update site for Eclipse.

The Build Servers

Jenkins

Each project has its own multibranch pipeline on the Jenkins build server. The server polls the git repositories every few minutes and automatically refreshes its build jobs: a new job is created for each new branch that is found, and the jobs of branches that do not exist anymore are deleted. This approach gives committers a very convenient way to test their changes on the server without affecting the main development streams (master and maintenance branches).

The actual build job description is written in a Jenkinsfile. This file is included in the git repositories and thus can be modified per branch. Basically it defines the commands to execute for each build, where to find test results, and which artifacts to make available.

The build artifacts of each project are made available as a Maven repository, a p2 repository, or both. The build jobs of downstream projects are configured to consume the build artifacts of the projects they depend on. Gradle and Maven plug-in builds pick up their dependencies from the Maven repositories generated for their upstream projects, while Tycho builds use the p2 repositories.

Jenkins

The actual publishing is done on a Jenkins build server by two build jobs: xtext-snapshots for nightly snapshots and xtext-release for milestones and releases. Both employ the publishing project, which does the following:

  1. Download Maven artifacts with a specified version from the repositories that are made available by the various Jenkins build jobs as described above.
  2. Sign the jar files with the remote service of the Eclipse Foundation.
  3. Create additional signature files for publishing to remote Maven repositories.
  4. Upload the results either to a snapshot repository or to a staging repository.
  5. Download the common p2 repository from the Jenkins build job of xtext-umbrella as a zip file and unzip it.
  6. Sign the jar files that have not already been signed in step 2 (results of the signing service are reused if the input files are equal).
  7. Copy the signed jars and p2 repository metadata to the build-result directory and zip it.
  8. Generate ant scripts for deploying the p2 repository to the Eclipse public download area (this might be improved in the future).

The Release Process

Overview of the Process

  1. Development phase
    1. Work on new features and issues.
    2. Publish milestones as required, e.g. when important new features have been implemented.
  2. Release phase
    1. Stop implementing new features, but work on bug fixes.
    2. Create a release branch and publish release candidate milestones if required.
    3. Lift the version number on the master branch and start the development phase for the next major/minor release (in parallel to the following steps).
    4. Create a test plan and test both old and new features.
    5. Prepare release notes.
    6. Merge or cherry-pick the bug fixes between the master branch and the release branch.
    7. Publish the release.
    8. Create and push git tags.
    9. Close the GitHub milestone and create new milestones.
  3. Maintenance phase
    1. Work on bug fixes on the master branch.
    2. When it appears appropriate, create a service release branch and cherry-pick the bug fixes.
    3. Publish a service release.
    4. Create and push git tags.
    5. If necessary, repeat the maintenance phase.

When major or minor releases are done, there should be a time period of at least one week when only important fixes are added to the code base to be released. This can be achieved by creating the release branch and committing the fixes directly to that branch. The release branch is merged back into master just before the actual release is done so the fixes are available there, too.

Preparing Milestones and Releases

To initiate a release build start the build job release-prepare-branches. This job automates the process to a large degree. Once it is triggered it takes about 4-5 hours to complete. If you are interested in the details, read the section "What happens when the release-prepare-branches job is executed".

After everything above has run smoothly, it is finally time for some manual steps again. We will try to integrate some of them into the automated release pipeline from time to time.

  1. Check that everything was promoted correctly:
  2. Promote staged release on s01.oss.sonatype.org. Can only be done by Xtext release engineer with the right credentials at sonatype
    • Log in with your sonatype user.
    • Select Staging Repositories
    • Search for orgeclipsextext-NNNN and orgeclipsextend-NNNN with status open
    • Select both and perform the Close toolbar action
    • Wait until the checks have run successfully
    • Select both repositories again the perform the Release toolbar action
  3. The sign-and-deploy job waits for manual confirmation until the staging repositories are promoted. Open the console and look for 
    [Pipeline] input
Only continue after ensuring OSSRH staging repositories are closed an promoted
Delete Branches! or Abort
    Click on Delete Branches to continue
  4. Contribute release to Simrel Aggregation Build
    • Make sure the commit message is correct! (Correct SimRel Target etc)
    • Merge the open Gerrit change for the simrel/org.eclipse.simrel.build repository
  5. Release Notes (Only on final release)
    • Create and review release notes
    • Update committer stats for Xtext (without Xtend)
      • Run committer-activity-count job
      • Parameters are release tags (last one and current)
      • Look into the build artifact committer-activity-xtext.txt and put stats into the release post
      • Mark first-time contributors (look at older release notes if they appear, or run the committer-activity-count job with an old tag as FROM_VERSION param)
    • Update committer stats for Xtend
      • Run git shortlog -s -n -e v<PREV_RELEASE>..v<NEW_RELEASE> in the xtext-xtend repo
  6. Publish websites (Only on final release)
    • Remove published: false from release post
    • Create PR to merge branch website-master into website-published
  7. Update Marketplace entries (Only on final release)
  8. Update Marketplace composite site (Only on final release)
  9. Check availability of artifacts on Maven Central
    • It can take some hours until the artifacts are mirrored to Maven Central. Note: search.maven.org might not show the new release immediately when it is available. Check the availability of the artifacts for example with mvn dependency:get -U -DgroupId=org.eclipse.xtext -DartifactId=org.eclipse.xtext -Dversion=<VERSION>
  10. As soon as Maven Central is updated - send notifications
    • Newsgroup / Forum
    • Mailing list
    • Gitter
    • Twitter
    • Blog (for releases)
  11. Add / Update xtext-reference-projects (Only on final release)
  12. Adjust the bootstrap version to use the newly produced milestone / release
  • Note: The builds will fail if the newly promoted artifacts are not available on maven central yet.
  • Run the bot-updates job
  • Use XTEXT_BOOTSTRAP_VERSION for the UPDATE_TYPE parameter
  1. xtext-apidiff (Only on final release)

What happens when the release-prepare-branches job is executed

Read this section when you are interested in some details about the steps that are performed when triggering the release-prepare-branches job. The job will perform the following:

  1. Check out a clean copy of all Xtext repositories
  2. Create branches for the release on all repositories (milestone_«version» for milestones, and release_«version» for releases)
  3. Adjust target platforms and pom.xml files to fetch from upstream job builds
  4. Set the Xtext version into gradle/versions.gradle files
  5. Set the Xtext version into version property of pom.xml files where needed
  6. Commit all changes
  7. Tag the commit with the tag name v«version»
  8. Push the release branches

When the job finishes, all release branches are prepared. Now Jenkins will pick up the new branches and initiate builds for all repositories. Note that the first build for all jobs except for xtext-lib will fail then, since the downstream jobs won't find the required artifacts from a successful upstream build at that time. But the jobs will trigger their downstream jobs automatically. So when xtext-lib was successful, it will trigger xtext-core, which triggers xtext-extras on success and so on. It will take some hours until the last build, xtext-umbrella is triggered for the release build. Usually this is just a matter of waiting, but keep an eye on the builds. A build might fail due to a flaky test, or a missed performance goal in a test. Then just re-trigger the build.

Finally, when xtext-umbrella was build successfully, all release artifacts were build and they are ready for publishing. The xtext-umbrella job will now trigger the sign-and-deploy job with the appropriate parameters. This will do:

  1. Sign all artifacts with Eclipse Signing Service (for p2 artifacts) and with GPG (for Maven artifacts)
  2. Upload all artifacts to OSSRH
  3. Deploy the p2 repository to the Xtext Downloads location on the project storage
  4. Unzip the repository to its location on the p2 update site location (directory below the composite site location)
  5. Update the p2 composite site descriptors
  6. Trigger an update of the SimRel repository contribution with the release-simrel-update job
  7. For the final release: Trigger lifting of the Xtext version to the next minor release version in the projects and raise Pull Requests for the update. This is done with the bot-updates job.

Lifting the Version Number

Lifting the version number is performed with the bot-updates job. Choose the option XTEXT_VERSION for parameter UPDATE_TYPE and enter the version in for parameter UPDATE_VALUE.

When performing a release this will be automatically triggered by the sign-and-deploy job and increment the current version from the master branch to the next minor version.

For bugfix or major releases the job has to be triggered manually.

GitHub Milestones

We use GitHub milestones to communicate when bug fixes and new features will be available and to generate a list of resolved issues for the release notes. When a major or minor release is done, the corresponding GitHub milestone should be closed. If there are open issues left in that milestone, they should be removed from it or assigned to another milestone.