-
Notifications
You must be signed in to change notification settings - Fork 59
Maintenance
This is partly for community build maintainers on the Scala team, and partly for project authors interested in having their projects in the community build and keeping the version included current.
You'll want to be familiar with the basics of our CI infrastructure, as documented at https://github.com/scala/scala-jenkins-infra/blob/master/doc/overview.md. Most community build maintenance tasks only involve making commits in the scala/community-builds repo, but some types of improvements may require scala/scala-jenkins-infra changes.
You'll need an account on our Jenkins instance (https://scala-ci.typesafe.com) with enough power to start jobs, cancel jobs, and edit configs. (Permanent config changes are made through scala-jenkins-infra, but it's sometimes helpful to make temporary changes on the fly.)
Getting set up to make scala-jenkins-infra is harder; you might wait and see if you need it. See the doc in that repo.
dbuild is the meta-build-tool that is the foundation of the community build. (It was also used to build version 1 of Lightbend's Reactive Platform.)
You'll need at least a basic grounding in what dbuild is about; so at least skim through the dbuild manual at https://typesafehub.github.io/dbuild/ . If it isn't clear 1) why something like dbuild is needed for this, and 2) basically how dbuild does its job, talk to someone on the Scala or RP teams about it.
Don't forget to login into Jenkins, or you won't see all of the options that are available. Jenkins is exceptionally eager to log you out again, so prepare to find yourself clicking that "login" link a lot. (Help wanted: find out why that is and fix it?)
There are four community builds: the 2.11.x (JDK6) and 2.11.x-jdk8 (JDK8) builds in the 2.11.x tab in Jenkins, and the 2.12.0 and 2.12.x builds in their respective tabs.
Except for 2.11.x-jdk8, the builds run nightly as part of the release-main flows on each branch. When you're actively maintaining the builds, you'll be triggering a lot of runs manually. The "rebuild last" button is handy for this.
Each build has parameters, with an appropriate set of defaults. When manually triggering a build, you'll often want to change some of the following:
-
repo_user: if you're testing a change in your fork of the community-builds repo, change this fromscalato your own GitHub user name -
repo_ref: your change in your fork is presumably in a feature branch, so put the branch name here -
version: this is a Scala version number such as2.12.x-933bab2-nightly- dbuild caches results from previous runs of the community build, so when testing changes, your first choice will to be use a version number for which a recent fully-green build exists.
Manually aborting a community build through the Jenkins UI doesn't lose any completed and cached builds; so if you have additional changes you want to test, it's OK to abort an existing run and start a new one with your additional changes.
As mentioned already, there are four builds: 2.11.x on JDK6 (2.11.x branch in this repo), 2.11.x on JDK8 (2.11.x-jdk8 branch in this repo), and 2.12.0 and 2.12.x on JDK8 (2.12.0 and 2.12.x branches in this repo).
The contents of the various branches have now diverged substantially. Only the 2.12.x build is still actively developed; the older builds now receive only minimal maintenance.
On the 2.12.0 branch, all included projects are frozen at fixed SHAs. This speeds rebuilding and narrows down the possible causes of failures.
We manually merge changes from 2.11.x to 2.11.x-jdk8 and from there to 2.12.x. There isn't any schedule or automation for this, it's just something we do when we remember to.
After Scala 2.12.0 is released, we'll create a 2.13.x branch in the scala/scala repo and a corresponding community build.
Both 2.11 builds run on jenkins-worker-behemoth-1 only; the 2.12 builds run on jenkins-worker-behemoth-2 only. They are assigned to particular nodes to get the greatest advantage of dbuild's build caching.
You'll want to look over the issues in https://github.com/scala/community-builds/issues to get an idea of what shortcomings currently exist. The community build is capable of absorbing a practically unlimited amount of time and effort if you let it, so it's often necessary to make cost/benefit tradeoffs between getting things right and getting them "good enough".
In an ideal world:
- every significant Lightbend/EPFL library would be included
- every popular third-party open-source Scala library would be included
- and perhaps also some important Scala codebases that aren't themselves libraries
- every included project would be tracked from a recent yet stable branch
- every time we forked a project to make it work, we would notify the maintainers and make sure they incorporate the needed changes, so we can eliminate the fork and go back to tracking a branch
- every included project's test suite would be run
and so on. In practice, counterexamples are abundant.
Regardless, as long as we include lots of projects, with lots of code, in reasonably recent versions, we're doing pretty well. We're not trying to provide the entire Scala community with CI.
If you're a library author, yes.
If you're maintaining the community build as a whole, go the Jenkins route first and get familiar with that as it's usually more practical. You can decide later whether to supplement Jenkins with local runs.
You might want to set JAVA_HOME first. The 2.11.x community build assumes Java 6; the others, Java 8.
After picking a Scala version you want to test against, do e.g.:
version=2.12.x-933bab2-nightly ./run.sh
While you wait, make yourself a sandwich. (Even better, make yourself a sandwich-making machine.)
You can build just a subset if you want:
version=2.12.x-933bab2-nightly ./run.sh scalatest,scalacheck,specs2,utest
This will build only the listed projects and their dependencies. (In the Jenkins GUI, use the projects setting for this.)
Every so often, check to see if we're using the latest Scala and the latest sbt to do extractions and builds, and bump the versions if not. Use git grep to find all the places that need to be updated.
What typically goes wrong with the community build?
Over time, the builds get slower: as we add more projects, as the projects themselves grow, as we add more resolvers to the config file, etc.
In PRs such as https://github.com/scala/scala-jenkins-infra/pull/129 we try to set the timeouts high enough so that the builds will normally pass, but low enough so that if builds start taking abnormally long, we are made aware of it, and are forced to make a conscious decision about whether to just increase the timeouts again, or try to figure out whether there's some performance problem we ought to tackle directly.
If a build does time out, usually you can just run it again with the same parameters, and the new run will finish much faster thanks to dbuild's caching.
Keep an occasional (weekly?) eye on the free space numbers at https://scala-ci.typesafe.com/computer/. (You need to be logged in to Jenkins to see the numbers. Use "Launch slave agent" to fire up the behemoths if needed.)
It is normal for disk usage to creep up slowly over time, and eventually a behemoth will need cache clearing, but if we're running out of disk space more than very occasionally, the root cause should be found and fixed.
As you'll see in project-refs.conf, we actively track current development branches of many projects, so if a build fails, the cause isn't always your last change to the build config. It might just be because some project's code changed since the last time you built and tested it.
In response, you can either:
- continue tracking but add something to
community.dbuildto address the problem (e.g. by disabling a subproject, changing the build settings, etc.) - freeze at an older rev (perhaps also forking to make a needed change)
- when depending on an old rev it's better to depend on a version tag than a raw SHA
- if you fork, include in your comment what the fork point was and why; be nice to future you and future maintainers
We encourage upstream maintainers not to leave intermittently failing tests in their suites, but it happens sometimes. Try to report the problem upstream.
If you must disable tests, try to do it just for an offending subproject, if there is one, rather than the whole build.
Some people's builds are a grab bag of stable vs unstable, official vs experimental stuff. If a failure can be isolated to a particular subproject, and if other people don't depend on it, you can just skip it (see examples in common.conf).
If a particular project regresses often, it may be worth contacting the project maintainer and asking (or just looking for yourself on GitHub) if there's a less volatile branch we could track. We're not trying to provide the entire Scala community with CI, after all.
e.g., they might be using -Xfatal-warnings. It's rarely worth the effort to maintain that high a standard in the context of the community build. There are examples in community.dbuild of how to disable this.
or the build might be picky about Scala version numbers and get confused by the unusual/unexpected scalaVersion and scalaBinaryVersion values that are used in the community build.
or the build might be sensitive to sbt version. normally we use the latest sbt version to build everything, but that might fail for certain projects. common.conf has examples of how to force a lower sbt version for a certain project
If a dependency of a project's build definition (typically an sbt plugin, or a library that plugin depends on) can't be found, you'll need to add a resolver to the Artifactory config on scala-ci.
You might also need to add a resolver if a project has a binary Java dependency that isn't on Maven Central.
if you see an error message like "org.scalacheck#scalacheck from scalacheck and scalacheck, both visible in space “default”” a typical cause is that you have an upgraded to a version of a library that has added Scala.js support, and dbuild is seeing both JVM and JS artifacts. usual fix: override extra.projects and/or extra.exclude
It's often possible to resolve a version incompatibility by forking a project and patching its source a little bit. If that happens, try to contribute the change upstream, so your fork won't be needed anymore.
It's difficult to know in advance whether upgrading library A will trigger a cascade of upgrades and regressions. Sometimes upgrades multiply, problems multiply, and you end up stuck. If you fail, tell the story of your failed attempt in a ticket, in case someone wants to try again eventually.
If an intractable version conflict occurs, the build may need to split into multiple spaces. See next section.
We try to use dbuild's "spaces" feature as little as possible, since it makes the build configuration a lot harder to understand and troubleshoot, and it increases build times.
Sometimes, though, it may be necessary to support two different versions of the same project, if the versions are source-incompatible and both used by important projects. In that case, putting the different versions in separate spaces is the only way forward.
Currently (October 2016) all four builds use just one space. For a while, the 2.12.x build used multiple spaces in order to accommodate different versions of Specs2, but we were able to get rid of that.
If some project is needed as a dependency but building it from source is too difficult or annoying, consider adding it as a binary dependency. dbuild supports this with system: "ivy"; there are a few examples in the history.
At present (October 2016) all forks in the 2.12.x build have recently been checked for being reasonably current, but we'll probably backslide on that as time passes.
It's not your job to troubleshoot every failure yourself. Open a ticket, paste in relevant section of the error log (with a link to the full log on Jenkins), @-mention the maintainer, and ask for their help. It's their code, so they'll probably spot the problem faster, and usually maintainers are happy to be included in the community build and happy to help.
Toni, Eugene, etc. They know more about dbuild, sbt, resolvers, Artifactory, etc than you ever will. Everyone on the Scala team has a history with these things, too.
The community build is important, but it is usually not urgent.
Now that the community build is part of our nightly release flows, it's fairly urgent to find some solution if something is failing, perhaps by freezing, forking, or commenting out the offending project.
After that, it's fine if it takes a while to go back and find a longer-term solution: because you're waiting for a project maintainer, because you're waiting for slow builds to finish, because a change needs multiple rounds of testing to get right, etc. It's common for this to drag on for days and weeks. It's fine. The community build is a moving target.
When you fix an issue, leave clues for future-you by including the error message in the commit comment. And if an error message seems mysterious, look for it in git log, a previous maintainer might have seen it before and left clues for you.
Every Jenkins run begins with extracting the dependencies of all of the projects, even if none of them need to be rebuilt. This takes a few minutes, which is too long and sit and watch, so inevitably you'll start working on something else. And actually getting feedback on the change you made might take 30 minutes, or an hour, or 8+ hours if everything has to be rebuilt! By the time you get back to it, it's easy to forget what you were even doing.
It's tempting to maintain mental flow by keeping a tight watch on your builds, but doing so is a huge waste of time. It's better to always write down everything you're thinking and doing as you go, so it's never too costly to set the community build aside while you wait for results. Get something else done, and once your results are in, if you have good notes on where you were, it won't be that hard to reconstruct your mental state and figure out what your next move is.
It's basically the same advice as "leave bread crumbs", but you're leaving the crumbs for yourself in 10 minutes or 1 hour or 1 day, not just some other sad sack months or years later.
Don't just tackle one open ticket, considering tackling 3 or 4 at once, at least if you're reasonably confident the changes are independent and won't interfere with each other. A test run can take a long time, so why not maximize the amount of information you get from each run?
Experts only! This is definitely bad advice at least some of the time.
Just because a project was in at one time doesn't mean we need to keep it forever. Keep cost/benefit in mind.
The same goes for questions of whether to run tests, what subprojects to include, and so forth.