Decouple dependency on clojars.org when building Clojars uberjar

[danielcompton]

To build Clojars the app, we need Clojars the site. This is a nasty circular dependency we need to break.

If we're trying to bootstrap a new clojars server, it stands to reason that there may not be a running one in place at the time. There are a few options here for building the app, and two main scenarios to think about: rebuilding the server from scratch when there is no Clojars server, and everyday deployments.

Use a mirror for building Clojars on the clojars server

We can set a mirror in the Leiningen config, and use this for both building from scratch, and everyday builds.

Pros:

• Minimal changes to current build system required
• JAR will still be built on server, simple to understand
• Keeps all config in Ansible

Cons:

• Relies on mirror support to work well in Leiningen
• We would need to modify the clojars-web project.clj to use these mirrors; currently mirrors in profiles.clj won't work for :plugin-repositories and we need the supersport plugin.
• Relies on the mirror server/block store to be available at the time we are rebuilding
• Needs to download all of the JARs at build time for bootstrapping, this can take a while. On an established server this isn't likely to be as big an issue.
• Takes a while to build and test the JAR
• Puts some amount of load on the server

Build Clojars locally ahead of time and copy it to the server

I think @tobias is in favour of this option.

Pros:

• All deps likely to be downloaded so it will be fast in all cases
• Likely to be building with a machine with more resources than a VPS
• Pretty simple

Cons:

• Still needs a mirror available in case the main server is down
• Kind of gets away from the ansible ideal (I think?) as it relies on local state to have a working build toolchain and it is managed out of band from the rest of the Ansible config. May not be as easily reproducible between different people's machines, e.g. profiles.clj differences.
• Build scripts would probably need to reach out from their own folders to the other project

Build Clojars on the server ahead of time and write an artifact to S3 or similar

How this would work:

1. When a new version is ready, an admin would run an ansible play to build the new version on the clojars server (or optionally on a new VM in EC2, but that's probably not necessary).
2. Once the uberjar is built it is copied into S3 (and another redundant object store?) for deployment.
3. When it comes to deploy that version (either straight away, or far off in the future) it doesn't rely on any of our infrastructure being up and running.

This is my personal preference as it seems to be the most fault tolerant, however after listing the cons, I'm not so sure it's the right solution.

Pros:

• Deploys from scratch are much faster as they just need to download a JAR
• Possibly most fault tolerant design
• Doesn't rely on a working Clojure environment on the deployers machine
• Keeps everything in Ansible

Cons:

• It adds a dependency on the object store
• It adds additional complexity around orchestration
• It adds additional complexity keeping more application secrets
• Requires another account

Thoughts welcome!

[tobias]

I think I still prefer the simplicity of building locally, and not having the build step be handled by ansible. How about a build script in the repo that sets $LEIN_HOME to a temp dir, then writes a profiles.clj that sets :local-repo to another temp dir, then builds? That should eliminate any differences in the local environment (other than java version). It wouldn't solve the issue of bootstrapping with no repo available, but if read access to the repo isn't available for some reason, we would want to get that back up as a priority anyway before restoring the app at clojars.org.