Support for building a project multiple times (multiple artifacts) and publishing as one version

[jverbus]

I’m using shipkit for a Scala/Spark open source project (https://github.com/linkedin/isolation-forest).

For every commit, I'd like to build this project multiple times with different dependencies (e.g. build with both Spark 2.3.0 and Spark 2.4.3) and create an artifact for each build. All of these artifacts would then be published under the same library version number.

The output on Bintray would look something like this. This output was achieved by using the auto shipkit process to publish isolation-forest v0.3.1 built with Spark 2.3.0 via Travis CI. I then manually uploaded isolation-forest v0.3.1 built with Spark v2.4.3 from my laptop using bintrayUpload. Ideally, I'd like to automate this entire process using Travis CI + Shipkit.

[mockitoguy]

Thank you for documenting this use case! We'll take a look at it.

@koral--, what do you think about this idea?

[koral--]

I haven't deployed yet such artifacts. However, for me it look like a valid use case for maven POM classifier:
https://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html
http://maven.apache.org/pom.html#Dependencies
On linked docs there is an example with jdk8 and jdk11 classifiers so it seems that it also can be applied to versions of the Spark.

If this assumption is true then probably it should be enough to configure maven publication in the project, it may be even opaque to Shipkit. Haven't checked if bintray supports classifiers though.

[mockitoguy]

@koral--,

I suspect that classifiers will not help here. @jverbus needs to run the build multiple times with different variant. To goal is to make something like that possible with Shipkit:

./gradlew build -Pvariant=2.11
./gradlew build -Pvariant=2.12
./gradlew release  // releases all variants

[mockitoguy]

@jverbus, we haven't had time yet to look into this. We'll try to make more progress sometime this month.

[jverbus]

@mockitoguy, thanks for the update!

[jverbus]

@mockitoguy, @koral-- : Checking in to see if there are any updates on this feature request.

[mockitoguy]

Hey @jverbus, we did not make tangible progress on this yet. I have a couple of ideas but did not have a chance yet.

[jverbus]

Thanks for the update, @mockitoguy!

[jverbus]

Hey, @mockitoguy!

I was recently thinking about this some more. Here is a bit more detail on my current process and an idea for a solution.

Current process

I'm now building and publishing three artifacts (with different Scala/Spark version dependencies) under the same project version. I expect the number of different builds under the same project version will continue to increase.

Here are the artifacts from a recent commit: https://bintray.com/linkedin/maven/isolation-forest/0.3.3#files/com/linkedin/isolation-forest

I have Travis CI build all three variants on each push to validate that all dependency combinations work. These builds are done in parallel. Here is the Travis CI config:

https://github.com/linkedin/isolation-forest/blob/master/.travis.yml

Due to the Shipkit limitations described earlier in this issue thread, only one artifact variant is automatically released to Bintray via the Travis CI build. I currently publish the others manually via ./gradlew bintrayUpload on my local machine.

Once all artifact variants for a particular project version have been uploaded to Bintray, I manually sync them to Maven Central.

https://oss.sonatype.org/#nexus-search;quick~isolation-forest

What I'd like

If we can have all of those builds in the Travis CI build matrix publish their results to Bintray under the same project version (0.3.3 in the example links above), that would be awesome. I think this would also allow me to pretty easily automate the subsequent sync with Maven Central in Travis CI as well.

Potential solution

The thing that seems tricky is that having multiple Travis CI build processes running in parallel could potentially result in race conditions where the Shipkit publishing / auto version number updates step on each other.

If the user could have more granular control of the version.properties update process with Shipkit, this might be a solution. For example, if the user could turn off the auto version update during the build + ciPerformRelease jobs in the build matrix, then all of the build tasks would upload a different artifact with the same project version. I could then use a Travis build stage (https://docs.travis-ci.com/user/build-stages#how-to-define-build-stages) to run a Shipkit command to update the version.properties in the git repo after all of the jobs in the build matrix succeed.

Perhaps something like this is already possible given the current Shipkit / bintray plugin functionality? Looking forward to hearing what you think. Thanks!

[mstachniuk]

Hey, @jverbus

Could you look at #869 please? Maybe it will help in your case?

[jverbus]

Thanks, @mstachniuk!

It seems like that solution would require creating a separate sub-module in the project for each additional artifact variant? I could imagine having builds for multiple different combinations of Spark/Scala dependencies, so this doesn't scale too well.

I think I found an inelegant, but seemingly working, solution for my use case using multiple Travis build stages. I should be able to scale up to more artifact variants by adding additional jobs to the "upload" Travis stage in the link below.

Here is the Travis config: https://github.com/linkedin/isolation-forest/blob/master/.travis.yml

Here is an example of a successful run: https://travis-ci.org/github/linkedin/isolation-forest/builds/691954282

It would be nice if there was a built in Shipkit or bintray-gradle-plugin task that could perform something similar to what I am doing for each job in the "upload" Travis stage.

[mockitoguy]

If the user could have more granular control of the version.properties update process with Shipkit, this might be a solution

The vision of for the "version.properties" is to avoid version bump commits - like the https://github.com/shipkit/shipkit-auto-version

@jverbus, I created #870 to track cleaning up isolation-forest

[geoHeil]

#869 I am interested in this feature as well, as I want to use the shadow plugin to additionally upload a fat jar besides the regular publications.