- Kubeflow Pipelines Release Process
Kubeflow Pipelines has weekly patch releases and monthly minor releases. Patch releases only contain bug fixes, while minor releases have new features additionally.
Releases are tagged with tags like X.Y.Z
, e.g. 1.0.2
. A special format like
1.0.0-rc.2
is a pre-release. It is the second release candidate before
releasing the final 1.0.0
.
A release branch has the name structure release-X.Y
where X.Y
stands for the
minor version. Releases like X.Y.Z
will all be released from the branch
release-X.Y
.
For example, 1.0.2
release should be on release-1.0
branch.
The following section targets contributors. No need to read further if you only want to use Kubeflow Pipelines.
After the release-X.Y
release branch is cut, pull requests (PRs) merged to
master will be only get released in the next minor release X.(Y+1).0
.
If you want your PR released earlier in a patch release X.Y.(Z+1)
:
- The PR must already get merged to master branch.
- The PR should be a bug fix.
- The PR should be cherry picked to corresponding release branch
release-X.Y
.
Choose one of the following options for cherry picking your PR to release branch.
Contributors should ask OWNERS who approved the PR to add a cherrypick-approved
label if they want the PR cherry picked to release branch.
Release manager will periodically or before release, search all merged PRs with
cherrypick-approved
and cherry pick them into current release branch.
-
Find the commit you want to cherry pick on master as $COMMIT_SHA.
-
Find the active release branch name $BRANCH, e.g. release-1.0
-
Cherry pick a commit:
git checkout $BRANCH git checkout -b <cherry-pick-pr-branch-name> git cherry-pick $COMMIT_SHA
-
Resolve merge conflicts if any
-
git push origin HEAD
-
Create a PR and remember to update PR's destination branch to
release-$MINOR_VERSION
-
Ask the same OWNERS that would normally need to approve this PR
The following sections target release managers. You don't need to read further if you only want to use or contribute to this repo.
- OS: Linux or MacOS
- Permissions needed
- Can create a branch in github.com/kubeflow/pipelines.
- (Before #4840 is resolved) one would need the admin access to kubeflow/pipelines repo.
- Can trigger cloudbuild jobs in
google.com/ml-pipeline-test
GCP project.
- Tools that should be in your
$PATH
- docker
- python3
- Preparations
- Clone github.com/kubeflow/pipelines repo into
$KFP_REPO
. cd $KFP_REPO
- Clone github.com/kubeflow/pipelines repo into
-
Choose a good commit on master branch with commit hash as
$COMMIT_SHA
. -
Choose the next release branch's
$MINOR_VERSION
in formatx.y
, e.g.1.0
,1.1
... -
Make a release branch of format
release-$MINOR_VERSION
, e.g.release-1.0
,release-1.1
. Branch from the commit and push to kubeflow pipelines upstream repo.git checkout $COMMIT_SHA git checkout -b release-$MINOR_VERSION git push upstream HEAD
Do the following things before a release:
-
(Do this step only when releasing from a NON-master release branch)
cherry pick all merged PRs with
cherrypick-approved
label:-
Search all merged PRs with
cherrypick-approved
label, but nocherrypicked
label using this link -
Use the git cherry-pick option to pick these PR commits into the release branch one by one in a batch and add
cherrypicked
label to these PRs.NOTE: if there are merge conflicts for a PR, ask the PR author or area OWNER to create a cherry pick PR by themselves following other two options.
-
git push upstream release-$VERSION
directly to the release branch.
There's an automated script that can help you do the above:
# Prepare your env cd ~/kubeflow/pipelines git fetch upstream git checkout release-1.0 git pull git checkout -b <your-cherry-pick-branch-name> # The following command shows usage info ./hack/cherry-pick.sh # The following command cherry picks PRs #123 #456 #789 for you. # It runs git cherry-pick for each merged commit, then adds `cherrypicked` # label on the PR. # # If there's a merge conflict in the middle, it will stop there waiting for # you to resolve. You need to add the `cherrypicked` label by yourself in # this case. After the issue resolved, you can rerun the same command and # PRs already cherrypicked (with the label `cherrypicked`) will be skipped. ./hack/cherry-pick.sh 123 456 789 # After cherry pickings are done, they are still in your local repo. Push # them to your remote branch to create a PR. git push origin HEAD
You can get the list of PRs waiting to be cherrypicked by:
-
Open cherrypick-approved PRs that haven't been cherrypicked sorted by updated order.
-
Open browser console (usually by pressing F12).
-
Paste the following command into the console.
console.log(Array.from(document.querySelectorAll('[id^="issue_"][id*="_link"]')).map(el => /issue_(.*)_link/.exec(el.id)[1]).join(' '))
-
-
Verify cloudbuild and postsubmit tests are passing: visit https://github.com/kubeflow/pipelines/commits/master for master branch.
If not, contact the KFP team to determine if the failure(s) would block the release.
Note, when releasing from master, all the below mentions of "release branch" means master branch.
-
Choose the release's complete
$VERSION
following semantic versioning, e.g.1.0.0-rc.1
1.0.0-rc.2
1.0.0
1.0.1
1.1.0
- ... Contact @Bobgy if you are not sure what next version should be.
-
Update all version refs in release branch by
cd ./test/release && TAG=$VERSION BRANCH=$BRANCH make release
It will prompt you whether to push it to release branch. Press
y
and hitEnter
.Note, the script will clone kubeflow/pipelines repo into a temporary location on your computer, make those changes and attempt to push to upstream, so that it won't interfere with your current git repo.
If you see error "docker.sock: connect: permission error", you need to allow managing docker as a non-root user.
-
View related cloudbuild jobs' statuses by clicking the latest commit's status icon in the release branch. The page will look like https://github.com/kubeflow/pipelines/runs/775788343. Refer to the above screenshot to find the page.
-
Wait and make sure the
build-each-commit
cloudbuild job that builds all images in gcr.io/ml-pipeline-test succeeded. If it fails, please click "View more details on Google Cloud Build" and then "Retry".NOTE: you can find your latest release commit in https://github.com/kubeflow/pipelines/commits/master and select your release branch.
-
Select the
release-on-tag
cloudbuild job that copies built images and artifacts to public image registry and gcs bucket. This job should have already failed because artifacts haven't been built. Now, please click "View more details on Google Cloud Build" and then "Retry", because after waiting for previous step, artifacts are now ready.NOTE: (To be verified) Do not click the "Re-run" button from GitHub Actions status page. It will create a build with "Branch: $BRANCH" instead of "TAG: $VERSION". Open "View more details on Google Cloud Build", and rerun from there.
TODO: we should have an automation KFP cluster, and the waiting and submiting
release-on-tag
cloudbuild task should happen automatically.NOTE: postsubmit tests will most likely fail for the release commit, this is expected, postsubmit tests start right after the commit is in GitHub repo, but some artifacts they depend on are still being built by the processes in these two steps.
-
Search "PyPI" in Google internal release doc for getting password of kubeflow-pipelines user.
-
Release
kfp-server-api
python packages to PyPI.git checkout $BRANCH git pull upstream cd backend/api/python_http_client rm -r dist python3 setup.py --quiet sdist python3 -m twine upload --username kubeflow-pipelines dist/*
-
Release
kfp
python packages to PyPI.pip3 install twine --user gsutil cp gs://ml-pipeline/release/$VERSION/kfp.tar.gz kfp-$VERSION.tar.gz python3 -m twine upload --username kubeflow-pipelines kfp-$VERSION.tar.gz
!!! The file name must contain the version. See kubeflow#1292
-
Create a GitHub release using
$VERSION
git tag and titleVersion $VERSION
, fill in the description. Detailed steps:-
Typing in version tag field to search and select the "$VERSION" tag published in release instructions above. Its format is like
X.Y.Z
orX.Y.Z-rc.N
. -
Use this template for public releases and replace the
$VERSION
with real values.To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in [here](https://www.kubeflow.org/docs/pipelines/standalone-deployment-gcp/) or via UI [here](https://console.cloud.google.com/ai-platform/pipelines) Install python SDK (python 3.6 above) by running: ```bash python3 -m pip install kfp kfp-server-api --upgrade ``` See the [Change Log](https://github.com/kubeflow/pipelines/blob/$VERSION/CHANGELOG.md)
Use this template for prereleases (release candidates) and PLEASE CHECK the This is a prerelease checkbox in the GitHub release UI.
To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in [here](https://www.kubeflow.org/docs/pipelines/standalone-deployment-gcp/). Install python SDK (python 3.6 above) by running: ```bash python3 -m pip install kfp kfp-server-api --pre --upgrade ``` See the [Change Log](https://github.com/kubeflow/pipelines/blob/$VERSION/CHANGELOG.md)
-
(Do this step only when releasing from a NON-master release branch) Update master branch to the same version and include latest changelog:
git checkout master git pull git checkout -b <your-branch-name> # This avoids line break at end of line. echo -n $VERSION > VERSION # This takes a while. pushd test/release make release-in-place popd git checkout $VERSION -- CHANGELOG.md git add -A git commit -m "chore(release): bump version to $VERSION on master branch"
-
If current release is not a prerelease, create a PR to update version in kubeflow documentation website: https://github.com/kubeflow/website/blob/master/layouts/shortcodes/pipelines/latest-version.html
Note, there MUST NOT be a line ending in the file. Editing on GitHub always add a line ending for you so you cannot create a PR on GitHub UI. Instead, you can checkout the repo locally and
echo -n 1.0.0 > layouts/shortcodes/pipelines/latest-version.html
and create a PR to update the version, e.g. kubeflow/website#1942.
Please refer to ./test/release.