Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: rename workflow and streamline catalog validation process #63

Merged
merged 4 commits into from
Dec 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 10 additions & 135 deletions .github/workflows/validate-catalog.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: catalog-entry
name: leap-catalog

on:
pull_request:
Expand All @@ -20,137 +20,12 @@ concurrency:
cancel-in-progress: true

jobs:
validate-and-generate:
timeout-minutes: 10
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: validate feedstock entry and generate catalog
uses: leap-stc/data-catalog-actions/leap-catalog@main
with:
generation-single: "./feedstock/catalog.yaml"
output-directory: "./"

- name: Format JSON
run: |
jq -r '.' output/single-feedstock-web-catalog.json | sed '$s/^$//;/^$/d' > formatted.json
mv formatted.json output/single-feedstock-web-catalog.json
cat output/single-feedstock-web-catalog.json | jq

- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: single-feedstock-web-catalog.json
path: output/single-feedstock-web-catalog.json

update:
needs: validate-and-generate
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
environment:
name: ${{ github.head_ref }}
url: https://catalog.leap.columbia.edu/?catalog=https://raw.githubusercontent.com/${{ github.repository }}/${{ github.head_ref }}/output/single-feedstock-web-catalog.json
steps:
- uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.PRIVATE_KEY }}

- uses: actions/checkout@v4
with:
ref: ${{ github.head_ref || github.ref_name }}
token: ${{ steps.app-token.outputs.token }}

- uses: actions/download-artifact@v4
with:
name: single-feedstock-web-catalog.json
path: output

- run: |
git config --global user.name '${{ steps.app-token.outputs.app-slug }}[bot]'
git config --global user.email '${{ steps.get-user-id.outputs.user-id }}+${{ steps.app-token.outputs.app-slug }}[bot]@users.noreply.github.com>'

- name: Commit and Push
run: |
git add output/single-feedstock-web-catalog.json
git commit -m "Auto-update single-feedstock-web-catalog.json" || true # ignore if there is nothing to commit

- name: Push changes
if: github.event_name == 'pull_request'
uses: ad-m/github-push-action@master
with:
github_token: ${{ steps.app-token.outputs.token }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
branch: ${{ github.event.pull_request.head.ref }}

- name: Create Pull Request
id: cpr
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || github.event_name == 'push'
uses: peter-evans/create-pull-request@v7
with:
token: ${{ steps.app-token.outputs.token }}
commit-message: "build: update catalog entry"
committer: GitHub <[email protected]>
author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
title: Update catalog entry
body: |
- Updates the single-feedstock web catalog for changes on `main`
- Auto-generated by [create-pull-request][1]

[1]: https://github.com/peter-evans/create-pull-request
branch: update-single-feedstock-web-catalog
delete-branch: true

create-or-update-issue:
needs: validate-and-generate
if: failure() && github.event_name == 'schedule'
runs-on: ubuntu-latest
steps:
- name: Create or Update Issue
uses: actions/github-script@v7
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const issueTitle = 'Validate and Generate Job Failed';
const issueBody = `The validate-and-generate job failed in the scheduled run on ${new Date().toISOString()}.

Please check the [workflow run](${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}) for more details.`;

// Search for an existing open issue
const issues = await github.rest.issues.listForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
creator: 'github-actions[bot]',
title: issueTitle
});

if (issues.data.length > 0) {
// Update existing issue
await github.rest.issues.update({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issues.data[0].number,
body: issueBody
});
console.log(`Updated issue #${issues.data[0].number}`);
} else {
// Create new issue
const newIssue = await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: issueTitle,
body: issueBody
});
console.log(`Created issue #${newIssue.data.number}`);
}
single-feedstock:
uses: leap-stc/data-catalog-actions/.github/workflows/reusable-catalog-entry.yml@main
with:
python-version: "3.12"
feedstock-path: "./feedstock/catalog.yaml"
output-directory: "./"
secrets:
APP_ID: ${{ secrets.APP_ID }}
PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
2 changes: 1 addition & 1 deletion feedstock/catalog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ stores:
- id: "small"
name: "The cool small Proto Dataset"
url: "gs://leap-scratch/data-library/feedstocks/proto_feedstock/small.zarr"
"ncviewjs:rechunking":
rechunking:
- path: "gs://some-bucket/small.zarr"
use_case: "multiscales"

Expand Down
4 changes: 2 additions & 2 deletions output/single-feedstock-web-catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
"id": "small",
"name": "The cool small Proto Dataset",
"url": "gs://leap-scratch/data-library/feedstocks/proto_feedstock/small.zarr",
"ncviewjs:rechunking": [
"rechunking": [
{
"path": "gs://some-bucket/small.zarr",
"use_case": "multiscales"
Expand All @@ -51,7 +51,7 @@
"id": "large",
"name": "The even cooler large Proto Dataset",
"url": "gs://leap-scratch/data-library/feedstocks/proto_feedstock/large.zarr",
"ncviewjs:rechunking": null,
"rechunking": null,
"public": false,
"geospatial": null,
"xarray_open_kwargs": {
Expand Down
Loading