Dev/catalogupload (#83)

* Updated to use catalog upload range 3.0.X. --------- Co-authored-by: Michiel Oda <[email protected]>
SkylineCommunications · Nov 14, 2024 · fb2ccf4 · fb2ccf4
1 parent 9f3a9dc
commit fb2ccf4
Show file tree

Hide file tree

Showing 2 changed files with 144 additions and 10 deletions.
diff --git a/README.md b/README.md
@@ -4,6 +4,50 @@
 > This GitHub action no longer runs under its own docker image. The docker image has been deprecated and is replaced by .NET tools which makes it easier to create workflows/pipelines/... outside GitHub and still be able to deploy packages to DataMiner.
 > You can still use this GitHub action in GitHub workflows. It will perform the dotnet tool calls on the current runner.
 
+## **Important Changes Since Version 2.0.0**
+
+- The **catalog identifier** (GUID identifying the catalog item on [catalog.dataminer.services](https://catalog.dataminer.services/)) is now required. It must be specified in a `catalog.yml` file as described [here](https://docs.dataminer.services/user-guide/Cloud_Platform/Catalog/Register_Catalog_Item.html#manifest-file).
+
+- If a `README.md` file or an `Images` folder exists in the same directory (or a parent directory) as the `.dmapp` file, they will be automatically registered alongside the package.
+
+- Either the repository’s name or a GitHub topic must be used to infer the catalog item type.
+
+### **Inferring Catalog Item Type:**
+
+The GitHub action can automatically determine the artifact type in two ways:
+
+1. **Repository Naming Convention:**
+   - If the repository follows the naming conventions outlined in the [GitHub Repository Naming Convention](https://docs.dataminer.services/develop/CICD/Skyline%20Communications/Github/Use_Github_Guidelines.html#repository-naming-convention), the tool can infer the type from the repository name itself without needing a GitHub topic.
+
+2. **GitHub Topic:**
+   - If the repository does not follow the naming convention, the tool relies on the presence of a GitHub topic that matches one of the [Artifact Types](#artifact-types) to determine the type.
+
+If neither the repository name follows the convention nor the appropriate GitHub topic is present, the tool will fail to detect the type and return an error.
+
+### Artifact Types
+
+- **AS**: automationscript
+- **C**: connector
+- **CF**: companionfile
+- **CHATOPS**: chatopsextension
+- **D**: dashboard
+- **DISMACRO**: dismacro
+- **DOC**: documentation
+- **F**: functiondefinition
+- **GQIDS**: gqidatasource
+- **GQIO**: gqioperator
+- **LSO**: lifecycleserviceorchestration
+- **PA**: processautomation
+- **PLS**: profileloadscript
+- **S**: solution
+- **SC**: scriptedconnector
+- **T**: testingsolution
+- **UDAPI**: userdefinedapi
+- **V**: visio
+
+> [!IMPORTANT]
+> Currently only Automation Script solutions are supported. This will limit the types to 'automationscript', 'gqidatasource', 'gqioperator', 'lifecycleserviceorchestration', 'profileloadscript' and 'userdefinedapi'.
+
 ## Transition to .NET Tools
 
 Our workflow has evolved from relying solely on GitHub actions to embracing the versatility of .NET tools. This transition offers enhanced flexibility, enabling seamless integration across various widely utilized platforms such as GitHub, GitLab, Azure DevOps, and more.
@@ -37,7 +81,7 @@ Below, we present an example detailing the migration process from the GitHub act
 
        - name: Upload to Catalog
          id: uploadToCatalog
-         run: echo id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ github.workspace }}/${{ steps.packageName.outputs.name }}.dmapp" --dm-catalog-token ${{ secrets.api-key }} --uri-sourcecode "${{ github.server_url }}/${{ github.repository }}" --artifact-version ${{ inputs.referenceName }}) >> $GITHUB_OUTPUT
+         run: echo id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ github.workspace }}/${{ steps.packageName.outputs.name }}.dmapp" --dm-catalog-token ${{ secrets.api-key }} --artifact-version ${{ inputs.referenceName }}) >> $GITHUB_OUTPUT
 
        - name: Deploy to DataMiner
          run: dataminer-package-deploy from-catalog --artifact-id "${{ steps.uploadToCatalog.outputs.id }}" --dm-catalog-token ${{ secrets.DATAMINER_DEPLOY_KEY }}
@@ -80,6 +124,10 @@ This action currently only supports the creation of artifacts with Automation sc
 
 **Required**. The path to the .sln file of the solution. At present, only DataMiner Automation Script solutions are supported. E.g. `'./Example/AutomationScript.sln'`. Required for stages `'Upload'` and `'All'`.
 
+### `github-token`
+
+**Optional**. The secrets.GITHUB_TOKEN.  Required for stages `'Upload'` and `'All'`.
+
 ### `artifact-name`
 
 **Optional**. The chosen name for the artifact. E.g. `'MyPackageName'`. Required for stages `'Upload'` and `'All'`.
@@ -125,10 +173,10 @@ jobs:
     name: Deploy the artifact on the DataMiner System job
     steps:
       # To use this action, the repository must be checked out 
-      - name: Checkout	
-        uses: actions/checkout@v3
+      - name: Checkout
+        uses: actions/checkout@v4
       - name: Set up NuGet
-        uses: nuget/setup-nuget@v1.1.1      
+        uses: nuget/setup-nuget@v2.0.1     
       - name: NuGet restore solution
         run: nuget restore "AutomationScript.sln" -OutputDirectory ${{ github.workspace }}/packages
       - name: Deploy the artifact on the DataMiner System step
@@ -137,6 +185,7 @@ jobs:
         with:
           api-key: ${{ secrets.NAME_OF_YOUR_APIKEY_SECRET }}
           solution-path: './Example/AutomationScript.sln'
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           artifact-name: 'MyArtifactName'
           version: '1.0.1'
           timeout: '300'
@@ -155,10 +204,10 @@ jobs:
     outputs:
       ARTIFACT_ID: ${{ steps.Build_and_upload_artifact_step.outputs.ARTIFACT_ID }}
     steps:
-      - name: Checkout	
-        uses: actions/checkout@v3
+      - name: Checkout
+        uses: actions/checkout@v4
       - name: Set up NuGet
-        uses: nuget/setup-nuget@v1.1.1      
+        uses: nuget/setup-nuget@v2.0.1     
       - name: NuGet restore solution
         run: nuget restore "AutomationScript.sln" -OutputDirectory ${{ github.workspace }}/packages
       - name: Deploy the artifact on the DataMiner System step
@@ -167,6 +216,7 @@ jobs:
         with:
           api-key: ${{ secrets.NAME_OF_YOUR_APIKEY_SECRET }}
           solution-path: './Example/AutomationScript.sln'
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           artifact-name: 'MyArtifactName'
           version: '1.0.1'
           stage: Upload

diff --git a/action.yml b/action.yml
@@ -7,6 +7,9 @@ inputs:
   api-key: # id of input
     description: "The API-key generated in the DCP Admin app as authentication for a certain DataMiner System, saved in a GitHub secret"
     required: true
+  github-token:
+    description: "The secrets.GITHUB_TOKEN. Required for stages: All and Upload."
+    required: false
   solution-path:
     description: "The path to the .sln file of the solution. Atm only DataMiner Automation Script solutions are supported. E.g ./Example/AutomationScript.sln. Required for stages: All and Upload."
     required: false
@@ -62,6 +65,11 @@ runs:
       if: ${{ inputs.stage == 'Upload' || inputs.stage == 'All' }}
       run: |
 
+        if [[ -z "${{ inputs.github-token }}" ]]; then
+          echo "Error: 'github-token' is required but was not provided. This is usually secrets.GITHUB_TOKEN."
+          exit 1
+        fi
+
         if [[ -z "${{ inputs.version }}" && -z "${{ inputs.build-number }}" ]]; then
           echo "Error: 'version' & 'build-number' are both empty."
           exit 1
@@ -86,9 +94,19 @@ runs:
         echo "All required inputs for the Deploy stage are provided."
       shell: bash
 
+    - name: Validate auto-generated catalog YAML file existence
+      if: ${{ github.ref_type == 'tag' }}
+      shell: pwsh
+      run: |
+        if (!(Test-Path "${{ github.workspace }}/.githubtocatalog/auto-generated-catalog.yml")) {
+          Write-Host "Error: The auto-generated catalog YAML file was not found."
+          Write-Host "Run the workflow on a branch before releasing, and ensure the tag points to the latest commit containing the auto-generated-catalog.yml file."
+          exit 1
+        }
+
     - name: Install .NET Tools
       run: |
-        dotnet tool install -g Skyline.DataMiner.CICD.Tools.Packager
+        dotnet tool install -g Skyline.DataMiner.CICD.Tools.Packager --version 2.0.3
         dotnet tool install -g Skyline.DataMiner.CICD.Tools.CatalogUpload
         dotnet tool install -g Skyline.DataMiner.CICD.Tools.DataMinerDeploy
       shell: bash
@@ -142,18 +160,84 @@ runs:
       run: dataminer-package-create dmapp "${{ steps.solutionFolder.outputs.path }}" --type automation --build-number ${{ inputs.build-number }} --output "${{ steps.solutionFolder.outputs.path }}" --name "${{ steps.packageName.outputs.name }}"
       shell: bash
 
+    # Always try to auto-generate a catalog.yml
+    - name: Auto-Generating Catalog from GitHub
+      id: autoGenerateCatalogFromGithubTag
+      if: ${{ (inputs.stage == 'Upload' || inputs.stage == 'All') }}
+      run: |
+        dotnet tool install -g Skyline.DataMiner.CICD.Tools.GitHubToCatalogYaml
+        github-to-catalog-yaml --workspace "${{ github.workspace }}" --github-token "${{ inputs.github-token }}" --github-repository "${{ github.repository }}"
+      shell: bash
+
+    - name: Check if catalog.yml or manifest.yml exists
+      id: check_files
+      shell: pwsh
+      run: |
+        if (Test-Path "${{ github.workspace }}/catalog.yml") {
+          echo "catalogDetails=catalog.yml" >> $env:GITHUB_ENV
+        }
+        elseif (Test-Path "${{ github.workspace }}/manifest.yml") {
+          echo "catalogDetails=manifest.yml" >> $env:GITHUB_ENV
+        }
+        else {
+          Write-Host "No valid file found!"
+          exit 1
+        }
+    
+    - name: Sanitize Artifact Name
+      id: sanitize-artifact-name
+      if: env.catalogDetails != ''
+      shell: bash
+      run: |
+        # Set the sanitized artifact name by replacing invalid characters
+        CLEANED_NAME=$(echo "${{ inputs.artifact-name }}" | sed 's/[":<>|*?\\\/\r\n]//g')
+        echo "SANITIZED_NAME=$CLEANED_NAME" >> $GITHUB_ENV
+  
+    - uses: actions/upload-artifact@v4
+      if: env.catalogDetails != ''
+      with:
+        name: Catalog Details ${{ env.SANITIZED_NAME }}
+        path: "${{ github.workspace }}/${{ env.catalogDetails }}"
+
+    # Only try to commit for non-tag
+    - name: Commit .githubtocatalog/auto-generated-catalog
+      if: ${{ (inputs.stage == 'Upload' || inputs.stage == 'All') && github.ref_type != 'tag' }}
+      shell: pwsh
+      run: |
+        git config --global user.name "github-actions[bot]"
+        git config --global user.email "github-actions[bot]@users.noreply.github.com"
+        git add "${{ github.workspace }}/.githubtocatalog/auto-generated-catalog.yml"
+        
+        # Check if there are any changes to be committed
+        git diff --staged --quiet
+        if ($LASTEXITCODE -ne 0) {
+          git commit -m "auto-generated"
+        }
+        else {
+          Write-Host "No changes to commit."
+        }
+
+    # Only try to push for non-tag
+    - name: Push .githubtocatalog/auto-generated-catalog
+      if: ${{ (inputs.stage == 'Upload' || inputs.stage == 'All') && github.ref_type != 'tag' }}
+      run: |
+        git push
+      env:
+        GITHUB_TOKEN: ${{ inputs.github-token }}
+      shell: pwsh
+
     # Use branch name from 'findBranch' step for the branch name when a tag
     - name: Upload to CatalogUpload
       id: uploadToCatalogTag
       if: ${{ (inputs.stage == 'Upload' || inputs.stage == 'All') && github.ref_type == 'tag' }}
-      run: echo "id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ steps.solutionFolder.outputs.path }}/${{ steps.packageName.outputs.name }}.dmapp" --uri-sourcecode "${{ github.server_url }}/${{ github.repository }}" --branch "${{ steps.findBranch.outputs.branch }}" --dm-catalog-token ${{ inputs.api-key }})" >> $GITHUB_OUTPUT
+      run: echo "id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ steps.solutionFolder.outputs.path }}/${{ steps.packageName.outputs.name }}.dmapp" --artifact-version "${{ github.ref_name }}" --branch "${{ steps.findBranch.outputs.branch }}" --dm-catalog-token ${{ inputs.api-key }})" >> $GITHUB_OUTPUT
       shell: bash
 
     # Use github.ref_name for the branch name when not a tag
     - name: Upload to CatalogUpload
       id: uploadToCatalog
       if: ${{ (inputs.stage == 'Upload' || inputs.stage == 'All') && github.ref_type != 'tag' }}
-      run: echo "id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ steps.solutionFolder.outputs.path }}/${{ steps.packageName.outputs.name }}.dmapp" --uri-sourcecode "${{ github.server_url }}/${{ github.repository }}" --branch "${{ github.ref_name }}" --dm-catalog-token ${{ inputs.api-key }})" >> $GITHUB_OUTPUT
+      run: echo "id=$(dataminer-catalog-upload with-registration --path-to-artifact "${{ steps.solutionFolder.outputs.path }}/${{ steps.packageName.outputs.name }}.dmapp" --artifact-version "${{ github.ref_name }}" --branch "${{ github.ref_name }}" --dm-catalog-token ${{ inputs.api-key }})" >> $GITHUB_OUTPUT
       shell: bash
 
     - name: Set artifact Id