To use pre-release versions or not? Want to be cutting edge with "stability"

[BenjaminEngeset]

Hi project maintainers and contributors.

PSRule is great and under constant development. I am debating whether to use pre-release verions or not. Want to be cutting edge, use the newest functionality and give back to the community in terms of feedback etc.

Pre-release module versions are created on major commits and can be installed from the PowerShell Gallery. Pre-release versions should be considered experimental. Modules and change log details for pre-releases will be removed as stable releases are made available.

Since they are based on major bumps, what is the project thinking about consumers leveraging this functionality for non-experimental Bicep and ARM projects? Take into consideration you want the newest functionality and leverage the advantages. Projects will respect the disadvantages it can give as well.

Was thinking of just locking it with require on PSRule and PSRule.Rules.Azure to specific pre-release versions in yml.

Disadvantage is to manually have to update yml when the CI fails (due to not meeting requirements since modules might have gotten new pre-release versions -AllowPrerelease).

Advantage is that then you can actually see what new pre-release versions have included in change log details (as this is where you will look for new versions most likely).

Any thoughts?

[BernieWhite]

@bengeset96 Pre-release versions have a higher chance that they might introduce a bug or issue that breaks a pipeline, because they are typically less tested then stable versions.

We release pre-release versions to gain higher confidence that our changes doesn't break existing functionality and to allow the community to provide feedback or try out new features before they release in a stable version.

I'd suggest not requiring a pre-release version within a project pipeline where it might prevent a PR from being merged or a deployment.

Options you might consider to get the best out of both:

1. Run two pipelines one with pre-release version and one with a stable version. It's fairly easy within GitHub Actions or Azure Pipelines to trigger and optional and required pipeline. Make the stable a requirement to merge PR and report status for the pre-release but make it optional. If it fails it's not going to break anything but may help improve code quality.

2. Run the pre-release version as a scheduled pipeline so that you can see if an upcoming change may break or add new rules that need to be integrated.

[BenjaminEngeset]

@BernieWhite

Thank you for the feedback.

Agreed, optional and not required CI for pre-release will be nice and what I will opt for.

Do you know if there is a way to have two seperate ps-rule.yml in a repo? One for stable CI and another one for pre-release CI.

[BernieWhite]

@bengeset96 No worries.

Yes, running via PowerShell you can use -Option ps-rule.preview.yaml to specify an alternative options file for Assert-PSRule or Invoke-PSRule. PSRule defaults to ps-rule.yaml but if you specify an alternative options file it will use this instead.

For GitHub Actions there is an option: property and this is also planned for Azure Pipelines tasks but it hasn't been shipped yet.

[BenjaminEngeset]

@BernieWhite

Did some testing today with your suggestion and I'd really like your feedback.

Preview task

- task: AzurePowerShell@5
        displayName: "Validate with Well-Architected Framework Preview"
        inputs:
          azureSubscription: $(serviceConnectionName)
          scriptType: "inlineScript"
          inline: |
            Write-Output "##vso[task.setvariable variable=waf]results"
            Install-Module -Name PowerShellGet -Repository PSGallery -Scope CurrentUser -Force;
            Install-Module PSRule -AllowPrerelease -Scope CurrentUser -Force;
            Install-Module PSRule.Rules.Azure -AllowPrerelease -Scope CurrentUser -Force;
            git diff --diff-filter=d HEAD^ HEAD --name-only | ForEach-Object { Get-ChildItem $_ } | Assert-PSRule -Option 'ps- 
            rule.preview.yml' -Format File -Module 'PSRule.Rules.Azure' -Baseline 'Azure.Preview' -OutputFormat NUnit3 -OutputPath 
           'reports/ps-rule-preview-results.xml'
        azurePowerShellVersion: latestVersion

Preview configuration file

#
# PSRule Preview for Azure configuration
#

# Please see the documentation for all configuration options:
# https://aka.ms/ps-rule-azure

# Use rules from the following modules/
include:
  module:
    - "PSRule.Rules.Azure"

# Require a minimum version of modules that include referenced baseline.
requires:
  PSRule: "@pre >=2.2.0-B0175"
  PSRule.Rules.Azure: "@pre >1.17.0-B0035"

output:
  culture:
    - "en-US"

execution:
  # Ignore warnings for resources and objects that don't have any rules.
  notProcessedWarning: false

configuration:
  # Enable expansion for Bicep source files.
  AZURE_BICEP_FILE_EXPANSION: true

  # Expand Bicep module from Azure parameter files.
  AZURE_PARAMETER_FILE_EXPANSION: true

  # Set timeout for expanding Bicep source files.
  AZURE_BICEP_FILE_EXPANSION_TIMEOUT: 30

input:
  pathIgnore:
    # Ignore common files that don't need analysis.
    - "**/bicepconfig.json"
    - "*.md"
    - "*.png"
    - ".azuredevops/"

    # Exclude Bicep module files.
    - "modules/**/*.bicep"

    # Exclude JSON module files.
    - "modules/**/*.json"

    # Include test/main.test files from modules.
    - "!modules/**/test/*.bicep"

  ignoreObjectSource: true

rule:
  exclude:
    # Ignore these recommendations for this repo.
    - Azure.Resource.UseTags
    - Azure.ACR.ContentTrust

This works well, but I am not able to use -Baseline 'Azure.GA_2022_06','Azure.Preview_2022_06'. I can only use one at time it seems like?

Reason for wanting to use Azure.GA_2022_06 (which I am using on stable) is to have it included if it should give any other behavior on pre-release versions. Seems like Azure.Preview does the same job?

I had some issues with build compilation time on this setup with pre-release versions, but 30 seconds seems to work fine (10-15 nested modules..). It does not really have anything to do with PSRule, but great to have the expansion timeout setting.

Any thoughts on the setup?

[BernieWhite]

@bengeset96 Yes, PSRule only supports specifying a single baseline at time. The quarterly baselines include a fixed rule set that does not have any new rules added to it after stable release. The Azure.Preview would include all the rules from Azure.GA_2022_06 and Azure.Preview_2022_06 but would also include any rules added in the future.

Alternatively you could create a custom baseline combining:

PSRule.Rules.Azure/src/PSRule.Rules.Azure/rules/Baseline.Rule.yaml

Lines 257 to 300 in 8f413df

	---
	# Synopsis: Include rules released June 2022 or prior for Azure GA features.
	apiVersion: github.com/microsoft/PSRule/v1
	kind: Baseline
	metadata:
	name: Azure.GA_2022_06
	spec:
	configuration:
	# Configure minimum AKS cluster version
	AZURE_AKS_CLUSTER_MINIMUM_VERSION: '1.22.6'
	rule:
	tag:
	release: GA
	ruleSet:
	- '2020_06'
	- '2020_09'
	- '2020_12'
	- '2021_03'
	- '2021_06'
	- '2021_09'
	- '2021_12'
	- '2022_03'
	- '2022_06'
	---
	# Synopsis: Include rules released June 2022 or prior for Azure preview only features.
	apiVersion: github.com/microsoft/PSRule/v1
	kind: Baseline
	metadata:
	name: Azure.Preview_2022_06
	spec:
	rule:
	tag:
	release: preview
	ruleSet:
	- '2020_06'
	- '2020_09'
	- '2020_12'
	- '2021_03'
	- '2021_06'
	- '2021_09'
	- '2021_12'
	- '2022_03'
	- '2022_06'

In terms of options files, if you only need to change a few options but most are the same for stable/ preview pipelines an alternative would be to just override options via environment variable that should be different.

See example: Requires

[BenjaminEngeset]

@BernieWhite

Thank you! I have gotten it to work it as I hoped for with your help.

Truly grateful.