[Bug]: Launchpad can not upgrade itself

[rswrz]

Bug Description

When a change is made to the virtual_machine_scale_set in the Launchpad module, the Launchpad inadvertently updates itself. This creates a problematic situation because the update is initiated by the Launchpad’s GitHub Private Runner.

In many cases, this triggers a rollout of updates to the VMSS instance, which hosts the Launchpad currently executing Terraform. As a result, the GitHub Runner is terminated mid-execution, causing Terraform to fail. This failure leaves behind:

• A locked Terraform state
• An incomplete and inconsistent Terraform state

This behavior disrupts workflows and requires manual intervention to recover the environment.

Terraform-Version

any

Relevant log output

Relevant Error Messages

Additional Information

Root Cause and Suggested Solution

The primary issue lies in the upgrade_mode setting of the VMSS. Currently, it is set to Automatic, which causes problems when Terraform execution is running on the Launchpad itself, as any updates to the Launchpad VMSS trigger a self-termination of the instance executing Terraform.

To resolve this issue, the following changes are recommended:

1. Set upgrade_mode to Manual (the default) to prevent automatic updates.
2. Configure the azurerm provider feature flag reimage_on_manual_upgrade to false. This ensures that manual upgrades of the VMSS do not re-image the currently running instances (the GitHub Runner).

With these changes, the Launchpad VMSS will no longer automatically re-image itself during Terraform execution.

Next Steps

We need a method for rolling out VMSS upgrades without manual intervention. This process should not be handled by Terraform, running on the Private GitHub Runner (Launchpad).

Public runners are unsuitable due to the strict and private nature of the Storage Account holding the Terraform Remote State.

A potential solution could involve using the az CLI in a dedicated GitHub Actions job to re-image the Launchpad VMSS instances after the Terraform Apply job completes successfully.

Privacy Statement

• I agree

Code of Conduct

• I agree to follow this repository's Code of Conduct

[rswrz]

Another potential solution could involve using VMSS Instance Scale Protection