ACR with Private Endpoint deployment fails on second run as it tries to delete subnets

[nad-au]

I'm trying to get started with Bicep and create a Container Registry with Private Endpoint using the guide https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link but having a few problems.

Firstly, there seems to be no sample to do this so I created everything manually as per the guide and then exported the template and decompiled to Bicep. Here's the script:

targetScope = 'resourceGroup'

param acrPepName string = 'pep-sbox-hub-01'
param acrName string = 'acrsboxhub01'
param vnetName string = 'vnet-sbox-hub-01'
param location string = 'australiaeast'

resource acr 'Microsoft.ContainerRegistry/registries@2023-08-01-preview' = {
  location: location
  name: acrName
  properties: {
    adminUserEnabled: false
    anonymousPullEnabled: false
    dataEndpointEnabled: false
    networkRuleBypassOptions: 'AzureServices'
    networkRuleSet: {
      defaultAction: 'Allow'
    }
    publicNetworkAccess: 'Disabled'
    zoneRedundancy: 'Disabled'
  }
  sku: {
    name: 'Premium'
  }
}

resource acrDnsZone 'Microsoft.Network/privateDnsZones@2018-09-01' = {
  location: 'global'
  name: 'privatelink.azurecr.io'
}

resource vnet 'Microsoft.Network/virtualNetworks@2023-05-01' = {
  location: location
  name: vnetName
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
  }
}

resource acrSubnet 'Microsoft.Network/virtualNetworks/subnets@2023-05-01' = {
  parent: vnet
  name: 'endpoints-subnet'
  properties: {
    addressPrefixes: [
      '10.0.0.0/24'
    ]
    privateEndpointNetworkPolicies: 'Disabled'
    privateLinkServiceNetworkPolicies: 'Enabled'
  }
}

resource vnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2018-09-01' = {
  parent: acrDnsZone
  name: 'acr-vnet-link'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: vnet.id
    }
  }
}

resource pep 'Microsoft.Network/privateEndpoints@2023-05-01' = {
  location: location
  name: acrPepName
  properties: {
    privateLinkServiceConnections: [
      {
        name: 'acr-service-connection'
        properties: {
          groupIds: [
            'registry'
          ]
          privateLinkServiceId: acr.id
        }
      }
    ]
    subnet: {
      id: acrSubnet.id
    }
  }
}

resource privateDnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-04-01' = {
  parent: pep
  name: 'dnsgroupname'
  properties: {
    privateDnsZoneConfigs: [
      {
        name: 'config1'
        properties: {
          privateDnsZoneId: acrDnsZone.id
        }
      }
    ]
  }
}

This works but when I create a second deployment it fails with:

{
  "status": "Failed",
  "error": {
    "code": "DeploymentFailed",
    "target": "/subscriptions/6242290a-ddbf-48a4-9d01-85245644aaee/resourceGroups/rg-sbox-hub-01/providers/Microsoft.Resources/deployments/sundaytest",
    "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
    "details": [
      {
        "code": "InUseSubnetCannotBeDeleted",
        "message": "Subnet endpoints-subnet is in use by /subscriptions/6242290a-ddbf-48a4-9d01-85245644aaee/resourceGroups/rg-sbox-hub-01/providers/Microsoft.Network/networkInterfaces/pep-sbox-hub-01.nic.0a58cbc2-78c7-43bb-92eb-3fc27fdfef0a/ipConfigurations/privateEndpointIpConfig.966e77c8-7d55-486e-964d-3b4355fb1768 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
        "details": []
      }
    ]
  }
}

However, when I do a what-if, there should be no changes. I'm still learning Bicep but should I be able to safely create multiple deployments and if there's no reported changes it shouldn't change anything? Why does it attempt to delete the resources?

[nad-au]

Seems if I define subnets within the vnet then I don't get the error

resource vnet 'Microsoft.Network/virtualNetworks@2023-04-01' = {
  name: vnetName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'endpoints-subnet'
        properties: {
          addressPrefix: '10.0.0.0/24' 
          privateEndpointNetworkPolicies: 'Disabled'
        } 
      }
    ]
  }
  tags: tags
}

There's some mention of this https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-modes#incremental-mode

[johnlokerse]

Hi @nad-au,

This error appears because of the resource declarationvnet.

The deployment wont fail on the first run, because the resources are new. But after the first run the virtual network (vnet) does not have the state of the subnets during deployment. The Azure Resource Manager thinks that the virtual network declaration "state" is different in your code than it is in Azure. So the Azure Resource Manager tries to delete the subnet that is not present in your vnet declared resource.

The solution to resolving this error is to declare the subnet inside the vnet resource declaration. This way the "state" of your code is the same as in Azure.

I hope this clear for you. If not, let me know! 💪

[ahelland]

@johnlokerse is right about what's happening here, and how to avoid it.

To add my two cents - it is possible to deploy vnets and snets independently of each other, but this depends on the bigger picture of what you are trying to achieve. If all you want to do is deploy an ACR connected to a private vnet you don't have to complicate things.

If you structure your resources by level you can have a hierarchy where you are provided a vnet by the networking team and have to do your own subnetting. You would then create the subnets like this:

resource subnet 'Microsoft.Network/virtualNetworks/subnets@2023-05-01' = {
  name: '${vnetName}/${subnetName}'
  properties: { ... }

This works slightly different than referencing the parent directly, but achieves the same result. You could put it into modules for more flexibility and you can roll out ACR and other resources separately. (In Azure architecture the networking is often said to be at a lower level than ACR, web apps, etc. so you can remove the upper layers as you wish, but not the other way around.)