Azure Authorization Failed when using a second provider

[kiweezi]

What happened?

When performing updates to resources in a secondary subscription added through a provider, we have an Azure Authorization Failed error. This is despite the service principle having ownership over the subscription.

The issue started after I changed my own user permissions for the subscription to be read access.

Steps to reproduce

Change your user permissions from contributor to read over the subscription.

Run pulumi destroy or pulumi up

Expected Behavior

All stack resources to be deleted.

Actual Behavior

This example is pulumi destroy

1. Pulumi will delete resources that are executed in the main subscription, before the first resource found in the second subscription. Any resources after this will not be deleted as an error has occurred.

2. The error below is output once Pulumi tries to update the resource in our second subscription.
   Diagnostics: azure-native:network:VirtualNetworkPeering (peer-core-dev-westeurope): error: Code="AuthorizationFailed" Message="The client '[email protected]' with object id 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete' over scope '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-core-westeurope-network-7b11163a/providers/Microsoft.Network/virtualNetworks/vnet-core-westeurope-69129d85/virtualNetworkPeerings/peer-core-dev-westeurope' or the scope is invalid. If access was recently granted, please refresh your credentials."

Versions used

PS> pulumi about

CLI
Version      3.36.0
Go Version   go1.17.12
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  1.67.0
azuread       5.26.1
mailgun       3.4.0
opsgenie      1.1.6
python        unknown - (3.10)
random        4.8.0

Host
OS       Microsoft Windows 11 Pro
Version  10.0.22000 Build 22000
Arch     x86_64

Additional context

It looks like to me that Pulumi is instead trying to authenticate as my user account.

We use the same service principle in the two subscriptions. We have populated the azure-native:clientId and azure-native:clientSecret in the stack config, as well as some other information about the primary subscription.
In the primary subscription, the service principle has contributor. In the secondary it has owner.

When creating the second provider, we specify the subscription and details for the service principle:

core_provider = pulumi_azure_native.Provider(
    "core-provider",
    subscription_id=core_subscription,
    client_id=azure_config.require("clientId"),
    client_secret=azure_config.require_secret("clientSecret"),
    tenant_id=tenant_id
)

We do not explicitly specify the main provider.

I've tried my best to fill this out as much as possible, without creating an essay. Sorry if I've missed anything, just let me know if you need any more information.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).