Registry experimental feature in 0.4.717 (5fc1202b95)

[Agazoth]

I'm playing around with the experimental registry feature in this build: https://github.com/Azure/bicep/actions/runs/1176207276

Initially bicep publish will not run properly. the command:

bicep publish .\mymodule.bicep --target myacr.azurecr.io/demo/mymodule:1.0

just returns

The specified module target "myacr.azurecr.io/demo/mymodule:1.0" is not valid.

In order to make bicep accept my publish command, I had to add this environment variable: $env:BICEP_REGISTRY_ENABLED_EXPERIMENTAL = $true
and that certainly changed the output. Now I get:

Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 401 (Unauthorized)

Content:
{"errors":[{"code":"UNAUTHORIZED","message":"retrieving permissions failed"}]}

Headers:
Server: openresty
Date: Sat, 28 Aug 2021 05:07:30 GMT
Connection: keep-alive
X-Ms-Correlation-Request-Id: REDACTED
x-ms-ratelimit-remaining-calls-per-second: REDACTED
Strict-Transport-Security: REDACTED
Content-Type: application/json
sage, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)  
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryBlobRestClient.StartUploadAsync(String name, CancellationToken cancellationToken)        at Bicep.Core.RegistryClient.BicepRegistryBlobClient.UploadBlobAsync(Stream stream, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PushArtifactAsync(OciArtifactModuleReference moduleReference, StreamDescriptor config, StreamDescriptor[] layers)
   at Bicep.Core.Registry.OciModuleRegistry.PublishModule(ModuleReference moduleReference, Stream compiled)   at Bicep.Core.Registry.ModuleDispatcher.PublishModule(ModuleReference moduleReference, Stream compiled)
   at Bicep.Cli.Commands.PublishCommand.RunAsync(PublishArguments args)   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

I have successfully pushed and pulled modules to my registry with oras (found here: https://github.com/oras-project/oras) authenticating with a mix of either AzureCLI/Docker or PowerShell/Docker. There seems to be some sort of dependency to docker in both cases, but that is for a discussion in another forum I assume.

I would like to know, what I have to do to take me past the authentication error when running bicep publish and would love to share my further findings in this discussion.

[majastrz]

In the current builds, auth is implemented using this class: https://docs.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential?view=azure-dotnet. The logic walks down the list of credential types until finds something that works. We don't have any dependency on docker cli being installed.

ACR returns 403 if you authenticated correctly but don't have permissions to push or pull. Since you received a 401 you might not be authenticated. Did you do az login or Login-AzAccount with a user or SPN that has at least the AcrPush role assigned to it in the scope of the registry or above? (Or any other role that has those actions like Contributor for example.)

[Agazoth]

Yes. I logged in with both, actually.

I'll be away from my machine for the next 8 hours. I'll give it another try when I get back and list the pa and cli versions (they are less then 14 days old)

[Agazoth]

Name                           Value
----                           -----
PSVersion                      7.1.4
PSEdition                      Core
GitCommitId                    7.1.4
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     6.3.0                 Az                                  Core,Desk

azure-cli                         2.27.2

core                              2.27.2
telemetry                          1.0.6

[Agazoth]

I think I have located the reason for the error in my setup.

I have Visual Studio setup with my corporate account. The acr I try to access is located in a tenant, where my corporate account is setup as a B2B account with owner access to the acr.

Removing my corporate profile from Visual Studio skips that step and takes the next TokenCredential:

Configuring an account in Visual Studio, that is local to the tenant holding the acr and has contributor access to the acr solves the issue.

The same issue exists in VSCode. My corporate account generates the error, the tenant local account succeeds.

When developing modules, this might cause issues for several users having a similar setup as I. I use VS Code and Visual Studio for several tasks that require me to sign in with my corporate credentials. Having to "sing out" of both applications prior to working with bicep publishing seems cumbersome. At the very least there should be a warning that VSCode and Visual Studio credentials didn't work.

Ideally the sort order of credentials to try would be sorted differently. In the pipeline, AzureCLI or PowerShell credentials would be the most likely TokenCredentials used. Likewise when working with code locally.

Another way to avoid this error could be to continue with the other authentication types, if the Visual Studio or VSCode fails.

[majastrz]

Thanks for creating a new issue to track this (#4211). I'll reply there.

[Agazoth]

There seems to be a similar issue with the vscode extension. I get the same 401 unauthorized as before when publishing.

The difference is, that I get the error both when configuring a tenant local account in both visual studio and vscode and when authenticating with PowerShell or AzureCLI. Looking at the progmon trace, there is a lot of action from Microsoft.Asal.TokenService.exe

[Agazoth]

I am actually running 6.3.0 in the Windows terminal and 6.4.0 in the Linux terminal.

The pipeline is locked to 5.7.0 (latestVersion)

[brwilkinson]

The issue you said was on WSL, which is 6.4.0 in the Linux terminal and likely az.resources 4.3.1, so try that az.resources to 4.3.0.

Also if you can create a MD table of the machines that you are troubleshooting and what works and what doesn't that would be best, since it confusing to troubleshoot across X number of machines in Y number of environments. Perhaps we can dedicate a new thread in this discussion to the different scenarios that you have.

Something like this:

OS+info	Bicep Cli	Az Module	Az.Resources	Az.Accounts	public ACR works	private ACR works	Deployment method	Current Error	Description or info
win10 vm in Azure	0.4.613	6.3.0	4.3.0	2.5.2			AZ Powershell	Service request failed.\r\nStatus: 403 (Forbidden)\r\n\r\nContent:\r\n{"errors":[{"code":"DENIED","message":"retrieving permissions failed|
WSL	0.4.613	6.4.0	4.3.1	2.5.2			AZ Powershell	Cannot retrieve the dynamic parameters for the cmdlet	trydowngrade az.resources to 4.3.0

[Agazoth]

No WSL has no issues with bicep 0.4.777, Az 6.4.0 and Az.Accounts 2.5.3. It retrieves from both public and private registries. I'll make a table

[brwilkinson]

I tried running PowerShell on WSL2 Ubuntu with 0.4.777 and was able to run a bicep deployment pointing at the public registry. 0.4.854, however, gave me New-AzDeployment: Cannot retrieve the dynamic parameters for the cmdlet. Cannot find path '/tmp/46180d3e-5b02-411e-ac28-3e05a55b7bba/main.json' because it does not exist.

This is the issue related to Az 6.4.0, with az.resources 4.3.1.

Cool, thanks yeah a table will help.

[Agazoth]

So bicep 0.4.854 does not work with 6.4.0 but bicep 0.4.777 does. Seems like a bicep issue to me then. Added a table below

[Agazoth]

OS+info	Bicep Cli	Az Module	Az.Resources	Az.Accounts	public ACR works	private ACR works	Deployment method	Current Error	Description or info
W11-Dev	0.4.777	6.3.0	4.3.0	2.5.2	false	true	PowerShell	New-AzDeployment: Cannot retrieve the dynamic parameters for the cmdlet. Cannot find path 'C:\Users\agazoth\AppData\Local\Temp\69cec19a-e8b9-4968-98bc-abb7cefa217a\main.json' because it does not exist.	Error only occours when referencing public ACR. Works when referencing a private ACR
W11-Dev	0.4.777	6.4.0	4.3.1	2.5.3	false	true	PowerShell	New-AzDeployment: Cannot retrieve the dynamic parameters for the cmdlet. Cannot find path 'C:\Users\agazoth\AppData\Local\Temp\cf0ac245-2734-42ca-8bcf-731f3282312b\main.json' because it does not exist.	Error only occours when referencing public ACR. Works when referencing a private ACR
W11-Dev	0.4.777	6.4.0	4.3.0	2.5.3	false	false	PowerShell	New-AzDeployment: Unexpected character encountered while parsing value: @. Path '', line 7, position 0.	No error in VS Code on private ACR but 403 on public ACR
WSL2 (on W11-Dev)	0.4.777	6.4.0	4.3.1	2.5.3	true	true	PowerShell	null	It just works
WSL2 (on W11-Dev)	0.4.854	6.4.0	4.3.1	2.5.3	false	true	PowerShell	New-AzDeployment: Cannot retrieve the dynamic parameters for the cmdlet. Cannot find path '/tmp/46180d3e-5b02-411e-ac28-3e05a55b7bba/main.json' because it does not exist.	Private ACR works
W10 Azure	0.4.777	6.3.0	4.3.0	2.5.2	false	false	PowerShell	Error BCP096: Expected a module identifier at this location. Status: 403 (Forbidden)	Nothing works here - tried to publish to private with same login as in all other terminals. VS Code not accepthing either registry
Azure DevOps Pipeline ubuntu-latest	0.4.777	5.7.0	3.4.0	2.3.0	true	true	PowerShell	null	It just works

[brwilkinson]

seems like your visual studio token is not allowing you to access the new ACR. I would either log out or refresh the VS token and then test again.

[Agazoth]

That does not fit with being able to push to the old acr in the same shell. I think something is off with the registry. I'll keep hunting the registry.

In any case, I can push from WSL. It's only bicep.exe acting strange.

[brwilkinson]

I think it will work, if you try it : ) I am thinking that WSL does not read the token from VS, it's using the AZ cli (or vscode) token. That is the difference, not bicep.exe vs bicep.

Other than that, seems that you are in an okay place for now overall. So we can see how things progress once the anonymous capability is enabled and check back in then. However if you find any other anomalies let us know.

[Agazoth]

Signing out from Visual Studio absolutely works. I think what got me is, that I have not had Visual Studio opened since I logged out of all accounts last, and when I opened it now, an account with insufficient credentials was logged in.

Getting a different priority in what credentials are used in bicep would solve this. az or pwsh first, followed by VS Code would be great.

Thanks again for support and sparring! Even VS Code works now, so I'm really good.

[brwilkinson]

Great news... I will mark the summary below as answered for now.

As you continue to test, feel free to either continue the discussion here or open a new discussion, depending on what you find.

Also I listed the highlights from our discussions below and these items will be addressed as development continues. There are Stories already open for most of those items including having a bicep configuration for more control over which Creds may be used, so keep an eye out for updates Etc.

[brwilkinson]

Given this thread is so long, I will summarize a few things that were discovered with early/experimental testing of the Module Gallery.

Related to: #2128

• Currently the anonymous Pull feature is still in development.
• Currently you have to set the environment variable to use the module gallery feature, I forgot this a few times, upcoming updates will remove this requirement (see [Story] Bicep Registry #2128 for setup e.g. you also need the nightly build Etc.)
• If you have an Azure or Azure ARC machine, that you have setup with a Managed Identity, the DefaultAzureCredential will use the token from the managed identity as first preference, so you would need to delegate RBAC access for that Identity to the ACR or remove the managed identity configured on the machine.
  • maybe consider having an Opt-in setting for using the managed identity and skip it by default? Consider the same for Visual Studio.
• It will be helpful to add extra logging on which Credential was used, this will be used in troubleshooting auth failures.
  • Below is an example of an error you might get if you log out of EVERYTHING and you can see the precedence in which the creds are checked/validated.
  • If you successfully obtain a token, it's not clear which provider/option was used.

(Inner Exception #1) Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.


(Inner Exception #2) Azure.Identity.CredentialUnavailableException: Process "D:\VisualStudio\2022\Preview\Common7\IDE\Extensions\pyqi1uze.0fz\TokenService\Microsoft.Asal.TokenService.exe" has failed with unexpected error: 
	TS003: Error, TS005: No accounts found.  Please go to Tools->Options->Azure Services Authentication, and add an account to be to authenticate to Azure services during development..


(Inner Exception #1) Azure.Identity.CredentialUnavailableException: Process "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\rahssdlq.j5o\TokenService\Microsoft.Asal.TokenService.exe" has failed with unexpected error: 
	TS003: Error, TS005: No accounts found.  Please go to Tools->Options->Azure Services Authentication, and add an account to be to authenticate to Azure services during development..


(Inner Exception #3) Azure.Identity.CredentialUnavailableException: VisualStudioCodeCredential authentication unavailable. Token acquisition failed. Ensure that you have authenticated in VSCode Azure Account.


(Inner Exception #4) Azure.Identity.CredentialUnavailableException: Please run 'az login' to set up account


(Inner Exception #5) Azure.Identity.CredentialUnavailableException: Please run 'Connect-AzAccount' to set up account.

[Agazoth]

credentialPrecedence has been implemented in #4762 and and makes it possible to define what credentials to use.

Adding this to bicepconfig.json is all it takes:

    "cloud": {
        "currentProfile": "AzureCloud",
        "credentialPrecedence": [
            "AzurePowerShell",
            "AzureCLI"
        ]
    }

Thanks, @majastrz !

[majastrz]

Btw, the current profile doesn't work completely right yet, so I don't recommend changing it until #3986 is done.

[JeroenvdBurg]

thanks for sharing you experience. Was really usefull when starting to experiment with the bicep registery modules.
I have some extra info to share:

• installing the latest build via : https://github.com/Azure/bicep/actions/runs/1318617099

• install the Visual studio Code Extension can't be done by double clicking the VSIX file. You should go to the extensions in VScode and then select Install from VSIX from the menu. Then it should work.

• publishing syntax is slightly changed in the latest verison (0.4968)
  bicep publish .\app-service-plan.bicep --target br:mybicepmodules.azurecr.io/demo/appservice-plan:1.0

• using the module in your main.bicep
  module appServicePlanModule 'br:mybicepmodules.azurecr.io/demo/appservice-plan:1.0' = {

• finally you can also use bicep registery alias. Add to your bicepconfig.json

"moduleAliases": { "ts": {}, "br": { "mymodules": { "registry": "mybicepmodules.azurecr.io", "modulePath": "demo" } } }

use the alias in your main

module appServicePlanModule 'br/mymodules:appservice-plan:1.0' =

Thanks to the bicep team for making this awesome feature ❤️