Azure auth token expiry

[petervandivier]

Anecdotally, workloads which are eligible for tagging cannot run for more than 24 hours and fail with the following error.

error

An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.
Unhandled exception. Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.)
---> System.AggregateException: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.)
---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: The read operation failed, see inner exception.
---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Security.SslStream'.
  at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|123_0(ExceptionDispatchInfo e)
  at System.Net.Security.SslStream.DecryptData(Int32 frameSize)
  at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
  --- End of inner exception stack trace ---
  at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
  at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
  at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
  at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
  --- End of inner exception stack trace ---
  at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
  at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
  at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
  at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
  at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  --- End of inner exception stack trace ---
  at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
  at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  --- End of inner exception stack trace ---
  at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
  at Azure.Core.HttpPipelineMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
  at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, Boolean retry, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, ILoggerAdapter logger, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, IDictionary`2 bodyParameters, ILoggerAdapter logger, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders, Func`2 onBeforePostRequestData)
  at Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(Uri endPoint, RequestContext requestContext, Boolean addCommonHeaders, Func`2 onBeforePostRequestHandler)
  at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
  at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.SilentRequestHelper.RefreshAccessTokenAsync(MsalRefreshTokenCacheItem msalRefreshTokenItem, RequestBase request, AuthenticationRequestParameters authenticationRequestParameters, CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.TryGetTokenUsingFociAsync(CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.RefreshRtOrFailAsync(CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.ExecuteAsync(CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.Requests.Silent.SilentRequest.ExecuteAsync(CancellationToken cancellationToken)
  at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
  at Microsoft.Identity.Client.ApiConfig.Executors.ClientApplicationBaseExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenSilentParameters silentParameters, CancellationToken cancellationToken)
  at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
  at Azure.Identity.MsalPublicClient.AcquireTokenSilentCoreAsync(String[] scopes, String claims, IAccount account, String tenantId, Boolean async, CancellationToken cancellationToken)
  at Azure.Identity.MsalPublicClient.AcquireTokenSilentAsync(String[] scopes, String claims, IAccount account, String tenantId, Boolean async, CancellationToken cancellationToken)
  at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
---> (Inner Exception #1) System.Threading.Tasks.TaskCanceledException: The operation was canceled.
  at Azure.Core.CancellationHelper.ThrowOperationCanceledException(Exception innerException, CancellationToken cancellationToken)
  at Azure.Core.CancellationHelper.ThrowIfCancellationRequested(CancellationToken cancellationToken)
  at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
  at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
  at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---

  --- End of inner exception stack trace ---
  at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
  at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
  at Azure.Identity.SharedTokenCacheCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
  at Microsoft.Azure.PowerShell.Authenticators.MsalAccessToken.Renew()
  at Microsoft.Azure.PowerShell.Authenticators.MsalAccessToken.AuthorizeRequest(Action`2 authTokenSetter)
  at Microsoft.WindowsAzure.Commands.Storage.Common.Cmdlet.NewAzureStorageContext.GetTokenStrFromAccessToken(IAccessToken accessToken)
  at Microsoft.WindowsAzure.Commands.Storage.Common.Cmdlet.NewAzureStorageContext.<>c__DisplayClass99_0.<<GetTokenRenewer>b__0>d.MoveNext()
--- End of stack trace from previous location ---
  at Microsoft.Azure.Storage.Auth.TokenCredential.RenewTokenAsync(Object state)
  at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
  at System.Threading.QueueUserWorkItemCallback.Execute()
  at System.Threading.ThreadPoolWorkQueue.Dispatch()
  at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
VERBOSE: 2023-09-09T19:54:46.4242261-04:00: Awaiting Operation: '9fe5c7a4-f30c-412f-8c37-ee6fb03dd83e', Start: '09/08/2021 00:00:00 +00:00', End: '09/08/2021 06:00:00 +00:00'. Current wait time: 00:06:05.6266995
2023-09-09T19:54:46.4246362-04:00: Completed Operation: '32a7ebc2-9ebb-4a60-a1a0-c22a13b75ade', Start: '09/07/2021 06:00:00 +00:00', End: '09/07/2021 12:00:00 +00:00'.
VERBOSE: 2023-09-09T19:54:46.4300062-04:00: Awaiting Operation: '735dd511-af3a-4019-9e84-26505c865209', Start: '09/08/2021 06:00:00 +00:00', End: '09/08/2021 12:00:00 +00:00'. Current wait time: 00:06:05.3766995
2023-09-09T19:54:46.4406450-04:00: Completed Operation: 'dbbfda0f-5588-49b8-b872-bcf648481e71', Start: '09/07/2021 18:00:00 +00:00', End: '09/08/2021 00:00:00 +00:00'.
2023-09-09T19:54:46.4416992-04:00: Completed Operation: 'd3c58a7e-17ff-4802-9285-083faad86dba', Start: '09/07/2021 00:00:00 +00:00', End: '09/07/2021 06:00:00 +00:00'.
2023-09-09T19:54:46.7713522-04:00: Completed Operation: '2afeced1-e4fa-448f-a326-5de2c878211f', Start: '09/08/2021 12:00:00 +00:00', End: '09/08/2021 18:00:00 +00:00'.
VERBOSE: 2023-09-09T19:54:46.7713336-04:00: Awaiting Operation: 'c933d373-ff88-4906-9b86-0a5dd8ea4b48', Start: '09/08/2021 18:00:00 +00:00', End: '09/09/2021 00:00:00 +00:00'. Current wait time: 00:06:05.0485740
VERBOSE: 2023-09-09T19:54:46.8777916-04:00: Awaiting Operation: '82987175-b3db-4ecb-b975-9a562f55711a', Start: '09/09/2021 06:00:00 +00:00', End: '09/09/2021 12:00:00 +00:00'. Current wait time: 00:06:04.9235739
2023-09-09T19:54:46.8777916-04:00: Completed Operation: 'dc80bb7c-1bc8-47b1-a9b7-5ee1f6b9c096', Start: '09/09/2021 00:00:00 +00:00', End: '09/09/2021 06:00:00 +00:00'.

[process exited with code 3762504530 (0xe0434352)]
You can now close this terminal with Ctrl+D, or press Enter to restart.

I suspect this is due to Azure auth token expiry. Spike & debug.