Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT and ENABLE_ORYX_BUILD. #86

bmc-at-hfi · 2021-05-26T21:34:11Z

First observed yesterday, our deployment action is having some issues. We're using node.js, our action is based on this template.

After Client.updateAppSettingViaKudu is called and returns 204, we get stuck in a timeout / retry cycle for about 3 minutes. See Github job log below.

Run Azure/functions-action@v1
  with:
    app-name: REDACTED
    package: .
    publish-profile: ***
    slot-name: REDACTED
    respect-pom-xml: false
    respect-funcignore: false
    scm-do-build-during-deployment: false
    enable-oryx-build: false
  env:
    AZURE_FUNCTIONAPP_NAME: REDACTED
    AZURE_FUNCTIONAPP_PACKAGE_PATH: .
    NODE_VERSION: 10.x
Successfully parsed SCM credential from old publish-profile format.
Using SCM credential for authentication, GitHub Action will not perform resource validation.
Sucessfully acquired app settings from function app (with SCM credential)!
Will archive . into /home/runner/work/_temp/temp_web_package_47007169786336567.zip as function app content
Will use Kudu https://<scmsite>/api/zipdeploy to deploy since publish-profile is detected.
Setting SCM_DO_BUILD_DURING_DEPLOYMENT in Kudu container to false
Update using Client.updateAppSettingViaKudu
Response with status code 204
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 19
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 19
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 18
...truncated...
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 1
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 1
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 0
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 0
Warning: App setting SCM_DO_BUILD_DURING_DEPLOYMENT fails to propagate to Kudu container
Setting ENABLE_ORYX_BUILD in Kudu container to false
Update using Client.updateAppSettingViaKudu
Response with status code 204
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 19
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 19
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 18
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 18
...truncated...
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 1
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 1
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 0
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 0
Warning: App setting ENABLE_ORYX_BUILD fails to propagate to Kudu container
Package deployment using ZIP Deploy initiated.
Deploy logs can be viewed at REDACTED
Successfully deployed web package to App Service.
Successfully updated deployment History at REDACTED
Response with status code 204
Response with status code 204
Response with status code 204

Here's some additional information.

The deploy itself seems to go as expected. But the retry cycle is significantly slowing down our releases.
We've observed this happening on both production & staging deploy slots.
We've observed this with both Windows & Linux Function Apps.
No similar issues are observed when deploying to the function app via VS Code (running in a dev docker container).
No changes were made to our workflow files since the last time it was working properly.

Here's a few things I've tried, to no effect.

Explicitly setting the scm-do-build-during-deployment and enable-oryx-build in the workfile (both true & false).
Knew it was probably unrelated, but tried tweaking the NODE_VERSION from 12.x to 10.x

I'd be happy to provide more information on request. Thanks for your help!

The text was updated successfully, but these errors were encountered:

flcdrg · 2021-05-28T02:13:03Z

Seeing the same issue with a .NET Core 3.1 Function deployment. Wasn't seeing this a couple of weeks ago.

JordanBurns27 · 2021-05-28T03:13:27Z

Also seeing the same issues on .NET 5.0 Function deploy. I attempted most of the changes you did, I even changed those settings in Azure to see if it would read them from the function and skip setting them but no luck. It's adding a ton of time to our deploys.

keesschollaart81 · 2021-05-30T13:02:08Z

Same here with a Python using this template and a brand new Azure Functions with Python runtime 3.8 (although other versions behave similar).

It seems due to recent changes as reverting back to v.1.3.1 fixes it:

    - name: 'Run Azure Functions Action'
      uses: Azure/[email protected]

github-actions · 2021-06-02T12:23:46Z

This issue is marked need-to-triage for generating issues report.

crgarcia12 · 2021-06-04T13:05:01Z

+1 - I can repro if I create the pipeline from the Azure Portal directly

brianrodecap · 2021-06-08T15:04:27Z

+1 - Also experiencing the same issue. Function deployment duration has slowed from 1 minute to 8 - 10 minutes.

tmakowka-tc · 2021-06-08T16:48:11Z

+1 - We are also experiencing this issue, switching back to v1.3.2 fixes it.

We are using the action to deploy a Python 3.9 project,

Hazhzeng · 2021-06-08T17:57:47Z

This seems like a backend issue where the Kudu lite container is not able to pick up the app settings change in time.
I will reduce the retry count to 3 in the next release. Thanks for reporting the bug.

mikesprague · 2021-06-08T18:27:53Z

Not to sound negative bc I appreciate the issue getting attention but that doesn't sounds like it's fixing the actual problem. And all of us who are having the issue will still see increased build times (just not quite as bad, bc of 3 vs 10 retries).

Speaking only for myself, I would like to see the core issue get resolved or to have these attempts to apply settings removed altogether if they're not actually needed.

bmc-at-hfi · 2021-06-08T18:59:10Z

This seems like a backend issue where the Kudu lite container is not able to pick up the app settings change in time.
I will reduce the retry count to 3 in the next release. Thanks for reporting the bug.

Reducing the retry count will treat the symptom (delay in function app deployment) but not the root cause (settings not syncing with kudu container).

It is never successfully syncing these settings with the Kudu container, unless I revert to v1.3.2 as suggested above.

Thank you for looking into this!

This is a bandage solution to alleviate the symptoms of the AppSettings not propogating to Kudu as mentioned in Azure#86 . Because the propagation is broken both these settings retry 20 times at a 5 second interval and 200 seconds (over 3 minutes) are added to any builds using this action because of it. Setting the retry count still allows 15 seconds of redundancy but wastes far less time (just 30s) if the setting just fails to copy. Ultimately, I feel the settings propagation should either be fixed, or this setting removed if the propagation is unnecessary, but I don't know enough about those settings to understand how to fix it or if they are required.

IronSean · 2021-06-14T20:08:44Z

I opened #90 as a bandage solution to limit the retries while a proper solution is figured out.

Norrch2 · 2021-06-17T01:35:20Z

Can confirm that I am experiencing this issue as well. Azure Functions 3.1 new GitHub workflow.

- avoid a problem relates to this issue Azure/functions-action#86

* Add or update the App Service deployment workflow configuration from Azure Portal. * Specify the version of `Azure/functions-action` - avoid a problem relates to this issue Azure/functions-action#86 * Separate slot by target branch * Update code

lloydjatkinson · 2021-06-25T17:48:45Z

Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 10
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 10
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 9
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 9
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 8
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 8
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 7
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 7
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 6
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 6
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 5
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 5
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 4
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 4
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 3
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 3
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 2
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 2
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 1
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 1
Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 0
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 0
Warning: App setting SCM_DO_BUILD_DURING_DEPLOYMENT fails to propagate to Kudu container
Setting ENABLE_ORYX_BUILD in Kudu container to false
Update using Client.updateAppSettingViaKudu
Response with status code 204
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 19
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 19
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 18
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 18

This is what I get as of today. It looks like a complete failure of the deployment process.

Azure/functions-action#86

tomkerkhove · 2021-06-26T20:59:47Z

Reverting to v1.3.2 fixed it for me.

Azure/functions-action#86

PhakornKiong · 2021-07-08T08:13:02Z

Reverting to v1.3.2 fixed it for me as well.

Was using v1.0.0 and v1.4.0.

kulov · 2021-07-11T10:50:10Z

Reverting to v1.3.2 of the action fixed the error for me also.
Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT and ENABLE_ORYX_BUILD

I can provide debug logs if needed.

kpm-at-hfi · 2021-07-29T21:11:31Z

Should this issue be "idle"? I think we're just waiting around to see it get fixed.

OnlyOnePro · 2021-08-04T08:23:23Z

Yes, a lot of time and no response whatsoever and it is even properly identified by the repo maintainers. This is a bug that considerably slows downs release times on our Prod/Dev instances for each Azure Function that we have, and it is a lot of time to be wasting on a CI/CD pipeline.

morrisonbrett · 2021-08-06T03:15:12Z

Just discovered this today too. Deploying Azure Functions. 19 retries and then succeeds. Slows down deploy time dramatically.

mattchatterley · 2021-08-06T11:47:27Z

Also getting the same - action completes successfully, however spends ages retrying over the two variables.

mdhom · 2021-08-10T05:22:35Z

Same here:

Warning: Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT, remaining retry 19
App setting SCM_DO_BUILD_DURING_DEPLOYMENT has not been propagated to Kudu container yet, remaining retry 19

after that:

Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 19
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 19

and finally:

Package deployment using ZIP Deploy initiated.

gabrielgarciagava · 2021-08-12T12:34:38Z

On my case, I do the builds using ubuntu-latest.

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

3 days ago my deployments stopped working. The github actions logs were saying that the deployment was succesful, but my functions were NOT deployed.

App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 1
Warning: Failed to check app setting propagation for ENABLE_ORYX_BUILD, remaining retry 0
App setting ENABLE_ORYX_BUILD has not been propagated to Kudu container yet, remaining retry 0
Warning: App setting ENABLE_ORYX_BUILD fails to propagate to Kudu container
Package deployment using ZIP Deploy initiated.
Deploy logs can be viewed at <redacted>
Successfully deployed web package to App Service.
Successfully updated deployment History at <redacted>
Response with status code 204
Response with status code 204
Response with status code 204

Rolling back to v1.3.2 fixed the issue to me.

I assume that my case was being a failure because I need these 2 settings to be true since I'm building on ubuntu instead of windows.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-deployment-technologies#remote-build-on-linux

    - name: 'Run Azure Functions Action'
      uses: Azure/[email protected]
      id: fa
      with:
        app-name: <redacted>
        package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
        publish-profile: <redacted>
        scm-do-build-during-deployment: true
        enable-oryx-build: true

anirudhgarg · 2021-08-25T16:22:02Z

Hello folks - thanks for reporting it. we are looking into this.

patelchandni · 2021-08-25T17:59:21Z

This bug is fixed in the latest version v1.4.2 (also v1).

mikesprague · 2021-08-25T19:31:17Z

This bug is fixed in the latest version v1.4.2 (also v1).

@patelchandni While no longer getting retry messages about the settings, I have yet to have a deploy complete successfully since trying the updates (previously pinned at v1.3.2, I changed to v1 to get latest version).

My latest attempt has spent over 18 minutes on the functions-action step and never output anything meaningful in the logs. Last output on that run was Successfully deployed web package to App Service. and yet it continued on that step until I finally canceled the job.

Just like before, reverting to v1.3.2 allows it to complete successfully (and does so in less than 5 minutes for all build steps combined).

morrisonbrett · 2021-08-25T19:35:06Z

Just tried. Same results as @mikesprague. 18 minutes for me with @v1 and @v.1.4.2.

balag0 · 2021-08-25T20:00:07Z

@mikesprague @morrisonbrett
Could you share the appname please. ( you can share the name privately using
https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately#option-2 )

mikesprague · 2021-08-25T20:05:29Z

@balag0 Unless you require option 2, I manually ran one of my functions in the function app and got the following:

GUID: 75eb2ca2-213d-4d0a-9f91-f3217b88a179

Full output:

2021-08-25T20:02:56.106 [Information] Executing 'Functions.redactedFunctionName' (Reason='This function was programmatically called via the host APIs.', Id=75eb2ca2-213d-4d0a-9f91-f3217b88a179)

morrisonbrett · 2021-08-25T20:11:17Z

You can see the action runs here: https://github.com/morrisonbrett/TrueVote.Api/actions

You can also see in the commit history where I used Azure/[email protected] and then Azure/functions-action@v1 and then Azure/[email protected], and then back to Azure/[email protected]. and the run times of each attempt.

patelchandni · 2021-08-25T23:19:06Z

The hanging deployment issue should be fixed in the latest v1.4.3 (v1). Please give it a try and let us know.

morrisonbrett · 2021-08-25T23:28:54Z

Confirmed fixed. I just pushed using Azure/functions-action@v1 and it works. Thanks.

mikesprague · 2021-08-25T23:43:11Z

@patelchandni Working for me now as well with the latest (via v1). Thanks for the attention to this!

zhughes3 · 2021-08-26T05:33:00Z

Worked as well for me when changing to v1. Do you guys recommend keeping v1 in my deployment scripts or locking it to 1.4.3?

patelchandni · 2021-08-26T13:48:07Z

@zhughes3 It is recommended to use v1 to get the latest version.

flcdrg · 2021-08-27T00:17:38Z

As a general rule, using a v1 tag should be ok - though we've seen an example in this action where there was a regression so using v1 meant you got hit with the bug.

An alternative is to use a specific version and enable Dependabot to provide PRs to upgrade your actions. That way you're more intentional about changing things.

github-actions · 2021-09-10T03:03:07Z

This issue is idle because it has been open for 14 days with no activity.

bbevers2 · 2024-10-18T16:59:19Z

FYI for apps with flex consumption see #245 (comment)

bmc-at-hfi added the need-to-triage label May 26, 2021

github-actions bot removed the need-to-triage label Jun 2, 2021

github-actions bot added the need-to-triage label Jun 2, 2021

kanika1894 added bug Something isn't working and removed need-to-triage labels Jun 2, 2021

IronSean mentioned this issue Jun 14, 2021

Update retryCount to 3 #90

Closed

dzeyelid added a commit to dzeyelid/azure-functions-with-python that referenced this issue Jun 22, 2021

Specify the version of Azure/functions-action

cbb2cbb

- avoid a problem relates to this issue Azure/functions-action#86

tomkerkhove added a commit to tomkerkhove/azure-autoscaling-toolbox that referenced this issue Jun 26, 2021

fix: Work around issue

9bdc22e

Azure/functions-action#86

dfar-io added a commit to dfar-io/ynab-sync that referenced this issue Jul 5, 2021

Fixes build warnings

eb2743d

Azure/functions-action#86

redmer mentioned this issue Jul 7, 2021

Downgrade Azure-Functions to v1.3.2 Stichting-CROW/data.crow.nl#4

Closed

econaxis added a commit to econaxis/transit that referenced this issue Jul 8, 2021

https://github.com/Azure/functions-action/issues/86

ef2a33a

hshaosf added a commit to SFDigitalServices/address-microservice-fn-py that referenced this issue Jul 15, 2021

using v1.3.2 to fix action see Azure/functions-action#86

14a4335

github-actions bot added the idle label Jul 29, 2021

github-actions bot removed the idle label Jul 30, 2021

anirudhgarg assigned patelchandni Aug 25, 2021

patelchandni closed this as completed Aug 25, 2021

patelchandni reopened this Aug 25, 2021

morrisonbrett mentioned this issue Aug 26, 2021

Azure Function actions are running longer than usual #101

Closed

github-actions bot added the idle label Sep 10, 2021

patelchandni closed this as completed Sep 22, 2021

tinuoommen mentioned this issue Apr 3, 2022

Action stuck indefinitely on deploy #122

Closed

Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT and ENABLE_ORYX_BUILD. #86

Failed to check app setting propagation for SCM_DO_BUILD_DURING_DEPLOYMENT and ENABLE_ORYX_BUILD. #86

Comments

bmc-at-hfi commented May 26, 2021

flcdrg commented May 28, 2021

JordanBurns27 commented May 28, 2021

keesschollaart81 commented May 30, 2021 • edited Loading

github-actions bot commented Jun 2, 2021

crgarcia12 commented Jun 4, 2021

brianrodecap commented Jun 8, 2021

tmakowka-tc commented Jun 8, 2021 • edited Loading

Hazhzeng commented Jun 8, 2021

mikesprague commented Jun 8, 2021

bmc-at-hfi commented Jun 8, 2021

IronSean commented Jun 14, 2021

Norrch2 commented Jun 17, 2021

lloydjatkinson commented Jun 25, 2021

tomkerkhove commented Jun 26, 2021

PhakornKiong commented Jul 8, 2021

kulov commented Jul 11, 2021 • edited Loading

kpm-at-hfi commented Jul 29, 2021

OnlyOnePro commented Aug 4, 2021

morrisonbrett commented Aug 6, 2021

mattchatterley commented Aug 6, 2021

mdhom commented Aug 10, 2021

gabrielgarciagava commented Aug 12, 2021 • edited Loading

anirudhgarg commented Aug 25, 2021

patelchandni commented Aug 25, 2021

mikesprague commented Aug 25, 2021 • edited Loading

morrisonbrett commented Aug 25, 2021

balag0 commented Aug 25, 2021

mikesprague commented Aug 25, 2021 • edited Loading

morrisonbrett commented Aug 25, 2021

patelchandni commented Aug 25, 2021

morrisonbrett commented Aug 25, 2021

mikesprague commented Aug 25, 2021

zhughes3 commented Aug 26, 2021

patelchandni commented Aug 26, 2021

flcdrg commented Aug 27, 2021

github-actions bot commented Sep 10, 2021

bbevers2 commented Oct 18, 2024

keesschollaart81 commented May 30, 2021 •

edited

Loading

tmakowka-tc commented Jun 8, 2021 •

edited

Loading

kulov commented Jul 11, 2021 •

edited

Loading

gabrielgarciagava commented Aug 12, 2021 •

edited

Loading

mikesprague commented Aug 25, 2021 •

edited

Loading

mikesprague commented Aug 25, 2021 •

edited

Loading