Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@4 version broken by a patch release #4193

Open
githubcdr opened this issue Nov 18, 2024 · 13 comments · May be fixed by #4198
Open

@4 version broken by a patch release #4193

githubcdr opened this issue Nov 18, 2024 · 13 comments · May be fixed by #4198

Comments

@githubcdr
Copy link

Version

Hi Team,

After upgrading our software to version 4.0.6594, we encountered severe issues where the application failed to start entirely. Despite extensive debugging efforts, we were unable to resolve the issue, and the only viable solution was to revert and pin the version back to 4.0.6280.

Given the significant impact and breaking changes introduced, we’re left wondering why this was released as a patch release, which typically implies non-breaking, backward-compatible changes. Could you provide insight into the reasoning behind this release versioning, and are there any known issues or changes in 4.0.6594 that could explain this behavior?

We appreciate any guidance or information you can offer to help us understand and address this problem.

Thank you.

Description

After upgrading our software to version 4.0.6594, we encountered severe issues where the application failed to start entirely. Despite extensive debugging efforts, we were unable to resolve the issue, and the only viable solution was to revert and pin the version back to 4.0.6280.

Steps to reproduce

Change version to 4.0.6594

@kshyju
Copy link
Member

kshyju commented Nov 18, 2024

@githubcdr

  1. What operating system are you using?
  2. How/where did you update the core tools to the latest version from? Are you using NPM?
  3. What is your function project type? dotnet-isolated or dotnet in-proc or some other language? If dotnet/dotnet-isolated, What version of .NET (TargetFramework value)?
  4. Can you share the error message you are seeing in the terminal, if any?

@githubcdr
Copy link
Author

  1. We are using this docker container, see below.
  2. NPM is used
  3. TFM, in proc? What does that mean? We had to revert the runtime for most of our services.
  4. There are no messages in stdout, the container just crashloops
FROM node:18-slim@sha256:b175cd7f3358c399f7bcee9b1032b86b71b1afa4cfb4dd0db55d66f871475a3e

# Disable ICU
# https://github.com/Azure/azure-functions-core-tools/issues/2554#issuecomment-820935686
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true

# install func CLI required to run Azure Functions
RUN apt update && \
  apt install  --no-install-recommends -y \
  ca-certificates \
  libssl1.1 \
  && \
  rm -rf /var/lib/apt/lists/* && \
  npm install -g [email protected] --unsafe-perm true

@Zemfm
Copy link

Zemfm commented Nov 19, 2024

I'm also having issues when running a Function App from a docker container. In my case, I have a docker-compose.yml file that I was using in Visual Studio until recently without issues.

Recently, when launching from Visual Studio Pro v17.12.0 (which I believe uses the latest version of this tool), the container logs this instead of normal startup/application logs:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

This is happening to all our Function App projects and only in Visual Studio. This does not occur when running docker-compose up from the command line for the same projects.

Update: I was able to fix this issue by moving the setting "FUNCTIONS_WORKER_RUNTIME" from our docker-compose.yml environment variables to local.settings.json. This was not needed in the past.

@kshyju
Copy link
Member

kshyju commented Nov 19, 2024

Azure Functions Core Tools are designed for local development and testing of function apps on your computer. The Docker images published by the Functions team already include the Function runtime, which executes your function app when the container is started. Could you share more about your use case? What’s the specific reason you need Core Tools inside the container?

@Zemfm
Copy link

Zemfm commented Nov 19, 2024

I don't need Core Tools and am not adding it to the container. Here's my dockerfile:

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0 AS base
WORKDIR /home/site/wwwroot
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["src/Job/our.func.app.Job.csproj", "src/Job/"]
COPY ["src/Application/our.func.app.Application.csproj", "src/Application/"]
COPY ["src/Domain/our.func.app.Domain.csproj", "src/Domain/"]
RUN dotnet restore "src/Job/our.func.app.Job.csproj"
COPY . .
WORKDIR "/src/src/Job"
RUN dotnet build "our.func.app.Job.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
RUN dotnet publish "our.func.app.Job.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /home/site/wwwroot
COPY --from=publish /app/publish .
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

The issue was that previously, we were defining FUNCTIONS_WORKER_RUNTIME in our docker-compose.yml environment variables and did not need a local.settings.json file.
Currently, if you set FUNCTIONS_WORKER_RUNTIME only in docker-compose.yml, it seems Core Tools does not read this environment variable and asks the user to specify the runtime on startup.

Here's how to recreate:

  1. Create a new function app project in Visual Studio with Docker support.
  2. Right-click function app project -> Add.. -> Container Orchestrator Support -> Docker Compose.
  3. Delete local.settings.json file.
  4. Add this environment section (FUNCTIONS_WORKER_RUNTIME) and Azurite image to the generated yml file:
    environment:
      AzureWebJobsStorage: UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://azurite:10000/;
      FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
    depends_on:
      - azurite
    
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite:latest
    command: "azurite --loose --blobHost 0.0.0.0 --blobPort 10000 --queueHost 0.0.0.0 --queuePort 10001 --tableHost 0.0.0.0 --tablePort 10002 --location /workspace --debug /workspace/debug.log"
    ports:
       - "20000:10000"
       - "20001:10001"
       - "20002:10002"
    environment: 
      - AZURE_STORAGE_ACCOUNT=devstoreaccount1
      - AZURE_STORAGE_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

This used to work but if you try and check the Function App container logs you will see:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

And your function app will not be executed.

@kshyju
Copy link
Member

kshyju commented Nov 20, 2024

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.

https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12

I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.

Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

@ksgopal4
Copy link

I'm also having issues when running a Function App from a docker container. In my case, I have a docker-compose.yml file that I was using in Visual Studio until recently without issues.

Recently, when launching from Visual Studio Pro v17.12.0 (which I believe uses the latest version of this tool), the container logs this instead of normal startup/application logs:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

This is happening to all our Function App projects and only in Visual Studio. This does not occur when running docker-compose up from the command line for the same projects.

Update: I was able to fix this issue by moving the setting "FUNCTIONS_WORKER_RUNTIME" from our docker-compose.yml environment variables to local.settings.json. This was not needed in the past.

I have exactly the same problem. I dont use docker, my test starts the function app. The test has been successfully starting Function app fine until now and with the new functions runtime, it no longer works. It now needs local.settings.json in the functions folder and for it to contain FUNCTIONS_WORKER_RUNTIME setting, before I can run func start. Although FUNCTIONS_WORKER_RUNTIME was being passed in as an environment variable, its no longer enough.

@Zemfm
Copy link

Zemfm commented Nov 21, 2024

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.

https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12

I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.

Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

Hi @kshyju ,

I have the exact same repository as you and I still have the issue. Did you run this from Visual Studio's latest version? As I said, I have no issues when running docker-compose up from the command line. Only when running in Visual Studio with "docker-compose" as the startup project.

@Zemfm
Copy link

Zemfm commented Nov 21, 2024

I may have found the cause for this issue. In this PR #4142 , the added method EnsureWorkerRuntimeIsSet() only checks local.settings.json through ISecretsManager.GetSecrets(). I believe it should also check environment variables and return immediately if the runtime is set there (or use it to set GlobalCoreToolsSettings.CurrentWorkerRuntime?).

@Zemfm
Copy link

Zemfm commented Nov 25, 2024

/bot not-stale

@kshyju
Copy link
Member

kshyju commented Nov 26, 2024

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.
https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12
I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.
Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

Hi @kshyju ,

I have the exact same repository as you and I still have the issue. Did you run this from Visual Studio's latest version? As I said, I have no issues when running docker-compose up from the command line. Only when running in Visual Studio with "docker-compose" as the startup project.

As specified in the README on my working repo, I used a command line/terminal window to run the docker commands. Haven't tried with VS.

@Zemfm
Copy link

Zemfm commented Nov 26, 2024

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.
https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12
I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.
Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

Hi @kshyju ,
I have the exact same repository as you and I still have the issue. Did you run this from Visual Studio's latest version? As I said, I have no issues when running docker-compose up from the command line. Only when running in Visual Studio with "docker-compose" as the startup project.

As specified in the README on my working repo, I used a command line/terminal window to run the docker commands. Haven't tried with VS.

But you are not running Azure Functions Tools when you only run docker commands with that simple docker-compose. However, when running from Visual Studio, other compose files are added that execute this tool.

When I read #3544 , it seems that providing FUNCTIONS_WORKER_RUNTIME as an environment variable was planned as per @mattchenderson 's comment. However, in the linked PR #4142 to address it, the code seems to only look at local.settings.json.

Which is why I created #4198 to look for the environment variable FUNCTIONS_WORKER_RUNTIME on startup.

Maybe a simpler way to replicate is:

  • Clone the latest version of this repository
  • Set the environment variable in your system FUNCTIONS_WORKER_RUNTIME=dotnet-isolated
  • Run the tool against your function app with this command (from your contributing.md page): dotnet run --project dotnet run --project PATH_TO_FUNCTIONS_CLI/src/Azure.Functions.Cli start
  • You will see the wizard to select the runtime, which in my opinion is incorrect as it is set in the environment variable.
  • Repeat the process but use Add env var check for worker runtime in StartHostAction #4198 as your Azure Function Tool code and the wizard will not appear and the function will start executing normally.

@Zemfm
Copy link

Zemfm commented Nov 28, 2024

Just as a final information and because I was sure this used to be a feature, I checked out a4bf0e4 (v4.0.6543) and was able to run the function using the steps in my previous comment (with FUNCTIONS_WORKER_RUNTIME in the environment variables).

When I tried to run that version of the tool without the env variable, instead of the wizard, this warning was shown in the output:
"The 'FUNCTIONS_WORKER_RUNTIME' setting is required. Please specify a valid value. See https://go.microsoft.com/fwlink/?linkid=2257963 for more information. The application will continue to run, but may throw an exception in a future release."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants