Precedence of environment variables not respected in v3.20.0

[blakewatters]

I suspect that this is related to the newly merge task level dotenv support.

What I am seeing that environment variables specified explicitly at the task level no longer take precedence over ones defined by dotenv. For example, given a Taskfile of the form:

# dotenv file exports a `DATABASE_NAME` env var
dotenv: ['.env', '{{.HOME}}/.adapt.env']

task generate:config:
  env:
    DATABASE_NAME: "whatever"
  cmds:
    - echo "${DATABASE_NAME}"

You will always get the value of DATABASE_NAME defined by dotenv rather than the override defined at the local task level

• Task version: v3.20.0
• Operating System: macOS Ventura

[andreynering]

Hi @blakewatters,

I could not reproduce this issue.

I tried with global dotenv and also local dotenv and in both scenarios it's working as expected: the local env in considered.

# .env
FOO=foobar

version: '3'

dotenv: ['.env']

tasks:
  default:
    env:
      FOO: override
    cmds:
      - echo $FOO

$ task
task: [default] echo $FOO
override

version: '3'

tasks:
  default:
    dotenv: ['.env']
    env:
      FOO: override
    cmds:
      - echo $FOO

$ task
task: [default] echo $FOO
override

[blakewatters]

Okay let me try being a little more specific:

I have a task that generates a dotenv configuration file at .env from a .env.tmpl file. From time to time, new configuration keys are added to the template and configuration needs to be regenerated. This task is called generate:config and looks something like this:

generate:config:
  desc: Generate a config file
  method: none
  preconditions:
    - sh: "! test -f .env"
      msg: "Existing config file found at `.env`, halting..."
  sources:
    - .env.tmpl
  generates:
    - .env
  vars:
    # Replace slashes with underscores to create URL safe branch identifier
    GIT_BRANCH_SAFE:
      sh: git branch --show-current | tr / _
  env:
    TZ: UTC

    DATABASE_NAME: { { .GIT_BRANCH_SAFE } }
    DATABASE_USER: { { .GIT_BRANCH_SAFE } }
    DATABASE_HOST: localhost
    DATABASE_PORT: 5432

    NODE_ENV: development
    WEBAPP_PORT: 5001
    API_BASE_URL: http://localhost:3000

  cmds:
    - cmd: envsubst < .env.tmpl > .env
      silent: true
    - cmd: echo "Wrote new configuration to $PWD/.env:"
      silent: true
    - task: config

I have another task called reinit that destroys the .env file (among other things) and then calls down to generate:config to build a new configuration from the template.

What I am seeing that the values that wind up being written out to the file are the ones from the previously deleted .env file rather than the explicit overrides specified in the generate:config task (or dynamically retrieved/generated values from parts of the example task ommitted for brevity.

In previous versions of Task, this all worked perfectly. Now I can only get the tasks to run by using an env -i execution to clean the environment like so:

generate:worktree-from-remote:
    desc: Generate a new worktree tracking a remote branch
    aliases: ['gwt']
    preconditions:
      - sh: "[ '{{.BRANCH}}' != '<no value>' ]"
        msg: "Error: BRANCH not set.\nUsage: task generate:worktree-from-remote BRANCH=<WORKTREE_NAME>"
    dir: '{{.ADAPT_WORKTREES_DIR}}'
    cmds:
      - cd main && git pull origin main
      - git fetch origin
      - git worktree add --track -B {{.BRANCH}} {{.BRANCH}} origin/{{.BRANCH}}
      # NOTE: workaround for Task v3.20.0 regression (see https://github.com/go-task/task/issues/993)
      # - task --dir {{.BRANCH}} init
      - env -i USER="$USER" HOME="$HOME" PATH="$PATH" bash -l -c "source $(brew --prefix nvm)/nvm.sh && task --dir {{.BRANCH}} init"

Once the values from the previous .env are loaded by Task no overrides or local references instructing it to load from .env work -- it's like the environment is being hopelessly polluted

[zbindenren]

I have a similar problem if a variable is already defined in the shell.

For the taskfile:

version: '3'

tasks:
  default:
    env:
      FOO: override
    cmds:
      - echo $FOO

everything works as expected:

$ task
task: [default] echo $FOO
override

But when I do the following:

$ export FOO=foo
$ task
task: [default] echo $FOO
foo

In my opinion it should print override not foo. This leads to hard to debug issues in task files.

[bcdonadio]

Indeed. The documentation explicitly informs that the expected behavior should be the opposite of what we get in the, as of now latest, v3.24.0 release and main branch. Like @zbindenren demonstrated in their comment, the observed output generated by the current implementation is that the predefined environment where Task is invoked is getting a greater precedence than any defined content of an env: section in any hierarchy position of a Taskfile.yml file. Doesn't matter if the definition is inside a task: or in the file root.

I believe there's no "correct" way of defining this order, there are arguments and uses for both ways, but the documentation should reflect the actual behavior. What was the original intent? Should we fix the code or the docs?

[andreynering]

@bcdonadio That documentation for Task variables (vars:), and not environment variables (env:) inside Task. The behavior differs slightly (which I agree may be confusing).

See discussion at #1038 (comment). Change that would be a breaking change, but maybe a setting would be nice to make all users happy (some people like the current behavior).

[bcdonadio]

@andreynering Yeah, I believe you're right. This line further reinforces the idea that they all follow the same rules, but doesn't specifically states that they behave the same in every possible case.

I think it would be a good idea just to make this behavior clear in the docs of this major release (with a mere patch bump), and only after that discuss and ponder whether should that be implemented as a breaking change in the next major, if there should be a config statement to change the default behavior in the next minor or whatever the solution might ultimately be.

Just trying to follow the principle of least surprise.

[krystian-panek-vmltech]

I have the same problem...

I want to override JAVA_HOME env var defined by the OS because I am managing my own Java installation which should be used instead of OS-wide. I am unable to set the global env override in Taskfile but I want to :(

as a workaround, I am forced to override env var right before executing the Maven, for example:

 cmd: JAVA_HOME=$AEM_BUILD_JAVA_HOME mvn clean package $AEM_BUILD_ARGS

setting

env:
  JAVA_HOME: '{{ .AEM_BUILD_JAVA_HOME }}'

anywhere (global or task level) is not effective; maybe some new section like env_override will do the trick? :)

or sth like:

env:
  JAVA_HOME:
    override: true
    value: 'anything'

[blakewatters]

lol I find this entire discussion pretty amusing. We are 18 months deep and there is apparently some disagreement about whether or not a task such as:

  init:workspace:
    internal: true
    silent: true
    deps:
      - task: generate:config:workspace
        vars:
          OUTPUT_PATH: .env.development
          OSIRIS_RUNTIME: development
          DB_HOST: localhost
          DB_PORT: 5432
          DB_USER: osiris
          DB_NAME: "{{.DB_BASE_NAME}}_test"
          TYPESENSE_COLLECTION_PREFIX: "{{.TYPESENSE_COLLECTION_PREFIX_BASE}}_test"

Should just ignore the variable values explicitly provided via the vars stanza and defer to the environment?

How is this even a discussion or a potentially breaking change? It is clearly incorrect. I am telling the task explicitly what I want the value to be and it is just ignoring it. It's a straight up bug and invalid behavior. Who in the universe is going to have their code broken by providing an explicit input value that is magically ignored based on the environment?

[pd93]

Hi all, there are many open issues related to variables in v3. To help centralise and focus our discussions we have created a new megathread. The env precedence experiment is now stable and there are open tickets linked from the megathread to discuss inheritance and scoping so I'm going to close this issue.