Variables containing special characters coming from env file are not preserved

[hpello]

Description of the issue

Variables containing special characters coming from env file are not preserved

Context information (for bug reports)

• Using Compose V2 docker compose ...
• Using Compose V1 docker-compose ...

Output of docker(-)compose version

Docker Compose version v2.0.0-rc.2

(Also, Output of docker-compose-v1 version)

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Output of docker version

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  main:
    build:
      context: ***/dollar
      dockerfile: ***/dollar/Dockerfile
    environment:
      VARIABLE_0: $ax
      VARIABLE_1: $ax
      VARIABLE_2: $a$0$12$_x
      VARIABLE_3: $ax
    env_file:
    - .env
    networks:
      default: null
networks:
  default:
    name: dollar_default

Steps to reproduce the issue

1. Create a simple Dockerfile

FROM alpine:3.13.6

CMD env | grep VARIABLE

2. Create a simple docker-compose.yml file that reads from an env file

version: '3'

services:
  main:
    build: .
    env_file: .env

3. Create a .env file with the following content

VARIABLE_0=$a$0$12$_x
VARIABLE_1="$a$0$12$_x"
VARIABLE_2='$a$0$12$_x'
VARIABLE_3=       $a$0$12$_x

Observed result

Some parts of the environment variables are missing:

• dollar signs preceding numeric value or underscore
• double quotes
• single quotes (seeming to preserve the rest, but they are missing)
• leading spaces

Indeed, docker-compose up --build produces

main_1  | VARIABLE_0=$ax
main_1  | VARIABLE_1=$ax
main_1  | VARIABLE_2=$a$0$12$_x
main_1  | VARIABLE_3=$ax

Expected result

As the documentation states

There is no special handling of quotation marks. This means that they are part of the VAL.

So I expect the original values to be preserved, including all "special" characters (' " $0 $_...).
As a consquence, from what I understand, docker-compose up --build should produce:

main_1  | VARIABLE_0=$a$0$12$_x
main_1  | VARIABLE_1="$a$0$12$_x"
main_1  | VARIABLE_2='$a$0$12$_x'
main_1  | VARIABLE_3=       $a$0$12$_x

Additional information

For information, the v1 version of docker-compose produces (docker-compose-v1 up --build):

main_1  | VARIABLE_0=$a$0$12$_x
main_1  | VARIABLE_1=$a$0$12$_x
main_1  | VARIABLE_2=$a$0$12$_x
main_1  | VARIABLE_3=$a$0$12$_x

and docker-compose-v1 config:

services:
  main:
    build:
      context: ***/dollar
    environment:
      VARIABLE_0: $$a$$0$$12$$_x
      VARIABLE_1: $$a$$0$$12$$_x
      VARIABLE_2: $$a$$0$$12$$_x
      VARIABLE_3: $$a$$0$$12$$_x
version: '3'

Which is not exactly the expected result either (quotes and leading spaces are NOT being preserved), but seems more reasonable when it comes to dollar signs.

Documentation issue

Whether it is chosen to preserve or not quotation marks or special characters (including spaces), I think it would be important that the documentation reflects the actual behaviour.

[ndeloof]

Unfortunately the dotEnv file format is a de facto standard with no established specification. On Compose V2 we rely on godotenv library which has been designed to align with Ruby's implementation of dotEnv support, but has obvious differences with the python one. We can't be 100% compatible with the Python implementation (until we design our own dotEnv parser for this single reason)

Basically, we recommend you wrap any value that might be ambiguous with single quotes.
Docs on https://docs.docker.com/compose/ is for Compose V1, we will need to update this for Compose V2 to reflect this incompatibility.

[ndeloof]

Proposed update to docs to clarify differences between v1 and v2 regarding dotEnv file format: https://deploy-preview-13474--docsdocker.netlify.app/compose/env-file/#syntax-rules

[hpello]

Thank you! It is a lot clearer like that.

I noticed a small typo:

which will be remived from actual value

[hpello]

As for the differences I noticed with ruby's dotenv, I filed an issue in the godotenv repo: joho/godotenv#155

[zikaeroh]

I hit this as well and was very confused when all of my traefik basic auth rules started not applying.

Can this be noted somewhere more prominently, e.g. on the releases or readme with other differences? This seems like a significant change which is likely to hit people who are using things like basic auth (where the username/salt/etc are $ separated as per htaccess convention).

[ToshY]

I was beating myself senseless with a similar issue because I had a basic auth entry in my dotenv for Traefik but it just kept giving me a 403.

Variables defined by dolar sign $ inside value are substitued, unless value is wrapped by single quotes

So using the following was enough

BASIC_AUTH_USER='test'
BASIC_AUTH_PWD='$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/'

[markus-ksg]

Unfortunately the dotEnv file format is a de facto standard with no established specification.

In such cases, the first published version of a product that uses the format is the specification. Any later versions must adhere to the interpretation of the first version. This is unfortunate for the developer, but should teach them not to rely on unspecified behavior.

[markus-ksg]

Proposed update to docs to clarify differences between v1 and v2 regarding dotEnv file format: https://deploy-preview-13474--docsdocker.netlify.app/compose/env-file/#syntax-rules

Apparently, docker-compose V1 already supported single quotes around the values, so this bullet point in the old docs was already wrong:

• There is no special handling of quotation marks. This means that they are part of the VAL.

If the breaking changes cannot be avoided, at least the docs should mention that single-quoting the value will block variable substitution and is compatible with both V1 and V2.

[markus-ksg]

Apparently, docker-compose V1 already supported single quotes around the values

The behavior with single quotes around the value in docker compose V1 is actually platform-dependent:

• on Linux, single or double quotes are passed verbatim into the value of the environment variable (i.e. including the quotes)
• on Windows, single or double quotes are removed from the value of the environment variable (i.e. no quotes in the variable)

If anything changes with respect to quotes or other escape character processing when reading .env files in compose V2, then this needs to be announced loudly (and correctly) in the migration docs.

https://github.com/compose-spec/compose-spec/blob/master/spec.md sounds like the behavior of compose V2 matches V1 on Linux. That would be great!

[ndeloof]

The behavior with single quotes around the value in docker compose V1 is actually platform-dependent:

Didn't know, that's bad :'(

https://github.com/compose-spec/compose-spec/blob/master/spec.md#env_file-format is definitively wrong regarding quotes, this is not how Compose v2 behaves AFAICT, and this is not relevant from a yaml syntax point of view. Need to be fixed (cc @glours @ulyssessouza)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.