[Kamal 2] Are ENV variables set in .env available in .kamal/secrets?

[jessevdp]

After initializing a new project with Kamal 2.0.0.rc3, I have a setup like the following:

# config/deploy.yml

# ...

registry:
  username: my-username
  password:
    - KAMAL_REGISTRY_PASSWORD

# .kamal/secrets
KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD

# .env
KAMAL_REGISTRY_PASSWORD="<REDACTED>"

Running kamal registry login results in the following error:

$ kamal registry login
  INFO [078e4b73] Running docker login -u [REDACTED] -p [REDACTED] as me@localhost
 DEBUG [078e4b73] Command: docker login -u [REDACTED] -p [REDACTED]
 DEBUG [078e4b73]       flag needs an argument: 'p' in -p
 ERROR (SSHKit::Command::Failed): docker exit status: 256
docker stdout: Nothing written
docker stderr: flag needs an argument: 'p' in -p

---

This leads me to believe that $KAMAL_REGISTRY_PASSWORD is never filled. Do we have access to those ENV variables in .kamal/secrets?

[nickhammond]

@jessevdp There are some WIP docs for 2.0, here are a few helpful pages:

.env to Kamal secrets: https://github.com/basecamp/kamal-site/blob/kamal-2/docs/upgrading/secrets-changes.md
Kamal Secrets: https://github.com/basecamp/kamal-site/blob/kamal-2/docs/commands/secrets.md

[jessevdp]

Hey!

Thanks for taking the time to write back!

I found the same WIP documentation. It mentions that you can use values from ENV inside of the .kamal/secrets file. And that .kamal/secrets is in-and-of-itself loaded through dotenv.

I guess that this means that you can only load variables from your shell ENV. Not from dotenvs .env file.

But I wasn't completely sure...

Re-reading through those docs a bit more I suppose the section on environment variables in deploy.yml solidifies my assumption even more. The .env file is NEVER loaded in Kamal 2.

---

I want to move to 1password stuff through the kamal secrets CLI. I just hadn't gotten around to it. Perhaps I should just bite that bullet.

[tillcarlos]

For me it works to just call it with dotenv kamal deploy

[jessevdp]

Yeah fair!

That adds to my conclusion:

Kamal 2 does not load the .env file. You can only access ENV variables from your shell.

You can of course add another tool into the chain; as you suggest by prefixing with the dotenv CLI, or by using a tool like https://direnv.net/, etc.

Thanks!

[nickhammond]

Kamal just loads .kamal/secrets, .kamal/secrets.production, .kamal/secrets-common instead of .env. You can interpolate or pull from your actual ENV, it's up to you.

The upgrade guide is also out which goes over this https://kamal-deploy.org/docs/upgrading/overview/

[laptopmutia]

@nickhammond I still cannot make it works, can you help me where to put this .env files? did we need to add a gem to make it works? did I need to execute some command to make it works?

the doc is not really helping me at all

I have my old .env that works on kamal 1 that I put on my rails project roots folder
and I have set my .kamal/secrets

and it still not working

[buncis]

@jessevdp I don't what to be that guy who nitpicking

but could you just mark @tillcarlos answer as the answer, because its more straight forward, and practically solve the question.

so people that googling the answer could just find the solution,

by that we could lessen the confusions and avoiding bad approach like putting the pass to .secrets or using cat commands to copy values in the env files

[charnould]

Hi,
Kind of same problem here; I don't get how environnement variables and/or secrets work.

With Kamal 1.x, everything was working:

• simple .env file with my secrets
• in deploy.yml:

env:
   secret:
    - API_KEY

With Kamal 2.x:

• in .kamal/secrets:
  API_KEY=$API_KEY

• in .env:
  API_KEY=my-super-secret-key

• in deploy.yml:

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - <%= ENV["API_KEY"] %>

... and when I run kamal config: CRASH

 ERROR (Psych::SyntaxError): (<unknown>): found character that cannot start any token while scanning for the next token at line 51 column 7

(There is nothing weird at line 51: just another secret: - <%= ENV["PASSWORDS"] %>)

Any clue?
Docs are not that clear for me :-(
Thanks!

[jessevdp]

I don’t think you want to use ERB interpolation under env.secret. Just put the name of the ENV variables there. Does that work?

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - API_KEY

[charnould]

I don’t think you want to use ERB interpolation under env.secret.
Just put the name of the ENV variables there. Does that work?

<% require "dotenv"; Dotenv.load(".env") %>
env:
   secret:
      - API_KEY

Thanks, I've made some progress: I've got a new kind of error :-)

.env:
A list of env var.

deploy.yml:

<% require "dotenv"; Dotenv.load(".env") %>
env:
  secret:
    - OPENAI_API_KEY
    - REGISTRY_TOKEN
    - AUTH_SECRET
    - PASSWORDS
    - TELEMETRY

.kamal/secret:

OPENAI_API_KEY=$OPENAI_API_KEY
ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
REGISTRY_TOKEN=$REGISTRY_TOKEN
AUTH_SECRET=$AUTH_SECRET
PASSWORDS=$PASSWORDS
TELEMETRY=$TELEMETRY
ACCOUNT_SID=$ACCOUNT_SID
AUTH_TOKEN=$AUTH_TOKEN

...and when I run:

• dotenvx get PASSWORDS.... my passwords print correctly!
• kamal registry login... it succeeds!
• dotenvx run -- kamal upgrade (or just kamal upgrade, it outputs:

ERROR (SSHKit::Command::Failed): Exception while executing on host 138.201.246.225: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Hard to upgrade!
Anu other clue?
Thanks

[jessevdp]

This is a new kind of issue indeed.

Completely unrelated to the ENV variables.
BTW, does the setup you have now work without the “dotenv load” ERB tag in your deploy.yaml? Since you’re running the dotenvx command you shouldn’t need that line.

---

I’m actually running into a very similar issue. In my case it seems to have something to do with a docker healthcheck. My container isn’t reporting .State.Health or something like that.

Perhaps best to look for a discussion around health checks, or start a new one if there isn’t one already there.

Could you link that discussion here for me? Many thanks!

[cpburm]

ERROR (SSHKit::Command::Failed): Exception while executing on host 138.201.246.225: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

I had the same issue after fixing my initial failed deployment due to KAMAL_REGISTRY_PASSWORD not being set. I fixed it by running kamal restore to wipe the failed first deployment from the server and then running kamal setup again

[nickhammond]

@jessevdp Are you using ENV at all in your deploy recipes and do you use destinations?

[jessevdp]

Not yet, I’m not upgrading or anything I’m migrating a pretty old Rails app from Capistrano to Kamal. So for now, things that might need to be an ENV variable, or different between stages is just not setup yet.

[antarr]

This is the solution I went with. Nothing else seemed to work other than setting them in ~/.zshrc but that conflicted with my development env.

.kamal/secret

APPSIGNAL_PUSH_API_KEY=$(cat .env.production | grep APPSIGNAL_PUSH_API_KEY | cut -d '=' -f 2)
ELASTIC_APM_SERVER_URL=$(cat .env.production | grep ELASTIC_APM_SERVER_URL | cut -d '=' -f 2)
ELASTICSEARCH_URL=$(cat .env.production | grep ELASTICSEARCH_URL | cut -d '=' -f 2)
KAMAL_REGISTRY_PASSWORD=$(cat .env.production | grep KAMAL_REGISTRY_PASSWORD | cut -d '=' -f 2)
LEGISCAN_API_KEY=$(cat .env.production | grep LEGISCAN_API_KEY | cut -d '=' -f 2)
LEGISCAN_DATABASE_URL=$(cat .env.production | grep LEGISCAN_DATABASE_URL | cut -d '=' -f 2)
RAILS_MASTER_KEY=$(cat config/credentials/production.key)
VOTER_DATA_URL=$(cat .env.production | grep VOTER_DATA_URL | cut -d '=' -f 2)
S3_ACCESS_KEY_ID=$(cat .env.production | grep AWS_ACCESS_KEY_ID | cut -d '=' -f 2)
S3_SECRET_ACCESS=$(cat .env.production | grep AWS_SECRET_ACCESS_KEY | cut -d '=' -f 2)

[jessevdp]

@antarr you need the -f flag in the dotenv CLI since you're using a non-standard .env file.

dotenv -f ".env.production" kamal deploy

https://github.com/bkeepers/dotenv?tab=readme-ov-file#cli

[tillcarlos]

Yeah, I'd rather use dotenv kamal deploy (or with env.production - but then you could also scope the variables in secrets.production:

RAILS_MASTER_KEY=$PRODUCTION_MASTER_KEY

Just an example. Another idea could be to use https://direnv.net/ - then you would read it from the directory and it loads the ENVs directly into every call.

[jessevdp]

Scoping the ENV variables names is a good idea yeah!

[antarr]

is there something that prevent you to prepend your kamal command with dotenv?

Just didn't work

[tillcarlos]

@antarr - would be awesome if you could provide a bit context.

Some ideas how to debug:

Compare these different outputs:

rails runner ' ENV.each { |key, value| puts "#{key}=#{value}" }'
dotenv rails runner ' ENV.each { |key, value| puts "#{key}=#{value}" }'

I installed dotenv in my rubygems, not in the Gemfile. That might also be a hint.

And I just checked my .env file - looks like this, so I don't need to prepend that -f production file.

➜  cat .env                                                                                                                                                  
PORT=3000

# for .kamal/secrets
KAMAL_REGISTRY_PASSWORD=.....

# for .kamal/secrets-production
PRODUCTION_RAILS_MASTER_KEY=....
PRODUCTION_LITESTREAM_REPLICA_URL=s3://....roduction-replica
PRODUCTION_LITESTREAM_ACCESS_KEY_ID=....
PRODUCTION_LITESTREAM_SECRET_ACCESS_KEY=....

[paicha]

I prefer using Rails credentials to store secrets. To facilitate this, I created a custom Rails task for reading credentials:

# lib/tasks/credentials.rake
namespace :credentials do
  desc "Read a specific credential"
  task read: :environment do
    key_path = ENV['KEY'].to_s.split(',').map(&:to_sym)
    value = Rails.application.credentials.dig(*key_path)
    puts value
  end
end

This task is then utilized in the Kamal secrets file:

# .kamal/secrets

# Read secrets based on the specific environment
KAMAL_REGISTRY_PASSWORD=$(RAILS_ENV=development KEY=kamal,registry_password bin/rails credentials:read)
MYSQL_ROOT_PASSWORD=$(RAILS_ENV=production KEY=database,password bin/rails credentials:read)

# From bin/rails credentials:edit --environment=production
RAILS_MASTER_KEY=$(cat config/credentials/production.key)

This approach offers several advantages:

1. The MYSQL_ROOT_PASSWORD only appears in the accessories configuration.
2. The value for MYSQL_ROOT_PASSWORD is read from Rails.application.credentials, which is the same source used by config/database.yml.
3. It eliminates the need for additional .env files, centralizing all sensitive information in the encrypted credentials.yml.enc.

[tillcarlos]

Interesting solution. If it works for you - cool!

The only caveat for me is that usually you want to share basic development credentials (not anything being able to access production). But here you store your production credentials in the development file. If at one point the developers share the development.key you expose production.

[paicha]

The credentials are encrypted, and only I possess the production.key, so only I can decrypt the credentials for the production environment (and only I can deploy to the production server). Other developers have their own master.key to edit the credentials for their development environments.

[ACPK]

In Kamal 2, the following line causes confusion, as it implies that the key is being loaded from the .env file. This is due to the text referencing "SECRET_FROM_ENV".

# .kamal/secrets

SECRET_FROM_ENV=$SECRET_FROM_ENV

[wdiechmann]

I'm not sure I'm "at the right desk" but anyways here goes:

I found .env -> .kamal/secrets -> deploy.yml to be a very potent way to handle variables but since like yesterday (after taking a hint from dependabot to bump rails & friends to the very shiny new, I find myself faced with issues that taste like the ones mentioned in this thread - kind'a

My deploys have started to fail - and I don't really know where to start debug 😢

#18 [build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
#18 1.124 bin/rails aborted!
#18 1.125 KeyError: key not found: "MS_AD_ID" (KeyError)
#18 1.125 /rails/config/initializers/devise.rb:337:in `fetch'
#18 1.125 /rails/config/initializers/devise.rb:337:in `block in <main>'
#18 1.125 /rails/config/initializers/devise.rb:14:in `<main>'
#18 1.125 /rails/config/environment.rb:15:in `<main>'
#18 1.125 Tasks: TOP => assets:precompile => environment
#18 1.125 (See full trace by running task with --trace)
#18 ERROR: process "/bin/sh -c SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile" did not complete successfully: exit code: 1
------
 > [build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile:
1.124 bin/rails aborted!
1.125 KeyError: key not found: "MS_AD_ID" (KeyError)
1.125 /rails/config/initializers/devise.rb:337:in `fetch'
1.125 /rails/config/initializers/devise.rb:337:in `block in <main>'
1.125 /rails/config/initializers/devise.rb:14:in `<main>'
1.125 /rails/config/environment.rb:15:in `<main>'
1.125 Tasks: TOP => assets:precompile => environment
1.125 (See full trace by running task with --trace)
------
Dockerfile:50
--------------------
  48 |     
  49 |     # Precompiling assets for production without requiring secret RAILS_MASTER_KEY
  50 | >>> RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
  51 |     
  52 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile" did not complete successfully: exit code: 1

grep MS_AD_ID .env tells me that MS_AD_ID=9cf--8<--0d and likewise does grep MS_AD_ID .kamal/secrets inform me that MS_AD_ID=$(cat .env | grep MS_AD_ID | cut -d'=' -f2) and finally grep MS_AD_ID config/deploy.yml offers me - MS_AD_ID

the config/deploy.yml actually looks more like this:

service: app
image: meondocker/app
servers:
  web:
    - 2.2.2.2
proxy: 
  ssl: true
  host: app.mortimer.pro
registry:
  username: meondocker
  password:
    - KAMAL_REGISTRY_PASSWORD
builder:
  remote: ssh://[email protected]
  arch: arm64
env:
  secret:
    - SECRET_KEY_BASE
    - RAILS_MASTER_KEY
    - MS_AD_ID
    - SMTP_HOST
  clear:
    SOLID_QUEUE_IN_PUMA: true
    PORT: 3000
    RAILS_ENV: production
    PDF_HOST: 4.4.4.4
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"
volumes:
  - "storage:/rails/storage"

In config/initializers/devise.rb I apply one of the ENV's like this

  APP_ID = ENV.fetch("MS_AD_ID")

I suspect the missing ENV's are due to Kamal not loading the ENV "early on" but that's just guesswork...

[jessevdp]

It looks like you're missing an ENV variable at build time.

Kamal doesn't inject your .env nor kamal/secrets when running docker build. And for good reason! You don't want to put secrets in at the build step.

What's likely happening is that some Rails plugin, during the Rails boot process (in an initializer of some sort) is testing for that ENV variable and failing when it's not there.

This is usually a good idea. You don't want to start your app when you know you're missing an essential setting.

Except for the fact that all initializers run on rails assets:precompile.

I faced similar problems with some of the code that I owned. One of my very own initializers was doing something like raise "missing a thing" unless ENV["A_THING"].present?.

I ended up hooking into the already existing SECRET_KEY_BASE_DUMMY ENV variable which is set right there in the Dockerfile to detect this situation.

Hope this helps!

(This is a rails & built-time thing, not a Kamal thing.)

[wdiechmann]

@jessevdp thx for noting but with R8 and 'no-build' - and your point (This is a rails & built-time thing, not a Kamal thing.) - where does that leave me? If all is well in development^* then I reasoned that it had to be during the 'docker build' step (which I believe is a Kamal thing)

Solution:
I did my best to escape the config/credentials.yml.enc - but decided that battle would have to wait to another day so I begged my way back to the encrypted altar 😎

APP_ID = "#{Rails.application.credentials.dig(:microsoft, :ad_id)}"

I hope, some day, I'll find a way to avoid the credentials alltogether

Again @jessevdp - thx for pointing the catalyzing finger forcing me on my way 🫶

^* I don't do assets:precompile - only to drive my point

SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
Rebuilding...
Done in 760ms.

[tillcarlos]

I'm not exactly sure I have grasped the whole problem here. Some things you might want to check (may be redundant / already said)

• Are the ENVs in your build step declared as such in Dockerfile?
• Do you pass them in Kamal build settings? If not: are there defaults?
• Can you possibly get rid of that ENV at build step, or set a dummy? What @jessevdp said is correct: don't put secrets into your image.

My gut feeling here is that this can be figured out in a debugging session. When I have issues like this it's 99% me trying to apply standards on something that shouldn't be done in that way, in the first place.

[wdiechmann]

well - you are right, of cause 😄

I woke one morning with an urge to get rid of credentials - it has been an itch since day zero to me

Debugging my commits backwards I have learned that the moment I rid the initializers of credentials variables my deploy troubles started - so, yeah, you're right -

I like the .env -> .kamal/secrets -> config/deploy.yml so much that I (wrongly) thought I could rely on that construct all over the place but as the rest of the Internet apparently knows - that will not fly until the framework is fully loaded (or at least it will not afford me ENV's in my initializers

That - I think is/was the problem 😃

[jessevdp]

One though I have here... I'm not sure putting the value in credentials solves anything...

You're not putting your RAILS_MASTER_KEY in your docker build step right?

If you don't have the master key... you can't decrypt the credentials. I'm pretty sure that raises an error of sorts. Which crashes the docker build.

The SECRET_KEY_BASE ENV variable is the best example here. Usually the secret key base is stored in encrypted credentials.

Rails included that SECRET_KEY_BASE_DUMMY variable to get around this issue: don't rely on encrypted credentials during assets:precompile.

[wdiechmann]

nope - and I have all files tasting like secrecy added to .gitignore (.kamal/*, master.key, credentials*, more)

(it works btw - using APP_ID = "#{Rails.application.credentials.dig(:microsoft, :ad_id)}" in config/initializers/devise.rb)

but in line with the 12-factor rites IMHO .env is where "configuration" should live - no matter if this configuration pertains to setting up gems (and their initialization) or usage of the app in general with that one exception that all configuration decided by the user should "live" in a CRUD resource like /settings or like construct

From the list of issues here I sense not being the only one to 'fight' this part of the framework - I reckon Rails is in effect paying some kind of technical debt here, showing some signs of design fatigue probing various ways to solving what is not a trivial issue.