-
Subject of the issueThe step-ca container health state is shown as Your environment
Steps to reproduce
version: '3.7'
services:
# Smallstep Step CA
step-ca:
image: smallstep/step-ca:0.15.6
restart: alwaysExpected behaviourAs the CA service runs correctly, the health check should pass and the container state should become Actual behaviourHealth check needs too long ( |
Beta Was this translation helpful? Give feedback.
Replies: 24 comments 3 replies
-
|
@strarsis Thanks for the report. I have a couple questions.
When I try to reproduce this with our Docker tutorial, the container health check works. |
Beta Was this translation helpful? Give feedback.
-
|
@tashian: One factor could be that WSL 2 with Docker for Desktop is used. |
Beta Was this translation helpful? Give feedback.
-
|
@strarsis Please provide more details about the environment and steps to reproduce, so we can test this. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
I am using WSL 2 with Docker for Desktop and getting the same issue. |
Beta Was this translation helpful? Give feedback.
-
|
Hi folks, I am having an issue with this container's healthcheck on docker swarm as well. I can reproduce very easily and I think I know what the problem may be (for me, at least). docker run -it -e STEPDEBUG=1 smallstep/step-ca:0.16.0 sh
~ $ step ca init
✔ (e.g. Smallstep): Deisel█
What DNS names or IP addresses would you like to add to your new CA?
✔ (e.g. ca.smallstep.com[,1.1.1.1,etc.]): ca.diesel.net
What IP and port will your new CA bind to?
✔ (e.g. :443 or 127.0.0.1:4343): :443
What would you like to name the CA's first provisioner?
✔ (e.g. [email protected]): test
Choose a password for your CA keys and first provisioner.
✔ [leave empty and we'll generate one]: █
✔ Password: gj%Dyy-[BuKp#.EP(%vl,!#{`fF4$cH,
Generating root certificate...
all done!
Generating intermediate certificate...
all done!
✔ Root certificate: /home/step/certs/root_ca.crt
✔ Root private key: /home/step/secrets/root_ca_key
✔ Root fingerprint: 2675480ce53fa83431099ddafe152f532ad0a197a6784a1a4641be32969f2578
✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
✔ Database folder: /home/step/db
✔ Default configuration: /home/step/config/defaults.json
✔ Certificate Authority configuration: /home/step/config/ca.json
Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.
FEEDBACK 😍 🍻
The step utility is not instrumented for usage statistics. It does not
phone home. But your feedback is extremely valuable. Any information you
can provide regarding how you’re using `step` helps. Please send us a
sentence or two, good or bad: [email protected] or join
https://github.com/smallstep/certificates/discussions.
~ $ step ca health
Get "https://ca.diesel.net/health": x509: certificate is valid for 9721f7d721878f7496b87c17dcab760d.2868b98699e09c78a80c69bee273ddd8.traefik.default, not ca.diesel.net
client.Health; client GET https://ca.diesel.net/health failed
github.com/smallstep/certificates/errs.Wrapf
/go/pkg/mod/github.com/smallstep/[email protected]/errs/error.go:122
github.com/smallstep/certificates/ca.(*Client).Health
/go/pkg/mod/github.com/smallstep/[email protected]/ca/client.go:612
github.com/smallstep/cli/command/ca.healthAction
/src/command/ca/health.go:79
github.com/urfave/cli.HandleAction
/go/pkg/mod/github.com/urfave/[email protected]/app.go:526
github.com/urfave/cli.Command.Run
/go/pkg/mod/github.com/urfave/[email protected]/command.go:174
github.com/urfave/cli.(*App).RunAsSubcommand
/go/pkg/mod/github.com/urfave/[email protected]/app.go:407
github.com/urfave/cli.Command.startApp
/go/pkg/mod/github.com/urfave/[email protected]/command.go:373
github.com/urfave/cli.Command.Run
/go/pkg/mod/github.com/urfave/[email protected]/command.go:102
github.com/urfave/cli.(*App).Run
/go/pkg/mod/github.com/urfave/[email protected]/app.go:279
main.main
/src/cmd/step/main.go:98
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371I use traefik as a reverse proxy in front of step-ca. It is set up as a simple TCP relay, and is letting the step-ca container handle all of the TLS itself. This was working perfectly in version 0.15.4 (BEFORE the healthcheck was added). For those unfamiliar, Traefik simply looks at docker labels in order to know how to route to the container and do all of its proxying, due to its simplicity it has become a very popular choice among docker swarm and Kubernetes stacks. TLDR; The problem is that Traefik will not set up the routing until the Healthcheck is passed, however the healthcheck relies on dns resolution and any proxy configuration to already be set up correctly and working in order to succeed. Is there some way we can disable the health check for more compatibility with a setup like mine? Another thought I had was to perhaps change the healthcheck to check against |
Beta Was this translation helpful? Give feedback.
-
|
@tomdaley92 the problem with your health check is that the dnsNames in the ca.json should have Looking at the error it looks like the domain |
Beta Was this translation helpful? Give feedback.
-
|
Hi @tomdaley92, The health check just runs |
Beta Was this translation helpful? Give feedback.
-
Right, so this is exactly my point. Since I have a DNS record pointing to the VM that the proxy lives on, Traefik is throwing up the |
Beta Was this translation helpful? Give feedback.
-
I will try this, but I would assume the domain I feed step-ca with is needed for it to know what certificate to generate/serve when a client hits |
Beta Was this translation helpful? Give feedback.
-
|
@tashian no luck with setting Here is my output: docker run -it -e STEPDEBUG=1 smallstep/step-ca:0.16.0 sh
~ $ step ca init
What would you like to name your new PKI?
✔ (e.g. Smallstep): Diesel
What DNS names or IP addresses would you like to add to your new CA?
✔ (e.g. ca.smallstep.com[,1.1.1.1,etc.]): localhost█
What IP and port will your new CA bind to?
✔ (e.g. :443 or 127.0.0.1:4343): :443█
What would you like to name the CA's first provisioner?
✔ (e.g. [email protected]): [email protected]
Choose a password for your CA keys and first provisioner.
✔ [leave empty and we'll generate one]:
✔ Password: b^MAT=f9<v=c$IMzRBz[!253V/,k;u7C
Generating root certificate...
all done!
Generating intermediate certificate...
all done!
✔ Root certificate: /home/step/certs/root_ca.crt
✔ Root private key: /home/step/secrets/root_ca_key
✔ Root fingerprint: a19b0ce1f59fc67f36e362675f77e8c46687e4bbe9e66d3ca8439b45159b1d07
✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
✔ Database folder: /home/step/db
✔ Default configuration: /home/step/config/defaults.json
✔ Certificate Authority configuration: /home/step/config/ca.json
Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.
FEEDBACK 😍 🍻
The step utility is not instrumented for usage statistics. It does not
phone home. But your feedback is extremely valuable. Any information you
can provide regarding how you’re using `step` helps. Please send us a
sentence or two, good or bad: [email protected] or join
https://github.com/smallstep/certificates/discussions.
~ $
~ $ step ca health
Get "https://localhost/health": dial tcp 127.0.0.1:443: connect: connection refused
client.Health; client GET https://localhost/health failed
github.com/smallstep/certificates/errs.Wrapf
/go/pkg/mod/github.com/smallstep/[email protected]/errs/error.go:122
github.com/smallstep/certificates/ca.(*Client).Health
/go/pkg/mod/github.com/smallstep/[email protected]/ca/client.go:612
github.com/smallstep/cli/command/ca.healthAction
/src/command/ca/health.go:79
github.com/urfave/cli.HandleAction
/go/pkg/mod/github.com/urfave/[email protected]/app.go:526
github.com/urfave/cli.Command.Run
/go/pkg/mod/github.com/urfave/[email protected]/command.go:174
github.com/urfave/cli.(*App).RunAsSubcommand
/go/pkg/mod/github.com/urfave/[email protected]/app.go:407
github.com/urfave/cli.Command.startApp
/go/pkg/mod/github.com/urfave/[email protected]/command.go:373
github.com/urfave/cli.Command.Run
/go/pkg/mod/github.com/urfave/[email protected]/command.go:102
github.com/urfave/cli.(*App).Run
/go/pkg/mod/github.com/urfave/[email protected]/app.go:279
main.main
/src/cmd/step/main.go:98
runtime.main
/usr/local/go/src/runtime/proc.go:225
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371Again, that's just an adhoc command to debug, but my actual stack is on docker swarm: https://github.com/Diesel-Net/step-ca |
Beta Was this translation helpful? Give feedback.
-
|
If I replace
|
Beta Was this translation helpful? Give feedback.
-
|
Again thanks for the help everyone, don't mean to blow up this thread. Just posting my findings. It looks like Traefik doesn't have a way to "not require" health checks in order to set up proxy configurations either Looks like my only option might be to either enable TLS termination on traefik (but traefik uses Step-ca as acme client so another chicken and egg problem) or to build a custom docker image without the healthcheck. It would be awesome if the healthcheck was made configurable but I can see why this might dismissed pretty quick |
Beta Was this translation helpful? Give feedback.
-
|
The name in the health check URL has to match a name in When you got |
Beta Was this translation helpful? Give feedback.
-
Ahh I will try that, thanks again. FYI that command output |
Beta Was this translation helpful? Give feedback.
-
|
Wewt that was it @tashian I now have a succesfull healthcheck! thank you taking time out of your day to help me with this, really appreciate it 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Happy to help. The command to start |
Beta Was this translation helpful? Give feedback.
-
|
@tomdaley92 Ok I see what's going on with your last output, step-ca is not running. Looking at your ansible configuration in your github, you're mounting a pre-created configuration, good. So to imitate this using Then start the ca with the volume mounted, using the default command and running the health check: And in another terminal, exec in and try to health: And if you look at the output of the step-ca, you will see that health check (the one in the docker file) is running every 30s |
Beta Was this translation helpful? Give feedback.
-
|
Follow up with recent |
Beta Was this translation helpful? Give feedback.
-
|
I ran into the same DNS resolution problem when using docker swarm. Instead of modifying ca.json and defaults.json, I used the extra_hosts service option to provide DNS resolution in the container of ca.diesel.net to 127.0.0.1. This is my stack compose file. Note that it takes around 30 seconds for the service to finish coming up. version: '3.4'
networks:
step:
external: true
volumes:
step_home_step:
external: true
services:
step:
extra_hosts:
- 'ca.diesel.net:127.0.0.1'
image: smallstep/step-ca:0.20.0
networks:
- step
volumes:
- source: step_home_step
target: /home/step
type: volume
volume:
nocopy: true |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a ton for this! I've spent literally hours trying to figure out why my container wasn't starting properly. In hindsight I think what's happening is the docker swarm doesn't publish ports directly (unless you use host networking), but rather uses its ingress network. So when using swarm the ports aren't forwarded to the container per se. They go through the ingress network. You can see that by running The ingress network won't forward traffic however until the container is "started", which won't happen until it passes the health check. So there's a bit of a catch 22 where the swarm is waiting for the service to become healthy to start forwarding traffic to it, but the service can't become healthy until traffic is forwarded to it. Maybe it'd be good to add to the documentation that using swarm mode may require disabling health checks or doing this hosts workaround. |
Beta Was this translation helpful? Give feedback.
-
|
@sunvalleyfoods that's a nice approach! |
Beta Was this translation helpful? Give feedback.
-
|
@sunvalleyfoods: Thanks for this sample configuration! However, when I add |
Beta Was this translation helpful? Give feedback.
-
|
So the health check command Docker uses is defined here: When I use
Now the paths to these configuration files are passed to the smallstep CA daemon command in command: "step-ca /step/config/ca.json --password-file=/step/secrets/password"These arguments are not set using environment variables, hence the health check command won't re-use them (and it doesn't attach itself to a running smallstep CA daemon where it could get its starting arguments from). To make this all work, the path to the step CA directory must be passed by environment variable instead. version: '3.7'
services:
# Smallstep Step CA
step-ca:
image: smallstep/step-ca:0.21.0
restart: always
tty: true # currently needed for step-ca in docker
environment:
STEPPATH: "/step"
command: "step-ca --password-file=/step/secrets/password"
# health workaround (@see https://github.com/smallstep/certificates/discussions/1020#discussioncomment-3418228)
extra_hosts:
- "ca.localhost:127.0.0.1"
Now the health check command will also use the same step CA directory and find the required configuration. Another issue arose as the command is now able to run, but can't connect to the step CA:
This appears to uncover a bug, as the step CA health check command states: This looks pretty close already, but the health check command is unable to connect to the step CA, maybe a port issue? In the used configuration file "address": ":443"So the
The extra host mapping of extra_hosts:
- "ca.localhost:127.0.0.1"The current, working version: '3.7'
services:
# Smallstep Step CA
step-ca:
image: smallstep/step-ca:0.21.0
restart: always
tty: true # currently needed for step-ca in docker
environment:
STEPPATH: "/step"
command: "step-ca --password-file=/step/secrets/password"
volumes:
- "./step:/step"
- "./db:/db"
{
"ca-url": "https://ca.localhost:443",
"ca-config": "/step/config/ca.json",
"fingerprint": "[...]",
"root": "/step/certs/root_ca.crt"
}
{
[...]
"address": ":443",
"dnsNames": [
"ca.localhost"
],
[...] |
Beta Was this translation helpful? Give feedback.
-
|
I'm glad to hear you got this sorted out! |
Beta Was this translation helpful? Give feedback.
-
|
Here's my humble thoughts in case it helps someone: When running in docker swarm mode, the problem that Tom and I had was with the health check using the DNS name when connecting to the server and not being able to resolve it to that container. Like Tom, I'm also running Step behind Traefik. Since multiple instances can be running behind the load balancer and on different nodes, the resolution to the specific instance has to be done very close to the container. In swarm, when the service fails the health check, the container is removed and a new one is created. Without the resolution it loops forever. One proposed way to solve it was to create custom edits to the configuration files so it would look for the server at "localhost" and be able to resolve it. I proposed to instead give the container a way to resolve the DNS name without customizing the configuration files and having Docker inject the DNS name resolution to the loopback IP address. Strarsis, Using a persistent overlay volume at /home/step makes everything flow nicely. Run the vanilla container with the volume mapped and then run "step ca init" within the container to generate the config files. Create a Docker Secret to provide the password and you should be good to go. I'm curious how you're resolving ca.localhost to the container? |
Beta Was this translation helpful? Give feedback.
-
When I understand this correctly, it would be better to use a Docker volume instead of a file system mount?
I assigned |
Beta Was this translation helpful? Give feedback.
-
|
The magic is the extra_hosts entry that resolves ca.localhost to 127.0.0.1. Same as adding the entry to /etc/hosts without all the pain of doing it to the container. Looks like its called --add-host for the regular Docker client interface. BTW, I'm using a single node Docker swarm and stacks which feels so much simpler to me. Also, you can get the same effect or a file system mount with volume. I just bind the volume to a directory in my Git repo and all is well. I put an example below. |
Beta Was this translation helpful? Give feedback.
So the health check command Docker uses is defined here:
https://github.com/smallstep/certificates/blob/master/docker/Dockerfile.step-ca#L25
(
step ca health 2>/dev/null | grep "^ok" >/dev/null)When I use
docker compose execto invoke that command on the running container the command demands additional CLI arguments.The
ca healthcommand doesn't attach to a running step CA daemon, instead it does a standalone check (whether it can connect to it).docker compose exec step-ca step ca health'step ca health' requires the '--ca-url' flagWith the
--ca-urlflag is passed (--ca-url=https://ca.localhost):'step ca health' requires the '--root' flagWith the
-rootflag also passed (--root):flag…