Issues with fresh OSS install

[gonzalesMK]

I executed the following command from a fresh repository clone

sudo docker compose --profile puppet up

but I was unable to run the testing command:

docker compose --profile test run testing puppet agent -t

because I got those errors in the command line:

sudo docker compose --profile test run testing puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/crafty-testing. (method :get). Please see the server logs for details.
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.20.0.4)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/crafty-testing. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/crafty-testing. (method :put). Please see the server logs for details.

Would you mind help me fix whatever I'm missing? I checked auth.conf and it seems alright.

Here is some debug info

found those errors on the log:

puppet-1       | 2024-09-03 21:12:20,219 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.20.0.7) access to /puppet/v3/catalog/crafty-testing. (method :post) (authenticated: true) denied by rule 'puppetlabs v3 catalog from agents'.
puppet-1       | 172.20.0.7 - - - 03/Sep/2024:21:12:20 +0000 "POST /puppet/v3/catalog/crafty-testing.?environment=production HTTP/1.1" 403 109 172.20.0.7 172.20.0.7 8140 6
puppet-1       | 2024-09-03 21:12:20,279 ERROR [p.t.a.rules] Forbidden request: crafty-testing.(172.20.0.7) access to /puppet/v3/report/crafty-testing. (method :put) (authenticated: true) denied by rule 'puppetlabs report'.

I also tried to run this command on the puppet container:

sudo docker container exec puppet-puppet-1  puppet agent -t

and I got those errors:

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.20.0.5)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/puppet. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/puppet. (method :put). Please see the server logs for details.

and the logs:

puppet-1       | 2024-09-03 21:15:21,841 ERROR [p.t.a.rules] Forbidden request: puppet.(172.20.0.5) access to /puppet/v3/catalog/puppet. (method :post) (authenticated: true) denied by rule 'puppetlabs v3 catalog from agents'.
puppet-1       | 172.20.0.5 - - - 03/Sep/2024:21:15:21 +0000 "POST /puppet/v3/catalog/puppet.?environment=production HTTP/1.1" 403 101 172.20.0.5 172.20.0.5 8140 9

[rwaffen]

hmm strange. i used the same commands and it succeeded for me. but i also updatet the container images of the server and puppetdb recently. may you retry again please?

did you wait until you got these lines in the log?

puppet-1       | 2024-09-06 14:06:17,915 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
...
puppetdb-1     | 2024-09-06 14:06:24,571 INFO  [p.p.pdb-routing] PuppetDB finished starting, disabling maintenance mode
...
puppetboard-1  | INFO:puppetboard.utils:PuppetDB version: 8.7.0
puppetboard-1  | [2024-09-06 14:06:26 +0000] [1] [INFO] Starting gunicorn 21.2.0
puppetboard-1  | [2024-09-06 14:06:26 +0000] [1] [INFO] Listening at: http://0.0.0.0:8088 (1)

and checked with docker compose ps if all containers are healthy?

NAME                IMAGE                                         COMMAND                  SERVICE       CREATED         STATUS                   PORTS
oss-postgres-1      docker.io/postgres:16-alpine                  "docker-entrypoint.s…"   postgres      7 minutes ago   Up 7 minutes (healthy)   5432/tcp
oss-puppet-1        ghcr.io/voxpupuli/puppetserver:8.6.1-latest   "dumb-init /docker-e…"   puppet        7 minutes ago   Up 7 minutes (healthy)   0.0.0.0:8140->8140/tcp
oss-puppetboard-1   ghcr.io/voxpupuli/puppetboard:latest          "/bin/sh -c 'gunicor…"   puppetboard   7 minutes ago   Up 6 minutes (healthy)   80/tcp, 0.0.0.0:8088->8088/tcp
oss-puppetdb-1      ghcr.io/voxpupuli/puppetdb:8.7.0-latest       "dumb-init /docker-e…"   puppetdb      7 minutes ago   Up 7 minutes (healthy)   8080/tcp, 0.0.0.0:8081->8081/tcp

[johnduarte]

I am also experiencing 403 (forbidden) errors for the catalog and report endpoints when running the testing command.

The containers were all in a healthy state prior to running the testing command.

root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker container ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED         STATUS                    PORTS
                                              NAMES
efef62966719   ghcr.io/voxpupuli/puppetserver:8.6.3-latest   "dumb-init /docker-e…"   4 minutes ago   Up 4 minutes (healthy)    0.0.0.0:
8140->8140/tcp, :::8140->8140/tcp             oss-puppet-1
dcedb4707df0   ghcr.io/voxpupuli/puppetdb:8.7.0-latest       "dumb-init /docker-e…"   4 minutes ago   Up 4 minutes (healthy)    8080/tcp
, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   oss-puppetdb-1
1be2517d34cf   ghcr.io/voxpupuli/puppetboard:latest          "/bin/sh -c 'gunicor…"   4 minutes ago   Up 16 seconds (healthy)   80/tcp,
0.0.0.0:8088->8088/tcp, :::8088->8088/tcp     oss-puppetboard-1
c78c3508a8a6   postgres:17-alpine                            "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes (healthy)    5432/tcp
                                              oss-postgres-1
root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker compose --profile test run testing puppet agent -t
[+] Creating 1/0
 ✔ Volume "oss_agent-ssl"  Created                                                                                                 0.0s
[+] Running 9/9
 ✔ testing Pulled                                                                                                                 46.3s
   ✔ 6414378b6477 Already exists                                                                                                   0.0s
   ✔ ee424688b5cb Pull complete                                                                                                    0.3s
   ✔ 9b2d84335313 Pull complete                                                                                                    0.4s
   ✔ 95a17f9f6d59 Pull complete                                                                                                    0.4s
   ✔ 9db99164c2d3 Pull complete                                                                                                   42.7s
   ✔ 220735b9dc96 Pull complete                                                                                                   43.7s
   ✔ 24efd0fb060e Pull complete                                                                                                   45.4s
   ✔ b3e1c9a498fd Pull complete                                                                                                   45.4s
Info: Creating a new RSA SSL key for crafty-testing.
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for crafty-testing.
Info: Certificate Request fingerprint (SHA256): F3:42:49:45:9B:82:F5:48:CD:84:45:46:4D:8E:90:7F:1D:89:C9:D2:30:92:47:72:81:B9:1D:89:3F:28:28:D7
Info: Downloaded certificate for crafty-testing. from https://puppet:8140/puppet-ca/v1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/crafty-testing. (method :get).  Please see the server logs for details.
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet:8140 (172.18.0.4)
Notice: Catalog compiled by puppet.
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/crafty-testing. (method :post). Please see the server logs for details.
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/crafty-testing. (method :put). Please see the server logs for details.

Environment

I am running on a Digital Ocean droplet for the Docker host:

• "size":"s-1vcpu-2gb"
• "image":"ubuntu-24-04-x64"

I am using a crafty clone at SHA 9fa6ee7 .

Docker version 27.3.1, build ce12230

Container versions pulled by docker compose --profile puppet:

root@ubuntu-s-1vcpu-2gb-sfo3-01:~/crafty/puppet/oss# docker image ls
REPOSITORY                       TAG            IMAGE ID       CREATED         SIZE
ghcr.io/betadots/pdc             latest         6a12d656eeff   2 days ago      843MB
ghcr.io/voxpupuli/puppetserver   8.6.3-latest   695634975b25   4 days ago      715MB
postgres                         17-alpine      bb46dc8bfad7   3 weeks ago     248MB
ghcr.io/voxpupuli/puppetdb       8.7.0-latest   fe14b3d33c54   6 weeks ago     589MB
ghcr.io/voxpupuli/puppetboard    latest         3498a78ecea7   7 months ago    253MB

Puppet Server container log

oss-puppet-1.log

[johnduarte]

The condition I documented above appears to be specific to DigitalOcean. I presume this is a side effect of Droplets not having a domain name, causing the puppet container's certname to be puppet. and the testing container's certname to be crafty-testing.. The empty trailing dot may be the cause of this behavior on this platform.

Attempting to reproduce this condition on an AWS EC2 instance allows the catalog and report endpoints for the puppet and crafty-testing container to be delivered as expected.

[rwaffen]

ah okay, now i see. the domain is not set and so an invalid one is generated 🤦