docker-compose up is not starting icinga2

[pnunn]

Hi,

I've cloned the repo and changed the root mysql password and generally followed the bouncing ball.

However, when I try and bring up the machines icinga2 fails with errors about not being able to log in with the root user.

I started the mariadb and graphite containers on their own and the root mysql password seems to be correct so I'm at a bit of a loss.

The errors are

icinga2_1 | warning/cli: Feature 'command' already enabled.
mysql_1 | 2020-04-11 1:59:56 29 [Warning] Access denied for user 'root'@'172.18.0.3' (using password: YES)
icinga2_1 | ERROR 1045 (28000): Access denied for user 'root'@'172.18.0.3' (using password: YES)
icinga2_1 | run-parts: /opt/setup/50-icinga2 exited with return code 1
graphite | 11/04/2020 01:59:57 :: [tagdb] Tagging carbon.agents.graphite-a.updateOperations
graphite | 11/04/2020 01:59:57 :: [tagdb] Tagged carbon.agents.graphite-a.updateOperations in 0.008032083511352539
mysql_1 | 2020-04-11 2:00:00 30 [Warning] Access denied for user 'root'@'172.18.0.3' (using password: NO)
mysql_1 | 2020-04-11 2:00:00 31 [Warning] Access denied for user 'root'@'172.18.0.3' (using password: YES)

and then icinga fails with an exit code of 1

Any ideas?

[jjethwa]

Hi @pnunn

I think it might be because MySQL/MariaDB only permits root logins from localhost by default. You'll need to enable root login from the remote host or in this case, from the pod IP set: https://mariadb.com/kb/en/configuring-mariadb-for-remote-client-access/

[pnunn]

hi @jjethwa,

I'm a bit new to this I have to admit... how do I change that? I'm using the docker-compose file from the repo... should it be set in there somewhere?

Sorry if this is obvious.

OK, worked out from the link that it was a database operation.. sorry, my lack of reading properly.

So, privileges for root were already in place for '%' but I added '172.18.0.%' as well just in case, sadly, some errors still.

[jjethwa]

Hi @pnunn

No problem at all and sorry that it did not work. Are you seeing the same error in the container startup?

Access denied for user 'root'@'172.18.0.3

[pnunn]

Hi @jjethwa

Don't be sorry, probably my lack of knowledge.

The logs look like this

Generating locales (this might take a while)... Generation complete. Clearing symlinks in /etc/ssl/certs... done. Updating certificates in /etc/ssl/certs... 128 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. Site 000-default already enabled Site icingaweb2-ssl already disabled Site icingaweb2-ssl-redirect already disabled Pinging mysql:3306 [2020-04-12 22:07:11 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted" [2020-04-12 22:07:11 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted" warning/cli: Feature 'ido-mysql' already enabled. warning/cli: Feature 'livestatus' already enabled. warning/cli: Feature 'compatlog' already enabled. warning/cli: Feature 'command' already enabled. ERROR 1045 (28000): Access denied for user 'root'@'172.18.0.4' (using password: YES) run-parts: /opt/setup/50-icinga2 exited with return code 1

I tried to attache to the machine to see if I could get to the mysql server, but it dies before I can get connected unfortunately.

Happy to help debug this but a bit at a loss as to how. :)

[pnunn]

Looking further at the logs.. it seems to be trying to log into itself as root and that's what is failing. The mysql server is on 0.2 not 0.4.

[jjethwa]

Ah nice find! The variable DEFAULT_MYSQL_HOST should be set within the docker-compose file to mysql which is what the container running MariaDB is named. Somehow it's defaulting to localhost though and I'm not sure why. As a test, try hardcoding the IP of the MariaDB container as the value for DEFAULT_MYSQL_HOST in your docker-compose and see if it works.

[pnunn]

Hmm... maybe I'm wrong here... I made that change to the compose file, but I'm still getting

icinga2_1 | done.
icinga2_1 | Site 000-default already enabled
icinga2_1 | Site icingaweb2-ssl already disabled
icinga2_1 | Site icingaweb2-ssl-redirect already disabled
icinga2_1 | Pinging 172.18.0.2:3306
icinga2_1 | [2020-04-13 05:48:25 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
icinga2_1 | [2020-04-13 05:48:25 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
icinga2_1 | warning/cli: Feature 'ido-mysql' already enabled.
icinga2_1 | warning/cli: Feature 'livestatus' already enabled.
icinga2_1 | warning/cli: Feature 'compatlog' already enabled.
icinga2_1 | warning/cli: Feature 'command' already enabled.
icinga2_1 | ERROR 1045 (28000): Access denied for user 'root'@'172.18.0.4' (using password: YES)
icinga2_1 | run-parts: /opt/setup/50-icinga2 exited with return code 1
icinga2_1 | Generating locales (this might take a while)...
icinga2_1 | Clearing symlinks in /etc/ssl/certs...
icinga2_1 | Generation complete.
graphite | 13/04/2020 05:48:28 :: [listener] MetricLineReceiver connection with 127.0.0.1:44264 established
graphite | 13/04/2020 05:48:28 :: [listener] MetricLineReceiver connection with 127.0.0.1:44264 closed cleanly
icinga2_1 | done.
icinga2_1 | Updating certificates in /etc/ssl/certs...
icinga2_1 | 128 added, 0 removed; done.
icinga2_1 | Running hooks in /etc/ca-certificates/update.d...
icinga2_1 | done.
icinga2_1 | Site 000-default already enabled
icinga2_1 | Site icingaweb2-ssl already disabled
icinga2_1 | Site icingaweb2-ssl-redirect already disabled
icinga2_1 | Pinging 172.18.0.2:3306
mysql_1 | 2020-04-13 5:48:29 117 [Warning] Access denied for user 'root'@'172.18.0.4' (using password: NO)
icinga2_1 | [2020-04-13 05:48:29 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
icinga2_1 | [2020-04-13 05:48:29 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
icinga2_1 | warning/cli: Feature 'ido-mysql' already enabled.
icinga2_1 | warning/cli: Feature 'livestatus' already enabled.
icinga2_1 | warning/cli: Feature 'compatlog' already enabled.
icinga2_1 | warning/cli: Feature 'command' already enabled.
mysql_1 | 2020-04-13 5:48:29 118 [Warning] Access denied for user 'root'@'172.18.0.4' (using password: YES)
icinga2_1 | ERROR 1045 (28000): Access denied for user 'root'@'172.18.0.4' (using password: YES)
icinga2_1 | run-parts: /opt/setup/50-icinga2 exited with return code 1
mysql_1 | 2020-04-13 5:48:33 119 [Warning] Access denied for user 'root'@'172.18.0.4' (using password: NO)
mysql_1 | 2020-04-13 5:48:34 120 [Warning] Access denied for user 'root'@'172.18.0.4' (using password: YES)
icinga2_icinga2_1 exited with code 1
graphite | 13/04/2020 05:48:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:44284 established
graphite | 13/04/2020 05:48:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:44284 closed cleanly
graphite | 13/04/2020 05:48:39 :: [tagdb] Tagging carbon.agents.graphite-a.errors
graphite | 13/04/2020 05:48:39 :: [tagdb] Tagged carbon.agents.graphite-a.errors in 0.008699655532836914

But.. it does seem to be mysql1 that is refusing access to icinga2.

[pnunn]

OK.. I installed a ubuntu container on the same network as icinga2 and installed mysql client into it.

I then tried to connect to the mysql server with the following results.

Processing triggers for libc-bin (2.27-3ubuntu1) ...
root@3cf153a62d9b:/# mysql -p 172.18.0.2 -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root@3cf153a62d9b:/# mysql -p mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Not sure if that helps or confuses things.

[pnunn]

But this does work

root@83d5cb8d81d8:/# mysql -h 172.18.0.2 --protocol=tcp -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 23
Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic mariadb.org binary distribution

Interesting.

[jjethwa]

Thanks for the additional troubleshooting. So the last comment does show that MySQL is up and running and allowing remote access as you were able to connect to it from the Ubuntu container. I really think it's something with the DEFAULT_MYSQL_HOST var not being set properly so it's defaulting to trying to connect to a local MySQL/MariaDB which doesn't exist in this case. For another test, can you try running the container in a standalone mode? Something like

sudo docker run -p 80:80 -h icinga2 -t jordan/icinga2:latest

and see if it's able to come up properly? This will run MariaDB inside the same container as Icinga2.

[pnunn]

OK, I think the stand alone one started OK, especially when I put it on the same network as the graphite machine..

I then blew away the other instances and simplified the root password thinking that it might be due to the # in the password. Unfortunately I still get mysql trying to open on the icinga2 host regardless of the host name in the env file, and now I can't even log into root mysl on the actual mysql server (access denied errors). Sigh.. no idea where to go now.

[pnunn]

Having said that... I just looked at the end of the logs in the icinga instance and there are a bunch of these

Pinging 172.18.0.2:3306
Pinging 172.18.0.2:3306
Pinging 172.18.0.2:3306

which would seem to suggest it now knows where the mysql server is, just can't log in.

[pnunn]

Right.. I removed all of the containers and volumes and tried again with a simpler password and it seems to have come up now (I can see things in the logs that indicate its all working).. however.. I had to redirect the ports 4443->443 and 4080->80 because of other machines running on the docker host and now I'm getting connection refused when I go to either http://192.168.44.144:4080 or https://192.168.44.144:4443.

I get connection refused even from the host console itself with curl.

Is this because its on its own network perhaps? Although if that were the case it wouldn't have complained about the clashing ports.. hmm..

Got to be close now :)

[pnunn]

OK, this is a bit hinky... I just logged into the icinga2 container and there doesn't seem to be any web server running on here unless I'm missing something.

[pnunn]

OK, seems it a permissions issue.

Fatal error: Uncaught Icinga\Exception\ConfigurationError: Can't save session, path '/var/lib/php/sessions' is not writable. in /usr/share/php/Icinga/Web/Session/PhpSession.php:98 Stack trace: #0 /usr/share/php/Icinga/Web/Session/PhpSession.php(48): Icinga\Web\Session\PhpSession->__construct(Array) #1 /usr/share/php/Icinga/Web/Session.php(32): Icinga\Web\Session\PhpSession::create() #2 /usr/share/php/Icinga/Application/Web.php(363): Icinga\Web\Session::create() #3 /usr/share/php/Icinga/Application/Web.php(89): Icinga\Application\Web->setupSession() #4 /usr/share/php/Icinga/Application/ApplicationBootstrap.php(369): Icinga\Application\Web->bootstrap() #5 /usr/share/php/Icinga/Application/webrouter.php(99): Icinga\Application\ApplicationBootstrap::start() #6 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...') #7 {main} thrown in /usr/share/php/Icinga/Web/Session/PhpSession.php on line 98

I'm going to blow it all away again and restart as root and see what happens.

[pnunn]

OK, no change.. apache2 fails to start and manually running apache2 results in the error above even when started as sudo.

Sigh.

[pnunn]

So.. change data pemissions to a+rw for all directories and files and I've managed to manually start apache2 in the icinga2 container.

I now get

Should I now manually create a token and set it up or does this just mean something is still broken.

ps -A gives me

root@icinga2:/# ps -A
PID TTY TIME CMD
1 ? 00:00:00 run
10 ? 00:00:00 run-parts
1340 ? 00:00:00 50-icinga2
1344 pts/0 00:00:00 bash
1368 ? 00:00:00 apache2
1371 ? 00:00:00 apache2
1372 ? 00:00:00 apache2
1373 ? 00:00:00 apache2
1374 ? 00:00:00 apache2
1375 ? 00:00:00 apache2
1395 ? 00:00:00 apache2
1396 ? 00:00:00 apache2
1397 ? 00:00:00 apache2
1399 ? 00:00:00 mysqladmin
1400 pts/0 00:00:00 ps

[jjethwa]

It should have set up the token for you automatically and you shouldn't have needed to do anything with the permissions either. This is really strange. What version of docker are you running and what platform? Starting the container with just the docker run command like I suggested should have given you a fully up and running icinga2 instance with MariaDB and Apache... all self-contained 😟

[pnunn]

pnunn@dockerhost:~$ docker -v
Docker version 19.03.6, build 369ce74a3c

on linux.

Should clarify.. the above message is back to the multi machine setup using docker-compose, that's the one I'm trying to debug. I'll go ahead and create the token and see what happens. Just a bit concerned that other things may not have been set up along with the token.

[jjethwa]

I'll download that version and give it a go tomorrow. I agree with you, let's hope nothing else missed the setup step. It should have errored out when the container started up, but since we're in uncharted territory, anything is possible 😛

[pnunn]

OK... there don't appear to be any more errors in the logs.. but apache didn't start when the container did so something is grumpy still by the looks.. I'll go ahead and see if I can work out how to get this thing running from here later today.

[jjethwa]

Oh no! Really sorry about this. Maybe try the docker run command on a clean VM too if you can:

sudo docker run -p 80:80 -h icinga2 -t jordan/icinga2:latest

[pnunn]

OK, so the docker-compose system is obviously still borked... I tried manually setting up the icingaweb2 configuration, but ended up with this.

Might just dump all of this and go with the single machine setup instead.

[pnunn]

OK, I've just worked out that the docker-compose up didn't create any database schema's either so I guess it didn't get through the setup process.

[jjethwa]

Yeah, that makes sense, the step that failed with connecting to the database as root is so it can set up the schema 😢

[pnunn]

OK, got it...

So, I think it was because there was no container name for the mysql container.

I added container_name:mysql to the docker-compose file.

I also created the data directory after the git clone and set it to a+w.

I the fixed the rest of the config items in docker-compose and started everything and its now working (does a little happy dance).

I think I needed both of the fixes above to make it run, but I've got it going, blown it away, re-cloned and stared it again a number of times now and it seems to be good.

Now all I need to do is work out how to use icinga and graphite :)

[jjethwa]

That's awesome! Do you mind sharing the change in the docker-compose file for the MySQL/MariaDB as an example?

[pnunn]

Sure.. here it is.

docker-compose.yml.txt

(had to add the .txt to upload).

[jjethwa]

Thanks, @pnunn

So looks like the docker-compose file was missing

container_name: mysql

I'll add it now. Let me know if anything else was missing 😄

[jiri-lenik]

OK, got it...

So, I think it was because there was no container name for the mysql container.

I added container_name:mysql to the docker-compose file.

I also created the data directory after the git clone and set it to a+w.

I the fixed the rest of the config items in docker-compose and started everything and its now working (does a little happy dance).

I think I needed both of the fixes above to make it run, but I've got it going, blown it away, re-cloned and stared it again a number of times now and it seems to be good.

Now all I need to do is work out how to use icinga and graphite :)

Hi @pnunn, what exactly you did with "I also created the data directory after the git clone and set it to a+w."?
Because I am having same problem as you had, checked all the envs, bindings etc, but still have the same issues....

[r0b1ndot]

Hi @jjethwa ;
same issue still for:

root@XXXXXX:~/icinga2# docker-compose -v
docker-compose version 1.28.5, build c4eb3a1f
root@XXXXX:~/icinga2# docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:17:50 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

i guess it's related to the jordan/icinga2 version, i used an older tag:

jordan/icinga2:2.10.2

with that the database connection worked liked a charm

[jangeador]

I was having problems as described on this post with the latest image 2.13.5. I tried an older image as @reyerdam suggested and it worked. I then tried several images on the 2.13 family, and it seems that 2.13.1 and 2.13.2 work. Then 2.13.3 and 2.13.5 give me the following error:

icinga2     | /opt/setup/55-cron: line 11: /etc/cron.d/icinga/icinga-certs: Operation not permitted
icinga2     | run-parts: /opt/setup/55-cron exited with return code 1

Looking at the commit history, it seems that the cron support is causing this problem as the 55-cron file was introduced between the release of 2.13.2 and 2.13.3 . Let me know if there is something I could do to help to fix this.

EDIT:
Correction: 2.13.2 version does not run either, so the problem must have been introduced there. 2.13.1 runs fine.

[jjethwa]

Hey @jangeador I'm on vacation, but will take a look as soon as I get back home 😄

[jjethwa]

Hi @jangeador

Thanks for your patience, I'm back! 😄

I ran through a quick test following the instructions here: https://github.com/jjethwa/icinga2#docker-compose and it seemed to work without throwing the error you received. For reference, I am on docker-compose version 1.29.2, build 5becea4c Do you mind sharing your docker-compose file if you have modified it?