Prometheus not starting up

[matus-barta]

In logs getting error:
prometheus | level=error ts=2021-01-10T02:42:00.437Z caller=main.go:293 msg="Error loading config (--config.file=/etc/prometheus/config.yml)" err="read /etc/prometheus/config.yml: is a directory"

I am not running on rpi, I am on Ubuntu 20.10 x64
When I check "./services/prometheus/" config.yml is folder
I did try to do clean git clone of the repo and to bulid and start up only Prometheus but still the same issue.

If I can provide any more let me know

[Paraphraser]

Try:

$ cd ~/IOTstack
$ docker-compose stop prometheus
$ docker-compose rm -f prometheus
$ sudo rm -rf ./services/prometheus/config.yml
$ cp ./.templates/prometheus/config.yml ./services/prometheus/
$ docker-compose up -d prometheus

The basic problem is that docker-compose assumes a volume mapping is a directory and it auto-creates anything missing from a container's volumes list. At some point your config.yml went walkabout so the folder got created in its place.

Personally, I think it's a daft idea to set up volumes definitions that point to files, and this behaviour is one of the reasons I started to think that. I've noticed a few of these in IOTstack (and fixed a couple too) so it's not exactly "unusual", just not a good idea. If you agree, maybe you can develop a Pull Request to fix it.

Instead of

     - ./services/prometheus/config.yml:/etc/prometheus/config.yml

this would have the same effect:

     - ./services/prometheus:/etc/prometheus:ro

If the config file needs to be writable by prometheus then it'd be better to put it in ./volumes/prometheus/etc and add a directoryfix.sh script to copy it into place at menu time.

[matus-barta]

Your steps solved the issue, not sure why in my case there was a problem. If there is someone more experienced with docker and shell scripts it would be great to develop fix for future. If this issue is some edge case that people cant bother to fix, then it can be closed. @Paraphraser thank you for help!

[Paraphraser]

I don't think there's any way to "fix" this kind of problem. The real problem is that something happened on your RPi that deleted the config.yml file. If you don't remember doing that yourself then it's gremlins but, if it were me, I'd be trying to nail down the cause so it didn't happen again.

docker-compose doesn't have the ability to run self-repair scripts before bringing up a container but suppose it did, what would it do in this situation? If it copied the default config.yml from the template then that would lose any customisation you might have put in place and you'd wonder why prometheus was working but not quite right. The alternative would be to chuck up an error saying, "hey, where's my config?"

That's pretty much what happened - an error message pointing you to the right place. 😎

[matus-barta]

I am just if there is some kind of bug in IOTstack because I just did git clone and run menu.sh to build docker-compose. Nothing more, so I want to do just heads up to developers to check if there is bug and if so to fix it. If there is nothing to fix then issue can be closed.

[StohanzlMart]

I had the same problem and just skipped the Prometheus container out of desperation. Thanks for coming in clutch again Paraphraser!

[Paraphraser]

I'm going to add some cross-references here in case they help with issues which have the same basic underlying cause (new menu not really supporting service definitions which have not yet been updated with a "build"). I think that's the most likely reason why config.yml is not where it is meant to be.

• Discord comment
• Issue 210
• Issue 224 on gcgarner

[KlausHans]

Just want to add, that i got the same problem and when i tried the fix from @Paraphraser i got the error:

pi@raspberrypi:~/IOTstack $ cp ./.templates/prometheus/config.yml ./services/prometheus/
cp: reguläre Datei './services/prometheus/config.yml' kann nicht angelegt werden: Keine Berechtigung

(it translates to: ... could not be created. No permission.)
With sudo it works. Maybe this is a hint why the config.yml could not be created in the first place?