Error reading journal position

[taha-yassine]

Describe the bug
Recently Promtail stopped sending data to my Loki instance.

To Reproduce
Start Promtail addon.

Expected behavior
Promtail sends data to the configured Loki endpoint.

Logs

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/00-banner.sh
-----------------------------------------------------------
 Add-on: Promtail
 Promtail for Home Assistant
-----------------------------------------------------------
 Add-on version: 2.2.0
 You are running the latest version of this add-on.
 System: Home Assistant OS 10.1  (aarch64 / raspberrypi4-64)
 Home Assistant Core: 2023.5.2
 Home Assistant Supervisor: 2023.04.1
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
cont-init: info: /etc/cont-init.d/00-banner.sh exited 0
cont-init: info: running /etc/cont-init.d/01-log-level.sh
Log level is set to INFO
cont-init: info: /etc/cont-init.d/01-log-level.sh exited 0
cont-init: info: running /etc/cont-init.d/02-set-timezone.sh
[21:22:46] INFO: Configuring timezone
cont-init: info: /etc/cont-init.d/02-set-timezone.sh exited 0
cont-init: info: running /etc/cont-init.d/promtail_setup.sh
[21:22:46] INFO: Setting base config for promtail...
cont-init: info: /etc/cont-init.d/promtail_setup.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun promtail (no readiness notification)
s6-rc: info: service legacy-services successfully started
[21:22:47] INFO: Starting Promtail...
[21:22:48] INFO: Promtail log level set to info
[21:22:48] INFO: Handing over control to Promtail...
level=error ts=2023-05-07T19:22:48.362033984Z caller=journaltarget.go:246 msg="received error reading saved journal position" err="failed to get realtime timestamp: cannot assign requested address"
level=info ts=2023-05-07T19:22:48.367774421Z caller=server.go:288 http=[::]:9080 grpc=[::]:36197 msg="server listening on addresses"
level=info ts=2023-05-07T19:22:48.368369211Z caller=main.go:121 msg="Starting Promtail" version="(version=2.6.1, branch=HEAD, revision=6bd05c9)"

Environment:

• Add-on version: 2.2.0
• Supervisor version: supervisor-2023.04.1
• Operating system Home Assistant OS 10.1

Additional context
It could be related to grafana/loki#8163, but I'm not very familiar with Promtail so I can't tell for sure. I tried deleteting positions.yaml and restarting the addon as suggested in grafana/loki#8163 (comment), but that didn't solve the problem for me.

[taha-yassine]

systemd was updated to 252 on the latest release of HA OS (https://github.com/home-assistant/operating-system/releases/tag/10.1), I think that's the culprit.

[johntdyer]

Yep, seem to be getting the same error

[johntdyer]

@mdegat01 any chance you have an idea on how we might get around this ? I've personally come to rely on this quite a bit in my personal workflows w/ Home Assistant

[LucidityCrash]

looks like the promtail program used in this is built from https://github.com/mdegat01/promtail-journal which uses an earlier version of the lib systemd dev packages - (libsystemd-dev=247.3-7+deb11u1)
The docker image for promtail on docker hub works with systemd, any reason not to just grab the latest version of that ?

Edit : hmm ... my host that runs that docker image is running systemd 247 so using the official version may not work for that either

[LucidityCrash]

Not sure what the missing component is ... upload the repo to make a local add-on and changed it to get promtail from the latest build on docker hub (I'm using this image already for a different promtail instance) but it still isn't working. As reported in the promtail ticket ... silence in the logs. One thing missing from this add-on compared to the docs for the official docker image is when using the journal scrape the promtail docker image needs /etc/machine-id from the host which isn't possible easily with the restricted options for docker containers under Hassos ... I hacked the image and added the file but that still didn't fix anything.

I suppose the next step is to go back to building promtail from source but with the right systemd dev libraries

[LucidityCrash]

Hah, got it working !
added :
RUN echo "deb http://deb.debian.org/debian bullseye-backports main" >> /etc/apt/sources.list RUN apt-get update && apt-get install -t bullseye-backports -qy libsystemd-dev
to the dockerfile to get the newer libsystemd-dev package

[massive]

Planning to release a fork of the repo with your fix by any chance?

[LucidityCrash]

Yeah ... I was just getting the link but had a meeting :)
https://github.com/LucidityCrash/addon-promtail

Worth noting it doesn't need the /etc/machine-id that the promtail docs suggested it needs and I'd added yesterday, I took that back out today and it still worked without it :)

[LucidityCrash]

I should say this is not a proper replacement for the add-on, I have no idea how to publish one, to use it drop the promtail folder from my repo into the /addons folder in your HA instance using an ssh add-on or something like the samba add-on. I've done nothing except modify the Dockerfile to build based on the latest versions of the official Promtail image and install the backports version of the systemd-dev package, so all the links and docs etc point to this original repo.

[JeanMertz]

I've taken @LucidityCrash's fixes and published them in an add-on that you can add directly through the Home Assistant UI.

Add https://github.com/JeanMertz/addon-promtail as an add-on repository, refresh the page, and you should see "Promtail" show up.

This is a temporary forked repository, until @LucidityCrash's PR #222 is merged (once @mdegat01 is back from paternity leave 👶 🥳)

[efpe]

I couldn't use @JeanMertz's solution as for some reason the pre-built promtail version did not work on an Rpi2.
Instead, I submitted mdegat01/promtail-journal#140 to update the base image and had to create https://github.com/efpe/addon-promtail to use the package (under my user).

Once my PR to promtail-journal is merged, the original way can be used to build the base.

[mazzy89]

Is there any update to keep alive this addon? @mdegat01 can you please share your plan on this addon? Just to have a clear idea about the future. if you are not going to continue to keep it maintained then please move the ownership to someone else. Thank you

[mazzy89]

What about to vote having the forked https://github.com/JeanMertz/addon-promtail as the official addon. This repo won't see any love from the author so for the goodness of the community let's move on with a fork that it is mantained.

[LucidityCrash]

I think that would be sub optimal ... that repo was forked from my patch which uses the official promtail image and @efpe had issues with that on his pi2 ... that said I'm not sure even given the change he proposed even worked for him as it just built promtail for armv7 and this addon builds using amd64 ?

Even taking that out of the equation any repo should probably forked from this one rather than my one so there is less confusion when trying to chase down issues/things ?

Edit : @mdegat01 is clearly still active in other Home Assistant repos, just not this one, so it is not due to his recent addition to the family, this project just isn't high enough up on his priority list - nowt wrong with that.

Perhaps we could get the ownership of this one changed.
Also ideally if we do use a new repo it should be added to the community list of add-ons and have a different name to avoid confusion.

[efpe]

It worked for that arch and it’s a hacky way for sure. I also agree that the original repo should be forked and the patches added on top. I’m happy to test and review prs if needed :El 3 ene 2024, a las 11:44, LucidityCrash ***@***.***> escribió: I think that would be sub optimal ... that repo was forked from my patch which uses the official promtail image and @efpe had issues with that on his pi2 ... that said I'm not sure even given the change he proposed even worked for him as it just built promtail for armv7 and this addon builds using amd64 ? Even taking that out of the equation any repo should probably forked from this one rather than my one so there is less confusion when trying to chase down issues/things ? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[taha-yassine]

I agree that ideally gaining ownership of this repo would be the best move. It would be less confusing for newcomers. However, that entirely depends on @mdegat01. Maybe we should give it some additional time to see if he would ultimately agree. Otherwise, @LucidityCrash's repo + updating the community add-ons forum post to redirect people here seems like the second best option.

[Maarc]

Hey there! FYI, I tried setting up nearly all your forks without success, and ended forking @JeanMertz 's fork based on @LucidityCrash and doing a few changes to get promtail running on my HA instance. In case you are interested, here is the fork. HTH!