tls: failed to verify certificate: x509: certificate signed by unknown authority

[MSommer95]

What went wrong?

What happened:

• When trying to access OnCall after upgrading from 1.8.13 to 1.13.6 the requess fail with the following message:

error setting up request headers: error making request: Get "https://localdomain.local/grafana/api/org/users": tls: failed to verify certificate: x509: certificate signed by unknown authority 

• The Oncall and Grafana Containers run in seperate pods in a kubernetes environment.
• With version 1.8.13 the connection can be established just fine and OnCall works.
• After upgrading to 1.13.6 or setting up OnCall completely fresh the connection fails to establish

These are the logs from Grafana:

logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.665924351Z level=error msg="getting incident plugin settings" error="error making request: Get \"https://localdomain.local/grafana/api/plugins/grafana-incident-app/settings\": tls: failed to verify certificate: x509: certificate signed by unknown authority, https://localdomain.local/grafana/api/plugins/grafana-incident-app/settings"
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.668043731Z level=error msg="getting labels plugin settings" error="error making request: Get \"https://localdomain.local/grafana/api/plugins/grafana-labels-app/settings\": tls: failed to verify certificate: x509: certificate signed by unknown authority, https://localdomain.local/grafana/api/plugins/grafana-labels-app/settings"
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.67175386Z level=info msg=GetSyncData time=1
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.671766929Z level=error msg="Error getting sync data" error="error getting users = error making request: Get \"https://localdomain.local/grafana/api/org/users\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.68224683Z level=info msg=GetUser user="map[Email:[email protected] Login:admin Name:admin Role:Admin]"
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.684751198Z level=error msg="Error getting user" error="error making request: Get \"https://localdomain.local/grafana/api/org/users\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
logger=plugin.grafana-oncall-app t=2024-12-04T15:59:43.684763408Z level=error msg="Error validating oncall plugin settings" error="error setting up request headers: error making request: Get \"https://localdomain.local/grafana/api/org/users\": tls: failed to verify certificate: x509: certificate signed by unknown authority "
logger=context userId=1 orgId=1 uname=admin t=2024-12-04T15:59:43.684904621Z level=error msg="Request Completed" method=GET path=/api/plugins/grafana-oncall-app/resources/plugin/status status=500 remote_addr=192.168.1.97 time_ms=5 duration=5.632089ms size=196 referer=https://localdomain.local/grafana/plugins/grafana-oncall-app handler=/api/plugins/:pluginId/resources/* status_source=downstream

The cluster uses self-signed certificates behind an nginx reverse proxy, which are ofc. not verified by default. I there any workaround for this?

How do we reproduce it?

1. Deploy OnCall and Grafana behind a reverse proxy with self signed certificates.
2. Try to install OnCall in Grafana

Grafana OnCall Version

v1.13.6

Product Area

API, Other

Grafana OnCall Platform?

Docker

User's Browser?

No response

Anything else to add?

No response

[MSommer95]

Follow-up to the problem: If I embed the ca.crt of my self-signed certificate into the grafana container the Plugin installation requests grafana externally and not with smth like localhost:3000 or grafana:3000.

In my case this results in an error, because the request isnt authenticated.

[MSommer95]

Apparently I had to set the OnCall Plugin settings manually via a CURL:

curl -X POST 'https://admin:[email protected]/grafana/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://oncall:8080/", "grafanaUrl":"http://grafana:3000/"}}'

But I will be honest, this seems rather unintuitive, esp. since this wasnt necessary in earlier versions.