webhook error: Received response code 405 , expected 200

[bfitzelle]

configmap-reload container showing following logs when configmap is updated:

2021/10/26 18:13:56 Watching directory: "/etc/config"
2021/10/26 18:50:08 config map updated
2021/10/26 18:50:08 performing webhook request (1/1)
2021/10/26 18:50:08 error: Received response code 405 , expected 200
2021/10/26 18:50:18 error: Webhook reload retries exhausted

however curl from inside configmap-reload container to same webhook gets expected 200 response and prometheus configuration reloads successfully

./curl-amd64 -vvvvvv -X POST http://127.0.0.1:9090/prometheus/-/reload
Trying 127.0.0.1:9090...
Connected to 127.0.0.1 (127.0.0.1) port 9090 (#0)
POST /prometheus/-/reload HTTP/1.1
Host: 127.0.0.1:9090
User-Agent: curl/7.79.1
Accept: /

Mark bundle as not supporting multiuse
HTTP/1.1 200 OK
Date: Wed, 27 Oct 2021 15:14:31 GMT
Content-Length: 0

Connection #0 to host 127.0.0.1 left intact

[migueleliasweb]

Could you paste the command you have in your configmap-reload pod?

[bfitzelle]

sure - command running in configmap-reload container is:

/configmap-reload --volume-dir=/etc/config --webhook-url=http://127.0.0.1:9090/prometheus//-/reload

[migueleliasweb]

I would find very odd if the return is really a 405. The only different between your pod's command and the curl you pasted before is the url on your pod seems to have // two slashes.

Have you tried adding retries to the request?