Parity between the puppet HTTP client and the sensu http client

[asachs01]

In internal ref: https://secure.helpscout.net/conversation/2034967718/30087?folderId=5845954, we ran into an issue where version 0.4.0 of the handler doesn't seem to respect the --insecure-skip-tls-verify flag.

Upon investigation, the issue seems to arise from the fact that the --insecure-skip-tls-verify flag seems to be scoped only to the Puppet HTTP client:

https://github.com/sensu/sensu-puppet-handler/blob/master/main.go#L219-L224

And doesn't have an analog in the Sensu HTTP client. It's also worth noting that the Sensu HTTP client doesn't seem to respect the system's CA certs (see the code linked above).

So there are two problems:

• The Sensu HTTP client doesn't respect the --insecure-skip-tls-verify as it's scoped to the Puppet HTTP client
• The Sensu HTTP client doesn't seem to include a system's given CA certs

To reproduce the issue, do the following:

• Get a keepalive event (e.g., sensuctl event info sensu-backend-01 keepalive --format json > backend01_keepalive.json
• Pipe the event through the handler with the requisite configuration (cat backend01_keepalive.json | ./sensu-puppet-handler...)
• Ensure the handler is configured to use a self-signed cert for connecting to the Sensu API
• Observe the failure, should look like:

{
    "check_name": "keepalive",
    "check_namespace": "ts-sre",
    "component": "pipeline/legacy",
    "entity_name": "log01",
    "entity_namespace": "ts-sre",
    "event_id": "b845d7d7-bff6-4770-bff4-8ac003bb893f",
    "handler": "puppet",
    "level": "info",
    "msg": "event pipe handler executed",
    "output": "2023/02/07 19:14:28 Overriding default handler configuration with value of \"Entity.Annotations.sensu.io/plugins/sensu-puppet-handler/config/node-name\" (\"log01.example.com\")\n2023/02/07 19:14:28 puppet node \"log01.example.com\" does not exist\n2023/02/07 19:14:28 deleting entity ()\nUsage:\n  sensu-puppet-handler [flags]\n  sensu-puppet-handler [command]\n\nAvailable Commands:\n  help        Help about any command\n  version     Print the version number of this plugin\n\nFlags:\n      --ca-cert string             path to the site's Puppet CA certificate PEM file (default \"/etc/puppetlabs/puppet/ssl/certs/ca.pem\")\n      --cert string                path to the SSL certificate PEM file signed by your site's Puppet CA (default \"/etc/puppetlabs/puppet/ssl/certs/puppet_cert.pem\")\n  -e, --endpoint string            the PuppetDB API endpoint (URL). If an API path is not specified, /pdb/query/v4/nodes/ will be used (default \"https://puppet.example.com:8081\")\n  -h, --help                       help for sensu-puppet-handler\n      --insecure-skip-tls-verify   skip SSL verification\n      --key string                 path to the private key PEM file for that certificate (default \"/tmp/pupp_key.pem\")\n      --node-name string           node name to use for the entity when querying PuppetDB\n  -a, --sensu-api-key string       The Sensu API key (default \"f015322f-1741-47f4-9db7-add0e99a07a0\")\n  -u, --sensu-api-url string       The Sensu API URL (default \"https://sensu.example.com:8080\")\n  -c, --sensu-ca-cert string       The Sensu Go CA Certificate (default \"/etc/sensu/ssl/ca.der\")\n\nUse \"sensu-puppet-handler [command] --help\" for more information about a command.\n\nError executing sensu-puppet-handler: error executing handler: Delete https://sensu.example.com:8080/api/core/v2/namespaces/default/entities/log01: x509: certificate signed by unknown authority\n",
    "pipeline": "legacy-pipeline",
    "pipeline_workflow": "legacy-pipeline-workflow-puppet",
    "status": 1,
    "time": "2023-02-07T19:14:28Z"
}

[echlebek]

The Sensu HTTP client does respect the system cert pool, see https://github.com/sensu/sensu-plugin-sdk/blob/master/httpclient/client.go#L286