Vector fails to resolve metadata.google.internal

[luisloros]

Hi Vector team,

I am using Vector in a custom Kubernetes cluster running on GCP VM instances. I want to push logs to a Google Cloud Storage Bucket but my Vector pods were always failing.

I have all the correct permissions on my GCP instance VM node (roles: "Storage Object Creator" and " Storage Object Viewer ").

So after investigating the issue, I found out that the pod was not resolving the "metadata.google.internal" to 169.254.169.254. The GCP VM instance had the "metadata.google.internal" entry hardcoded in /etc/hosts but that entry was not being used by K8s containers. I can still reach the endpoint by using the IP. Here is the test on debug pod:

/ # curl http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -H "metadata-flavor: Google"
curl: (6) Could not resolve host: metadata.google.internal
/ # curl http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token -H "metadata-flavor: Google"
{"access_token":"[redacted]","expires_in":[redacted],"token_type":"Bearer"}/ #

The workaround was to add the hardcoded entry in coreDNS configmap by adding the following and then restarting the vector pods:

hosts /etc/coredns/customdomains.db google.internal
    {
     169.254.169.254 metadata.google.internal   
  }

Is it possible to fallback to the IP 169.254.169.254 if the DNS "metadata.google.internal" fails? Or just use the IP and not use the DNS?

Logs:

2023-07-12T16:14:34.824322Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-07-12T16:14:34.824411Z  INFO vector::app: Log level is enabled. level="debug"
2023-07-12T16:14:34.826068Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-07-12T16:14:34.827334Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-07-12T16:14:34.828552Z DEBUG vector::topology::builder: Building new source. component=k8s
2023-07-12T16:14:34.829273Z  INFO source{component_kind="source" component_id=k8s component_type=kubernetes_logs component_name=k8s}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="node-4vgd"
2023-07-12T16:14:34.842478Z  INFO source{component_kind="source" component_id=k8s component_type=kubernetes_logs component_name=k8s}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2023-07-12T16:14:34.842597Z DEBUG vector::topology::builder: Building new sink. component=bucket
2023-07-12T16:14:34.842898Z DEBUG sink{component_kind="sink" component_id=bucket component_type=gcp_cloud_storage component_name=bucket}: vector::gcp: Fetching implicit GCP authentication token.
2023-07-12T16:14:34.877233Z DEBUG sink{component_kind="sink" component_id=bucket component_type=gcp_cloud_storage component_name=bucket}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token method=GET version=HTTP/1.1 headers={"metadata-flavor": "Google", "user-agent": "Vector/0.31.0 (x86_64-unknown-linux-gnu 0f13b22 2023-07-06 13:52:34.591204470)", "accept-encoding": "identity"} body=[empty]
2023-07-12T16:14:34.880664Z DEBUG hyper::client::connect::dns: resolving host="metadata.google.internal"
2023-07-12T16:14:34.887297Z  WARN sink{component_kind="sink" component_id=bucket component_type=gcp_cloud_storage component_name=bucket}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name or service not known error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-07-12T16:14:34.889769Z ERROR vector::topology: Configuration error. error=Sink "bucket": Failed to get implicit GCP token: Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name or service not known

Configuration vector-values.yaml:

---
role: Agent
serviceHeadless:
  enabled: false
service:
  enabled: false
env:
  - name: "VECTOR_LOG"
    value: "debug"
customConfig:
  data_dir: /vector-data-dir
  sources:
    k8s:
      type: kubernetes_logs
  sinks:
    bucket:
      type: gcp_cloud_storage
      bucket: my_k8s_logs
      encoding:
        codec: "raw_message"
      inputs:
        - k8s

[jszwedko]

I'm curious why the DNS resolution doesn't work in the container using curl. Vector should just be using the native resolver and will match curls behavior. 🤔

[luisloros]

Note that doing a curl request on a debug container against the DNS also fails. Curl requests only works against the IP.
The DNS resolution does not work because the DNS "metadata.google.internal" is hardcoded in /etc/hosts on the GCP instance. Apparently the container runtime does not copy/use the hosts file from the instance.

[luisloros]

Hi @jszwedko, we have found the issue, it was an application that was changing the coreDNS configmap. Disregard what I said earlier and thank you.

[jszwedko]

Thanks for following up! Hopefully this helps other users that run into this issue 😄