IPv6 dual-stack not working on IPv6 first clusters due to IPv4 fixed listeners

[alessiodionisi]

Description:

I'm testing the latest release (v1.2.0-rc.1) that adds IPv6 and dual-stack support to Envoy Gateway. On my IPv6-first cluster the pods created for the gateway resource fail the startup probe because, by default, the listener is listening only on IPv4 instead of IPv6 or all interfaces.

gateway/internal/xds/bootstrap/bootstrap.go

Line 29 in 19eb5f5

EnvoyAdminAddress = "127.0.0.1"

gateway/internal/xds/bootstrap/bootstrap.go

Line 42 in 19eb5f5

envoyReadinessAddress = "0.0.0.0"

These IPs are not customizable and are used directly in the bootstrap.yaml.tpl template, example:

gateway/internal/xds/bootstrap/bootstrap.yaml.tpl

Line 68 in 19eb5f5

address: {{ .ReadyServer.Address }}

Repro steps:

1. Deploy Envoy Gateway on the cluster
2. Configure a resource of type Gateway

Environment:

• Kubernetes cluster configured in dual-stack with IPv6 as primary:

  apiVersion: kubeadm.k8s.io/v1beta4
  kind: ClusterConfiguration
  ...
  networking:
    podSubnet: fd00:10:244::/48,10.244.0.0/16
    serviceSubnet: fd00:10:96::/112,10.96.0.0/12

• Envoy Gateway version: v1.2.0-rc.1

Logs:

Startup probe failed: Get "http://[fd00:10:244:2::5b]:19001/ready": dial tcp [fd00:10:244:2::5b]:19001: connect: connection refused

[arkodg]

thanks for testing this out @alessiodionisi ! we'll get this resolved by v1.2.0

[arkodg]

ptal @juwon8891

[alessiodionisi]

Quick update:

I've switched to v0.0.0-latest and patched the bootstrap configuration to listen on IPv6 instead of IPv4, and everything works correctly.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
...
spec:
  ipFamily: DualStack
  bootstrap:
    value: |
      yaml content below

Default values with 127.0.0.1 replaced to ::1 and 0.0.0.0 to :::

admin:
  accessLog:
    - name: envoy.access_loggers.file
      typedConfig:
        "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
        path: /dev/null
  address:
    socketAddress:
      address: ::1
      portValue: 19000
dynamicResources:
  adsConfig:
    apiType: DELTA_GRPC
    grpcServices:
      - envoyGrpc:
          clusterName: xds_cluster
    setNodeOnFirstMessageOnly: true
    transportApiVersion: V3
  cdsConfig:
    ads: {}
    resourceApiVersion: V3
  ldsConfig:
    ads: {}
    resourceApiVersion: V3
layeredRuntime:
  layers:
    - name: global_config
      staticLayer:
        envoy.restart_features.use_eds_cache_for_ads: true
        re2.max_program_size.error_level: 4294967295
        re2.max_program_size.warn_level: 1000
overloadManager:
  refreshInterval: 0.250s
  resourceMonitors:
    - name: envoy.resource_monitors.global_downstream_max_connections
      typedConfig:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
        maxActiveDownstreamConnections: "50000"
staticResources:
  clusters:
    - connectTimeout: 0.250s
      loadAssignment:
        clusterName: prometheus_stats
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: ::1
                      portValue: 19000
      name: prometheus_stats
      type: STATIC
    - connectTimeout: 10s
      loadAssignment:
        clusterName: xds_cluster
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: envoy-gateway
                      portValue: 18000
                loadBalancingWeight: 1
            loadBalancingWeight: 1
      name: xds_cluster
      transportSocket:
        name: envoy.transport_sockets.tls
        typedConfig:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          commonTlsContext:
            tlsCertificateSdsSecretConfigs:
              - name: xds_certificate
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-certificate.json
                  resourceApiVersion: V3
            tlsParams:
              tlsMaximumProtocolVersion: TLSv1_3
            validationContextSdsSecretConfig:
              name: xds_trusted_ca
              sdsConfig:
                pathConfigSource:
                  path: /sds/xds-trusted-ca.json
                resourceApiVersion: V3
      type: STRICT_DNS
      typedExtensionProtocolOptions:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicitHttpConfig:
            http2ProtocolOptions:
              connectionKeepalive:
                interval: 30s
                timeout: 5s
    - connectTimeout: 10s
      loadAssignment:
        clusterName: wasm_cluster
        endpoints:
          - lbEndpoints:
              - endpoint:
                  address:
                    socketAddress:
                      address: envoy-gateway
                      portValue: 18002
                loadBalancingWeight: 1
            loadBalancingWeight: 1
      name: wasm_cluster
      transportSocket:
        name: envoy.transport_sockets.tls
        typedConfig:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          commonTlsContext:
            tlsCertificateSdsSecretConfigs:
              - name: xds_certificate
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-certificate.json
                  resourceApiVersion: V3
            tlsParams:
              tlsMaximumProtocolVersion: TLSv1_3
            validationContextSdsSecretConfig:
              name: xds_trusted_ca
              sdsConfig:
                pathConfigSource:
                  path: /sds/xds-trusted-ca.json
                resourceApiVersion: V3
      type: STRICT_DNS
      typedExtensionProtocolOptions:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicitHttpConfig:
            http2ProtocolOptions: {}
  listeners:
    - address:
        socketAddress:
          address: "::"
          portValue: 19001
      filterChains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typedConfig:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                httpFilters:
                  - name: envoy.filters.http.health_check
                    typedConfig:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
                      headers:
                        - name: :path
                          stringMatch:
                            exact: /ready
                      passThroughMode: false
                  - name: envoy.filters.http.router
                    typedConfig:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                routeConfig:
                  name: local_route
                  virtualHosts:
                    - domains:
                        - "*"
                      name: prometheus_stats
                      routes:
                        - match:
                            prefix: /stats/prometheus
                          route:
                            cluster: prometheus_stats
                statPrefix: eg-ready-http
      name: envoy-gateway-proxy-ready-0.0.0.0-19001

[zirain]

#4550 will fix some cases, but I'm stucked with some users case.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[zirain]

@arkodg I think this's fixed now?

[arkodg]

Yeah should be fixed in v1.2.3