Failed to reconcile Ingress - Istio 1.11.4

[dahateb]

Hi we have an issue that when rolling out a kservice with a public url we get this status message:

status:
  conditions:
  - lastTransitionTime: "2021-12-21T10:57:18Z"
    status: "True"
    type: LoadBalancerReady
  - lastTransitionTime: "2021-12-21T10:57:18Z"
    status: "True"
    type: NetworkConfigured
  - lastTransitionTime: "2021-12-21T10:57:22Z"
    message: Ingress reconciliation failed
    reason: ReconcileIngressFailed
    status: Unknown
    type: Ready

This seems to come down to an error in the net-istio-controller:

{"severity":"ERROR","timestamp":"2021-12-21T12:41:51.34794035Z","logger":"net-istio-controller.istio-ingress-controller","caller":"ingress/ingress.go:96","message":"Failed to reconcile Ingress: ","commit":"c9e88c5","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"92a2952b-940f-4a87-a2ec-cd7da80f4c79","knative.dev/key":"httpbin/httpbin-backend","error":"Gateway.networking.istio.io \"httpbin-backend-1166151354\" is invalid: spec.servers: Invalid value: \"null\": spec.servers in body must be of type object: \"null\"","stacktrace":"knative.dev/net-istio/pkg/reconciler/ingress.(*Reconciler).ReconcileKind\n\tknative.dev/net-istio/pkg/reconciler/ingress/ingress.go:96\nknative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.(*reconcilerImpl).Reconcile\n\tknative.dev/[email protected]/pkg/client/injection/reconciler/networking/v1alpha1/ingress/reconciler.go:250\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
{"severity":"ERROR","timestamp":"2021-12-21T12:41:51.348058443Z","logger":"net-istio-controller.istio-ingress-controller","caller":"ingress/reconciler.go:313","message":"Returned an error","commit":"c9e88c5","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"92a2952b-940f-4a87-a2ec-cd7da80f4c79","knative.dev/key":"httpbin/httpbin-backend","targetMethod":"ReconcileKind","error":"Gateway.networking.istio.io \"httpbin-backend-1166151354\" is invalid: spec.servers: Invalid value: \"null\": spec.servers in body must be of type object: \"null\"","stacktrace":"knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.(*reconcilerImpl).Reconcile\n\tknative.dev/[email protected]/pkg/client/injection/reconciler/networking/v1alpha1/ingress/reconciler.go:313\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
{"severity":"ERROR","timestamp":"2021-12-21T12:41:51.348117034Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"c9e88c5","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"92a2952b-940f-4a87-a2ec-cd7da80f4c79","knative.dev/key":"httpbin/httpbin-backend","duration":"4.533843ms","error":"Gateway.networking.istio.io \"httpbin-backend-1166151354\" is invalid: spec.servers: Invalid value: \"null\": spec.servers in body must be of type object: \"null\"","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
{"severity":"INFO","timestamp":"2021-12-21T12:41:51.348371656Z","logger":"net-istio-controller.istio-ingress-controller","caller":"ingress/ingress.go:113","message":"Reconciling ingress: &v1alpha1.Ingress{TypeMeta:v1.TypeMeta{Kind:\"Ingress\", APIVersion:\"networking.internal.knative.dev/v1alpha1\"}, ObjectMeta:v1.ObjectMeta{Name:\"httpbin-backend\", GenerateName:\"\", Namespace:\"httpbin\", SelfLink:\"\", UID:\"b8915c7b-f179-4628-a525-3282c500ce58\", ResourceVersion:\"512674507\", Generation:2, CreationTimestamp:time.Date(2021, time.December, 21, 10, 57, 18, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string[]string{\"app\":\"httpbin\", \"app.kubernetes.io/instance\":\"httpbin\", \"app.kubernetes.io/managed-by\":\"Helm\", \"app.kubernetes.io/name\":\"httpbin-backend\", \"app.kubernetes.io/version\":\"0.22.0\", \"helm.sh/chart\":\"backend-1.0.1\", \"serving.knative.dev/route\":\"httpbin-backend\", \"serving.knative.dev/routeNamespace\":\"httpbin\", \"serving.knative.dev/service\":\"httpbin-backend\", \"tags.datadoghq.com/env\":\"staging\", \"tags.datadoghq.com/service\":\"httpbin-backend\", \"tags.datadoghq.com/version\":\"latest\", \"tier.io/importance\":\"high\", \"version\":\"latest\"}, Annotations:map[string[]string{\"meta.helm.sh/release-name\":\"httpbin\", \"meta.helm.sh/release-namespace\":\"httpbin\", \"networking.internal.knative.dev/rollout\":\"{\\\"configurations\\\":[{\\\"configurationName\\\":\\\"httpbin-backend\\\",\\\"percent\\\":100,\\\"revisions\\\":[{\\\"revisionName\\\":\\\"httpbin-backend-00001\\\",\\\"percent\\\":100}],\\\"stepParams\\\":{}}]}\", \"networking.knative.dev/ingress.class\":\"istio.ingress.networking.knative.dev\", \"serving.knative.dev/creator\":\"TM-cross-account-admin\", \"serving.knative.dev/lastModifier\":\"TM-cross-account-admin\", \"serving.knative.dev/rolloutDuration\":\"0s\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1\", Kind:\"Route\", Name:\"httpbin-backend\", UID:\"d0b67a2c-32a0-4f97-88f0-c1836d472202\", Controller:(*bool)(0xc000cf7ec8), BlockOwnerDeletion:(*bool)(0xc000cf7ec9)}}, Finalizers:[]string{\"ingresses.networking.internal.knative.dev\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:\"controller\", Operation:\"Update\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", Time:time.Date(2021, time.December, 21, 10, 57, 22, 0, time.Local), FieldsType:\"FieldsV1\", FieldsV1:(*v1.FieldsV1)(0xc002b47080)}}}, Spec:v1alpha1.IngressSpec{TLS:[]v1alpha1.IngressTLS{v1alpha1.IngressTLS{Hosts:[]string{\"httpbin-backend.httpbin.kn.c.euc1.tier-staging.io\"}, SecretName:\"route-d0b67a2c-32a0-4f97-88f0-c1836d472202\", SecretNamespace:\"httpbin\"}}, Rules:[]v1alpha1.IngressRule{v1alpha1.IngressRule{Hosts:[]string{\"httpbin-backend.httpbin\", \"httpbin-backend.httpbin.svc\", \"httpbin-backend.httpbin.svc.cluster.local\"}, Visibility:\"ClusterLocal\", HTTP:(*v1alpha1.HTTPIngressRuleValue)(0xc002b47098)}, v1alpha1.IngressRule{Hosts:[]string{\"httpbin-backend.httpbin.kn.c.euc1.tier-staging.io\"}, Visibility:\"ExternalIP\", HTTP:(*v1alpha1.HTTPIngressRuleValue)(0xc002b470b0)}}, HTTPOption:\"\"}, Status:v1alpha1.IngressStatus{Status:v1.Status{ObservedGeneration:2, Conditions:v1.Conditions{apis.Condition{Type:\"LoadBalancerReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:time.Date(2021, time.December, 21, 10, 57, 18, 0, time.Local)}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"NetworkConfigured\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:time.Date(2021, time.December, 21, 10, 57, 18, 0, time.Local)}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:time.Date(2021, time.December, 21, 10, 57, 22, 0, time.Local)}, Reason:\"ReconcileIngressFailed\", Message:\"Ingress reconciliation failed\"}}, Annotations:map[string[]string(nil)}, PublicLoadBalancer:(*v1alpha1.LoadBalancerStatus)(0xc002b470c8), PrivateLoadBalancer:(*v1alpha1.LoadBalancerStatus)(0xc002b470e0)}}","commit":"c9e88c5","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"0ecb47a4-2cfb-48ce-8d4a-a0e64131ab24","knative.dev/key":"httpbin/httpbin-backend"}

We are currently using Istio 1.11.4 with specific revision labels on the namespaces and also an istio-ingressgateway that was renamed to istio-ingressgateway-1-11-4.

The config-istio configmap looks like this:

apiVersion: v1
data:
  gateway.knative-serving.knative-ingress-gateway: istio-ingressgateway-1-11-4.istio-system.svc.cluster.local
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: net-istio
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/version: 1.1.0
    networking.knative.dev/ingress-provider: istio
    serving.knative.dev/release: v1.1.0
  name: config-istio
  namespace: knative-serving
  ownerReferences:
  - apiVersion: operator.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: KnativeServing
    name: knative-serving
    uid: b6fbc490-2584-4e3a-8d62-6726ad3862ee
  resourceVersion: "512680507"
  uid: c213cac1-e435-4f1f-acf4-ec32ca2adea1

Any help on this would be highly appreciated

[nak3]

Hi @dahateb
It seems that net-istio controller tries to create an invalid Istio Gateway which does not have spec.servers.
Could you please share the following information?

• What Knative version you are using?
• What your config-network looks like?

I wonder you configured http-protocol: "Disabled" or using some old Knative version 🤔 .

[nak3]

I am still not 100% sure if it's root cause or not but https://github.com/knative-sandbox/net-istio/blob/2e3df687019db608f78d690e66b9b737f9cc488e/pkg/reconciler/ingress/ingress.go#L166
needs nil check for httpServer as MakeHTTPServer could return nil when httpOption is empty. (dahateb's httpOption is empty as per log.)

[dahateb]

Hey thanks for the quick reply and the pointer. I reenabled httpProtocol in config-network and it started to work again. Still a bit puzzled why that flag causes an error though.

[dahateb]

Just to complete this:

knative version: 0.25.0
config-network:

apiVersion: v1
data:
  autoTLS: Enabled
  httpProtocol: Disabled
kind: ConfigMap
metadata:
  labels:
    serving.knative.dev/release: v0.25.0
  name: config-network
  namespace: knative-serving
  resourceVersion: "269435874"
  uid: 2c1750fc-d27d-4572-890a-af23449b657b

[dahateb]

that version of config-istio was from a test cluster where I upgraded to the latest version

[nak3]

Great to hear that it works with reenabled htttpOption.
httpProtocol: Disabled is deprecated actually.

https://github.com/knative/networking/blob/62aefa40945342bc60eec11a79e37df78053dbc6/config/config-network.yaml#L129-L130

We are discussing it on knative/networking#417 but the option would be removed.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.