kafkacat error connecting to kafka in consume mode; list mode works

[bipinm]

Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT

What happened:
Installed kafkacat to look into messages generated by minio (in configured topic)

kafkacat -b [kafka service ip]:9092 -L --> This command works fine
_Metadata for all topics (from broker -1: 10.152.183.146:9092/bootstrap):
1 brokers:
broker 1001 at kafka-0.broker.kubeless.svc.cluster.local:9092
4 topics:
topic "_consumer_offsets" with 50 partitions:
partition 0, leader 1001, replicas: 1001, isrs: 1001

kafkacat -C -b _[kafka service ip]_:9092 -t minio
% ERROR: Local: Host resolution failure: kafka-0.broker.kubeless.svc.cluster.local:9092/1001: Failed to resolve 'kafka-0.broker.kubeless.svc.cluster.local:9092': Name or service not known

Managed to fix this by adding following line in /etc/hosts
[kafka service ip] kafka-0.broker.kubeless.svc.cluster.local

What you expected to happen:
It should work without need to modify /etc/hosts

How to reproduce it (as minimally and precisely as possible):
Install kafkacat and try to consume messages for particular topic

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Kubeless version (use kubeless version):
  Kubeless version: v1.0.0

• Cloud provider or physical cluster:
  microk8s channel=1.12/stable

[andresmgot]

Hi @bipinm,

I assume you are using kafkacat from a pod or similar within a cluster? I assume that because you can access private IPs like 10.152.183.146.

I think that your issue is that the Kafka service name (by default) is just kafka so the URL would be: kafka.kubeless.svc.cluster.local:9092. Note that the URL should be <service_name>.<service_namespace>.svc.cluster.local:<svc_port>.

[bipinm]

Hi @andresmgot,

i'm using kafkacat from host (Ubuntu 1810 in VMWarePlayer), not from within pods. Funny thing is that kafkacat -L works but -C throws error
[kafka service ip] = 10.152.183.146

With microk8s i can directly access the pod IPs from host. I had one issue though - kafka pod could not resolve zookeeper.kubeless:2181. sudo iptables -P FORWARD ACCEPT helped fix this issue.

I have not touched the pods, just started it following the guide https://kubeless.io/docs/pubsub-functions/. Pods and services are automatically created, so the issue must be in yaml definitions ?

[andresmgot]

I am not familiar with kafkacat nor microk8s but it seems that the issue is that it's not able to resolve services from the host. Is that a supported feature? Why do you think that may be an issue with the yaml definitions?

[bipinm]

That is correct, its not able to resolve service from host.

Found a similar issue, almost the same as my problem - Yolean/kubernetes-kafka#99

kafkacat is able to find the broker and list the topics but not consume (-C option). Solution listed there vs mine is different, though mine would potentially work only on microk8s.

[andresmgot]

I see, let us know if there is something we could do in our side.

[Michabitton]

I faced with the same issue, after creating new record in /etc/hosts for the Continer ID binding to his private IP address it's solved
/etc/hosts
cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
172.0.17.2 4e0ccc2d7ed9 >> bind

nslookup 4e0ccc2d7ed9
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: 4e0ccc2d7ed9
Address: 172.0.17.2