Source Connector消息队列kafka中查不到消息

[xnxz123]

按照官方文档的配置
https://docs.taosdata.com/third-party/collection/kafka/
root@pcserver006:/kafka_2.13-3.7.0/bin# curl http://localhost:8083/connectors
["TDengineSourceConnector"]root@pcserver006:/kafka_2.13-3.7.0/bin#
source-demo.json内如如下：
{
"name":"TDengineSourceConnector",
"config":{
"connector.class": "com.taosdata.kafka.connect.source.TDengineSourceConnector",
"tasks.max": 1,
"subscription.group.id": "source-demo",
"connection.url": "jdbc:TAOS://192.168.201.152:6041",
"connection.user": "root",
"connection.password": "***",
"connection.database": "test",
"connection.attempts": 3,
"connection.backoff.ms": 5000,
"topic.prefix": "tdengine",
"topic.delimiter": "-",
"poll.interval.ms": 1000,
"fetch.max.rows": 100,
"topic.per.stable": true,
"topic.ignore.db": false,
"out.format": "line",
"data.precision": "ms",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter"
}
}

tdengine中也写入按文档建了test库与超级表meters，初始化了数据，
执行 curl -X POST -d @source-demo.json http://localhost:8083/connectors -H "Content-Type: application/json"
{"name":"TDengineSourceConnector","config":{"connector.class":"com.taosdata.kafka.connect.source.TDengineSourceConnector","tasks.max":"1","subscription.group.id":"source-demo","connection.url":"jdbc:TAOS://192.168.201.152:6041","connection.user":"root","connection.password":"taosdata","connection.database":"test","connection.attempts":"3","connection.backoff.ms":"5000","topic.prefix":"tdengine","topic.delimiter":"-","poll.interval.ms":"1000","fetch.max.rows":"100","topic.per.stable":"true","topic.ignore.db":"false","out.format":"line","data.precision":"ms","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.storage.StringConverter","name":"TDengineSourceConnector"},"tasks":[],"type":"source"}root@pcserver006:~/kafka_2.13-3.7.0#

kafka中没有数据
root@pcserver006:~/kafka_2.13-3.7.0/bin# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic tdengine-test-meters
^CProcessed a total of 0 messages

[xnxz123]

tdengine 3.2.2 kafka_2.13-3.7.0 git clone --branch 3.0 https://github.com/taosdata/kafka-connect-tdengine.git

[yu285]

kafka / TDengine 的日志中有报错信息吗

[xnxz123]

    at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:357)
    at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:144)
    at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:122)
    at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)

Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: no taos in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr>
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
at java.base/java.lang.System.loadLibrary(System.java:1993)
at com.taosdata.jdbc.TSDBJNIConnector.(TSDBJNIConnector.java:34)
... 19 more
[2024-10-21 20:14:25,458] ERROR [TDengineSourceConnector|worker] [Worker clientId=connect-127.0.1.1:8083, groupId=connect-cluster] Failed to start connector 'TDengineSourceConnector' (org.apache.kafka.>
org.apache.kafka.connect.errors.ConnectException: Failed to start connector: TDengineSourceConnector
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$startConnector$41(DistributedHerder.java:2045)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:360)
at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:144)
at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:122)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to transition connector TDengineSourceConnector to state STARTED
... 9 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.taosdata.jdbc.TSDBJNIConnector
at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:175)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:681)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at com.taosdata.kafka.connect.db.TSDBConnectionProvider.getConnection(TSDBConnectionProvider.java:35)
at com.taosdata.kafka.connect.source.MonitorThread.init(MonitorThread.java:51)
at com.taosdata.kafka.connect.source.MonitorThread.(MonitorThread.java:41)
at com.taosdata.kafka.connect.source.TDengineSourceConnector.start(TDengineSourceConnector.java:30)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:192)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:217)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:376)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:357)
... 8 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: no taos in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr>
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
at java.base/java.lang.System.loadLibrary(System.java:1993)
at com.taosdata.jdbc.TSDBJNIConnector.(TSDBJNIConnector.java:34)
... 19 more

[xnxz123]

tdengine 和kafka在不同的机器上

[yu285]

kafka 那边没装TDengine客户端吧。

[2779382063]

kafka装了客户端还是报了以下错误，我看了kafka-connect-tdengine-3.0的lib库下taos-jdbc驱动是3.3.3，所以在上面也装了3.3.3的TDEngine客户端（装3.0.7.1的客户端也是一样报以下错误），但我的TDEngine服务端是3.0.7.1。
[2024-10-22 16:19:11,277] ERROR [tdengine-source-3|worker] [Worker clientId=connect-1, groupId=connect-cluster] Failed to start connector 'tdengine-source-3' (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1829)
org.apache.kafka.connect.errors.ConnectException: Failed to start connector: tdengine-source-3
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$startConnector$34(DistributedHerder.java:1800)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:346)
at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:143)
at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:121)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to transition connector tdengine-source-3 to state STARTED
... 9 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.taosdata.jdbc.TSDBJNIConnector
at com.taosdata.jdbc.TSDBDriver.connect(TSDBDriver.java:175)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.taosdata.kafka.connect.db.TSDBConnectionProvider.getConnection(TSDBConnectionProvider.java:35)
at com.taosdata.kafka.connect.source.MonitorThread.init(MonitorThread.java:51)
at com.taosdata.kafka.connect.source.MonitorThread.(MonitorThread.java:41)
at com.taosdata.kafka.connect.source.TDengineSourceConnector.start(TDengineSourceConnector.java:30)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:190)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:215)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:360)
at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:343)
... 8 more