storm-kafka and kafka_2.9.2 "0.8.x", "0.9.x" compatibility

[andrewmilkowski]

hello

seeking to run storm-kafka against newer kafka codebase (0.8.x, 0.9.x) codebease in contrib/storm-kafka is compatible with 0.7.x only..

any chance in the near term for a port to the newer kafka ?

thanks!

[asmaier]

This would be very useful indeed.

[jarretraim]

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

[jasonjckn]

I think 0.7 and 0.8 kafka will be in use for a long time, so it makes sense
to maintain two branches of storm-kafka for each respectively. It would be
awesome if someone did a PR for this, I personally haven't needed 0.8 yet.

Is it not? The current dependencies for the project seem to state that
it's working with newer storm versions:
Andrew is referring to kafka version. Yes the latest storm-kafka works with
0.9 storm.

On Tue, Sep 24, 2013 at 5:56 AM, Jarret Raim [email protected]:

Is it not? The current dependencies for the project seem to state that
it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/38#issuecomment-25000074
.

[asmaier]

See also here https://groups.google.com/forum/#!topic/storm-user/OXQ0a9ppcYM

[dschiavu]

Use this project for Kafka 0.8: https://github.com/wurstmeister/storm-kafka-0.8-plus/

[sujeetyadav]

Hi,

I am trying to integrate storm and kafka but I am facing issue such as I am able to deploy the topology but it is not listening to kafka messages.

Following are the maven dependencies I have used:

org.apache.storm
storm-core
0.9.1-incubating


org.apache.storm
storm-kafka
0.9.2-incubating


org.apache.kafka
kafka_2.10
0.8.0


org.apache.zookeeper
zookeeper


log4j
log4j





org.apache.kafka
kafka-clients
0.8.2.1

When I run this program it gives following log:
4670 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
4676 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /home/sujeet/Downloads/fieldvantage/TimeSeriesDataMigration/target/TimeSeriesDataMigration-0.0.1-SNAPSHOT.jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar
4691 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar
4691 [main] INFO backtype.storm.StormSubmitter - Submitting topology TestTopology in distributed mode with conf {"topology.workers":1,"topology.message.timeout.secs":600,"topology.debug":false,"kafka.broker.properties":{"bootstrap.servers":"localhost:9092","acks":"1","value.serializer":"org.apache.kafka.common.serialization.StringSerializer","key.serializer":"org.apache.kafka.common.serialization.StringSerializer"}}

But when I am sending messages from Kafka producer it is unable to listen it.

Any help would be appreciable.

Thanks & Regards,
Sujeet Yadav

[Dennis1122]

Hi,

I too have the same issuewith kafka storm integration. I am not able to read from kafka topic when running on a single node cluster. Everything works fine while running in local mode.
@sujeetyadav Are you able to solve the issue.. ??

Any help will be highly appreciated.

Thanks in advance
Dennis