Sink/Source Usage

[ghost]

Can elaborate on how to use the the Redis Sink and the Redis source. Am I suppose to modify the config properties file, or the java file and then package with mvn ? Thanks!

[chiwanpark]

Hi. This plugin helps you transfer data from a flume agent to the other flume agent using Redis. Here is a sample structure with this plugin.

               Sink    Source
A Flume agent --- Redis --- A Flume agent

As I wrote in documentation, You can build the plugin package using mvn package -DskipTests. The properties files are for configuration of each flume agent (sink/source).

In example configuration, a flume agent receives data from socket (netcat source), and send to redis using this plugin. Another flume agent receives data from redis using redis source and print data to console.

I hope to help you understand about this plugin. If you have another question, comment to this thread anytime.

[ghost]

Chwin, Thanks for your reply. Sorry I think I wasn’t clear in my question.

I am interested in how the data feed looks like when the flume source ingests that data as in the NetCat Source.

does the data have to be in CSV format, if not, then what format. Is there a serializer to split the event body into values with assigned keys ?

If this is still no clear, please notify me first, and I will put an example in.

Thanks agian!

[chiwanpark]

Hi. Currently, this plugin does not support custom format of data. The plugin supports only raw string data now. Flume has a interface called EventSerializer for dealing with this issue. I will add support for this interface. (or Could you add this feature and send a pull request?)

[ghost]

I am willing to work on the EventSerializer. It might take me a while though to implement a functional and configurable serializer as I am new to programming and development. Thanks again.

[fenggolang]

When I have modified the /flume-ng-redis/conf/example-SubscribeDrivenSource.properties of agent.sources.redis.redisHost=192.168.75.128,next,I am running testSubscribe of RedisSubscribeDrivenSourceTest, The console happen

[chiwanpark]

Hi @Wpengcheng, example configuration files don't affect unit tests. The unit tests uses Redis on localhost (see this).