About ElasticSearch OutPut pack can't pack.Recycle .

[lairdnote]

Hi.
my log:
2016/05/20 05:26:05 Diagnostics: 1 packs have been idle more than 120 seconds.
2016/05/20 05:26:05 Diagnostics: (input) Plugin names and quantities found on idle packs:
2016/05/20 05:26:05 Diagnostics: ElasticSearchOutput: 1
2016/05/20 05:26:05
2016/05/20 05:26:35 Diagnostics: 1 packs have been idle more than 120 seconds.
2016/05/20 05:26:35 Diagnostics: (input) Plugin names and quantities found on idle packs:
2016/05/20 05:26:35 Diagnostics: ElasticSearchOutput: 1
2016/05/20 05:26:35
2016/05/20 05:27:05 Diagnostics: 1 packs have been idle more than 120 seconds.
2016/05/20 05:27:05 Diagnostics: (input) Plugin names and quantities found on idle packs:
2016/05/20 05:27:05 Diagnostics: ElasticSearchOutput: 1
2016/05/20 05:27:05
2016/05/20 05:27:35 Diagnostics: 1 packs have been idle more than 120 seconds.
2016/05/20 05:27:35 Diagnostics: (input) Plugin names and quantities found on idle packs:
2016/05/20 05:27:35 Diagnostics: ElasticSearchOutput: 1
2016/05/20 05:27:35

my hekad version :
heka_0.10.0_amd64.deb

my hekad configre:

[ElasticSearchOutput]
message_matcher = "Type =~ /gunicorn.*/"
server = "http://192.168.1.12:9200"
flush_interval = 200
flush_count = 100
encoder = "ES_Encoder"
use_buffering = true
http_disable_keepalives = true

[ElasticSearchOutput.buffering]
full_action = "block"
max_buffer_size = 209715200 #200m
cursor_update_count = 1000

The input have 400M size file.
when hekad LogstreamerInput was complete text file. so cache dir was full and some record send to ES
I wait some time .I see heka dashboard just has ok. but cache dir don't reduce
root@lairdnote:/var/log# du -hs /var/cache/hekad/
205M /var/cache/hekad/

I think this is ESOutput issue .

[simonpasquier]

For some reason, hekad can't send data to ES (probably a network issue) so I would recommend to look into the ES logs too or use tcpdump on tcp port 9200. Since you configure the 'block' policy for the buffering, the Heka pipeline gets stuck when the output plugin has filled the local buffer. Hence the idle packs errors.

Next time, please use the mailing list ([email protected]) for such request.

[lairdnote]

I checked network . network is fine . the cache data has send to ES . I check ES data. find cache data into .
So .I thins maybe ES output plugin can't Recycle cache. I change hekad version to 0.9.2 so all things is good

[simonpasquier]

We use a similar configuration with buffering in our environment and everything works fine for us. What's in your checkpoint.txt file for the Elasticsearch output cache directory?

[lairdnote]

this is my checkpoint.txt file.

root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# ll
total 204812
drwxr--r-- 2 root root 4096 May 20 05:47 ./
drwxr--r-- 3 root root 4096 May 20 04:30 ../
-rw-r--r-- 1 root root 209715062 May 20 05:16 3.log
-rw-r--r-- 1 root root 0 May 20 05:23 4.log
-rw-r--r-- 1 root root 0 May 20 05:47 5.log
-rw-r--r-- 1 root root 11 May 20 05:17 checkpoint.txt
root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# cat checkpoint.txt
3 189562626

thanks for your help