td-agent takes long to shut down

[analytically]

When rebooting my Ubuntu 13.04 (same with 13.10) the 'Stopping td-agent td-agent' takes about 20-25 seconds. These are very fast machines. Is this normal?

[repeatedly]

It depends on your configuration.
If a plugin uses TimerWatcher with long interval, then shutdown takes same interval to stop event loop.

[analytically]

The only plugin I install is https://github.com/uken/fluent-plugin-elasticsearch, could that be the issue?

[analytically]

My config is here: https://github.com/analytically/hadoop-ansible/blob/master/roles/td_agent/templates/td-agent.conf

[repeatedly]

Hmm... that's strange.
fluent-plugin-elasticsearch doesn't use TimerWatcher.
Some users use fluent-plugin-elasticsearch, but we don't hear this situation.

Does this always occure or only high workload?

[analytically]

Always occurs, workload is never high.

[repeatedly]

Hmm... okay. I will try to reproduce this problem with Ubuntu 13.04.

[repeatedly]

Did you hit same problem on bare Fluentd on Ubuntu 13.04?
I want to reduce the problem for detecting the cause is td-agent packaging or not.

[cgough]

I am having a similar problem, td-agent will not shut down at all.

[app002.prod:/var/log/td-agent] root% service td-agent stop
Stopping td-agent: Timeout error occurred trying to stop td-agent... * td-agent

Version:

[app002.prod:/var/log/td-agent] root% dpkg -l | grep td-agent
ii  td-agent                             2.3.3-0                                 amd64        Treasure Agent: A data collector for Treasure Data

[app002.prod:/var/log/td-agent] root% lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
[app002.prod:/var/log/td-agent] root% uname -a
Linux app002 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[app002.prod:/var/log/td-agent] root%

Configuration:

## access log
<source>
  @type tail
  path /home/deploy/exchange/shared/log/api_access.log
  pos_file /var/log/td-agent/api_access.log.pos
  tag api_access.log
  format none
</source>

## rails log
<source>
  @type tail
  path /home/deploy/exchange/shared/log/production.log
  pos_file /var/log/td-agent/production.log.pos
  tag api_production.log
  format none
</source>

## filter rules
<filter *.*>
  @type record_transformer
  enable_ruby true
  <record>
    ## postpend server name to message
    message ${[message, ' ', Socket.gethostname].join}
  </record>
</filter>

## ignore healthchecks
#<filter *.*>
#  @type ignore
#  regexp1 message /v9/status
#</filter>

## forward rules
<match *.*>
  @type secure_forward
  shared_key secret_string
  self_hostname "#{Socket.gethostname}"
  secure true
  ca_cert_path /etc/td-agent/ssl/ca_cert.pem
  flush_interval 1s
  <server>
    host admin001.prod.tedc.co
    port 24284
  </server>
</match>

I cannot shut down td-agent without using kill -9.