Enrich stats items [JIRA: RCS-217]

[shino]

Update 2015-07-17

After #1165 #1180 #1189 , major stats items are implemented for 2.1.0.
Remaining items for future releases are as follows.

• admin api: user
• admin api: users
• admin api: storage
• admin api: access
• Pool exhaution counter
• Indicator for block retries
• GC activity/elapsed time/gc count/error count
• deleted object size and number per batch (indicator for disk space reclaim)
• timeout, disconnect, overload, notfound counter
• 'siblings' in manifest size indicators
• bucket number per user indicator
• notfound in delete_fsm
• bucket list size in user record
• manifest history count
• multipart manifest part count
• total thruput MB/sec in
• total thruput MB/sec out
• error count grouped by error code/reason (total, last minutes)
• stale manifest usage ... how much to be scheduled_delete?
• http status codes stats
• ad-hoc stats triggered by operation command that copies all
  stats item, focusing on a single user
• Good / better / not-bad stats for object get/put latency,
  because they depend on body size
• Good / better / not-bad stats for storage calc mapred
  because it depends on number of objects

Original description

• User related elapsed time (get, put)
• Bucket related elaplsed time (get)
• Add latency_min and latency_100
• Recent history information for pools
• Indicator for pool overflow, or pool history
• Indicator for block retries
• GC activity/elapsed time/gc count/error count
• Storage calculation activity/elapsed time/error count
• Access Usage flush count/ fail count
• List objects activity/elapsed time/error count

Recommendations and ideas are welcome.

[ksauzz]

My recommendations are :

• GC activity/elapsed time/gc count/error count
• Storage calculation activity/elapsed time/error count
• Access Usage flush count

[shino]

@ksauzz thanks, added to the list in the description.

[kuenishi]

• deleted object size and number per batch (indicator for disk space reclaim)
• disconnect, overload, notfound counter
• proxy_get counter
• 'siblings' in manifest size indicators

[kuenishi]

nice

• bucket number per user indicator
• notfound in delete_fsm

And I think we have to move onto Exometer like Riak 2.0.

[ksauzz]

• bucket history size
• manifest history size

[shino]

total thruput

• MB/sec in
• MB/sec out

[ksauzz]

meter_rate always looks to indicate about zero for me. We should clarify how it's useful, or remove it.

[shino]

• error count grouped by error code/reason (total, last minutes)

[ksauzz]

I'm not sure if get/put object latency should be included in metrics. it looks to depend on object size.

[kuenishi]

get/put latency has been included in older versions prior to 2.0. But it's for all uptime, not for last minute.

[kuenishi]

I believe we've covered necessary stats. Will be out in 2.1.

[Basho-JIRA]

Release note will be in RCS-11.

_[posted via JIRA by Kota Uenishi]_