Skip to content

Latest commit

 

History

History
240 lines (196 loc) · 13.6 KB

collectd-hadoopjmx.md

File metadata and controls

240 lines (196 loc) · 13.6 KB
Raw

collectd/hadoopjmx

Monitor Type: collectd/hadoopjmx (Source)

Accepts Endpoints: Yes

Multiple Instances Allowed: Yes

Overview

Collects metrics about a Hadoop 2.0+ cluster using using collectd's GenericJMX plugin. You may also configure the collectd/hadoop monitor to collect additional metrics about the hadoop cluster from the REST API

To enable JMX in Hadoop, add the following JVM options to hadoop-env.sh and yarn-env.sh respectively

hadoop-env.sh:

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=5677 $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=5679 $HADOOP_DATANODE_OPTS"

yarn-env.sh:

export YARN_NODEMANAGER_OPTS="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8002 $YARN_NODEMANAGER_OPTS"
export YARN_RESOURCEMANAGER_OPTS="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=5680 $YARN_RESOURCEMANAGER_OPTS"

This monitor has a set of built in MBeans configured for:

Sample YAML configuration:

Name Node

monitors:
- type: collectd/hadoopjmx
  host: 127.0.0.1
  port: 5677
  nodeType: nameNode

Resource Manager

monitors:
- type: collectd/hadoopjmx
  host: 127.0.0.1
  port: 5680
  nodeType: resourceManager

Node Manager

monitors:
- type: collectd/hadoopjmx
  host: 127.0.0.1
  port: 8002
  nodeType: nodeManager

Data Node

monitors:
- type: collectd/hadoopjmx
  host: 127.0.0.1
  port: 5679
  nodeType: dataNode

Configuration

To activate this monitor in the Smart Agent, add the following to your agent config:

monitors:  # All monitor config goes under this key
 - type: collectd/hadoopjmx
   ...  # Additional config

For a list of monitor options that are common to all monitors, see Common Configuration.

Config option Required Type Description
host yes string Host to connect to -- JMX must be configured for remote access and accessible from the agent
port yes integer JMX connection port (NOT the RMI port) on the application. This correponds to the com.sun.management.jmxremote.port Java property that should be set on the JVM when running the application.
name no string
serviceName no string This is how the service type is identified in the SignalFx UI so that you can get built-in content for it. For custom JMX integrations, it can be set to whatever you like and metrics will get the special property sf_hostHasService set to this value.
serviceURL no string The JMX connection string. This is rendered as a Go template and has access to the other values in this config. NOTE: under normal circumstances it is not advised to set this string directly - setting the host and port as specified above is preferred. (default: service:jmx:rmi:///jndi/rmi://{{.Host}}:{{.Port}}/jmxrmi)
instancePrefix no string Prefixes the generated plugin instance with prefix. If a second instancePrefix is specified in a referenced MBean block, the prefix specified in the Connection block will appear at the beginning of the plugin instance, and the prefix specified in the MBean block will be appended to it
username no string Username to authenticate to the server
password no string User password to authenticate to the server
customDimensions no map of strings Takes in key-values pairs of custom dimensions at the connection level.
mBeansToCollect no list of strings A list of the MBeans defined in mBeanDefinitions to actually collect. If not provided, then all defined MBeans will be collected.
mBeansToOmit no list of strings A list of the MBeans to omit. This will come handy in cases where only a few MBeans need to omitted from the default list
mBeanDefinitions no map of objects (see below) Specifies how to map JMX MBean values to metrics. If using a specific service monitor such as cassandra, kafka, or activemq, they come pre-loaded with a set of mappings, and any that you add in this option will be merged with those. See collectd GenericJMX for more details.
nodeType yes string Hadoop Node Type

The nested mBeanDefinitions config object has the following fields:

Config option Required Type Description
objectName no string Sets the pattern which is used to retrieve MBeans from the MBeanServer. If more than one MBean is returned you should use the instanceFrom option to make the identifiers unique
instancePrefix no string Prefixes the generated plugin instance with prefix
instanceFrom no list of strings The object names used by JMX to identify MBeans include so called "properties" which are basically key-value-pairs. If the given object name is not unique and multiple MBeans are returned, the values of those properties usually differ. You can use this option to build the plugin instance from the appropriate property values. This option is optional and may be repeated to generate the plugin instance from multiple property values
values no list of objects (see below) The value blocks map one or more attributes of an MBean to a value list in collectd. There must be at least one value block within each MBean block
dimensions no list of strings

The nested values config object has the following fields:

Config option Required Type Description
type no string Sets the data set used within collectd to handle the values of the MBean attribute
table no bool Set this to true if the returned attribute is a composite type. If set to true, the keys within the composite type is appended to the type instance. (default: false)
instancePrefix no string Works like the option of the same name directly beneath the MBean block, but sets the type instance instead
instanceFrom no list of strings Works like the option of the same name directly beneath the MBean block, but sets the type instance instead
attribute no string Sets the name of the attribute from which to read the value. You can access the keys of composite types by using a dot to concatenate the key name to the attribute name. For example: “attrib0.key42”. If table is set to true, path must point to a composite type, otherwise it must point to a numeric type.
attributes no list of strings The plural form of the attribute config above. Used to derive multiple metrics from a single MBean.

Metrics

These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.

This monitor will also emit by default any metrics that are not listed below.

Group data-node

All of the following metrics are part of the data-node metric group. All of the non-default metrics below can be turned on by adding data-node to the monitor config option extraGroups:

  • counter.hadoop-datanode-blocks-read (cumulative)
  • counter.hadoop-datanode-blocks-written (cumulative)
  • counter.hadoop-datanode-bytes-read (cumulative)
  • counter.hadoop-datanode-bytes-written (cumulative)
  • gauge.hadoop-datanode-fs-capacity (gauge)
  • gauge.hadoop-datanode-fs-dfs-remaining (gauge)
  • gauge.hadoop-datanode-fs-dfs-used (gauge)
  • gauge.hadoop-datanode-info-xceiver (gauge)
  • gauge.hadoop-datanode-jvm-heap-used (gauge)
  • gauge.hadoop-datanode-jvm-non-heap-used (gauge)
  • gauge.hadoop-datanode-rpc-call-queue-length (gauge)
  • gauge.hadoop-datanode-rpc-open-connections (gauge)
  • gauge.hadoop-datanode-rpc-processing-avg (gauge)
  • gauge.hadoop-datanode-rpc-queue-time-avg (gauge)

Group jvm

All of the following metrics are part of the jvm metric group. All of the non-default metrics below can be turned on by adding jvm to the monitor config option extraGroups:

  • gauge.jvm.threads.count (gauge)
    Number of JVM threads
  • gauge.loaded_classes (gauge)
    Number of classes loaded in the JVM
  • invocations (cumulative)
    Total number of garbage collection events
  • jmx_memory.committed (gauge)
    Amount of memory guaranteed to be available in bytes
  • jmx_memory.init (gauge)
    Amount of initial memory at startup in bytes
  • jmx_memory.max (gauge)
    Maximum amount of memory that can be used in bytes
  • jmx_memory.used (gauge)
    Current memory usage in bytes
  • total_time_in_ms.collection_time (cumulative)
    Amount of time spent garbage collecting in milliseconds

Group name-node

All of the following metrics are part of the name-node metric group. All of the non-default metrics below can be turned on by adding name-node to the monitor config option extraGroups:

  • counter.hadoop-namenode-files-total (cumulative)
  • counter.hadoop-namenode-gc-count (cumulative)
  • counter.hadoop-namenode-gc-time (cumulative)
  • counter.hadoop-namenode-rpc-total-calls (cumulative)
  • counter.hadoop-namenode-total-load (cumulative)
  • counter.hadoop-namenode-volume-failures (cumulative)
  • gauge.hadoop-namenode-blocks-with-corrupt-replicas (gauge)
  • gauge.hadoop-namenode-capacity-remaining (gauge)
  • gauge.hadoop-namenode-capacity-total (gauge)
  • gauge.hadoop-namenode-capacity-used (gauge)
  • gauge.hadoop-namenode-corrupt-blocks (gauge)
  • gauge.hadoop-namenode-current-heap-used (gauge)
  • gauge.hadoop-namenode-dead-datanodes (gauge)
  • gauge.hadoop-namenode-dfs-free (gauge)
  • gauge.hadoop-namenode-live-datanodes (gauge)
  • gauge.hadoop-namenode-max-heap (gauge)
  • gauge.hadoop-namenode-missing-blocks (gauge)
  • gauge.hadoop-namenode-percent-dfs-used (gauge)
  • gauge.hadoop-namenode-percent-remaining (gauge)
  • gauge.hadoop-namenode-rpc-avg-process-time (gauge)
  • gauge.hadoop-namenode-rpc-avg-queue (gauge)
  • gauge.hadoop-namenode-stale-datanodes (gauge)
  • gauge.hadoop-namenode-under-replicated-blocks (gauge)

Group node-manager

All of the following metrics are part of the node-manager metric group. All of the non-default metrics below can be turned on by adding node-manager to the monitor config option extraGroups:

  • counter.hadoop-nodeManager-containers-failed (cumulative)
  • counter.hadoop-nodeManager-containers-launched (cumulative)
  • gauge.hadoop-nodeManager-allocated-memory (gauge)
  • gauge.hadoop-nodeManager-allocated-vcores (gauge)
  • gauge.hadoop-nodeManager-available-memory (gauge)
  • gauge.hadoop-nodeManager-available-vcores (gauge)

Group resource-manager

All of the following metrics are part of the resource-manager metric group. All of the non-default metrics below can be turned on by adding resource-manager to the monitor config option extraGroups:

  • gauge.hadoop-resourceManager-active-apps (gauge)
  • gauge.hadoop-resourceManager-active-nms (gauge)
  • gauge.hadoop-resourceManager-active-users (gauge)
  • gauge.hadoop-resourceManager-allocated-containers (gauge)
  • gauge.hadoop-resourceManager-allocated-memory (gauge)
  • gauge.hadoop-resourceManager-allocated-vcores (gauge)
  • gauge.hadoop-resourceManager-available-memory (gauge)
  • gauge.hadoop-resourceManager-available-vcores (gauge)
  • gauge.hadoop-resourceManager-heap-max (gauge)
  • gauge.hadoop-resourceManager-heap-used (gauge)

Non-default metrics (version 4.7.0+)

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.