$ obinsectd
obinsectd version 0.01, using libmosquitto 1.5.7 and libjson-c 0.12.1
Usage: ./obinsectd [-d] [-c configfile] -s device
[-b hostname] [-p port] [-u username [-P password]]
[-i id] [-k keepalive] [--insecure]
[{--cafile file | --capath dir} [--cert file] [--key file]
[--unscaled | --units ]
-c : Configuration file. Default: /etc/obinsect.conf
-d : Enable debugging
-s : Serial device connected to M-Bus. E.g /dev/ttyUSB0. Use '-' to read from stdin
MQTT client options:
-b : Broker hostname or IP address. Default: localhost
-i : Client Id. A random id is generated by default
-k : Keepalive in seconds. Default: 60
-p : Broker TCP port. Default: 1883 (8883 when using MQTTS)
-P : Password. Default is no authentication
-u : Username
MQTT over TLS (MQTTS):
--insecure : Do not validate brokername against certificate
--cafile : Trusted CA certificates PEM file
--capath : Trusted CA certificates directory
--cert : Client certificate
--key : Client private key
Published value format:
--unscaled : Do not scale numbers. Default: false (scaling enabled)
--units : Include units. Implies value scaling. Default: false
Example: ./obinsectd -s /dev/ttyUSB0 -b broker.example.com
Turns on very verbose debug output to stderr. Note that this is independent of the MQTT debug channel described below.
Change location of the JSON formatted configuration file. See below for details
The serial device is the only mandatory option. This should normally be an USB to M-BUS bridge connected to a supported meter
Using "-" as serial device to read packes from stdin instead. This is useful for feeding it replayed recorded samples.
obinsectd will attempt an anonymous MQTT connection to a broker running on the same host by default. It will continue without MQTT if this fails for any reason. It will not subscribe to any MQTT topics, and it will only publish if configured to do so using the configuration file.
obinsectd will by default scale all numeric values according to the OBIS list definition. This means that many values end up as floats instead of integers, which might be undesirable. The scaling will also suffer from the usual floating point division issues, where the number is off by a small fraction way outside the precision.
Typical example of a published "alias" object (see below):
{
"ListId": "KFM_001",
"SerialNumber": "MA304H3E",
"Model": 1176,
"Power": 0,
"PowerExport": 131,
"ReactivePower": 0,
"ReactivePowerExport": 2208,
"CurrentL1": 3.841,
"CurrentL2": 3.794,
"CurrentL3": 2.3890000000000002,
"VoltageL1": 0,
"VoltageL2": 239.70000000000002,
"timestamp": 1559117502
}Setting this option will disable the scaling, making obinsectd publish the raw integer values as received.
Same example as above, with --unscaled set:
{
"ListId": "KFM_001",
"SerialNumber": "MA304H3E",
"Model": 1176,
"Power": 0,
"PowerExport": 131,
"ReactivePower": 0,
"ReactivePowerExport": 2208,
"CurrentL1": 3841,
"CurrentL2": 3794,
"CurrentL3": 2389,
"VoltageL1": 0,
"VoltageL2": 2397,
"timestamp": 1559117502
}By default all values are publish as JSON numbers, either integer or double. This makes the values usable on the other end without further parsing. This suits most usecases like updating gauges, databases etc best. The unit is not published at all in this case.
Setting "--units" will change all values with a unit defined in the OBIS list definiton, to strings. The strings are formatted with the number of decimals specified in the list definition, followed by a space and the unit.
This can be useful for usecase where published values are used as strings anyway, for example in a simple text log or in a text element on a web page.
Note that this setting is global. It is currently not possible to publish some topics or values with units and some without.
--units implies scaling, and will silently re-enable scaling even if --unscaled is set.
Reusing the "alias" object example, but this time with units enabled:
{
"ListId": "KFM_001",
"SerialNumber": "MA304H3E",
"Model": 1176,
"Power": "0 W",
"PowerExport": "131 W",
"ReactivePower": "0 Var",
"ReactivePowerExport": "2208 Var",
"CurrentL1": "3.841 A",
"CurrentL2": "3.794 A",
"CurrentL3": "2.389 A",
"VoltageL1": "0.0 V",
"VoltageL2": "239.7 V",
"timestamp": 1559117502
}obinsect can be used for debugging the HAN port without any configuration. Simply run it with the -d (--debug) option. The serial port option is mandatory:
obinsectd -d -s /dev/ttyUSB0
In debug mode, obinsectd prints a hexdump of all received packets and the native structure of all sucessfully parsed packets as JSON.
But its real advantage lies in the configurable mapping of data to MQTT topics, as well as fully configurable OBIS code lists. The configuration file has two sections:
- "topicmap" - mapping MQTT topics to values
- "obisdefs" - list of JSON files with OBIS code defintions
This is a complete configuration example:
{
"topicmap" : {
"/obinsect/log/debug" : "debug",
"/obinsect/log/info" : "info",
"/obinsect/log/error" : "error",
"/obinsect/websocket/power" : "Power",
"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
"/obinsect/json/realtime" : "alias",
"/obinsect/json/counters" : [
"timestamp",
"MeterTime",
"CumulativeEnergy",
"CumulativeEnergyExport",
"CumulativeReactiveEnergy",
"CumulativeReactiveEnergyExport"
]
},
"obisdefs" : [
"/etc/obinsect/aidon_v0001.json",
"/etc/obinsect/kfm_001.json",
"/etc/obinsect/kamstrup_v0001.json"
]
}Any number of MQTT topics can be given, including none. No specific topic structure is assumed. The example above is simply an example. You do not have to use a common prefix, or organize topics in a tree at all.
After parsing a packet, data will be published to all topics having one or more values received from the meter. I.e. metadata is not considered when deciding whether to publish to a topic or not.
The value specification can be
- an OBIS code in text representation, e.g "1-1:0.2.129.255"
- a code alias defined in the currently selected code list, e.g. "Power"
- the "date-time" field of the DLMS/COSEM packet
- a metadata variable (see below for spec), e.g. "timestamp"
- one of two predefined objects: "normal" or "alias"
- a list combining one or more of the above
- special debug values: "rawpacket", "parserdata" or one of the loglevels "debug", "info" or "error"
The value is converted to its matching JSON representation, considering scaling and unit conversion as described above. By default, numeric values are returned as numbers scaled according to the OBIS list definition.
Note that Kaifa meters transmits arrays of values only. These are mapped to codes by obinsectd using the "alias" section of the matching OBIS list ("KFM_001"). This list must therefore be defined in the correct order.
"date-time" values are converted to UNIX epoch time. Some meters a binary object type for "MeterTime". This is corrected to "date-time" by obinsectd.
The OBIS code lists include an "alias" section. These names can be used instead of the actual code. The value transformation rules are as described above.
These aliases are intended to be consistent across code lists, but this is not verified. They can be changed as you like by editing the list definition, provided a few rules are followed:
- one alias for each code must be defined, and in the exact order they sent used by the meter. See the list specs for this, or edit the tables of the included lists without changing the order
- an alias must be an atom. Space and other special characters are not allowed
- aliases must not collide with any other value code word. An easy way to ensure this is by capitalizing the first letter.
DLMS/COSEM packets include an optional "date-time" field in their header. This is available as a value, if sent by the meter. Note that many meters skip this. Dump the "parserdata" if you are unsure, and look at the keys "data-notification" field.
If there is no "date-time", then "notification-body" follows immediately after "long-invoke-id-and-priority":
"data-notification": {
"long-invoke-id-and-priority": 1073741824,
"notification-body": [
... list contents ...
A packet with "date-time" will look similar to this:
"data-notification": {
"long-invoke-id-and-priority": 0,
"date-time-bug": true,
"date-time": 1508474270,
"notification-body": {
... list contents ...
The "date-time-bug" field is informative only and can be ignored. Some meter firmwares prefix the "date-time" with an object type code. This is wrong, and is simply ignored by obinsectd. "date-time-bug" existing and being true indicates that this happened. This prepares for a possible future feature where the exact binary packet received from the meter can be recreated from the parsed JSON object, including this bug. Possibly in the other end of an MQTT publish-subscribe chain...
obinsectd generates a number of internal variables every time a packet is parsed:
- "timestamp" - local packet arrival time as UNIX epoch
- "framlength" - length of packet, excluding frame markers (0x7e)
- "framenumber" - frame count since obinsectd was started
- "srcprog" - application name ("obinsectd")
- "srchost" - host name where obinsectd is running
- "version" - obinsectd version
- "serialport" - configured serial port, or "stdin"
- "parsetime" - total processing time for this packet, in microseconds
All of these can be included at once as a separate JSON object by specifying the value "metadata".
obinsectd can publish all values received from the meter as a single JSON object, as a normalized and flattened list of key => value pairs. Each value is formatted as specified in the OBIS code section.
There are two such lists available:
- "normal" - using the original OBIS code as key
- "alias" - using the OBIS code alias as key
Both objects include the internal "timestamp" metadata field, but no other internal fields.
The number of keys will obviously depend on the received packets, and should match the OBIS list documentation. An example "normal" value for a 10 second list:
{
"1-1:0.2.129.255": "Kamstrup_V0001",
"1-1:0.0.5.255": "5706567274389702",
"1-1:96.1.1.255": "6841121BN243101040",
"1-1:1.7.0.255": 1.828,
"1-1:2.7.0.255": 0,
"1-1:3.7.0.255": 0,
"1-1:4.7.0.255": 0.444,
"1-1:31.7.0.255": 6.67,
"1-1:51.7.0.255": 2.22,
"1-1:71.7.0.255": 6.21,
"1-1:32.7.0.255": 233,
"1-1:52.7.0.255": 228,
"1-1:72.7.0.255": 234,
"timestamp": 1559049474
}The exact same packet published using the "alias" value:
{
"ListId": "Kamstrup_V0001",
"SerialNumber": "5706567274389702",
"Model": "6841121BN243101040",
"Power": 1.828,
"PowerExport": 0,
"ReactivePower": 0,
"ReactivePowerExport": 0.444,
"CurrentL1": 6.67,
"CurrentL2": 2.22,
"CurrentL3": 6.21,
"VoltageL1": 233,
"VoltageL2": 228,
"VoltageL3": 234,
"timestamp": 1559049474
}Specifying a list of values from the sets defined above makes obinsect publish all the values as a single JSON object. For example, this value definition:
"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
results in an object like this
{
"1-0:1.7.0.255": 4.821,
"timestamp": 1559045686
}published to "/obinsect/json/powerandtime" every time a "1-0:1.7.0.255" value is received.
Nesting of complex objects is not allowed, but the pre-defined objects "normal" and "alias" can be included in a complex object.
It is also possible to use the debug objects "rawpacket" or "parserdata" in a complex object, but the usefulness of this is questionable...
obinsectd can also use MQTT for log and debug messages. This includes
The received packet as a hexbyte string. This includes any unparseable packets. Packets are strings of bytes starting and ending with a frame marker, 0x7e. Valid packets will have two bytes defining HDLC frame type 3 and the frame length following the frame start marker. "HDLC type 3" is 1010b, or 0xa, so the first hex digit following the frame start marker should always be a
An example of a published "rawpacket" value, broken into lines for readability:
7e a0 e2 2b 21 13 23 9a e6 e7 00 0f 00 00 00 00
0c 07 d0 01 01 06 16 21 00 ff 80 00 01 02 19 0a
0e 4b 61 6d 73 74 72 75 70 5f 56 30 30 30 31 09
06 01 01 00 00 05 ff 0a 10 35 37 30 36 35 36 37
30 30 30 30 30 30 30 30 30 09 06 01 01 60 01 01
ff 0a 12 30 30 30 30 30 30 30 30 30 30 30 30 30
30 30 30 30 30 09 06 01 01 01 07 00 ff 06 00 00
00 00 09 06 01 01 02 07 00 ff 06 00 00 00 00 09
06 01 01 03 07 00 ff 06 00 00 00 00 09 06 01 01
04 07 00 ff 06 00 00 00 00 09 06 01 01 1f 07 00
ff 06 00 00 00 00 09 06 01 01 33 07 00 ff 06 00
00 00 00 09 06 01 01 47 07 00 ff 06 00 00 00 00
09 06 01 01 20 07 00 ff 12 00 00 09 06 01 01 34
07 00 ff 12 00 00 09 06 01 01 48 07 00 ff 12 00
00 5b e5 7e
This value specification makes obinsectd publish the parsed JSON object, with all the structure and most of the types and values as received from the meter. Scaling and units are not used, and fields have the type specified by the meter.
The only exception to the no-touch rule is that OBIS codes are converted from their 6 byte binary original to the more readable OBIS string format.
Unique field names for the DLMS/COSEM "notification-body" are generated by using the field type together with an index into the object.
The packet structure from some meters can be rather complex, using for example nested objects inside arrays. The "parserdata" objects are therefore not suitable for direct use, but can be very useful for debugging both meter and parser.
The internally generated "metadata" object is also included in the "parserdata" object.
An example:
{
"metadata": {
"timestamp": 1559050860,
"framelength": 226,
"framenumber": 1,
"srcprog": "./obinsectd",
"srchost": "miraculix",
"version": "0.01",
"serialport": "stdin",
"parsetime": 1029
},
"hdlc": {
"format": 10,
"segmentation": false,
"length": 226,
"src": 43,
"dst": 33,
"control": 19,
"hcs": 39459,
"fcs": 58715
},
"llc": {
"lsap": 230,
"dsap": 231,
"quality": 0
},
"data-notification": {
"long-invoke-id-and-priority": 0,
"date-time": 946762380,
"notification-body": {
"visible-string-0": "Kamstrup_V0001",
"obis-1": "1-1:0.0.5.255",
"visible-string-2": "5706567000000000",
"obis-3": "1-1:96.1.1.255",
"visible-string-4": "000000000000000000",
"obis-5": "1-1:1.7.0.255",
"double-long-unsigned-6": 0,
"obis-7": "1-1:2.7.0.255",
"double-long-unsigned-8": 0,
"obis-9": "1-1:3.7.0.255",
"double-long-unsigned-10": 0,
"obis-11": "1-1:4.7.0.255",
"double-long-unsigned-12": 0,
"obis-13": "1-1:31.7.0.255",
"double-long-unsigned-14": 0,
"obis-15": "1-1:51.7.0.255",
"double-long-unsigned-16": 0,
"obis-17": "1-1:71.7.0.255",
"double-long-unsigned-18": 0,
"obis-19": "1-1:32.7.0.255",
"long-unsigned-20": 0,
"obis-21": "1-1:52.7.0.255",
"long-unsigned-22": 0,
"obis-23": "1-1:72.7.0.255",
"long-unsigned-24": 0
}
}
}obinsectd will log some debug info to stderr by default, but can also publish the log messages to MQTT topics, using one of the loglevels "debug", "info" or "error" as value for the topic.
Log messages related to MQTT are never published, for obvious reasons...
Each meter vendor has defined specifics sets of OBIS codes and the intervals these are published. These lists define the relative order of the values, resolution and scaling, and units.
obinsect extends the list definitions with a code alias for each OBIS code. These aliases are examples only, and can be changed as required.
The order of the OBIS code to alias map is significant, however, and must not be changed. Values are sometimes sent by the meter without any key or code label. The relative index into the alias map is used to map such values to their respective OBIS codes.
obinsect comes with three OBIS code lists which can be used as examples when adding support for other meters:
1. aidon_v0001.json
2. kfm_001.json
3. kamstrup_v0001.json
Run one obinsectd process per meter if you want to monitor more than one meter. The configuration files may be shared between the processes, as long as the same publishing rules and OBIS lists are wanted.