diff --git a/.gitignore b/.gitignore index ab35261..89c263c 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ int docker-compose.yaml dist/ vuegraf.egg-info/ +vugraftobe.py diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..e3631e2 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,35 @@ +# 1.TBD.TBD + +## Breaking changes +- TBD + +## New features +- TBD + +## Other changes +- TBD + +# 1.6.0 + +## Breaking changes +- Replaced Minute with Hour as normal interval since history is limited to 7 days from Emporia on minute data +- argparse libary was added, must run `pip install -r requirements.txt` again in the src directory (or pip3 based on install) + +## New features +- Hour / Day historic data retrieval: allows for history of up to two years to be pulled. Assists in clean numbers/graphs to see daily monthly usage to compare against utilities reports. +- Hour data runs with the get details time, default is 1 hour (3600 seconds). Based on when the program is started, you may be almost 2 hours behind for get hour. +- Moved one-time parameters out of the json config file. Those parameters are now specified as command line arguments (--historydays, --resetdatabase). + +## Other changes +- Started Changelog for this and future releases +- Added project metadata to `vuegraf.py`, values can be updated through github automations +- Added command line pairing with help syntax for all values, via argparse lib. +- Updated `requirements.txt` and setup.py with `argparse>= 1.4.0` +- Updated `vuegraf.json.sample` as history and reset database was moved to command line +- Updated Readme.md with above changes +- ran pylint and fixed + Quote delimiter consistency to all ' + Whitespaces + Extra lines + +With special thanks to @gauthig for initiating these 1.6.0 changes! diff --git a/README.md b/README.md index c90a8b1..518214a 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,6 @@ The minimum configuration required to start Vuegraf is shown below. "org": "vuegraf", "bucket": "vuegraf", "token": "", - "reset": false }, "accounts": [ { @@ -88,9 +87,14 @@ The minimum configuration required to start Vuegraf is shown below. ### Ingesting Historical Data -If desired, it is possible to have Vuegraf import historical data. To do so, specify a new temporary parameter called `historyDays` inside the `influxDb` section, with an integer value greater than zero. Once restarted, One-minute data from the past `historyDays` days will be ingested into InfluxDB. Emporia currently retains this data for 7 days, and therefore `historyDays` must be less than or equal to `7`. If `historyDays` is set to `0`, no historical data will be ingested into InfluxDB. +If desired, it is possible to have Vuegraf import historical data. To do so, run vuegraf.py with the optional `--historydays` parameter with a value between 1 and 720. When this parameter is provided Vuegraf will start and collect all hourly data points up to the specified parameter, or max history available. It will also collect one day's summary data for each day, storing it with the timestamp 23:59:59 for each day. It collects the time using your local server time, but stores it in influxDB in UTC. -IMPORTANT - If you restart Vuegraf with historyDays still set to a non-zero value then it will _again_ import history data. This will likely cause confusion with your data since you will now have duplicate/overlapping data. For best results, only enable historyDays > 0 for a single run, and then immediately set it back to 0 to avoid this duplicated import data scenario. +IMPORTANT - If you restart Vuegraf with `--historydays` on the command line (or forget to remove it from the dockerfile) it will import history data _again_. This will likely cause confusion with your data since you will now have duplicate/overlapping data. For best results, only enable `--historydays` on a single run. + +For Example: +``` +python3 path/to/vuegraf.py vuegraf.json --historydays 365 +``` ### Channel Names @@ -136,10 +140,16 @@ Vuegraf can be run either as a container (recommended), or as a host process. A Docker container is provided at [hub.docker.com](https://hub.docker.com/r/jertel/vuegraf). Refer to the command below to launch Vuegraf as a container. This assumes you have created a folder called `/home/myuser/vuegraf` and placed the vuegraf.json file inside of it. +Normal run with docker ```sh docker run --name vuegraf -d -v /home/myuser/vuegraf:/opt/vuegraf/conf jertel/vuegraf ``` +Recreate database and load 25 days of history +```sh +docker run --name vuegraf -d -v /home/myuser/vuegraf:/opt/vuegraf/conf jertel/vuegraf --resetdatabase --historydays=24 +``` + ## Host Process Ensure Python 3 and Pip are both installed. Install the required dependencies: @@ -164,6 +174,25 @@ or, on some Linux installations: python3 src/vuegraf/vuegraf.py vuegraf.json ``` +Optional Command Line Parameters +``` +usage: vuegraf.py [-h] [--version] [-v] [-q] [--historydays HISTORYDAYS] [--resetdatabase] configFilename + +Retrieves data from cloud servers and inserts it into an InfluxDB database. + +positional arguments: + configFilename JSON config file + +options: + -h, --help show this help message and exit + --version Display version number + -v, --verbose Verbose output - summaries + --historydays HISTORYDAYS + Starts executing by pulling history of Hours and Day data for specified number of days. + example: --load-history-day 60 + --resetdatabase Drop database and create a new one +``` + ## Alerts The included dashboard template contains two alerts which will trigger when either a power outage occurs, or a loss of Vuegraf data. There are various reasons why alerts can be helpful. See the below screenshots which help illustrate how a fully functioning alert and notification rule might look. Note that the included alerts do not send out notifications. To enable outbound notifactions, such as to Slack, you can create a Notification Endpoint and Notification Rule. @@ -183,18 +212,21 @@ This notification rule provides an example of how you can have several alerts ch ## Per-second Data Details -By default, Vuegraf will poll every minute to collect the energy usage value over the past 60 seconds. This results in a single value being capture per minute per channel, or 60 values per hour per channel. If you also would like to see per-second values, you can enable the detailed collection, which is polled once per hour, and backfilled over the previous 3600 seconds. This API call is very expensive on the Emporia servers, so it should not be polled more frequently than once per hour. To enable this detailed data, add (or update) the top-level `detailedDataEnabled` configuration value with a value of `true`. +By default, Vuegraf will poll every minute to collect the energy usage value over the past 60 seconds. This results in a single value being capture per minute per channel, or 60 values per hour per channel. If you also would like to see per-second values, you can enable the detailed collection, which is polled once per hour, and backfilled over the previous 3600 seconds. This API call is very expensive on the Emporia servers, so it should not be polled more frequently than once per hour. To enable this detailed data, add (or update) the top-level `detailedDataEnabled` configuration value with a value of `true`. The details is also what pulls the Hourly datapoint. ``` detailedDataEnabled: true ``` -Again: +For every datapoint a tag is stored in InfluxDB for the type of measurement - `detailed = True` represents backfilled per-second data that is optionally queried from Emporia once every hour. - `detailed = False` represents the per-minute average data that is collected every minute. +- `detailed = Hour` represents the data summarized in hours +- `detailed = Day` represents a single data point to summarize the entire day -When building graphs that show a sum of the energy usage, be sure to only include either detailed=true or detailed=false, otherwise your summed values will be higher than expected. Detailed data will take more time for the graphs to query due to the extra data involved. By default, it is set to False so most users can ignore this note. +When building graphs that show a sum of the energy usage, be sure to only include the correct detail tag, otherwise your summed values will be higher than expected. Detailed data will take more time for the graphs to query due to the extra data involved. If you want to have a chart that shows daily data over a long period or even a full year, use the `detailed = Day` tag. +If you are running this on a small server, you might want to look at setting a RETENTION POLICY on your InfluxDB bucket to remove minute or second data over time. For example, it will reduce storage needs if you retain only 30 days of per-_second_ data. ## Vue Utility Connect Energy Monitor diff --git a/setup.py b/setup.py index 2e2e23a..2e2e876 100644 --- a/setup.py +++ b/setup.py @@ -23,6 +23,7 @@ install_requires=[ 'influxdb>=5.3.1', 'influxdb_client>=1.33.0', - 'pyemvue>=0.16.0' + 'pyemvue>=0.16.0', + 'argparse>= 1.4.0' ] ) diff --git a/src/requirements.txt b/src/requirements.txt index 83da0e2..dbc2021 100644 --- a/src/requirements.txt +++ b/src/requirements.txt @@ -1,3 +1,4 @@ influxdb >= 5.3.1 influxdb_client >= 1.33.0 pyemvue == 0.16.0 +argparse >= 1.4.0 \ No newline at end of file diff --git a/src/vuegraf/.gitignore b/src/vuegraf/.gitignore new file mode 100644 index 0000000..89c263c --- /dev/null +++ b/src/vuegraf/.gitignore @@ -0,0 +1,6 @@ +vuegraf.json +int +docker-compose.yaml +dist/ +vuegraf.egg-info/ +vugraftobe.py diff --git a/src/vuegraf/vuegraf.py b/src/vuegraf/vuegraf.py index 5296a56..7544ce3 100644 --- a/src/vuegraf/vuegraf.py +++ b/src/vuegraf/vuegraf.py @@ -1,5 +1,14 @@ #!/usr/bin/env python3 +__author__ = 'https://github.com/jertel' +__license__ = 'MIT' +__contributors__ = 'https://github.com/jertel/vuegraf/graphs/contributors' +__version__ = '1.6.0' +__versiondate__ = '2023/05/16' +__maintainer__ = 'https://github.com/jertel' +__github__ = 'https://github.com/jertel/vuegraf' +__status__ = 'Production' + import datetime import json import signal @@ -7,6 +16,8 @@ import time import traceback from threading import Event +import argparse +import pytz # InfluxDB v1 import influxdb @@ -24,10 +35,10 @@ def log(level, msg): print('{} | {} | {}'.format(now, level.ljust(5), msg), flush=True) def info(msg): - log("INFO", msg) + log('INFO', msg) def error(msg): - log("ERROR", msg) + log('ERROR', msg) def handleExit(signum, frame): global running @@ -50,17 +61,17 @@ def populateDevices(account): device = account['vue'].populate_device_properties(device) deviceIdMap[device.device_gid] = device for chan in device.channels: - key = "{}-{}".format(device.device_gid, chan.channel_num) + key = '{}-{}'.format(device.device_gid, chan.channel_num) if chan.name is None and chan.channel_num == '1,2,3': chan.name = device.device_name channelIdMap[key] = chan - info("Discovered new channel: {} ({})".format(chan.name, chan.channel_num)) + info('Discovered new channel: {} ({})'.format(chan.name, chan.channel_num)) def lookupDeviceName(account, device_gid): if device_gid not in account['deviceIdMap']: populateDevices(account) - deviceName = "{}".format(device_gid) + deviceName = '{}'.format(device_gid) if device_gid in account['deviceIdMap']: deviceName = account['deviceIdMap'][device_gid].device_name return deviceName @@ -70,7 +81,7 @@ def lookupChannelName(account, chan): populateDevices(account) deviceName = lookupDeviceName(account, chan.device_gid) - name = "{}-{}".format(deviceName, chan.channel_num) + name = '{}-{}'.format(deviceName, chan.channel_num) try: num = int(chan.channel_num) @@ -89,28 +100,28 @@ def lookupChannelName(account, chan): def createDataPoint(account, chanName, watts, timestamp, detailed): dataPoint = None if influxVersion == 2: - dataPoint = influxdb_client.Point("energy_usage") \ - .tag("account_name", account['name']) \ - .tag("device_name", chanName) \ - .tag("detailed", detailed) \ - .field("usage", watts) \ + dataPoint = influxdb_client.Point('energy_usage') \ + .tag('account_name', account['name']) \ + .tag('device_name', chanName) \ + .tag('detailed', detailed) \ + .field('usage', watts) \ .time(time=timestamp) else: dataPoint = { - "measurement": "energy_usage", - "tags": { - "account_name": account['name'], - "device_name": chanName, - "detailed": detailed, + 'measurement': 'energy_usage', + 'tags': { + 'account_name': account['name'], + 'device_name': chanName, + 'detailed': detailed, }, - "fields": { - "usage": watts, + 'fields': { + 'usage': watts, }, - "time": timestamp + 'time': timestamp } return dataPoint -def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEndTime=None): +def extractDataPoints(device, usageDataPoints, pointType=None, historyStartTime=None, historyEndTime=None): excludedDetailChannelNumbers = ['Balance', 'TotalUsage'] minutesInAnHour = 60 secondsInAMinute = 60 @@ -125,14 +136,20 @@ def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEnd kwhUsage = chan.usage if kwhUsage is not None: - watts = float(minutesInAnHour * wattsInAKw) * kwhUsage - timestamp = stopTime - usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, False)) + if pointType is None: + watts = float(minutesInAnHour * wattsInAKw) * kwhUsage + timestamp = stopTime + usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, False)) + elif pointType == 'Day' or pointType == 'Hour' : + watts = kwhUsage * 1000 + timestamp = historyStartTime + usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, pointType)) if chanNum in excludedDetailChannelNumbers: continue if collectDetails: + #Seconds usage, usage_start_time = account['vue'].get_chart_usage(chan, detailedStartTime, stopTime, scale=Scale.SECOND.value, unit=Unit.KWH.value) index = 0 for kwhUsage in usage: @@ -141,29 +158,84 @@ def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEnd timestamp = detailedStartTime + datetime.timedelta(seconds=index) watts = float(secondsInAMinute * minutesInAnHour * wattsInAKw) * kwhUsage usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, True)) + if args.verbose: + info('Get Details (Seconds); start="{}"; stop="{}"'.format(detailedStartTime,stopTime )) index += 1 - + + + # fetches historical minute data if historyStartTime is not None and historyEndTime is not None: - usage, usage_start_time = account['vue'].get_chart_usage(chan, historyStartTime, historyEndTime, scale=Scale.MINUTE.value, unit=Unit.KWH.value) + if args.verbose: + info('Hour-History - {} - start="{}" - stop="{}"'.format(chanName, historyStartTime,historyEndTime )) + #Hours + usage, usage_start_time = account['vue'].get_chart_usage(chan, historyStartTime, historyEndTime, scale=Scale.HOUR.value, unit=Unit.KWH.value) index = 0 for kwhUsage in usage: if kwhUsage is None: continue - timestamp = historyStartTime + datetime.timedelta(minutes=index) - watts = float(minutesInAnHour * wattsInAKw) * kwhUsage - usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, False)) + timestamp = historyStartTime + datetime.timedelta(hours=index) + watts = kwhUsage * 1000 + usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, 'Hour')) + index += 1 + #Days + usage, usage_start_time = account['vue'].get_chart_usage(chan, historyStartTime, historyEndTime, scale=Scale.DAY.value, unit=Unit.KWH.value) + index = 0 + for kwhUsage in usage: + if kwhUsage is None: + continue + timestamp = historyStartTime + datetime.timedelta(days=index-1) + timestamp = timestamp.replace(hour=23, minute=59, second=59,microsecond=0) + timestamp = timestamp.astimezone(pytz.UTC) + watts = kwhUsage * 1000 + usageDataPoints.append(createDataPoint(account, chanName, watts, timestamp, 'Day')) index += 1 startupTime = datetime.datetime.utcnow() try: - if len(sys.argv) != 2: - print('Usage: python {} '.format(sys.argv[0])) - sys.exit(1) + #argparse includes default -h / --help as command line input + parser = argparse.ArgumentParser( + prog='vuegraf.py', + description='Veugraf retrieves data from cloud servers and inserts it into an InfluxDB database.', + epilog='For more information visit: ' + __github__ + ) + parser.add_argument( + 'configFilename', + help='JSON config file', + type=str + ) + parser.add_argument( + '--version', + help='Display version number', + action='store_true') + parser.add_argument( + '-v', + '--verbose', + help='Verbose output - summaries', + action='store_true') + parser.add_argument( + '-q', + '--quiet', + help='Do not print anything but errors', + action='store_true') + parser.add_argument( + '--historydays', + help='Starts execution by pulling history of Hours and Day data for specified number of days. example: --historydays 60', + type=int, + default=0 + ) + parser.add_argument( + '--resetdatabase', + action='store_true', + default=False, + help='Drop database and create a new one. USE WITH CAUTION - WILL RESULT IN COMPLETE VUEGRAF DATA LOSS!') + args = parser.parse_args() + if args.version: + print('vuegraf.py - version: ', __version__) + sys.exit(0) - configFilename = sys.argv[1] config = {} - with open(configFilename) as configFile: + with open(args.configFilename) as configFile: config = json.load(configFile) influxVersion = 1 @@ -193,12 +265,12 @@ def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEnd write_api = influx2.write_api(write_options=influxdb_client.client.write_api.SYNCHRONOUS) query_api = influx2.query_api() - if config['influxDb']['reset']: + if args.resetdatabase: info('Resetting database') delete_api = influx2.delete_api() - start = "1970-01-01T00:00:00Z" - stop = startupTime.isoformat(timespec='seconds') - delete_api.delete(start, stop, '_measurement="energy_usage"', bucket=bucket, org=org) + start = '1970-01-01T00:00:00Z' + stop = startupTime.isoformat(timespec='seconds') + 'Z' + delete_api.delete(start, stop, '_measurement="energy_usage"', bucket=bucket, org=org) else: info('Using InfluxDB version 1') @@ -214,33 +286,32 @@ def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEnd influx.create_database(config['influxDb']['database']) - if config['influxDb']['reset']: + if args.resetdatabase: info('Resetting database') influx.delete_series(measurement='energy_usage') - historyDays = min(config['influxDb'].get('historyDays', 0), 7) + historyDays = min(args.historydays, 720) history = historyDays > 0 - running = True - signal.signal(signal.SIGINT, handleExit) signal.signal(signal.SIGHUP, handleExit) - pauseEvent = Event() - - intervalSecs=getConfigValue("updateIntervalSecs", 60) - detailedIntervalSecs=getConfigValue("detailedIntervalSecs", 3600) - detailedDataEnabled=getConfigValue("detailedDataEnabled", False); + intervalSecs=getConfigValue('updateIntervalSecs', 60) + detailedIntervalSecs=getConfigValue('detailedIntervalSecs', 3600) + detailedDataEnabled=getConfigValue('detailedDataEnabled', False) info('Settings -> updateIntervalSecs: {}, detailedEnabled: {}, detailedIntervalSecs: {}'.format(intervalSecs, detailedDataEnabled, detailedIntervalSecs)) - lagSecs=getConfigValue("lagSecs", 5) + lagSecs=getConfigValue('lagSecs', 5) detailedStartTime = startupTime + pastDay = datetime.datetime.now() + pastDay = pastDay.replace(hour=23, minute=59, second=59, microsecond=0) while running: now = datetime.datetime.utcnow() + curDay = datetime.datetime.now() stopTime = now - datetime.timedelta(seconds=lagSecs) collectDetails = detailedDataEnabled and detailedIntervalSecs > 0 and (stopTime - detailedStartTime).total_seconds() >= detailedIntervalSecs - for account in config["accounts"]: + for account in config['accounts']: if 'vue' not in account: account['vue'] = PyEmVue() account['vue'].login(username=account['email'], password=account['password']) @@ -255,44 +326,70 @@ def extractDataPoints(device, usageDataPoints, historyStartTime=None, historyEnd for gid, device in usages.items(): extractDataPoints(device, usageDataPoints) - if history: - for day in range(historyDays): - info('Loading historical data: {} day(s) ago'.format(day+1)) - #Extract second 12h of day - historyStartTime = stopTime - datetime.timedelta(seconds=3600*24*(day+1)-43200) - historyEndTime = stopTime - datetime.timedelta(seconds=(3600*24*(day))) - for gid, device in usages.items(): - extractDataPoints(device, usageDataPoints, historyStartTime, historyEndTime) - pauseEvent.wait(5) - #Extract first 12h of day - historyStartTime = stopTime - datetime.timedelta(seconds=3600*24*(day+1)) - historyEndTime = stopTime - datetime.timedelta(seconds=(3600*24*(day+1))-43200) - for gid, device in usages.items(): - extractDataPoints(device, usageDataPoints, historyStartTime, historyEndTime) - if not running: - break - pauseEvent.wait(5) - history = False - - if not running: - break - - info('Submitting datapoints to database; account="{}"; points={}'.format(account['name'], len(usageDataPoints))) - if influxVersion == 2: - write_api.write(bucket=bucket, record=usageDataPoints) - else: - influx.write_points(usageDataPoints) + if collectDetails: + pastHour = stopTime - datetime.timedelta(hours=1) + pastHour = pastHour.replace(minute=00, second=00,microsecond=0) + historyStartTime = pastHour + usages = account['vue'].get_device_list_usage(deviceGids, pastHour, scale=Scale.HOUR.value, unit=Unit.KWH.value) + if usages is not None: + usageDataPoints = [] + for gid, device in usages.items(): + extractDataPoints(device, usageDataPoints, 'Hour', historyStartTime) + if args.verbose: + info('Collected Previous Hour: {} '.format(pastHour)) + + if pastDay.day < curDay.day: + usages = account['vue'].get_device_list_usage(deviceGids, pastDay, scale=Scale.DAY.value, unit=Unit.KWH.value) + historyStartTime = pastDay.astimezone(pytz.UTC) + if usages is not None: + usageDataPoints = [] + for gid, device in usages.items(): + extractDataPoints(device, usageDataPoints,'Day', historyStartTime) + if args.verbose: + info('Collected Previous Day: {}Local - {}UTC, '.format(pastDay, historyStartTime)) + pastDay = datetime.datetime.now() + pastDay = pastDay.replace(hour=23, minute=59, second=00, microsecond=0) + + if history: + info('Loading historical data: {} day(s) ago'.format(historyDays)) + historyStartTime = stopTime - datetime.timedelta(historyDays) + historyStartTime = historyStartTime.replace(hour=00, minute=00, second=00, microsecond=000000) + while historyStartTime <= stopTime: + historyEndTime = min(historyStartTime + datetime.timedelta(20), stopTime) + historyEndTime = historyEndTime.replace(hour=23, minute=59, second=59,microsecond=0) + if args.verbose: + info(' {} - {}'.format(historyStartTime,historyEndTime)) + for gid, device in usages.items(): + extractDataPoints(device, usageDataPoints, 'History', historyStartTime, historyEndTime) + if not running: + break + historyStartTime = historyEndTime + datetime.timedelta(1) + historyStartTime = historyStartTime.replace(hour=00, minute=00, second=00, microsecond=000000) + pauseEvent.wait(5) + history = False + + if not running: + break + + info('Submitting datapoints to database; account="{}"; points={}'.format(account['name'], len(usageDataPoints))) + if influxVersion == 2: + write_api.write(bucket=bucket, record=usageDataPoints) + else: + influx.write_points(usageDataPoints,batch_size=5000) except: - error('Failed to record new usage data: {}'.format(sys.exc_info())) + error('Failed to record new usage data: {}'.format(sys.exc_info())) traceback.print_exc() if collectDetails: detailedStartTime = stopTime + datetime.timedelta(seconds=1) - pauseEvent.wait(intervalSecs) info('Finished') -except: - error('Fatal error: {}'.format(sys.exc_info())) - traceback.print_exc() +except SystemExit as e: + #If sys.exit was 2, then normal syntax exit from help or bad command line, no error message + if e.code == 0 or e.code == 2: + quit(0) + else: + error('Fatal error: {}'.format(sys.exc_info())) + traceback.print_exc() diff --git a/vuegraf.json.sample b/vuegraf.json.sample index defba70..54f25d7 100644 --- a/vuegraf.json.sample +++ b/vuegraf.json.sample @@ -8,8 +8,10 @@ "reset": false, "ssl_enable": false, "ssl_verify": true + }, "updateIntervalSecs": 60, + "detailedDataEnabled": "true", "accounts": [ { "name": "Primary Residence",