Energy Management Using Real-Time Non-Intrusive Load Monitoring

Introduction

The goal of non-intrusive load monitoring (NILM) is to recover the energy consumption of individual appliances from the aggregate mains signal, which is a measure of the total electricity consumption of a building or house. NILM is also known as energy disaggregation and both terms will be used interchangeably throughout.

R. Gopinath, et al., nicely summarizes the rational behind NILM in the paper Energy management using non-intrusive load monitoring techniques — State- of-the-art and future research directions¹:

In recent years, the development of smart sustainable cities has become the primary focus among urban planners and policy makers to make responsible use of resources, conserve the environment and improve the well-being of the society. Energy management is an integral part of the smart sustainable cities development programme which involves conscious and efficient use of available energy resources towards attaining sustainability and self-reliance on energy systems. Building sector is one of the key sectors that utilize more energy. Therefore, efforts are being made to monitor and manage energy consumption effectively in residential and commercial buildings. In recent years, non-intrusive load monitoring (NILM) technique has become a popular and emerging approach to monitor events (on/off) and energy consumption of appliances/electrical utilities in buildings using single energy meter. The information about the energy consumption at the appliance level would help consumers to understand their appliance usage behavior and take necessary steps for reducing energy consumption.

By using deep learning models trained on publicly available datasets it is now feasible to enable NILM in very cost effective ways by leveraging commodity hardware and open source software. Deploying the trained models at the edge of the power grid, i.e. the building level, results in additional cost savings since it obviates the need for always connected internet cloud services which can be expensive at scale and running the algorithms in real-time provides for low latency operation.

This Project will show you how NILM works by taking you through the steps I used to implement a prototype system at my house.

Architecture

The following first two diagrams illustrate the NILM concept and process steps at a high level. The last diagram shows my prototype system which is based on these concepts and is implemented using Arduino- and Raspberry Pi-based compute.

NILM Algorithm Selection and Models

Algorithm Selection

Energy disaggregation is a highly underdetermined and a single-channel Blind Source Separation (BSS) problem² which makes it difficult to obtain accurate predictions. Let $M$ be the number of household appliances and $i$ be the index referring to the $i$-th appliance. The aggregate power consumption $x$ at a given time $t$ is the sum of the power consumption of all appliances $M$, denoted by $y_i\forall{i=1,...,M}$. Therefore, the total power consumption $x$ at a given time $t$ can expressed by the equation below.

$$x(t)=\sum_{i=1}^My_i(t)+\epsilon_{noise}(t) \tag{1}$$

Where $\epsilon_{noise}$ is a noise term. The goal of this project is to solve the inverse problem and estimate the appliance power consumption $y_i$, given the aggregate power signal $x$, and to do so in a manner suitable for deployment at the edge.

Past approaches have included factorial hidden Markov models (FHMM)¹ and various event-based methods with some success⁶. You can also solve the single-channel BSS problem can by using sequence-to-sequence (seq2seq) learning with neural networks, and it can applied to the NILM problem using transformers, convolutional and recurrent neural networks³. Seq2seq learning involves training a neural network to map between an input time series, such as the aggregate power readings the case of NILM, and a output sequence, such as the estimated energy consumption of a single appliance. A sliding input window is typically used to training the network which generates a corresponding window of the output. This method produces multiple predictions for each appliance in the output so an average of the predictions is used for the final result. Some of the predictions will be more accurate than others, especially those near the midpoint of the input sliding window. The averaging will tend to lower the overall accuracy of the predictions.

Some of the disadvantages of of seq2seq leaning can mitigated by sequence-to-point learning (seq2point) for single-channel BSS⁴. You also use a sliding input signal window in this approach, however the network is trained to predict the output signal only at the midpoint of the window which makes the prediction problem easier on the network, leading to more accurate results.

I selected the seq2point learning approach for my prototype system and my implementation was inspired and guided by work described by Michele D'Incecco, Stefano Squartini and Mingjun Zhong⁵. I developed a variety of seq2point learning models using Tensorflow as shown in the Python module define_models.py but focussed my work on the models transformer and cnn.

Neural Network Models

The cnn model is depicted below for an input sequence length of 599 samples. The model generally follows traditional cnn concepts from vision use cases where several convolutional layers are used to extract features from the input power sequence at gradually finer details as the input traverses through the network. These features are the on-off patterns of the appliances as well as their power consumption levels. Max pooling is used to manage the complexity of the model after each convolutional layer. Finally, dense layers are used to output the final single point power consumption estimate for the window which is de-normalized before using in downstream processing. There are about 40 million parameters in this model using the default values.

The transformer model is depicted below for an input sequence length of 599 samples where the transformer block is a Bert-style encoder. The input sequence is first passed through a convolutional layer to expand it into a latent space which is analogous to the feature extraction in the cnn model case. These features are pooled and L2 normalized to reduce model complexity and to mitigate the effects of outliers. Next, the sequence features are processed by a Bert-style transformer lineup which includes positional embedding and transformer blocks that applies importance weighting. The output of the encoder is decoded by several layers which are comprised of relative position embedding which applies symmetric weights around the mid-point of the signal, average pooling which reduces the sequence to a single value per feature and then finally dense layers that output the final single point estimated power value for the window which again is de-normalized for downstream processing. There are about six million parameters in this model using the default values.

The Bert-style transformer encoder is depicted below.

NILM Datasets

There are a number of large-scale publicly available datasets specifically designed to address the NILM problem which were captured in household buildings from various countries. The table⁷ below shows several of the most widely used.

The datasets generally include many 10’s of millions of active power, reactive power, current, and voltage samples but with different sampling frequencies which requires you to pre-process the data before use. Most NILM algorithms utilize only real (aka active or true) power data. Five appliances are usually considered for energy disaggregation research which are kettle, microwave, fridge, dish washer and washing machine. These are the appliances I considered for my prototype and following the work of Michele DIncecco, et al.⁵, I mainly focused on the REFIT⁸ dataset but will eventually include UK-DALE and REDD.

Note that these datasets are typically very imbalanced because the majority of the time an appliance is in the off state.

Model Training and Results

I used TensorFlow to train and test the model. All code associated with this section can be found on the Machine Learning section of the project’s GitHub, NILM⁹. The seq2point learning models for the appliances were trained individually on z-score standardized REFIT data or normalized to $[0, P_m]$, where $P_m$ is the maximum power consumption of an appliance in its active state. Using normalized data tended to give the best model performance so it is used by default.

I used the following metrics to evaluate the model’s performance. You can view the code that calculates these metrics here.

• Mean absolute error ($MAE$), which evaluates the absolute difference between the prediction and the ground truth power at every time point and calculates the mean value, as defined by the equation below.

$$MAE = \frac{1}{N}\sum_{i=1}^{N}|\hat{x_i}-x_i|\tag{2}$$

• Normalized signal aggregate error ($SAE$), which indicates the relative error of the total energy. Denote $r$ as the total energy consumption of the appliance and $\hat{r}$ as the predicted total energy, then SAE is defined per the equation below.

$$SAE = \frac{|\hat{r} - r|}{r}\tag{3}$$

• Energy per Day ($EpD$) which measures the predicted energy used in a day, useful when the household users are interested in the total energy consumed in a period. Denote $D$ as the total number of days and $e=\sum_{t}e_t$ as the appliance energy consumed in a day, then EpD is defined per the equation below.

$$EpD = \frac{1}{D}\sum_{n=1}^{D}e\tag{4}$$

• Normalized disaggregation error ($NDE$) which measures the normalized error of the squared difference between the prediction and the ground truth power of the appliances, as defined by the equation below.

$$NDE = \frac{\sum_{i,t}(x_{i,t}-\hat{x_{i,t}})^2}{\sum_{i,t}x_{i,t}^2}\tag{5}$$

I also used accuracy ($ACC$), F1-score ($F1$) and Matthew’s correlation coefficient ($MCC$) to assess if the model can perform well with the severely imbalanced datasets used to train and test the model. These metrics depend on the on-off status of the device and are computed using the parameters in the common.py module. $ACC$ is equal to the number of correctly predicted time points over the test dataset. $F1$ and $MCC$ are computed according to the equations below where $TP$ stands for true positives, $TN$ stands for true negatives, $FP$ stands for false positives and $FN$ stands for false negatives.

$$F1=\frac{TP}{TP+\frac{1}{2}(FP+FN)}\tag{6}$$

$$MCC=\frac{TN \times TP-FN \times FP }{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}\tag{7}$$

$MAE$, $SAE$, $NDE$ and $EpD_e$ (defined as $ 100% \times (EpD_{predicted} - EpD_{ground_truth}) / EpD_{ground_truth}$) reflect the model's ability to correctly predict the appliance energy consumption levels. $F1$ and $MCC$ indicates the model's ability to correctly predict appliance activations using imbalanced classes. $ACC$ is less useful in this application because most of the time the model will correctly predict the appliance is off which dominates the dataset.

A sliding window of 599 samples of the aggregate real power consumption signal is used as inputs to seq2point model and the midpoints of the corresponding windows of the appliances are used as targets. You can see how the samples and targets are generated by an instance of the WindowGenerator Class defined in the window_generator.py module.

You can see the code I used to train the model in train.py which uses the tf.distribute.MirroredStrategy() distributed training strategy. I used the Keras Adam optimizer and to reduce over-fitting, early stopping is used. The training code can be configured to train the seq2point model from scratch, or given a fitted model, prune it or fine tune it with quantization aware training (QAT), both of which can improve inference performance especially on edge hardware.

The key hyper-parameters for training and the optimizer are summarized below.

• Input Window Size: 599 samples
• Global Batch size: 1024 samples.
• From scratch Learning Rate: 1e-04
• QAT and Prune Learning Rate: 1e-05
• Adam Optimizer: beta_1=0.9, beta_2=0.999, epsilon=1e-08.
• Early Stopping Criteria: 6 epochs.

The loss function¹¹ shown in the equation below is used to compute training gradients and to evaluate validation loss on a per-batch basis. It consists of a combination of Mean Squared Error, Binary Cross-Entropy and Mean Absolute Error losses, averaged over distributed model replica batches.

$$L(x, s) = (\hat{x} - x)^2 -(\hat{s}\log{s}+(1-\hat{s})\log{(1-s)}) + (\lambda|\hat{x}-x|, \hat{x}\in\set{O})\tag{8}$$

Where $x, \hat{x}\in[0, 1]$ are the ground truth and predicted power usage single point values divided by the maximum power limit per appliance and $s, \hat{s}\in\set{0, 1}$ are the appliance state label and prediction, and $O$ is the set of predictions when either the status label is on or the prediction is incorrect. The hyperparameter $\lambda$ tunes the absolute loss term on an a per-appliance basis.

You can find the training results for each appliance in the models folder. Typical performance metrics for the cnn model are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e\thinspace(%)\downarrow$
kettle	0.7309	0.7346	0.9960	8.788	0.2284	0.4696	-22.85
microwave	0.5671	0.5667	0.9933	7.564	0.0460	0.8845	-4.661
fridge	0.7929	0.6942	0.8645	12.33	0.0373	0.4111	3.733
dishwasher	0.6069	0.6483	0.9801	5.367	0.0347	0.3742	-3.473
washingmachine	0.8478	0.8441	0.9893	14.75	0.2929	0.3470	-29.29

Typical performance metrics for the transformer model are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e\thinspace(%)\downarrow$
kettle	0.8177	0.8176	0.9967	7.264	0.1348	0.3760	13.48
microwave	0.6530	0.6506	0.9952	6.428	0.1454	0.7496	-14.54
fridge	0.8138	0.7262	0.8799	11.02	0.0559	0.3743	5.590
dishwasher	0.6914	0.7119	0.9873	5.373	0.0690	0.3906	-6.904
washingmachine	0.8435	0.8420	0.9886	14.19	0.2440	0.2440	-24.40

Average metrics across all appliances for both model architectures are compared in the table below.

Architecture	$\overline{F1}\uparrow$	$\overline{MCC}\uparrow$	$\overline{ACC}\uparrow$	$\overline{MAE}$ $(W)$ $\downarrow$	$\overline{SAE}\downarrow$	$\overline{NDE}\downarrow$	$\overline{|EpD_e|}\thinspace(%)\downarrow$
cnn	0.7161	0.7065	0.9643	9.852	0.1957	0.4528	19.57
transformer	0.7639	0.7497	0.9695	8.855	0.1298	0.3781	12.98

You can see that the cnn and transformer models have similar performance even though the latter has about six times fewer parameters than the former. However, each transformer training step takes about seven times longer than cnn due to the transformer model's use of self-attention which has $O(n^2)$ complexity as compared to the cnn model's $O(n)$, where $n$ is the input sequence length. On the basis on training (and inference) efficiency, you can see that cnn is preferable with little loss in model performance.

You can find a variety of performance metrics in the literature to compare these results against, here are two examples. The middle column in the table⁵ below shows another cnn-based model's performance. You can see the results of this project compare very favorably.

The table¹² below shows the results from two transformer-based models (BERT4NILM and ELECTRIcity) and three other architectures. Although the data is incomplete, again this project's transformer-based model results compare favorably.

Model Quantization

I performed Post-training quantization on the cnn and transformer models using the TensorFlow Lite converter API with various quantization modes to improve inference speed on edge hardware, including the Raspberry Pi and the Google Edge TPU, while mitigating the impact on accuracy. You can see the quantization modes I used in the table below.

Mode	Description
convert_only	Convert to tflite but keep all parameters in Float32 (no quantization).
w8	Quantize weights from float32 to int8 and biases to int64. Leave activations in Float32.
w8_a8_fallback	Same as w8 but quantize activations from float32 to int8. Fallback to float if an operator does not have an integer implementation.
w8_a8	Same as w8 but quantize activations from float32 to int8. Enforce full int8 quantization for all operators.
w8_a16	Same as w8 but quantize activations to int16.

The cnn model was quantized using all modes to understand the best tradeoff between latency and accuracy. Only the weights for the transformer model were quantized to int8 using mode w8, the activations needed to be kept in Float32 to maintain acceptable accuracy. See convert_keras_to_tflite.py for the code that does this quantization which also uses TensorFlow Lite's quantization debugger to check how well each layer in the model was quantized. I also profiled the converted models using the TensorFlow Lite Model Benchmark Tool to quantify inference latencies.

The quantized inference results are shown in the tables below, where $L_{x86}$ is the average inference latency on a 3.8 GHz x86 machine using eight tflite interpreter threads and $L_{arm}$ is the average inference latency on the ARM aarch-64-based Raspberry Pi 4 using four threads with both computers using the TensorFlow Lite XNNPACK CPU delegate. $L_{tpu}$ is the average inference latency on the Google Coral Edge TPU. Model inputs and outputs were keep in float32 to maximize inference speed for the x86- and ARM-based machines but were set to int8 for the edge tpu.

CNN Model Results and Discussion

The quantized results for the cnn models are shown in the table below for quantization mode w8.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)	$L_{x86}$ (${\mu}s$)	$L_{arm}$ (${\mu}s$)
kettle	0.7428	0.7454	0.9966	7.371	0.2013	0.4500	-20.13	743.910	3809.41
microwave	0.6400	0.6373	0.9933	7.971	0.1578	0.7194	-15.78	736.549	3586.55
fridge	0.6491	0.5040	0.7935	17.78	0.0907	0.7000	-8.971	731.371	3520.54
dishwasher	0.5143	0.5787	0.9719	5.3955	0.0569	0.3647	-5.694	742.279	3519.52
washingmachine	0.8838	0.8791	0.9919	15.33	0.2930	0.3811	-29.30	751.801	3515.68

The quantized results for the cnn kettle model are shown below for the other quantization modes.

Mode	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)	$L_{x86}$ (${\mu}s$)	$L_{arm}$ (${\mu}s$)	$L_{tpu}$ (${\mu}s$)
convert_only	0.7119	0.7199	0.9964	7.812	0.2862	0.4780	-28.65	309.531	12890.10	NA
w8_a8_fallback	0.6584	0.6736	0.9959	8.677	0.3700	0.5448	-36.70	186.483	3435.56	NA
w8_a8	0.6584	0.6736	0.9960	8.768	0.3670	0.5447	-36.70	185.403	3258.19	27777.8
w8_a16	0.7474	0.7479	0.9966	7.431	0.1531	0.4516	-15.31	5095.38	24065.3	NA

The quantized results for the cnn microwave model are shown below for the other quantization modes.

Mode	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)	$L_{x86}$ (${\mu}s$)	$L_{arm}$ (${\mu}s$)	$L_{tpu}$ (${\mu}s$)
convert_only	0.6410	0.6384	0.9933	7.976	0.1630	0.7195	-18.30	302.750	13055.3	NA
w8_a8_fallback	0.6268	0.6238	0.9931	8.006	0.1796	0.7206	-17.96	187.232	3458.30	NA
w8_a8	0.6268	0.6238	0.9931	8.005	0.1796	0.7206	-17.96	183.910	3466.65	100000
w8_a16	0.6391	0.6365	0.9933	7.968	0.1590	0.7172	-15.89	5100.76	24052.0	NA

Results for the other appliance models are omitted for brevity but show similar characteristics as a function of quantization mode.

You can see the negative impact of activation quantization but weight quantization, because of regularization effects, has a moderate benefit on some model performance metrics. As expected, the full quantization modes lead to the lowest latencies. Quantizing activations to int16 by the w8_a16 mode results in the highest latencies because only non-optimized reference kernel implementations are presently available in TensorFlow Lite but this scheme leads to the best model metrics given the regularization benefits from weight quantization and better preservation of activation numerics.

You can also see that inference latency of the modes follows w8 ${>}$ convert_only ${>}$ w8_a8 for the x86 machine but convert_only ${>}$ w8 ${>}$ w8_a8 for the aarch64 machine, although the variation is larger for x86. To better understand this, I profiled the converted models using the TFLite Model Benchmark Tool. A summary of the profiling results for the cnn microwave model are shown below which are representative of the other models.

Model Profiling on x86 (slowest to fastest)

You can see that the Fully Connected and Convolution operations are taking the longest to execute in all cases but are much faster in the fully quantized mode of w8_a8.

w8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
FULLY_CONNECTED	2	0.397	56.5527%	56.5527%	0	2
CONV_2D	4	0.204	29.0598%	85.6125%	0	4
Copy (NC X32)	2	0.051	7.26496%	92.8775%	0	9
Max Pooling (NHWC F32)	1	0.038	5.41311%	98.2906%	0	4
Convolution (NHWC F32) IGEMM	1	0.012	1.7094%	100%	0	1
Fully Connected (NC F32) GEMM	1	0	0%	100%	0	1
EXPAND_DIMS	9	0	0%	100%	0	9

convert_only node type	count	avg_ms	avg %	cdf %	mem KB	times called
Fully Connected (NC F32) GEMM	3	0.151	53.3569%	53.3569%	0	3
Convolution (NHWC F32) IGEMM	1	0.062	21.9081%	75.265%	0	5
Max Pooling (NHWC F32)	1	0.037	13.0742%	88.3392%	0	4
Copy (NC X32)	1	0.033	11.6608%	100%	0	9
EXPAND_DIMS	9	0	0%	100%	0	9

w8_a8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
Convolution (NHWC QC8) IGEMM	1	0.061	37.1951%	37.1951%	0	5
Fully Connected (NC QS8) GEMM	3	0.037	22.561%	59.7561%	0	3
Max Pooling (NHWC S8)	1	0.034	20.7317%	80.4878%	0	4
Copy (NC X8)	1	0.032	19.5122%	100%	0	9
EXPAND_DIMS	9	0	0%	100%	0	9
Convert (NC QS8 F32)	1	0	0%	100%	0	1
Convert (NC F32 QS8)	1	0	0%	100%	0	1

Model Profiling on aarch64 (slowest to fastest)

You can see the copy and Max Pooling operations in particular are relatively slower on x86 than on aarch64 which is probably due to memory bandwidth and micro-architecture differences.

convert_only node type	count	avg_ms	avg %	cdf %	mem KB	times called
Copy (NC X32)	1	34.835	30.9136%	30.9136%	0	9
Convolution (NHWC F32) IGEMM	1	32.414	28.7651%	59.6787%	0	5
Max Pooling (NHWC F32)	1	23.008	20.418%	80.0967%	0	4
Fully Connected (NC F32) GEMM	3	22.425	19.9006%	99.9973%	0	3
EXPAND_DIMS	9	0.003	0.00266229%	100%	0	9

w8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
Max Pooling (NHWC F32)	1	42.281	46.9085%	46.9085%	0	4
CONV_2D	4	17.461	19.3721%	66.2806%	0	4
Copy (NC X32)	2	16.877	18.7241%	85.0047%	0	9
Convolution (NHWC F32) IGEMM	1	10.658	11.8245%	96.8292%	0	1
FULLY_CONNECTED	2	2.847	3.1586%	99.9878%	0	2
Fully Connected (NC F32) GEMM	1	0.007	0.00776613%	99.9956%	0	1
EXPAND_DIMS	9	0.004	0.00443779%	100%	0	9

w8_a8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
Copy (NC X8)	1	34.686	30.9647%	30.9647%	0	9
Convolution (NHWC QC8) IGEMM	1	33.259	29.6908%	60.6554%	0	5
Max Pooling (NHWC S8)	1	23.922	21.3555%	82.0109%	0	4
Fully Connected (NC QS8) GEMM	3	20.146	17.9846%	99.9955%	0	3
EXPAND_DIMS	9	0.002	0.00178543%	99.9973%	0	9
Convert (NC F32 QS8)	1	0.002	0.00178543%	99.9991%	0	1
Convert (NC QS8 F32)	1	0.001	0.000892714%	100%	0	1

Quantization Efficacy

The RMSE / scale is close to $1 / sqrt(12)$ (~ 0.289) when quantized distribution is similar to the original float distribution, indicating a well-quantized model. The larger the value is, it's more likely for the layer not being quantized well. The tables below show the RMSE / Scale metric for the cnn kettle and cnn washingmachine models and the Suspected? column indicates a layer that significantly exceeds 0.289. Other models are omitted for brevity but show similar results. These layers can remain in float to generate a selectively quantized model that increases accuracy at the expense of inference performance but doing so for the cnn models did not materially improve accuracy. See Inspecting Quantization Errors with Quantization Debugger.

Layer quantization efficacy metrics for the cnn kettle model using mode w8_a8 are shown below.

layer	op_name	range	rmse/scale	Suspected?
0	EXPAND_DIMS	23.929975	2.75E-01
1	CONV_2D	5.518012	2.22E-01
2	RESHAPE	5.518012	2.38E-06
3	EXPAND_DIMS	5.518012	2.38E-06
4	MAX_POOL_2D	5.518012	2.42E-06
5	RESHAPE	5.518012	2.42E-06
6	EXPAND_DIMS	5.518012	2.42E-06
7	CONV_2D	3.494575	1.79E-01
8	RESHAPE	3.494575	1.44E-06
9	EXPAND_DIMS	3.494575	1.44E-06
10	MAX_POOL_2D	3.494575	1.49E-06
11	RESHAPE	3.494575	1.49E-06
12	EXPAND_DIMS	3.494575	1.49E-06
13	CONV_2D	5.910911	1.22E-01
14	RESHAPE	5.910911	1.09E-06
15	EXPAND_DIMS	5.910911	1.09E-06
16	MAX_POOL_2D	5.910911	1.18E-06
17	RESHAPE	5.910911	1.18E-06
18	EXPAND_DIMS	5.910911	1.18E-06
19	CONV_2D	10.82671	1.00E-01
20	RESHAPE	10.82671	1.00E-06
21	EXPAND_DIMS	10.82671	1.00E-06
22	MAX_POOL_2D	10.82671	1.08E-06
23	RESHAPE	10.82671	1.08E-06
24	EXPAND_DIMS	10.82671	1.08E-06
25	CONV_2D	2.987664	5.77E-02
26	RESHAPE	2.987664	3.82E-07
27	FULLY_CONNECTED	1.725032	1.20E+00	Yes
28	FULLY_CONNECTED	1.512342	6.19E-01
29	FULLY_CONNECTED	0.990833	2.03E+00	Yes

Layer quantization efficacy metrics for the cnn washingmachine model using mode w8_a8 are shown below.

layer	op_name	range	rmse/scale	Suspected?
0	EXPAND_DIMS	30.954547	3.00E-01
1	CONV_2D	12.620701	1.81E-01
2	RESHAPE	12.620701	1.56E-06
3	EXPAND_DIMS	12.620701	1.56E-06
4	MAX_POOL_2D	12.620701	1.67E-06
5	RESHAPE	12.620701	1.67E-06
6	EXPAND_DIMS	12.620701	1.67E-06
7	CONV_2D	15.030776	1.94E-01
8	RESHAPE	15.030776	1.94E-07
9	EXPAND_DIMS	15.030776	1.94E-07
10	MAX_POOL_2D	15.030776	2.36E-07
11	RESHAPE	15.030776	2.36E-07
12	EXPAND_DIMS	15.030776	2.36E-07
13	CONV_2D	10.132236	1.68E-01
14	RESHAPE	10.132236	1.83E-06
15	EXPAND_DIMS	10.132236	1.83E-06
16	MAX_POOL_2D	10.132236	2.06E-06
17	RESHAPE	10.132236	2.06E-06
18	EXPAND_DIMS	10.132236	2.06E-06
19	CONV_2D	9.038437	1.45E-01
20	RESHAPE	9.038437	1.90E-06
21	EXPAND_DIMS	9.038437	1.90E-06
22	MAX_POOL_2D	9.038437	2.20E-06
23	RESHAPE	9.038437	2.20E-06
24	EXPAND_DIMS	9.038437	2.20E-06
25	CONV_2D	5.151932	9.58E-02
26	RESHAPE	5.151932	1.08E-06
27	FULLY_CONNECTED	7.269378	1.13E-01
28	FULLY_CONNECTED	3.79172	1.75E-01
29	FULLY_CONNECTED	1.102378	9.51E-01	Yes

Model Memory Footprint

I used the TFLite Model Benchmark Tool to get the approximate RAM consumption of the TFLite cnn microwave model at runtime which is shown in the table below for each quantization mode as well as the TFLite model disk space. The other cnn models show similar characteristics. The findings for the x86 architecture were identical to the arm architecture. Note that the Keras model consumes about 42.49 (MB) on disk. You can see that there is about a four times reduction in disk storage space due to the float32 to int8 weight conversions. Interestingly, RAM runtime usage varies considerably due to the TFLite algorithms that that optimize intermediate tensors usage. These are pre-allocated to reduce inference latency at the cost of memory space. See Optimizing TensorFlow Lite Runtime Memory.

Quant Mode	Disk (MB)	RAM (MB)
convert_only	42.485044	84.957
w8	10.642040	12.7109
w8_a8	10.648016	23.3633
w8_a16	10.660984	10.1836

Transformer Model Results and Discussion

Note that even though the XNNPACK delagate was enabled during transformer model inference evaluation nothing was actually accelerated because the transformer model contains dynamic tensors. The following warning is shown when using the TFLite interpreter for inference:

Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#94 is a dynamic-sized tensor).

This means that all operators are unsupported by XNNPACK and will fall back to the default CPU kernel implementations. A future effort will involve refactoring the transformer model use only static-size tensors. Note that a tensor could be marked dynamic at TFLite runtime when there is a control-flow operation (e.g., if, while etc.) prepared. In other words, even when the model graph itself doesn't have any tensors of dynamic shapes statically, at runtime a model could have dynamic tensors. The current transformer model uses if control-flow operations.

The quantized results for the transformer model are shown in the table below for quantization mode w8.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)	$L_{x86}$ (${\mu}s$)	$L_{arm}$ (${\mu}s$)
kettle	0.8460	0.8483	0.9977	5.117	0.1761	0.2760	-17.61	20520	61939
microwave	0.7375	0.7355	0.9952	6.969	0.3435	0.6049	-34.35	20905	60823
fridge	0.7437	0.6261	0.8381	14.52	0.0838	0.5518	-8.560	20986	61862
dishwasher	0.6671	0.6905	0.9860	6.214	0.1117	0.4521	-11.17	20983	63432
washingmachine	0.8910	0.8872	0.9926	15.08	0.3796	0.3442	-37.96	21426	60901

The quantized results for the transformer kettle and microwave models are shown in the table below for quantization mode convert_only.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)	$L_{x86}$ (${\mu}s$)	$L_{arm}$ (${\mu}s$)
kettle	0.8460	0.8483	0.9977	5.117	0.1761	0.2760	-17.61	7100	69746
microwave	0.6719	0.6736	0.9948	7.054	0.3930	0.6518	-39.30	7050	68667

Model Profiling on x86 (slowest to fastest)

w8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
FULLY_CONNECTED	14	16.243	79.5485%	79.5485%	0	14
ADD	41	0.667	3.26657%	82.815%	0	41
MUL	42	0.661	3.23718%	86.0522%	0	42
BATCH_MATMUL	4	0.582	2.85029%	88.9025%	0	4
TRANSPOSE	20	0.551	2.69847%	91.601%	0	20
RESHAPE	48	0.384	1.8806%	93.4816%	0	48
POW	2	0.371	1.81694%	95.2985%	0	2
SOFTMAX	2	0.202	0.989275%	96.2878%	0	2
MEAN	13	0.177	0.86684%	97.1546%	0	13
CONV_2D	1	0.152	0.744405%	97.899%	0	1
SQUARED_DIFFERENCE	6	0.119	0.582791%	98.4818%	0	6
TANH	2	0.11	0.538714%	99.0205%	0	2
AVERAGE_POOL_2D	1	0.086	0.421176%	99.4417%	0	1
FILL	14	0.059	0.288947%	99.7306%	0	14
SQUARE	1	0.036	0.176306%	99.9069%	0	1
SQRT	1	0.019	0.0930506%	100%	0	1
SUB	6	0	0%	100%	0	6
STRIDED_SLICE	14	0	0%	100%	0	14
SHAPE	24	0	0%	100%	0	24
RSQRT	6	0	0%	100%	0	6
REDUCE_PROD	16	0	0%	100%	0	16
PACK	22	0	0%	100%	0	22
GATHER	16	0	0%	100%	0	16
EXPAND_DIMS	2	0	0%	100%	0	2
CONCATENATION	8	0	0%	100%	0	8

convert_only node type	count	avg_ms	avg %	cdf %	mem KB	times called
FULLY_CONNECTED	14	2.211	31.0447%	31.0447%	0	14
ADD	41	1.344	18.8711%	49.9158%	0	41
MUL	42	0.693	9.73041%	59.6462%	0	42
RESHAPE	48	0.619	8.69138%	68.3375%	0	48
BATCH_MATMUL	4	0.529	7.42769%	75.7652%	0	4
TRANSPOSE	20	0.4	5.6164%	81.3816%	0	20
POW	2	0.394	5.53215%	86.9138%	0	2
MEAN	13	0.216	3.03286%	89.9466%	0	13
SOFTMAX	2	0.155	2.17636%	92.123%	0	2
SQUARED_DIFFERENCE	6	0.128	1.79725%	93.9202%	0	6
TANH	2	0.119	1.67088%	95.5911%	0	2
AVERAGE_POOL_2D	1	0.102	1.43218%	97.0233%	0	1
FILL	14	0.099	1.39006%	98.4134%	0	14
CONV_2D	1	0.051	0.716091%	99.1294%	0	1
SQUARE	1	0.037	0.519517%	99.649%	0	1
SQRT	1	0.025	0.351025%	100%	0	1
SUB	6	0	0%	100%	0	6
STRIDED_SLICE	14	0	0%	100%	0	14
SHAPE	24	0	0%	100%	0	24
RSQRT	6	0	0%	100%	0	6
REDUCE_PROD	16	0	0%	100%	0	16
PACK	22	0	0%	100%	0	22
GATHER	16	0	0%	100%	0	16
EXPAND_DIMS	2	0	0%	100%	0	2
CONCATENATION	8	0	0%	100%	0	8

Model Profiling on aarch64 (slowest to fastest)

convert_only node type	count	avg_ms	avg %	cdf %	mem KB	times called
FULLY_CONNECTED	14	30.777	44.8716%	44.8716%	0	14
MUL	42	8.571	12.4962%	57.3678%	0	42
ADD	41	7.425	10.8254%	68.1931%	0	41
POW	2	5.308	7.73885%	75.932%	0	2
BATCH_MATMUL	4	3.82	5.56941%	81.5014%	0	4
TRANSPOSE	20	3.352	4.88708%	86.3885%	0	20
RESHAPE	48	2.927	4.26745%	90.6559%	0	48
TANH	2	2.02	2.94508%	93.601%	0	2
SOFTMAX	2	1.182	1.72331%	95.3243%	0	2
MEAN	13	0.927	1.35153%	96.6758%	0	13
SQUARED_DIFFERENCE	6	0.761	1.10951%	97.7854%	0	6
AVERAGE_POOL_2D	1	0.517	0.753765%	98.5391%	0	1
SQUARE	1	0.326	0.475295%	99.0144%	0	1
CONV_2D	1	0.281	0.409687%	99.4241%	0	1
SQRT	1	0.148	0.215778%	99.6399%	0	1
PACK	22	0.105	0.153086%	99.793%	0	22
FILL	14	0.077	0.112263%	99.9052%	0	14
SUB	6	0.022	0.0320751%	99.9373%	0	6
RSQRT	6	0.022	0.0320751%	99.9694%	0	6
STRIDED_SLICE	14	0.012	0.0174955%	99.9869%	0	14
SHAPE	24	0.005	0.0072898%	99.9942%	0	24
EXPAND_DIMS	2	0.003	0.00437388%	99.9985%	0	2
CONCATENATION	8	0.001	0.00145796%	100%	0	8
REDUCE_PROD	16	0	0%	100%	0	16
GATHER	16	0	0%	100%	0	16

w8 node type	count	avg_ms	avg %	cdf %	mem KB	times called
FULLY_CONNECTED	14	22.759	38.6283%	38.6283%	0	14
MUL	42	7.718	13.0996%	51.7278%	0	42
ADD	41	6.952	11.7994%	63.5273%	0	41
POW	2	5.074	8.61197%	72.1392%	0	2
BATCH_MATMUL	4	3.773	6.40382%	78.5431%	0	4
TRANSPOSE	20	2.982	5.06127%	83.6043%	0	20
RESHAPE	48	2.902	4.92549%	88.5298%	0	48
TANH	2	1.95	3.30968%	91.8395%	0	2
SOFTMAX	2	1.222	2.07407%	93.9136%	0	2
MEAN	13	0.91	1.54452%	95.4581%	0	13
SQUARED_DIFFERENCE	6	0.79	1.34085%	96.7989%	0	6
CONV_2D	1	0.659	1.1185%	97.9174%	0	1
AVERAGE_POOL_2D	1	0.48	0.814692%	98.7321%	0	1
SQUARE	1	0.359	0.609321%	99.3414%	0	1
SQRT	1	0.126	0.213857%	99.5553%	0	1
PACK	22	0.107	0.181608%	99.7369%	0	22
FILL	14	0.082	0.139176%	99.8761%	0	14
SUB	6	0.025	0.0424319%	99.9185%	0	6
RSQRT	6	0.023	0.0390373%	99.9576%	0	6
STRIDED_SLICE	14	0.013	0.0220646%	99.9796%	0	14
EXPAND_DIMS	2	0.005	0.00848637%	99.9881%	0	2
SHAPE	24	0.004	0.0067891%	99.9949%	0	24
GATHER	16	0.002	0.00339455%	99.9983%	0	16
CONCATENATION	8	0.001	0.00169727%	100%	0	8
REDUCE_PROD	16	0	0%	100%	0	16

Quantization Efficacy

Layer quantization efficacy metrics for the transformer kettle model using mode w8_a8 are shown in the table below, although as noted above quantizing the transformer model's activations results in very poor model performance. You can see the RSQRT operator in particular does not quantize well, these operators are used in the Gaussian error linear activation functions which helps explain the poor performance of the model. The other transformer appliance models show similar efficacy metrics.

layer	op_name	range	rmse/scale	Suspect?
0	EXPAND_DIMS	23.929975	0.2751951
1	CONV_2D	6.793423	0.2952648
2	RESHAPE	12.080546	0.1608947
3	EXPAND_DIMS	12.080546	0
4	AVERAGE_POOL_2D	12.080546	0
5	ADD	11.598116	0.03801457
6	RESHAPE	3.405601	0.0580239
7	FILL	1	0
8	MUL	0.489979	0.2888043
9	ADD	3.72709	0.2890081
10	FILL	1	0
11	FILL	2E-06	0
12	RESHAPE	3.72709	0
13	TRANSPOSE	3.72709	0
14	MEAN	1.371433	0.3178885
15	SQUARED_DIFFERENCE	4.45292	0.2296307
16	MEAN	0.547054	0.2962779
17	ADD	0.548054	0.4589049
18	RSQRT	15.745848	41.5733	Yes
19	MUL	15.745848	0
20	MUL	9.606549	0.3037147
21	MUL	2.384117	0.2467738
22	SUB	2.384117	1.428423E-06
23	ADD	8.792654	0.3326802
24	TRANSPOSE	8.792654	0
25	RESHAPE	8.792654	0
26	MUL	8.606096	0.2844383
27	ADD	8.682471	0.2893911
28	RESHAPE	8.682471	0
29	FULLY_CONNECTED	13.589397	0.337322
30	RESHAPE	13.589397	0
31	ADD	13.66659	0.2940356
32	RESHAPE	13.66659	0
33	TRANSPOSE	13.66659	0
34	FULLY_CONNECTED	16.061866	0.3192651
35	RESHAPE	16.061866	0
36	ADD	16.14824	0.2843503
37	RESHAPE	16.14824	0
38	TRANSPOSE	16.14824	0
39	BATCH_MATMUL	791.061	0.2887679
40	MUL	69.920572	1.274111E-06
41	SOFTMAX	0.996094	0.1901133
42	FULLY_CONNECTED	15.370091	0.3185204
43	RESHAPE	15.370091	0
44	ADD	15.420575	0.273666
45	RESHAPE	15.420575	0
46	TRANSPOSE	15.420575	0
47	BATCH_MATMUL	13.788238	0.2886845
48	TRANSPOSE	13.788238	0
49	RESHAPE	13.788238	0
50	RESHAPE	13.788238	0
51	FULLY_CONNECTED	13.103168	0.3028338
52	RESHAPE	13.103168	0
53	ADD	13.229318	0.2886953
54	ADD	15.80188	0.288787
55	FILL	1	0
56	FILL	2E-06	0
57	RESHAPE	15.80188	0
58	TRANSPOSE	15.80188	0
59	MEAN	0.345499	0.4923606
60	SQUARED_DIFFERENCE	63.224075	0.2779759
61	MEAN	4.931316	0.3027504
62	ADD	4.932316	0.0311987
63	RSQRT	1.195838	0.304619
64	MUL	1.195838	0
65	MUL	10.555507	0.2823485
66	MUL	0.219467	0.4081438
67	SUB	0.219467	2.18255E-06
68	ADD	10.513915	0.2897937
69	TRANSPOSE	10.513915	0
70	RESHAPE	10.513915	0
71	MUL	11.164726	0.2816773
72	ADD	11.191346	0.2954707
73	RESHAPE	11.191346	0
74	FULLY_CONNECTED	20.655138	0.3090305
75	RESHAPE	20.655138	0
76	ADD	20.650391	0.3390533
77	MUL	99.871166	0.2299994
78	ADD	120.521553	0.2948375
79	MUL	96.162275	9.767714E-07
80	TANH	1.992188	0.2440195
81	ADD	2	0.2198295
82	MUL	10.325196	0
83	MUL	9.776829	0.4144
84	RESHAPE	9.776829	0
85	FULLY_CONNECTED	83.573343	0.2907445
86	RESHAPE	83.573343	0
87	ADD	83.595579	0.08025211
88	ADD	85.804338	0.2887958
89	FILL	1	0
90	FILL	2E-06	0
91	RESHAPE	85.804338	0
92	TRANSPOSE	85.804338	0
93	MEAN	0.671132	0.3675501
94	SQUARED_DIFFERENCE	2199.59073	0.2545643
95	MEAN	175.261857	0.2906021
96	ADD	175.262859	0.001332498
97	RSQRT	0.985917	inf	Yes
98	MUL	0.985917	0
99	MUL	10.02912	0.2857049
100	MUL	0.143655	0.2965899
101	SUB	0.143655	1.620293E-06
102	ADD	10.016886	0.273117
103	TRANSPOSE	10.016886	0
104	RESHAPE	10.016886	0
105	MUL	10.061665	0.2678465
106	ADD	10.07655	0.2902205
107	RESHAPE	10.07655	0
108	FULLY_CONNECTED	24.452702	0.305032
109	RESHAPE	24.452702	0
110	ADD	24.471723	0.2637686
111	RESHAPE	24.471723	0
112	TRANSPOSE	24.471723	0
113	FULLY_CONNECTED	24.602528	0.3068753
114	RESHAPE	24.602528	0
115	ADD	24.524895	0.2782782
116	RESHAPE	24.524895	0
117	TRANSPOSE	24.524895	0
118	BATCH_MATMUL	2375.91354	0.2896734
119	MUL	210.003057	2.292973E-06
120	SOFTMAX	0.996094	0.05637038
121	FULLY_CONNECTED	20.075173	0.3076454
122	RESHAPE	20.075173	0
123	ADD	20.075706	0.263613
124	RESHAPE	20.075706	0
125	TRANSPOSE	20.075706	0
126	BATCH_MATMUL	19.701535	0.2828629
127	TRANSPOSE	19.701535	0
128	RESHAPE	19.701535	0
129	RESHAPE	19.701535	0
130	FULLY_CONNECTED	93.741001	0.2932989
131	RESHAPE	93.741001	0
132	ADD	93.754804	0.07540143
133	ADD	98.071743	0.2887844
134	FILL	1	0
135	FILL	2E-06	0
136	RESHAPE	98.071743	0
137	TRANSPOSE	98.071743	0
138	MEAN	0.733764	0.2978907
139	SQUARED_DIFFERENCE	2799.43284	0.2259659
140	MEAN	47.061099	0.3333294
141	ADD	47.062096	0.00378661
142	RSQRT	0.887359	0.932645	Yes
143	MUL	0.887359	0
144	MUL	17.251803	0.2807441
145	MUL	0.272383	0.2891514
146	SUB	0.272383	1.761181E-06
147	ADD	17.311272	0.3122337
148	TRANSPOSE	17.311272	0
149	RESHAPE	17.311272	0
150	MUL	17.347451	0.2828182
151	ADD	17.347451	0.2636831
152	RESHAPE	17.347451	0
153	FULLY_CONNECTED	18.324211	0.3175219
154	RESHAPE	18.324211	0
155	ADD	18.343422	0.3259408
156	MUL	69.135855	0.2440944
157	ADD	87.47928	0.2834869
158	MUL	69.798363	4.807578E-07
159	TANH	1.992188	0.3179474
160	ADD	2	0.2156563
161	MUL	9.171711	0
162	MUL	9.578754	0.2946312
163	RESHAPE	9.578754	0
164	FULLY_CONNECTED	364.467573	0.2889795
165	RESHAPE	364.467573	0
166	ADD	364.406858	0.01272385
167	ADD	371.900848	0.2615761
168	FILL	1	0
169	FILL	2E-06	0
170	RESHAPE	371.900848	0
171	TRANSPOSE	371.900848	0
172	MEAN	5.672659	0.3749806
173	SQUARED_DIFFERENCE	37128.26265	0.1978192
174	MEAN	2799.73527	0.2974154
175	ADD	2799.73629	7.520764E-05
176	RSQRT	0.991143	inf	Yes
177	MUL	0.991143	0
178	MUL	14.455902	2.44185	Yes
179	MUL	0.352157	14.605	Yes
180	SUB	0.352157	2.75764E-06
181	ADD	14.478933	0.3361114
182	TRANSPOSE	14.478933	0
183	RESHAPE	14.478933	0
184	MUL	16.685117	0.2978353
185	ADD	16.703711	0.2075399
186	FILL	1	0
187	MUL	0.424291	0.2876463
188	ADD	16.933183	0.2892929
189	FILL	1	0
190	FILL	2E-06	0
191	RESHAPE	16.933183	0
192	TRANSPOSE	16.933183	0
193	MEAN	0.062335	48.13572	Yes
194	SQUARED_DIFFERENCE	85.610105	0.2268296
195	MEAN	1.226184	381.2859	Yes
196	ADD	1.227184	0.01558558
197	RSQRT	1.094414	0.3163588
198	MUL	1.094414	0
199	MUL	15.961633	0.2128369
200	MUL	0.06705	0.2904581
201	SUB	0.06705	2.78254E-06
202	ADD	15.949823	0.2522852
203	TRANSPOSE	15.949823	0
204	RESHAPE	15.949823	0
205	MUL	13.885122	0.2597474
206	ADD	13.780075	0.3595855
207	MEAN	4.198399	0.3065109
208	FULLY_CONNECTED	2.434557	0.3960356
209	FULLY_CONNECTED	0.733557	9.714286	Yes

Model Memory Footprint

Identical to the cnn case, I used the TFLite Model Benchmark Tool to get the approximate RAM consumption of the TFLite microwave model at runtime which is shown in the table below for each relevant quantization mode as well as the TFLite model disk space. The other transformer models show similar characteristics. Note that the Keras model consumes about 6.02 (MB) on disk. You can see that there is about a three times reduction in model size due to the weights being quantized from float32 to int8 which is less than the four times reduction seen in the cnn case likely because there are fewer layers with weights. You can also see that the x86 TFLite runtime is more memory efficient than its aarch64 counterpart for this model.

Quant Mode	Disk (MB)	aarch64 RAM (MB)	x86 RAM (MB)
convert_only	6.217276	20.8242	16.8438
w8	2.088312	16.4258	13.3789

NILM Prototype System Components

I built a NILM prototype at my home to test the energy disaggregation algorithms in real-world conditions and to understand where they can be improved. The prototype is comprised of the following main subsystems. You can see a photograph of the prototype in the Appendix.

Analog Signal Conditioning

I used two clip-on current transformers in one of the home’s sub-panels to sense the current flowing through each of split voltage phases and a voltage transformer plugged into an outlet near the circuit breaker panel that provides the voltage of one of the phases. These signals are level-shifted, amplified and low-passed filtered by this subsystem before being passed on to the analog-to-digital converters inside an Arduino MEGA 2560 that performs aggregate metrics computation. You can see a schematic for the Analog Signal Conditioning Subsystem in the Appendix and find more details in the Panel to Arduino section.

Aggregate Metrics Computation

I used an Arduino MEGA 2560 to host the signal processing algorithms that takes the voltage signals from the Analog Signal Conditioning Subsystem and generates aggregate RMS voltage, RMS current, Real Power and Apparent Power metrics in real-time. Presently, only Real Power is used in downstream processing. I leveraged emonLibCM¹⁰ for these signal processing algorithms. emonLibCM runs continuously in the background and digitizes the analog input channels of the Arduino, calculates these metrics and then informs the Arduino sketch that the measurements are available and should be read and processed by downstream processing. The sketch is configured to update the metrics every eight seconds and can be found in ard.ino

Disaggregated Energy Consumption Computation

The actual energy disaggregation computations are hosted on a Raspberry Pi 4 which is connected over USB to the Arduino to fetch the aggregate metrics. The computations are comprised of running the tflite appliance inference models, trained and quantized per the steps described above, with pre- and post-processing steps. See the infer.py module for the code that performs these computations. The inference models output predicted energy for each appliance from 599-sample sized windows of the aggregate real power input signal. These predictions and aggregate mains power are stored in a local CSV file and made available for downstream reporting and analysis.

Typical disaggregated energy prediction results from my home are shown in the plots and tables below using tflite models trained on another machine from the dataset as described above and quantized using mode 'w8'. Two sets of results are shown, the first is with no fine-tuning and second, with fine-tuning using local data via the program fine_tune.

In the plots below, the horizontal axis is datetime. The vertical axis is energy consumption in Watts. The first plot shows the aggregate mains power with each appliance in separate sub-plots. Ground truth power was obtained by logging appliance power data in real-time at the outlet. The second plat shows aggregate mains power and appliance power plotted together and the last plot is a zoomed-in area of the second plot. These plots and the metrics that follow are generated by the program predict.

NILM metrics were computed for the Float32 predictions on aggregate mains data that was captured in real-time and for the quantized model real-time predictions, both vs ground truth.

The Float32 model trained on the default dataset predictions vs ground truth are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)
kettle	0.0	NaN	0.9918	1.659	INF	0.9999	INF
microwave	0.5319	0.5406	0.9961	4.261	0.0302	0.8198	-3.018
fridge	0.6875	0.4672	0.7414	36.59	0.2458	0.7494	-24.56
dishwasher	0.2372	0.2165	0.9596	12.74	0.9586	0.9615	-95.62
washingmachine	0.6661	0.6696	0.9749	7.596	0.2213	1.407	-23.97

The corresponding quantized model real-time predictions vs ground truth are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)
kettle	0.0	NaN	0.9986	2.873	INF	1.594	INF
microwave	0.5258	0.5353	0.9961	4.237	0.0188	0.8208	-1.880
fridge	0.6257	0.4002	0.7125	39.52	0.3668	0.8189	-36.65
dishwasher	0.2571	0.2354	0.9469	14.70	0.9557	0.9558	-95.57
washingmachine	0.6760	0.6827	0.9758	7.560	0.2401	1.393	-24.01

You can see that model performance is comparable between the Float32 and real-time quantized results but overall they are poor with microwave being the exception. This indicates that the models can benefit from fine tuning with the ground truth data.

The Float32 model fine-tuned with local data predictions vs ground truth are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)
kettle	0.6205	0.6672	0.9958	7.808	0.5716	0.5555	-57.16
microwave	0.4989	0.5129	0.9912	10.508	0.6543	0.6795	-65.43
fridge	0.8286	0.6665	0.8312	24.705	0.2502	0.3717	-25.02
dishwasher	0.7607	0.7491	0.9544	21.911	0.4602	0.3957	-46.02
washingmachine	0.9155	0.9138	0.9905	3.288	0.1156	0.3535	-11.56

The corresponding quantized model real-time predictions vs ground truth are shown in the table below.

Appliance	$F1\uparrow$	$MCC\uparrow$	$ACC\uparrow$	$MAE$ $(W)$ $\downarrow$	$SAE\downarrow$	$NDE\downarrow$	$EpD_e$ ($%$)
kettle	0.5845	0.6345	0.9955	8.199	0.5918	0.5954	-59.18
microwave	0.5221	0.5196	0.9904	10.722	0.5781	0.6574	-57.81
fridge	0.8351	0.6758	0.8367	26.137	0.2477	0.3603	-24.27
dishwasher	0.6188	0.6230	0.9083	23.506	0.2313	0.3414	-23.13
washingmachine	0.7103	0.7257	0.9585	3.910	0.2454	0.3409	24.54

You can see that fine-tuning the model with local data greatly improves its performance.

Amazon Alexa

I plan to use Amazon Alexa as the user interface to the appliance energy data, however this work is not yet started.

Conclusion

By using large publicly available datasets to train seq2point learning models it is very feasible to perform energy disaggregation that is fairly accurate without fine-tuning the models with local data. These models are modest in size and with little loss in accuracy can be quantized to run efficiently on commodity edge hardware such as the Raspberry Pi 4. More work needs to be done to further improve the accuracy of the models and to test with more appliance types. This project demonstrates that a key component of a sustainable and scalable power grid is within reach of the mass consumer market.

References

1. Sustainable Cities and Society 62 (2020) 102411 | Energy management using non-intrusive load monitoring techniques — State- of-the-art and future research directions by R. Gopinath, Mukesh Kumar, C. Prakash Chandra Joshua and Kota Srinivas.
2. Wikipedia | Signal Separation.
3. arXiv:1507.06594 | Neural NILM: Deep Neural Networks Applied to Energy Disaggregation by Jack Kelly and William Knottenbelt.
4. arXiv:1612.09106 | Sequence-to-point learning with neural networks for non-intrusive load monitoring by Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang, Nigel Goddard and Charles Sutton.
5. arXiv:1902.08835 | Transfer Learning for Non-Intrusive Load Monitoring by Michele D'Incecco, Stefano Squartini and Mingjun Zhong.
6. Artificial Intelligence Techniques for a Scalable Energy Transition pp 109–131 | A Review on Non-intrusive Load Monitoring Approaches Based on Machine Learning by Hajer Salem, Moamar Sayed-Mouchaweh and Moncef Tagina.
7. 1st International Symposium on Signal Processing Applications in Smart Buildings at 3rd IEEE Global Conference on Signal & Information Processing | Dataport and NILMTK: A Building Data Set Designed for Non-intrusive Load Monitoring by Oliver Parson, Grant Fisher, April Hersey, Nipun Batra, Jack Kelly, Amarjeet Singh, William Knottenbelt and Alex Rogers.
8. Proceedings of the 8th International Conference on Energy Efficiency in Domestic Appliances and Lighting | A data management platform for personalised real-time energy feedback by David Murray and Jing Liao and Lina Stankovic and Vladimir Stankovic and Richard Hauxwell-Baldwin and Charlie Wilson and Michael Coleman and Tom Kane and Steven Firth. The REFIT dataset used in this project is is licensed under the Creative Commons Attribution 4.0 International Public License.
9. GitHub | NILM by Lindo St. Angel
10. GitHub | EmonLibCM by Trystan Lea, Glyn Hudson, Brian Orpin and Ivan Kravets.
11. BERT4NILM: A Bidirectional Transformer Model for Non-Intrusive Load Monitoring by Zhenrui Yue, et. al.
12. ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring by Stavros Sykiotis, Maria Kaselimi ,Anastasios Doulamis and Nikolaos Doulamis.

Please also see this project's companion Medium article Energy Management Using Real-Time Non-Intrusive Load Monitoring.

Appendix

Photograph of Prototype

A photograph of an early version of my prototype system is shown below.

Analog Signal Conditioning Schematic

The schematic for the Analog Signal Conditioning circuitry is shown below.