Sweden/Luxembourg smart meter telegram integration #1394

Bokotimon · 2021-05-24T08:38:12Z

Allow power to support decimals
Update MQTT
Update API

Hello, I have got a problem. No values show up in DSMR Reader but I got an error message in Timescale log. Do not know what to do?

2021-05-21 14:23:41.245 UTC [756] ERROR: null value in column “electricity_delivered_1” violates not-null constraint
2021-05-21 14:23:41.245 UTC [756] DETAIL: Failing row contains (435, 2021-05-21 13:23:40+00, null, null, 0.000, 0.000, 0.229, 0.000, null, null, f, 0.017, 0.025, 0.186, 0.000, 0.000, 0.000, 232.9, 230.4, 232.8, 0, 0, 0).
2021-05-21 14:23:41.245 UTC [756] STATEMENT: INSERT INTO “dsmr_datalogger_dsmrreading” (“processed”, “timestamp”, “electricity_delivered_1”, “electricity_returned_1”, “electricity_delivered_2”, “electricity_returned_2”, “electricity_currently_delivered”, “electricity_currently_returned”, “phase_currently_delivered_l1”, “phase_currently_delivered_l2”, “phase_currently_delivered_l3”, “extra_device_timestamp”, “extra_device_delivered”, “phase_currently_returned_l1”, “phase_currently_returned_l2”, “phase_currently_returned_l3”, “phase_voltage_l1”, “phase_voltage_l2”, “phase_voltage_l3”, “phase_power_current_l1”, “phase_power_current_l2”, “phase_power_current_l3”) VALUES (false, ‘2021-05-21T13:23:40+00:00’::timestamptz, NULL, NULL, ‘0.000’, ‘0.000’, ‘0.229’, ‘0.000’, ‘0.017’, ‘0.025’, ‘0.186’, NULL, NULL, ‘0.000’, ‘0.000’, ‘0.000’, ‘232.9’, ‘230.4’, ‘232.8’, 0, 0, 0) RETURNING “dsmr_datalogger_dsmrreading”.“id”

The text was updated successfully, but these errors were encountered:

Bokotimon · 2021-05-24T08:47:34Z

When using putty I get the information from the meter.

0-0:1.0.0(210524094210W)
1-0:1.8.0(00000424.413*kWh)
1-0:2.8.0(00000000.000*kWh)
1-0:3.8.0(00000002.116*kVArh)
1-0:4.8.0(00000170.745*kVArh)
1-0:1.7.0(0000.182*kW)
1-0:2.7.0(0000.000*kW)
1-0:3.7.0(0000.000*kVAr)
1-0:4.7.0(0000.069*kVAr)
1-0:21.7.0(0000.053*kW)
1-0:22.7.0(0000.000*kW)
1-0:41.7.0(0000.107*kW)
1-0:42.7.0(0000.000*kW)
1-0:61.7.0(0000.020*kW)
1-0:62.7.0(0000.000*kW)
1-0:23.7.0(0000.000*kVAr)
1-0:24.7.0(0000.016*kVAr)
1-0:43.7.0(0000.000*kVAr)
1-0:44.7.0(0000.023*kVAr)
1-0:63.7.0(0000.000*kVAr)
1-0:64.7.0(0000.029*kVAr)
1-0:32.7.0(230.7*V)
1-0:52.7.0(227.8*V)
1-0:72.7.0(232.3*V)
1-0:31.7.0(000.4*A)
1-0:51.7.0(000.5*A)
1-0:71.7.0(000.1*A)
!7D16
/KFM5KAIFA-METER

dennissiemensma · 2021-05-24T12:18:14Z

Hi, I'll need some more context to help you. It seems your telegram may differ a bit from what DSMR-reader supports. Are you Dutch and using a Dutch smart meter? And what meter are you reading? Did you use the default datalogger and settings?

Bokotimon · 2021-05-24T12:58:04Z

I use a Kaifa MA304H4E smart meter, what is different from what DSMR-reader supports? The meter is supposed to suport SSMR5_Annex 02-5 Dutch Smart Meter Requirements v5.0.2 Final P1. If not I need to talk to the vendor. :)

dennissiemensma · 2021-05-24T13:30:03Z

It depends on where you're living (#769, #750).

Based on the 1-0:1.8.0(... and 1-0:2.8.0(... in your telegram, you may want to try switching the DSMR-version in /dsmr_datalogger/dataloggersettings/1/change/ to Luxembourgh (#769) and see if that resolves your problem.

Bokotimon · 2021-05-24T13:46:51Z

I live in Sweden. I will give it a try. How do I change it running Home assistant?

dennissiemensma · 2021-05-24T13:59:41Z

Great, let me know if it resolves the problem.

I'm not sure how to change it in HA. I presume you have access to the interface of DSMR-reader somewhere, and then you'll need to append /dsmr_datalogger/dataloggersettings/1/change/ to the URL, which should prompt you to login as admin and take you to the datalogger settings.

Bokotimon · 2021-05-24T14:29:07Z

Tadaaa.. It worked. Thanks a lot!

dennissiemensma · 2021-05-24T14:53:43Z

Great to hear, thanks for testing!

Bokotimon · 2021-06-11T08:29:29Z

Hello.
I know now why I got this problem in the first place and it is because we don't use tariffs in Sweden. I see the standard DMSR has that in the specified output. Is it possible to make a setting for Sweden in DSMR reader? All 5.5 million electric meters in Sweden will be changed by 1 jan 2025. About 3-4 million will have the P1 port and I already get questions when out grid company can open up the port to our customers. It would be nice if it worked from the start. With Luxenburg I am missing some of the values like current per phase.

dennissiemensma · 2021-06-11T22:04:02Z

I can give it a shot, but I'm not sure whether it would map correctly to the application. It would still require DSMR-reader to fake using either tariffs and users would need to manually select "merge tariffs" in the GUI settings. But I guess it might work.

Would you be able to provide a full telegram? I'm not sure whether the one provided above lacks any lines before the first one containing the date. Lines such as DSMR version and smart meter serial number. If there is a serial number, you may change it to a random number.

Bokotimon · 2021-06-14T15:01:55Z

This is the P1 output from our temporary Kaifa MA304H4E meter:

0-0:1.0.0(210524094210W)
1-0:1.8.0(00000424.413*kWh)
1-0:2.8.0(00000000.000*kWh)
1-0:3.8.0(00000002.116*kVArh)
1-0:4.8.0(00000170.745*kVArh)
1-0:1.7.0(0000.182*kW)
1-0:2.7.0(0000.000*kW)
1-0:3.7.0(0000.000*kVAr)
1-0:4.7.0(0000.069*kVAr)
1-0:21.7.0(0000.053*kW)
1-0:22.7.0(0000.000*kW)
1-0:41.7.0(0000.107*kW)
1-0:42.7.0(0000.000*kW)
1-0:61.7.0(0000.020*kW)
1-0:62.7.0(0000.000*kW)
1-0:23.7.0(0000.000*kVAr)
1-0:24.7.0(0000.016*kVAr)
1-0:43.7.0(0000.000*kVAr)
1-0:44.7.0(0000.023*kVAr)
1-0:63.7.0(0000.000*kVAr)
1-0:64.7.0(0000.029*kVAr)
1-0:32.7.0(230.7*V)
1-0:52.7.0(227.8*V)
1-0:72.7.0(232.3*V)
1-0:31.7.0(000.4*A)
1-0:51.7.0(000.5*A)
1-0:71.7.0(000.1*A)
!7D16
/KFM5KAIFA-METER

but we will get 275 000 Kamstrup OMNIA eMeter when the meter is ready for production. Don't have any meter with an aktiv port to be able to read it yet. I will try to get an output readout from other grid companies in Sweden. I would be necessary to compare different meters to know exactly how they work. I will get back to you on that. Thanks!

dennissiemensma · 2021-06-15T19:02:15Z

Thanks for the update! I'll have a look some time later as DSMR-reader might be able to use these fields and hardcode it to a fixed tariff or so:

0-0:1.0.0(210524094210W)
1-0:1.8.0(00000424.413*kWh)
1-0:2.8.0(00000000.000*kWh)
...
1-0:1.7.0(0000.182*kW)
1-0:2.7.0(0000.000*kW)
...
1-0:21.7.0(0000.053*kW)
1-0:22.7.0(0000.000*kW)
1-0:41.7.0(0000.107*kW)
1-0:42.7.0(0000.000*kW)
1-0:61.7.0(0000.020*kW)
1-0:62.7.0(0000.000*kW)
...
1-0:32.7.0(230.7*V)
1-0:52.7.0(227.8*V)
1-0:72.7.0(232.3*V)
1-0:31.7.0(000.4*A)
1-0:51.7.0(000.5*A)
1-0:71.7.0(000.1*A)

I'm not sure if it will break anything, but I guess it's worth trying anyway.

dennissiemensma · 2021-09-19T18:58:28Z

@Bokotimon do you happen to have your definitive smart meter by now? And any luck getting the P1 telegram associated with it?

dennissiemensma · 2021-09-22T18:48:00Z

Possible help with Smart meters in Sweden in the reader-only-package of Nigel: ndokter/dsmr_parser#86

Bokotimon · 2021-09-23T12:18:28Z

I have started to translate the Swedish specification, that is not needed then. :)

dennissiemensma · 2021-09-23T17:17:29Z

@Bokotimon do you happen to have a final dump of a (production) telegram I can use for integration? Or did it not change?

#1394 (comment)

dennissiemensma · 2021-09-23T19:26:09Z

@Bokotimon is the telegram you provided a real one? It seems to be either invalid or ambiguous regarding DST/timezones. I'm writing a test and I'll need the exact timezone it should match to verify parsing works.

The date:

210524094210W

Is marked as W (winter time), but according to this it seems that Sweden is similar to the Netherlands regarding DST: https://visitsweden.com/about-sweden/time-and-daylight-hours/
Meaning that the given month 2105 (May 2021) should be in summer time, ending the telegram timestamp with an S instead.

It's really important to know whether this is correct or a side effect or simply different in Sweden, as it may or may not cause hourly differences twice a year or even offset the time data wrong. Any information you can provide?

dennissiemensma · 2021-09-23T19:52:12Z

@Bokotimon another question, the voltages seems to be fine, but the power/current per phase seems to differ from the Dutch DSMR as well. Again, is the telegram valid or do you have a production one I can use instead?

I've mapped them internally to integers, which may explain why you did not get any data regarding current.

Your telegram:

1-0:31.7.0(000.4*A)
1-0:51.7.0(000.5*A)
1-0:71.7.0(000.1*A)

My telegram:

1-0:31.7.0(000*A)
1-0:51.7.0(001*A)
1-0:71.7.0(000*A)

I'm not a 100% sure, but it seems that the Dutch DSMR does not allow decimals for Amps:

Definition volts / amps

It maps to tag 18 with F3(0,0):

Tag 18

`Fn(x,x)`

Which seems right, as the other Voltage field indeed does specify 1 decimal and its spec is F3(1,1).

dennissiemensma · 2021-09-23T19:55:18Z

Also, there do not seem to be (breaking) differences with the Luxembourgh config, as in how DSMR-reader allows fields to be omitted or not. So it should be fine to use anyway.

If you have a final production telegram I'll be able to check again for you.

dennissiemensma · 2021-11-13T10:11:55Z

Thanks for the sample. Would you be able to have a plain text dump here so I can copy, paste and dry run it? Also, the order of the data (body) does not affect most parsers, since each line is prependend with it unique field identifier. As long as the CRC checksum matches the same order of course.

Bokotimon · 2021-11-15T09:13:32Z

Here you go @dennissiemensma, here is a sample:

/KAM5
0-0:1.0.0(211115100940W)
1-0:1.8.0(00000003.997*kWh)
1-0:2.8.0(00000000.000*kWh)
1-0:3.8.0(00000000.000*kVArh)
1-0:4.8.0(00000000.000*kVArh)
1-0:1.7.0(0000.000*kW)
1-0:2.7.0(0000.000*kW)
1-0:3.7.0(0000.000*kVAr)
1-0:4.7.0(0000.000*kVAr)
1-0:21.7.0(0000.000*kW)
1-0:41.7.0(0000.000*kW)
1-0:61.7.0(0000.000*kW)
1-0:22.7.0(0000.000*kW)
1-0:42.7.0(0000.000*kW)
1-0:62.7.0(0000.000*kW)
1-0:23.7.0(0000.000*kVAr)
1-0:43.7.0(0000.000*kVAr)
1-0:63.7.0(0000.000*kVAr)
1-0:24.7.0(0000.000*kVAr)
1-0:44.7.0(0000.000*kVAr)
1-0:64.7.0(0000.000*kVAr)
1-0:32.7.0(238.0*V)
1-0:52.7.0(238.0*V)
1-0:72.7.0(240.7*V)
1-0:31.7.0(000.0*A)
1-0:51.7.0(000.0*A)
1-0:71.7.0(000.0*A)
!5FCE

dennissiemensma · 2021-11-16T20:33:31Z

@Bokotimon I've inserted your telegram into f5f7bf0 and it seems to parse. However, it does contain a lot of zero-values, not really allowing me to assert parsed values.

Also, this part still does not match Dutch specs, see #1394 (comment):

1-0:31.7.0(000.0*A)
1-0:51.7.0(000.0*A)
1-0:71.7.0(000.0*A)

The current state of the test mentioned above may pass because of the zero-values, causing Python to internally omit the decimals. So it's still of concern and I cannot verify whether it will fail unless you have a real telegram with a value in either of the fields above. For now I still presume it mismatches.

Bokotimon · 2022-01-20T13:08:23Z

@dennissiemensma At last, here are some readings from a Kamstrup OMNIA eMeter. Can't wait to see it in action in homeassistant. Tried a dongel that shows 1 sek values in my cellphone. Looks really nice.

han-output.txt

dennissiemensma · 2022-01-20T21:36:01Z

Great, I will give it a try later. My dev environment is in between versions at the moment.

Telegram for me te try:

/KAM5

0-0:1.0.0(220120135735W)
1-0:1.8.0(00000184.554*kWh)
1-0:2.8.0(00000000.000*kWh)
1-0:3.8.0(00000001.082*kVArh)
1-0:4.8.0(00000091.984*kVArh)
1-0:1.7.0(0004.795*kW)
1-0:2.7.0(0000.000*kW)
1-0:3.7.0(0000.000*kVAr)
1-0:4.7.0(0000.079*kVAr)
1-0:21.7.0(0002.619*kW)
1-0:41.7.0(0000.073*kW)
1-0:61.7.0(0002.103*kW)
1-0:22.7.0(0000.000*kW)
1-0:42.7.0(0000.000*kW)
1-0:62.7.0(0000.000*kW)
1-0:23.7.0(0000.000*kVAr)
1-0:43.7.0(0000.000*kVAr)
1-0:63.7.0(0000.000*kVAr)
1-0:24.7.0(0000.010*kVAr)
1-0:44.7.0(0000.028*kVAr)
1-0:64.7.0(0000.041*kVAr)
1-0:32.7.0(228.2*V)
1-0:52.7.0(230.2*V)
1-0:72.7.0(228.6*V)
1-0:31.7.0(011.7*A)
1-0:51.7.0(000.3*A)
1-0:71.7.0(009.2*A)
!232C

I see that these fields are still decimals though, so not sure what to do:

1-0:31.7.0(011.7*A)
1-0:51.7.0(000.3*A)
1-0:71.7.0(009.2*A)

It probably requires some kind of parsing hack, but we'll see later.

Bokotimon · 2022-02-25T14:00:03Z

Hey, I have a working Luxemburg setup now but its kind of slow. The meter is pushing values every second but it does not update the live values every second. Do you know if its possible too speed up the gui?

dennissiemensma · 2022-02-25T17:25:35Z

Readings are grouped by minute by default. You may try disabling it in the admin interface in Consumption, if I remember correctly. Also, you may try to decrease the backend process sleep or datalogger sleep configs in the admin as well.

dennissiemensma · 2022-02-28T19:12:28Z

@Bokotimon did changing the settings above fix it?

Bokotimon · 2022-03-01T20:28:01Z

Thanks for asking. I checked it and I already set logger to use the same resolution as the output from the meter. The backend process I had set to the lowest number 0.5. I will look into it again.

Did you figure out the decimal problem from the Swedish HAN-output?

dennissiemensma · 2022-03-01T20:49:04Z

I have not been able to look into it. Does the Luxembourg setup at least provide some data?

Bokotimon · 2022-03-03T11:24:18Z

Yes, I get power, power per phase and voltage. See attached photo

.
But in the setting all values are displayed.

Right now there are a couple of 100 000 meters with the Swedish HAN but 1 jan 2025 there will be about 5,5 million meters with HAN. It seems though that the refresh rate will be different, from 1 to 10 sek. Our 275 000 meters in Gothenburg area will be changed in 2023-2024.

dennissiemensma · 2022-03-03T18:35:47Z

It seems that the Luxembourgh meters have the same issue:

#1533

I will merge the issues and keep the current one, since it was created first. I think fixing the Luxembourgh parsing will fix it for your meters as well.

Bokotimon · 2022-03-30T09:04:38Z

I have looked into the GUI update issue with the best programmer I know at work Joris but he think there might be a data process limitation in the GUI. Do you think that it is possible to solve the update issue? The DSMR standard 5.0.2 says 1 s update and it would be nice if the GUI have the same refresh rate.

dennissiemensma · 2022-03-30T15:39:54Z

I need some more context regarding any issues you are facing. Which part of the GUI do you mean?

If you're having issues with the current consumption header values, they should update every second:
https://github.com/dsmrreader/dsmr-reader/blob/v5/dsmr_frontend/templates/dsmr_frontend/dashboard.html#L63

If you're having issues with the graphs, they currenctly update every 5 seconds:
https://github.com/dsmrreader/dsmr-reader/blob/v5/dsmr_frontend/templates/dsmr_frontend/live-graphs.html#L197

I could decrease the latter a bit, but I will have to check whether it does not affect performance too much. Or make it a config value. You could alter it locally temporarily in the given file(s), for now. Note that every graph has its own value.

dennissiemensma · 2022-03-31T19:59:02Z

It seems to affect performance too much, but I've added a new setting for it in the next release. Use at your own risk.

Bokotimon · 2022-04-01T07:49:27Z

It was the live data view in the GUI I was referring to. Using an other dongle with live data graph using 1 sek resolution give instant feedback when turning electrical equipment on and off. Even if it just a small led-bulb shows a difference in the graph. I really like graphs, it is so much easier to understand than looking at the numbers.

dennissiemensma · 2022-06-08T20:01:29Z

I've recreated a fresh issue #1641 to clear all the noise of this topic.
Even though I'm not sure it's doable at all, without affecting existing user/connections not expecting decimals.

Bokotimon added the bug label May 24, 2021

dennissiemensma assigned Bokotimon May 24, 2021

Bokotimon closed this as completed May 24, 2021

dennissiemensma removed the bug label May 24, 2021

dennissiemensma added this to the Other milestone May 24, 2021

dennissiemensma changed the title ~~[756] ERROR: null value in column “electricity_delivered_1” violates not-null constraint~~ Sweden smart meter: [756] ERROR: null value in column “electricity_delivered_1” violates not-null constraint May 24, 2021

dennissiemensma reopened this Jun 11, 2021

dennissiemensma removed this from the Other milestone Jun 11, 2021

dennissiemensma added this to the 5.2 milestone Sep 10, 2021

dennissiemensma modified the milestones: 5.2, Some future release Sep 23, 2021

dennissiemensma added a commit that referenced this issue Nov 16, 2021

Updated Swedish meter test #1394

f5f7bf0

dennissiemensma assigned dennissiemensma and unassigned Bokotimon Jan 20, 2022

dennissiemensma mentioned this issue Mar 3, 2022

💡phase_power_current = integer #1533

Closed

dennissiemensma changed the title ~~Sweden smart meter telegram integration~~ Sweden/Fluvius smart meter telegram integration Mar 3, 2022

dennissiemensma removed the review Not sure yet whether to implement this label Mar 3, 2022

dennissiemensma modified the milestones: Some future release, 6.0 Mar 3, 2022

dennissiemensma changed the title ~~Sweden/Fluvius smart meter telegram integration~~ Sweden/Luxembourg smart meter telegram integration Mar 3, 2022

dennissiemensma modified the milestones: 6.0, 7.0 May 23, 2022

dennissiemensma mentioned this issue Jun 8, 2022

Sweden/Luxembourg: Support decimals for phase_power_current_l* #1641

Open

dennissiemensma closed this as not planned Won't fix, can't repro, duplicate, stale Jun 8, 2022

Sweden/Luxembourg smart meter telegram integration #1394

Sweden/Luxembourg smart meter telegram integration #1394

Comments

Bokotimon commented May 24, 2021 • edited by dennissiemensma Loading

Bokotimon commented May 24, 2021 • edited by dennissiemensma Loading

dennissiemensma commented May 24, 2021 • edited Loading

Bokotimon commented May 24, 2021

dennissiemensma commented May 24, 2021

Bokotimon commented May 24, 2021

dennissiemensma commented May 24, 2021

Bokotimon commented May 24, 2021

dennissiemensma commented May 24, 2021

Bokotimon commented Jun 11, 2021

dennissiemensma commented Jun 11, 2021

Bokotimon commented Jun 14, 2021 • edited by dennissiemensma Loading

dennissiemensma commented Jun 15, 2021 • edited Loading

dennissiemensma commented Sep 19, 2021

dennissiemensma commented Sep 22, 2021

Bokotimon commented Sep 23, 2021

dennissiemensma commented Sep 23, 2021

dennissiemensma commented Sep 23, 2021

dennissiemensma commented Sep 23, 2021

Definition volts / amps

Tag 18

Fn(x,x)

dennissiemensma commented Sep 23, 2021

dennissiemensma commented Nov 13, 2021 via email

Bokotimon commented Nov 15, 2021 • edited by dennissiemensma Loading

dennissiemensma commented Nov 16, 2021

Bokotimon commented Jan 20, 2022

dennissiemensma commented Jan 20, 2022

Bokotimon commented Feb 25, 2022

dennissiemensma commented Feb 25, 2022 via email

dennissiemensma commented Feb 28, 2022 • edited Loading

Bokotimon commented Mar 1, 2022

dennissiemensma commented Mar 1, 2022

Bokotimon commented Mar 3, 2022

dennissiemensma commented Mar 3, 2022

#1533

Bokotimon commented Mar 30, 2022

dennissiemensma commented Mar 30, 2022

dennissiemensma commented Mar 31, 2022 • edited Loading

Bokotimon commented Apr 1, 2022

dennissiemensma commented Jun 8, 2022

Bokotimon commented May 24, 2021 •

edited by dennissiemensma

Loading

Bokotimon commented May 24, 2021 •

edited by dennissiemensma

Loading

dennissiemensma commented May 24, 2021 •

edited

Loading

Bokotimon commented Jun 14, 2021 •

edited by dennissiemensma

Loading

dennissiemensma commented Jun 15, 2021 •

edited

Loading

`Fn(x,x)`

Bokotimon commented Nov 15, 2021 •

edited by dennissiemensma

Loading

dennissiemensma commented Feb 28, 2022 •

edited

Loading

dennissiemensma commented Mar 31, 2022 •

edited

Loading