How to find the "right" property / register mapping for device XYZ

[mkaiser]

As you mentioned in #46, the elstertable.h is not very reliable.

I too am experiencing this and have a hard time to get some values like VORLAUFISTTEMP, which should be easy to spot by eye (between 25 and 35 °c with raw register values between 250 - 350). Although I tried multiple registers, I was not able to find the right one. This approach does is more poking around and not very structured :/

In some FHEM forum I found that they developed some can bus scanner, but they are linux-based.

My Question: Is there an easy way to extend your code to make some debug-range scans issued by the ESPHome dashboard?

Basic idea:

• Within HA user interface:
  • Add CanMember as dropdown menu
  • Add two input numbers: scan from and scan to
  • Add button scan CAN bus pressed
• On button click do:
  • issue a couple of read requests starting at scan_from and ending at scan_to
  • pre-process the returned value and print it in a list-style for easier comparison
    • if the value is not zero, not 0xFFFF and not 0x8000 (32768) do
      • If known in property.h print name and value
      • else: print UNKNOWN and value

What do you think? Is this reasonable, or am I missing other options (or am I just tired)? ;)

[kr0ner]

Scanning all the available IDs makes no sense. You'd now that an ID exists, but not what it represents. The easiest way is to just use the phone/tablet/laptop, go to ESPHome logging, head over to the display, navigate through the menu to the entry of interest and check the logs at the same time and try to match the value. Repeat that for all the entries you want to monitor. Adjust property.h accordingly. If new sensors have been added, also add them to the respective yaml. Try using the existing templates whenever possible

[kr0ner]

@mkaiser did you succeed getting the IDs fixed?

[mkaiser]

still fishing :)

I made some progress - most of the "easy" temperature sensors work, as they are transmitted on the CAN bus when you go through the heatpumps menu.

But I am having a hard time with the "PROGRAMMSCHALTER". It seems as if there are no CAN messages when I manually toggle the Betriebsart between the modes. In general, these modes differ from your heatpump. Mine are "Notbetrieb, Bereitschaft, Programmbetrieb, Komfort, Eco and Warmwasser" but I have not figured out the right address.

Could it be that the Programmschalter is only published on CAN after a request?

My next steps are to improve logging and just finished a small python log parser for the evaluation: I hope to see one address with only 6 different values (Programmschalter), but no luck so far:

p.id= 0x000A p.name = UNKNOWN                                    values = {44x 0x0000090B (2315)}
p.id= 0x000F p.name = UNKNOWN                                    values = {44x 0x00000000 (0)}
p.id= 0x0010 p.name = UNKNOWN                                    values = {44x 0x00000051 (81)}
p.id= 0x0029 p.name = UNKNOWN                                    values = {44x 0x00000A00 (2560)}
p.id= 0x002A p.name = UNKNOWN                                    values = {44x 0x00001E00 (7680)}
p.id= 0x0053 p.name = UNKNOWN                                    values = {44x 0x00000000 (0)}
p.id= 0x0058 p.name = UNKNOWN                                    values = {15x 0x00000100 (256), 147x 0x00000200 (512)}
p.id= 0x005A p.name = UNKNOWN                                    values = {44x 0x00000200 (512)}
p.id= 0x005E p.name = UNKNOWN                                    values = {44x 0x00000000 (0)}
p.id= 0x006E p.name = UNKNOWN                                    values = {88x 0x00000100 (256)}
p.id= 0x0073 p.name = UNKNOWN                                    values = {44x 0x00000000 (0)}
p.id= 0x00FD p.name = UNKNOWN                                    values = {20x 0x00000800 (2048), 20x 0x00000100 (256)}
p.id= 0x00FE p.name = UNKNOWN                                    values = {132x 0x00000100 (256)}
p.id= 0x019A p.name = UNKNOWN                                    values = {16x 0x0000000A (10)}
p.id= 0x01D7 p.name = UNKNOWN                                    values = {0x00000000 (0), 0x00000114 (276)}
p.id= 0x02CA p.name = UNKNOWN                                    values = {0x00000000 (0), 0x000000ED (237)}
p.id= 0x0A00 p.name = UNKNOWN                                    values = {44x 0x00000028 (40)}
p.id= 0x0A20 p.name = UNKNOWN                                    values = {0x00000100 (256), 0x0000014C (332), 0x0000004C (76), 2x 0x0000006C (108), 4x 0x00000120 (288), 8x 0x00000000 (0), 14x 0x0000016C (364), 44x 0x00000020 (32)}

btw. do you use the FEK, oder did you turn it completely off and use the "virtual" thermostat?

[kr0ner]

Could it be that the Programmschalter is only published on CAN after a request?

It could be that you have different IDs for some of the components. It might make sense to switch the log level to Debug. That should show all CAN IDs and not only the ones which we are explicitly listening to. Once you have the additional IDs, add callback handler as for the others. After that all the sensors should be pretty easy to find, one at a time using the Smartphone or the Laptop setting the Programmschalter to some value and you should see the change ... note down the id, then continue with the next sensor. I wouldn't try to click through many menus and later try to figure out which ID belonged to what. Tried that ... it does not really work ^^

btw. do you use the FEK, oder did you turn it completely off and use the "virtual" thermostat?

I never owned one :D I have a tablet running HA so I also have no need for that. Also HA controls are way nicer and it is anyways better to publish the avg. temperature of the whole house instead of just one room ;)

[mkaiser]

Yeah, I just pimped the logging and used my Phone to compare values ;)

Just found Out that my extended logging stuff was wrong:

:dec in the Code does not mean decimal (and unaltered) but is divided by 10 ( for becoming a float Sensor value)
Then many values got truncated without me noticing..... Will check next Weekend and hopefully will find the Betriebszustand then

I tried to add a "raw" value to the property value for more logging / debugging. But hell you are using some templating ;) need more time to get this into my head

[kr0ner]

I still don't get why you need to change sth. in order to find the new values 😅

This should be everything you need to know. Instead of RUECKLAUFISTTEMP there will be UNKOWN followed by the hex number that has to be added to property.h and the raw value. Or I'm just failing to understand what you want to achieve :D
Nevertheless if you need help ... reach out

@mkaiser : maybe that one helps: cc50f91

You can now directly copy the CAN id

[mkaiser]

I still don't get why you need to change sth. in order to find the new values 😅

yeah, its not easy with me - I guess it is more that I want to fully understand the code first :)

The "Betriebszustand" is not working out of the box with the TTF07. After changing the mode in the Heatpumps display, I could not figure which values changed. So I started adding debug messages in hexadecimal (to identify single bits in one-hot coded parameters) and stumbled over the Property templates, which kind of truncate / convert the incoming uint16_t parameters to float /10, when the type is not a string. After adding some "toString"-functions and some "hacky print function mixing snprintf and sstream" in communication.h my output now looks like this:

(my intention was that the Property always carries its raw value next to possible float values. Does not work at the moment - will take care of that later, but wanted to answer you first ;) )

This should be everything you need to know. Instead of RUECKLAUFISTTEMP there will be UNKOWN followed by the hex number that has to be added to property.h and the raw value. Or I'm just failing to understand what you want to achieve :D Nevertheless if you need help ... reach out

Yes exactly! This way I found some more different values of the heatpump models using my phone :)

@mkaiser : maybe that one helps: cc50f91

You can now directly copy the CAN id

hmm something is odd, see blue highlighted rows in the screenshot: In communication.h the CAN ID is reported as COM (0x100), in the yaml can: section is is reported as KSL(0x180)

[mkaiser]

Minor update: getting there, found many sensors. Now there are only ~ 20 different "UNKNOWN" sensors left, reducing the amount of possible PROGRAMMSCHALTER and Betriebsszustand register addresses.

Example of my current log output (known / tested values are skipped)

Findings so far: There are more things different from your "common.yaml" than expected. I think the common.yaml cannot be used for me (I wrote a new one from scratch based on yours).

Basically my issue here is answered. When I figured out the remaining stuff, I am glad to contribute my working configuration.

Do you want to close this issue, or shall I keep you updated?

Edit: Code is here (but far away from being mergable...): https://github.com/mkaiser/OneESP32ToRuleThemAll/tree/dbg_msg

[kr0ner]

Good job :)

There are more things different from your "common.yaml" than expected. I think the common.yaml cannot be used for me (I wrote a new one from scratch based on yours).

Not really a problem ... we keep the common part there and move the heatpump specific stuff to wp_bsae.yaml
That was expected 😉
Just keep it updated and I will try to include your changes

[kr0ner]

@mkaiser I cherry-picked some of your changes and tried to fit it in the current structure.

Overall I still don't get why you add all these log statements e.g. raw_value to simple_variant. Why not just print it before it get's converted?

static const CanMember FEK{
0x401,
"FEK"}; // was removed here - why? master...dynamic_ids

because it was never used ... the read/write IDs were the ones from HK1

You are deviating from the original code very often instead of trying to fit it in the current structure. Most of the commits are deleting stuff and adding it to a different file. Like that it will be hard to maintain a single codebase 😉

[mkaiser]

Overall I still don't get why you add all these log statements e.g. raw_value to simple_variant. Why not just print it before it get's converted?

Started this before I (almost) understood your code :) Will revert that in the future. But the main problem was, that after converting to the variant, some information got truncated and mostly the /10 value was stored... Now I know, that I can avoid this by using "et_default" in the propery definitions while testing

You are deviating from the original code very often instead of trying to fit it in the current structure. Most of the commits are deleting stuff and adding it to a different file. Like that it will be hard to maintain a single codebase 😉

When everything is working, I will rebase and revert all the testing stuff :)

[kr0ner]

sounds like a plan :) If you like, you can leave small comments on #53 so that I can add the gradually

[mkaiser]

sure thing.

Side question: Are you using a formatter for yaml files?

I accidentally invoked mine and the single-line descriptions were gone... Is there any way to preserve them and also having a defined file formatting?

[kr0ner]

Side question: Are you using a formatter for yaml files?

I disabled it for yaml files in the vscode settings. The formatting was weird

[mkaiser]

Update: More and more is working :)
The monitoring mainly works - and I figured out how bad my configuration was set by the installer (e.g., compressor started every 20 minutes...)

Now I can set the Programmschalter and read some Betriebszustand values.

But there is still some work. Setting heating stuff is not working, yet, but that will be a matter of some free evenings ;)

Regarding the property.h: There are so many different values between your standard configuration and my TTF07. Maybe it is better to have a "common" section at the beginning and then diverge for the different heatpumps.

Just for a quick peak: I stripped everything not supported by TTF07 in this version. I am pretty sure that there will be more changes

https://github.com/mkaiser/OneESP32ToRuleThemAll/blob/dbg_msg/src/property.h

Also I skipped your common.yaml and directly integrated it into my ttf07.yaml

Common would be from my point of view: Heatpump datetime, and the Energy / COP sensors

[kr0ner]

The monitoring mainly works - and I figured out how bad my configuration was set by the installer (e.g., compressor started every 20 minutes...)

yeah ... I think that happens to almost everyone since everything is just shipped with default values :-/

Regarding the property.h: There are so many different values between your standard configuration and my TTF07. Maybe it is better to have a "common" section at the beginning and then diverge for the different heatpumps.

I know that this does not seem to be the easiest way, but I have chosen that path to make it less likely to introduce errors. You can easily create kProperty1,0x1 and kProperty2, 0x1 and it might lead to very strange behavior that I have to figure out in the end 😉. By keeping the order it would be very obvious that the same value is used twice Just take it as is and do this one time exercise :) after you found the initial set of ids, you won't have to touch that file again hopefully.

Just for a quick peak: I stripped everything not supported by TTF07 in this version. I am pretty sure that there will be more changes
https://github.com/mkaiser/OneESP32ToRuleThemAll/blob/dbg_msg/src/property.h
Also I skipped your common.yaml and directly integrated it into my ttf07.yaml

IMO this is not the way to go. When it comes to IoT it is all about maintainability .... if you just quick fix everything for your needs, you will suffer later when stuff breaks. But that is your decision in the end ;)

[kr0ner]

I'll port the stuff in the evening so that others can use the mainline directly. Thanks for figuring out the Ids so far 👍

[kr0ner]

I think I found a way to prevent ambiguous IDs during compile time 👍🏼
So I will implemet the sections in property.h to group stuff together

[mkaiser]

... You can easily create kProperty1,0x1 and kProperty2, 0x1
Yes, and also the amount of pre-defined values gave me a really hard time debugging because they kind of lead me to false assumptions that some sensors were working, but in fact they showed completely different things.

know that this does not seem to be the easiest way, but I have chosen that path to make it less likely to introduce errors. You can easily create kProperty1,0x1 and kProperty2, 0x1
nice!

please note that all the values are still under investigation ;)
Wenn i figured more stuff out, I will write more comments on the sensors.

btw. do you see a way to put the information for the "requestable using CAN target", e.g., "Kessel" or "HK1" into the property? There might be some side effects, when a value can be requested by Canmember X and is also periodically (or on change) updated by Canmember Y

[kr0ner]

btw. do you see a way to put the information for the "requestable using CAN target", e.g., "Kessel" or "HK1" into the property? There might be some side effects, when a value can be requested by Canmember X and is also periodically (or on change) updated by Canmember Y

In theory all values are requestable by design. I assume what you mean is e.g. that when you set fan speed from HA and via the display of the heatpump it results in 2 different paths and you might end up reading and publishing the value from the request, which will be 0. From the id we can find out if it is a request and drop it. What is missing is most likely the callback for a different can member