ArduinoJson 7 Compatibility

[mathieucarbou]

ArduinoJson 7 now being released, a lot of projets will want to switch to it, but the move should start first with libraries.

I have opened a PR at esphome/ESPAsyncWebServer : esphome/ESPAsyncWebServer#25

And currently working on implementing that in my forks for ESP-DASH.

Note that this is just a reminder issue, since nothing can be merged until the libraries officially support ArduinoJson 7...

[mathieucarbou]

FYI - I was working also on a way to stream the websocket messages in packets in order to avoid building a big packet which crashes the ESP. It requires some UI adjustment in the websocket handler, but I have a working solution for both ArduinoJson 6 and 7. See bblanchon/ArduinoJson#2033 for more info.

[mathieucarbou]

Update: changes are done in my forks, and work fine with both ArduinoJson 6 and 7.

I did the changes as part of the refactoring to improve websocket batching to reduce memory usage, which involves a small UI change so I didn't isolated it.

[ayushsharma82]

I'll be merging it soon. I recently got a lot of reports that ESP-DASH is not showing anything on the webpage - this is the result of using ArduinoJSON v7.0.1 currently with ESP-DASH.

In the meantime, for those who are affected - please switch to ArduinoJSON v6.21.5

[mathieucarbou]

Please note that I got a lot of issues on dual core esp because Arduino Json heap allocation works differently and is more prone to crash on code mistake.

In my PR (work in progress - I need to update it) I have switched to the ESP webserver fork from younodebox which fixes a lot of concurrency issues and using a shared pointer for the websocket buffer. Also it avoids a memory copy so we gain double the heap size. The api begin a little different, I've put a define to support both this fork and original fork.

I also had to change the esp dash code to remove the Json doc clear() call : there is a bug I reported to the Arduino Json 7 project where a document cannot be reused after a clear().

Also, since arduino Json 7 does not allow to cap the allocation, there is no memory / overflow check so I had to redo the implementation to send batches of websocket events. Otherwise a memory crash will happen for big dashboards.

I am testing with a dashboard of over 200 cards and stats updated each second.

There is a concurrency issue also in esp dash code: the layout refresh is done on the websocket handler, but the updates or the components (and update calls) can happen at the same time from the loop. So esp dash as missing something to sync. Putting a muted causes too much slowdown for high refresh rate so I used a volatile boolean.

This is also in my pr which is in draft.

Overall, I have removed the layout refresh calls from the add remove cards - basically any call which can trigger a concurrent layout refresh, to let the user call the only one update method when layout changes are finished and of there is no async layout refresh.

[mathieucarbou]

Please see: #195 ;-)

[ayushsharma82]

Fixed with v4.4.0 of Pro version and v4.0.4 of Lite.