how to send big data to webserver?

[TexeCode07]

I created webserver to view the data in table format. also on esp i use to store sensor datas in json format in littleFs file.
When the CalcData file is small this sending of json data works fine and on webserver it properly fetch and shown in table format.
But when the CalcData become big the websocket stops and fail to get the data at esp webserver verified from console log.
i try sending the data with different length and found that the data length of file if less than 36kb it sends properly and bigger than that fails.
context :- When webserver connect this is how i get from little fs and send data.
events.onConnect([](AsyncEventSourceClient *client){
file = LittleFS.open("/CalcData.txt", "r");
if (!file) {
Serial.println("Failed to open /CalcData.txt file for reading");
return;
}
String fileContent;
while(file.available()){
fileContent += file.readStringUntil('\n') + '\n';
}
file.close();
Serial.println(fileContent.c_str());
events.send(fileContent.c_str(), "calculateData", millis());
}
Thanx : )

[mathieucarbou]

This is a memory management issue on your side. You need to stream the data, not pre-load and send it once.
If it is a table, an idea would be to send a WS message for each line, or group of lines for example.

[JackV2020]

Below are some parts you may use as an idea to download the json filedirect from the LittleFS into a variable in javascript in your html and then process it in javascript.

I have an html which I serve from a Wemos D1 mini pro and I use it to edit files up to 500.000 characters which are stored in LittleFS.

The page has an input field :

It has a text area to hold the data so I can edit it on screen : <textarea id="fileContent" name="fileContent">Editor</textarea>
It also has a status div :

It has some scripting and I think you can use and modify this to get your data downloaded into javascript in your html
The "_('fileContent').value=data;" below puts the received data in the text area. This is where you may want to process your json.
( I add the ?key part to make sure that I get the latest version of the file. This is a standard trick to work around caching)

// Short for document.getElementById function
function _(el) {
return document.getElementById(el);
}

function getFile() {
_('fileContent').value="Loading...";
("status").innerHTML="Loading...";
var d=new Date();
var h=d.getHours();
var m=d.getMinutes();
var s=d.getSeconds();
var ms=d.getMilliseconds();
var key="?" + h + m + s + ms;
const filename=('filename').value+key;
const url=${filename}; // Construct the URL for the file

fetch(url)
.then(response => response.text())
.then(data => {
console.log("Data length received:", data.length);
_("status").innerHTML="Loaded";
_('fileContent').value=data;
})
.catch(err => {
console.error(err);
_("status").innerHTML="Error loading file.";
});
}

[mathieucarbou]

Hello,
Look also at the Stream example in the project.
It shows how to stream a set of files without memory overhead directly from the file system to the browser.