JSON input out of order causing IncompleteInput for large json when using http.getStream()

[xsorifc28]

I do not think this is a bug within ArduinoJson, seeking help from anyone with similar experience

Note - I am able to parse the json by reading the stream char-by-char into an array (and deleting some code/functionaliy to have enough memory)

Describe the bug
When passing http.getStream() to deserialize a large payload, I am getting InvalidInput error.

The json that i am parsing is ~88k, which I understand exceeds the maximum string size for esp32 (65k) (using xiao esp32c3).
Therefore I tried to use http.getStream so that the json isn't duplicated in memory.

I tried Serial.println(http.getStream().readString()); and this failes validation, the complete json is there but there are some characters our of order (cURL to the API & json validation passes).

Given that getStream().readString() is returning a bad string, this doesn't seem to be an arduino json issue, but wanted to ask here to confirm and see if there was a work around.

Troubleshooter report
Here is the report generated by the ArduinoJson Troubleshooter:

1. The program uses ArduinoJson 7
2. The issue happens at run time
3. The issue concerns deserialization
4. deserializeJson() returns InvalidInput
5. Input comes from a stream
6. *jsonlint says the document is valid (when using cURL, invalid data returned by http.getStream()
7. Adding a buffer doesn't solve the issue
8. Input's first byte doesn't suggest a BOM

Environment
Here is the environment that I used:

• Microcontroller: XIAO ESP32C3
• Core/runtime: Arduino
• IDE: Arduino IDE 2.3.4

Reproduction
Here is a small snippet that reproduces the issue.

  HTTPClient http;
  client.setInsecure();
  client.setTimeout(20000);
  http.useHTTP10(true);
  http.begin(client, url);
  http.GET();

  // Parse response
  JsonDocument filter;
  JsonObject filter_fixtures_0 = filter["fixtures"].add<JsonObject>();
  filter_fixtures_0["id"] = true;
  filter_fixtures_0["time"] = true;
  filter_fixtures_0["status"] = true;
  filter_fixtures_0["date"] = true;

  DynamicJsonDocument doc(32768);
  ReadBufferingStream bufferedStream(http.getStream(), 64);
  DeserializationError error = deserializeJson(doc, bufferedStream, DeserializationOption::Filter(filter));

  http.end();

  if (error) {
    Serial.println("JSON parse failed: " + String(error.c_str()));
    return false;
  }

Compiler output
n/a

Program output
JSON parse failed: IncompleteInput

[bblanchon]

Hi @xsorifc28,

I suspect the server returns a chuck-encoded response, even if you asked for HTTP/1.0.
Please print the response headers.

Best regards,
Benoit

[xsorifc28]

I don't think so, here is the response headers:

> GET <redacted> HTTP/1.0
> Host: <redacted>
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.14.2
< Date: Wed, 29 Jan 2025 02:01:02 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 88575
< Connection: close
< Vary: Accept-Encoding
< X-Powered-By: Express
< Cache-Control: public, max-age=30
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff

[bblanchon]

User-Agent: curl/8.7.1

You got this header list with Curl.
Could you print the headers using the Arduino code?

Headers might change from one response to the other; only the ones where deserializeJson() returns InvalidInput matter.

See also: How to use ChunkDecodingStream with HTTPClient

[xsorifc28]

User-Agent: curl/8.7.1

You got this header list with Curl. Could you print the headers using the Arduino code?

Headers might change from one response to the other; only the ones where deserializeJson() returns InvalidInput matter.

See also: How to use ChunkDecodingStream with HTTPClient

I have already tried the ChunkDecodingStream solution, even with this change the same error occurs - regardless of having enough free space to fit the entire response or not.

I will get the headers from Arduino and post them in a follow up comment, curious to see if there is a difference.

[bblanchon]

Any progress on this issue?