How to manage big json using HTTPClient ? about arduinojson HOT 12 CLOSED

You're welcome, @Smanar!
Thank you for using ArduinoJson.
Don't forget to cast a star to support the project 😉

from arduinojson.

Hi @Smanar,

Indeed, on ESP32, String has a maximum capacity of 65,535 characters.

Even if there were no size limit, String would still require a long contiguous buffer, which might not be available if the heap is fragmented.

To answer your question, String fails to allocate a large buffer to store the whole response, so getString() returns an empty string.
getStream() works because deserializeJson() reads characters progressively.

Best regards,
Benoit

from arduinojson.

Ha yes, so it can explain too why the issue is random, depend of fragmentation and device memory.
So the only solution possible ATM is using "Deserialization in chunks" described here https://arduinojson.org/v7/how-to/deserialize-a-very-large-document/ ? (And found a way for the HTTPClient lib don't stream all the data in one time, but packet by packet)

Edit:
Have found a way using psram here https://www.upesy.fr/blogs/tutorials/get-more-ram-on-esp32-with-psram

from arduinojson.

ArduinoJson 7 works better than version 6 when the heap is fragmented.
Did you check your memory consumption with the ArduinoJson Assistant?

from arduinojson.

I m malking test ATM on a ESP32-2432S028R (ESP32-WROOM-32 4M flash) , so have selected a Denky32 board
(RAM: [=== ] 27.7% (used 90720 bytes from 327680 bytes))

The json take 40 ko

On the assistant I have

• This document suffers from the "JSON in JSON" syndrome, so you may need to call deserializeJson() multiple times. The ArduinoJson Assistant doesn't support settings scenario.
• This is too big to fit in the RAM. See How to deserialize a very large document?

With using a filter

But something strange the memory used don't seem so big ....

Edit:
Was with String, I m testing with Stream ATM.

Edit2:
All are fine "theoricaly" with Stream.

Need to make a try with version 7 (I m stil on 6)

from arduinojson.

This is too big to fit in the RAM

I noticed that this alert sometimes stays on screen when it shouldn't.
It's a bug I introduced recently.
I'll fix this soon.

from arduinojson.

This document suffers from the "JSON in JSON" syndrome

Do you think it's a false positive?

from arduinojson.

Do you think it's a false positive?

Honnestly I think, this json is not complex enought to be considered for "json in json"> https://www.domoticz.com/wiki/Domoticz_API/JSON_URL%27s#Get_status_of_all_devices.2C_including_hidden_ones

I m trying with V7, and for the moment I can't sucess,
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled

        //DynamicJsonDocument doc(buffer); //V6
        JsonDocument doc; //V7
        DeserializationError err;
       err = deserializeJson(doc, client.getStream());
        *JS = doc["result"].as<JsonArray>();

        if (!*JS)
        {
            Serial.println("Json not available\n");
            return false;
        }

    char buffer[4096];
    serializeJsonPretty(JS, buffer);
    Serial.println(buffer);

The buffer can be small (It's not the same json) but even for (auto i : JS) // Scan the array (only 1) make a Guru meditation.

from arduinojson.

Honnestly I think, this json is not complex enought to be considered for "json in json"

It doesn't have to be complex.
Please share the JSON document in an attached file.

*JS = doc["result"].as<JsonArray>();

I cannot think of any way this can be correct.
Please replace with:

JsonArray result = doc["result"];

You can use step 3 of the assistant, it will write most of the extraction code for you.

from arduinojson.

You have the json here https://pastebin.com/NFbB3uSb
The complete code is here https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/ip_engine.cpp#L26

I can make you a minimal code (like here #2061 (comment) have make your replacement) but atm the code give me a Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled on version 7 but work without problem on version 6.

It crash on serializeJsonPretty(JS, buffer); or for (auto i : JS) (but it's more far away in the code) and I don't use the debugger (IDK how it work with esp32 device)

Edit:
The calling fonction is here https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/data_setup.cpp#L247

But now after reflexion, I think I have found something.

from arduinojson.

"Color" : "{\"b\":109,\"cw\":0,\"g\":255,\"m\":3,\"r\":0,\"t\":0,\"ww\":0}",

That's, indeed, JSON in JSON 😉

You cannot use a JsonArray once the JsonDocument is destructed:
https://github.com/Smanar/CYD-Domoticz-Remote/blob/test/src/core/data_setup.cpp#L256

That's because JsonArray is a reference, which is one of ArduinoJson's fundamentals.
Please read the documentation, the tutorials, or the book.
ArduinoJson 6 had the same restriction; you were lucky that the code didn't crash.

from arduinojson.

That's, indeed, JSON in JSON

Ha yes, for me was just a String.

BTW thx a lot. I thought of this just after posting, but as it have worked with V6, it didn't shock me.
So have updated the code, and make tries with V6, and :

• No more crash
• It work fine, even with the bigger json I can generate.

I will send the code to an other user that have issue, but I m almost sure It will works fine.

Thx again for help.

from arduinojson.