Sorry, there was an issue loading your presets! reason shown

Issues
1

The current code base seems to sometimes leave out the preset number (see snippet below) which causes the preset file to be flagged as corrupt.

    },
    "30": {
        "on": true,
        "bri": 128,
        "transition": 0,
        "mainseg": 0,
        "seg": [
            {
...snip...
            }
        ],
        "n": "30-Running"
    },    <<<********** HERE
    {
        "on": true,
        "bri": 128,
        "transition": 0,
        "mainseg": 3,
        "seg": [
            {
                "id": 0,
                "start": 0,

I only started touching WLED three days ago, but I’m going to try to find out where this is happening.

2

I think…

bool writeObjectToFile(const char* file, const char* key, JsonDocument* content)

…may not be taking the length of the key in consideration.

3 
uint32_t contentLen = 0;
if (!content->isNull()) contentLen = measureJson(*content) + strlen(key);

I will test when I get to my hardware, but it may be as simple as adding in the text key. The code after this looks at the length of the content and compares that to the size of the existing key in the file. If it’s smaller, it replaces it in the file and pads with spaces. If it’s larger, it deletes and adds to the end.

But if it’s 50 bytes in the file, and the new json is 50, it would add it but there wouldn’t be room for the “50”: maybe?

Just a quick glance at the code to end up there. I have zero experience so I hope better eyes can look at this.

4

Additional:

This problem may be showing up for me because I have taken the JSON and reformatted it so I could make edits easier. When I sent it back to the system, it parsed fine, BUT that might break the update logic.

From what I think the code is doing, it gets the size of the JSON object that has been parsed form the file. This will not include any of the whitespace bytes that may have been added. It uses this size to calculate where to go in the file, which now may be off.

In the file the object may be 50 bytes, but represented by 100 bytes due to whitespace.

Maybe the original code is fine, because there would have already been a key there – so if you save a “50” to “50”, it doesn’t need to count those bytes.

But when the file has been reformatted, it creates the problem.

I am using https://jsonparser.org to compress/uncompress now, and will try uploading just the whitespace removed versions and see if that can be done.

5

Additional notes… not touching the json does work better, but we still see the corruption. In this case, it was when it had to delete and old key to put a larger replacement at the end. It left some garage in the whitespace it wrote over the old key.

Looking at the file:

Old key was “40”, and replacement “40” was larger and ended up at the end. Where the old “40” was is just spaces, so that’s fine.

Old 51 has been replaced by space, fine.

Old 54 has been replaced by spaces, fine.

SOMETHING was placed at the end earlier, and is now spaces. Fine.

A good 51 is after that, fine.

Then after that good 51, where something used to be, is now the corruption:

"n":"30-Running"}    <-- end of last tag

...322 characters (spaces?)...
{
...1131 characters (spaces?)...
?
...3 spaces...
?
...20 spaces...
?
...7 spaces...
,"40":{"on":true,"bri":128, <-- start of next tag

The “40” that ends the file looks fine.

The ? show up as a triangle with a question mark in them, but do not show up if I past the file in VS CODE.

6

There’s nothing wrong IMO with WLED functions as they are unchanged since 0.11.
Corruption happens during flash writes within FS core code.

7

I can agree with @blazoncek. I encounter similar preset JSON file corruption issues due to flash writes. On one (buggy?) ESP32, I seem to trigger/reproduce this problem whenever I exceed around 125 presets. A simple erase of the entire ESP32 flash seems to fix issue, e.g.:

esptool.py --chip esp32 --port /dev/ttyUSB0 erase_flash

Hope this helps!

8

For EEPROM corruption issues at my day job, I implemented a write/read verify to know if I just trashed the settings, and will retry.

Does WLED have the entire json tree in memory or does it pull them on demand from the file system? I am coming up to speed with the Arduino JSON library to better understand how it works.

9

Settings as well as preset (a single one) are kept in RAM as runtime variables and is converted into ArduinoJSON memory structures during file write operation. Checking if write was successful would require double amount of memory or relying on the immediate parsing of written file by ArduinoJSON library.
In both cases this would produce noticeable “hiccup” during file writes. Still, to avoid corruption this is better than nothing.

10

I’d rather have a hiccup than a corrupt system that can’t show anything. This sounds like that approach I’ll work on implementing. Always fun!

11

FYI I have re-checked the code and there is no logical error in WLED part (file.cpp).
Corruption happens within serializeJson(doc,f) or f.write() which are library functions.
Reading-after-write will cause significant overhead as well as plenty of code rewrites.

12

I can see how the glitch we saw being off the number of characters of the string was a coincidence.

We’ve put in a small change that has taken the corruption down from happening 80% of the time to about 5%. I hope to get rid of that 5% when I get a moment. I’ve learned so much in the past few days.

Another ESP32 project I’ve worked with has issues that are far worse - it simply cannot be build with current versions of the Arudino ESP stuff for it. It ends up crashing, The author is using files from several years ago to make it work.

Open source is always fun. I’d rather be doing it this way than dealing with blue screens of death :slight_smile: Really enjoying learning WLED.

1 Like
13

Has this issue been resolved? I would really like to get back to using a Wifi connection on my Dig-uno and Dig-Quad controllers with WLED.

14

no and yes.
WLED now tries to mitigate as much as possible. do not edit presets.json file using smart editors or try to “prettify” it. if you do run the resulting JSON using one of the minimizers before saving it to WLED.