eraseConfig() causes crashes

[adrian-dybwad]

Platform

• Hardware: WROOM02
• Core Version: 2.5.0
• Development Env: Arduino
• Operating System: Mac

Settings in IDE

• Module: [Generic ESP8266]
• Flash Mode: [dio]
• Flash Size: [2MB]
• lwip Variant: [v2 Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [80Mhz]
• CPU Frequency: [160MHz]
• Upload Using: [SERIAL]
• Upload Speed: [other] (serial upload only)

Problem Description

After working on an issue related to #2186 (comment), I added an eraseConfig() into my code. After doing that, I started to see some weird behavior. One instance, the ESP will crash. This post is a result of working on understanding why it crashes and the very simple MCVE produces crashes in various ways. Any insight as to why is welcomed.

MCVE Sketch

//SDK:3.0.0-dev(c0f7b44)/Core:2.5.0=20500000/lwIP:STABLE-2_1_2_RELEASE/glue:1.1/BearSSL:6778687


#include <ESP8266WiFi.h>

void setup() {
  Serial.begin(115200);
  Serial.println();

  //This eraseConfig causes the sketch to crash
  ESP.eraseConfig();

  //Including this mode line (while leaving out this setup's disconnect line)
  //makes ESP crash once and then give crazy heap numbers
  //WiFi.mode(WIFI_STA);
  
  //Including this disconnect causes the heap values to be crazy
  WiFi.disconnect();
}

void loop() {

    //Try connecting to a non existent access point
    WiFi.begin("nonexistentssid", "nonexistentpassword"); 
    delay(2000); 

    //Leaving out this disconnect prevents crashing
    WiFi.disconnect();
    
    Serial.printf_P(PSTR("%d %d/%d/%d/%d\r\n"),
    millis(),
    ESP.getFreeHeap(), 
    ESP.getHeapFragmentation(), 
    ESP.getMaxFreeBlockSize(), 
    ESP.getFreeContStack()
    );

}

Debug Messages

Notice the getFreeHeap values in the last lines.

3724 51600/3/50408/3008
6229 51600/3/50408/3008
8737 51600/3/50408/3008
11251 51600/3/50408/3008

Exception (28):
epc1=0x4000df2f epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000033 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffb50 end: 3fffffc0 offset: 01a0
3ffffcf0:  40216e8b 4000444e 00002035 402263e6  
3ffffd00:  00000033 00000010 402177a5 3ffecdbc  
3ffffd10:  3ffef20c 00000000 3ffe8638 3ffef954  
3ffffd20:  3ffef20c 402176cd 3ffecdbc 001ff000  
3ffffd30:  3ffe8638 0000022b 0000022b 401005b8  
3ffffd40:  3ffef2ad 3ffef6ac 0000000f 3ffed4fc  
3ffffd50:  3ffe8638 3ffe8638 00000033 40219643  
3ffffd60:  3ffed4b0 40219214 4021b884 3ffed4b0  
3ffffd70:  40218d90 4022c93b 3ffef20c 000001ff  
3ffffd80:  00000001 00000001 3ffffd90 4022c8d2 <
3ffffd90:  4021924f 3ffed5a8 3ffed5a8 00000020  
3ffffda0:  4021b1cc 00000024 00000072 aa55aa55  
3ffffdb0:  3ffece40 4021b190 3ffecdbc 00000012  
3ffffdc0:  40219204 402191d8 3ffece40 00000012  
3ffffdd0:  402191a3 3ffecdbc 00000012 4022cff3  
3ffffde0:  40219754 3ffecdbc 00000000 4021b884  
3ffffdf0:  40219181 3ffecf38 00000002 00000000  
3ffffe00:  4021b63d 0000010f 3ffe856c 00000000  
3ffffe10:  00000013 0000010f 3ffecfd8 4021b491  
3ffffe20:  4021b49c 3ffe84dc 3ffe856c 00000003  
3ffffe30:  4022d624 3ffe84dc 3ffe856c 3ffe856c  
3ffffe40:  3ffef6ac 40202961 3ffe857c 40202897  
3ffffe50:  332f3800 656e6f6e 74736978 70746e65  
3ffffe60:  77737361 0064726f 3ffee32c 00000000  
3ffffe70:  00000000 3ffee32c feefeffe feefeffe  
3ffffe80:  3fffff50 3fffff20 00000008 0000c990  
3ffffe90:  401050f8 00332155 3ffee494 00000000  
3ffffea0:  3ffedcf0 3ffee494 3ffe8508 3ffee494  
3ffffeb0:  ffffff00 00ffffff 3ffffed0 40202bd0  
3ffffec0:  4022d600 656e6f6e 74736978 73746e65  
3ffffed0:  00646973 31352031 2f303036 30352f33  
3ffffee0:  2f383034 656e6f6e 74736978 70746e65  
3ffffef0:  77737361 0064726f 3ffee32c 00000000  
3fffff00:  00000000 3ffee32c feefeffe feefeffe  
3fffff10:  3fffff50 3fffff20 00000008 0000c990  
3fffff20:  00000000 00000000 00002bf3 0000c981  
3fffff30:  00000002 0000c401 4bc6a7f0 0000c990  
3fffff40:  00000004 00000000 3ffee2f8 00000001  
3fffff50:  40202a03 000007d0 000007d0 3ffee438  
3fffff60:  3fffdad0 00000000 3ffee2f8 40202512  
3fffff70:  00000bc0 001ff000 3ffee32c 40202d24  
3fffff80:  00002bf3 0000001c 3ffee32c 40202b46  
3fffff90:  00000000 00000000 00000001 3ffee438  
3fffffa0:  3fffdad0 00000000 3ffee408 40203408  
3fffffb0:  feefeffe feefeffe 3ffe8508 40100a59  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v951aeffa
~ld

3433 51600/1/51408/3152
5955 51600/1/51408/3008
8472 51600/1/51408/3008
10806 500824/94/22104/2944
12928 335216/100/6472/2944

[earlephilhower]

Please decode the stack dump. Those hex numbers are useless without the decoded info as we don't have your exact binary to reverse them against.

That said, I was able to repro the failure on GIT head, occasionally. Most times I could get 3 loops then crash, but one time it just ram forever (or at least the 10 mins and 600 loops or so before I tried something else).

When the Debug port == Serial is enabled in the IDE you get a WDT on the 3rd try and no crashes. When Debug port == none crashes do occur relatively repeatably.

When I did the stack trace it was somewhere in the SDK ROM calling an alloc or free (seemed to vary), but there was memory corruption so the UMM memory manager crashed.

Adding IRQ disables around all our own calls into the SDK had no effect. Crashes occurring in ROM code on memory callbacks, always.

Exception 3: LoadStoreError: Processor internal physical address or data error during load or store
PC: 0x402036b0: umm_make_new_block(unsigned short, unsigned short, unsigned short, unsigned short) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.cpp line 1158
EXCVADDR: 0x40020b22

Decoding stack results
0x4010025c: pvPortZalloc(size_t, char const*, int) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/heap.cpp line 176
0x40203843: calloc(size_t, size_t) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/umm_malloc/umm_malloc.cpp line 1704
0x4010025c: pvPortZalloc(size_t, char const*, int) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/heap.cpp line 176
0x40100119: __wrap_spi_flash_read(uint32_t, uint32_t*, size_t) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_phy.cpp line 309
0x40100119: __wrap_spi_flash_read(uint32_t, uint32_t*, size_t) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_phy.cpp line 309
0x40201304: ESP8266WiFiSTAClass::disconnect(bool) at /home/earle/Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp line 369
0x40202088: esp_yield() at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 91
0x4020264a: delay(unsigned long) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_wiring.cpp line 57
0x4020263f: delay(unsigned long) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_wiring.cpp line 54
0x4020109c: loop() at /home/earle/Arduino/sketch_may10b/sketch_may10b.ino line 34
0x402018cc: HardwareSerial::begin(unsigned long, SerialConfig, SerialMode, unsigned char) at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/HardwareSerial.cpp line 51
0x402016ce: EspClass::eraseConfig() at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/Esp.cpp line 482
0x40202138: loop_wrapper() at /home/earle/Arduino/hardware/esp8266com/esp8266/cores/esp8266/core_esp8266_main.cpp line 125

[earlephilhower]

And I think the getFreeHeap() stuff is just an artifact of the memory corruption and not something alloc'ing 30KB...

[adrian-dybwad]

Exception (28):
epc1=0x4000df2f epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000033 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffb50 end: 3fffffc0 offset: 01a0
3ffffcf0:  40216e8b 4000444e 00002035 402263e6  
3ffffd00:  00000033 00000010 402177a5 3ffecdbc  
3ffffd10:  3ffef20c 00000000 3ffe8638 3ffef954  
3ffffd20:  3ffef20c 402176cd 3ffecdbc 001ff000  
3ffffd30:  3ffe8638 0000022b 0000022b 401005b8  
3ffffd40:  3ffef2ad 3ffef6ac 0000000f 3ffed4fc  
3ffffd50:  3ffe8638 3ffe8638 00000033 40219643  
3ffffd60:  3ffed4b0 40219214 4021b884 3ffed4b0  
3ffffd70:  40218d90 4022c93b 3ffef20c 000001ff  
3ffffd80:  00000001 00000001 3ffffd90 4022c8d2 <
3ffffd90:  4021924f 3ffed5a8 3ffed5a8 00000020  
3ffffda0:  4021b1cc 00000024 00000072 aa55aa55  
3ffffdb0:  3ffece40 4021b190 3ffecdbc 00000012  
3ffffdc0:  40219204 402191d8 3ffece40 00000012  
3ffffdd0:  402191a3 3ffecdbc 00000012 4022cff3  
3ffffde0:  40219754 3ffecdbc 00000000 4021b884  
3ffffdf0:  40219181 3ffecf38 00000002 00000000  
3ffffe00:  4021b63d 0000010f 3ffe856c 00000000  
3ffffe10:  00000013 0000010f 3ffecfd8 4021b491  
3ffffe20:  4021b49c 3ffe84dc 3ffe856c 00000003  
3ffffe30:  4022d624 3ffe84dc 3ffe856c 3ffe856c  
3ffffe40:  3ffef6ac 40202961 3ffe857c 40202897  
3ffffe50:  332f3800 656e6f6e 74736978 70746e65  
3ffffe60:  77737361 0064726f 3ffee32c 00000000  
3ffffe70:  00000000 3ffee32c feefeffe feefeffe  
3ffffe80:  3fffff50 3fffff20 00000008 0000c990  
3ffffe90:  401050f8 00336102 3ffee494 00000000  
3ffffea0:  3ffedcf0 3ffee494 3ffe8508 3ffee494  
3ffffeb0:  ffffff00 00ffffff 3ffffed0 40202bd0  
3ffffec0:  40100a00 656e6f6e 74736978 73746e65  
3ffffed0:  00646973 40100a54 402033e8 00000000  
3ffffee0:  00000000 656e6f6e 74736978 70746e65  
3ffffef0:  77737361 0064726f 00000060 3ffe8508  
3fffff00:  00000000 3fffdad0 3ffee438 00000030  
3fffff10:  3fffff50 3fffff20 00000008 0000c990  
3fffff20:  00000000 00000000 00002c31 0000c981  
3fffff30:  00000002 0000c401 4bc6a7f0 0000c990  
3fffff40:  00000004 00000000 3ffee2f8 00000001  
3fffff50:  40202a03 000007d0 000007d0 3ffee438  
3fffff60:  3fffdad0 00000000 3ffee2f8 40202512  
3fffff70:  00000bc0 001ff000 3ffee32c 40202d24  
3fffff80:  00002c31 0000001c 3ffee32c 40202b46  
3fffff90:  00000000 00000000 00000001 3ffee438  
3fffffa0:  3fffdad0 00000000 3ffee408 40203408  
3fffffb0:  feefeffe feefeffe 3ffe8508 40100a59  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
PC: 0x4000df2f
EXCVADDR: 0x00000033

Decoding stack results
0x401005b8: _umm_free at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/umm_malloc/umm_malloc.c line 1300
0x40202961: ESP8266WiFiSTAClass::begin(char const*, char const*, int, unsigned char const*, bool) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp line 177
0x40202897: ESP8266WiFiSTAClass::begin(char const*, char const*, int, unsigned char const*, bool) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp line 135
0x40202bd0: HardwareSerial::write(unsigned char const*, unsigned int) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/HardwareSerial.h line 174
0x40100a00: free at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/umm_malloc/umm_malloc.c line 1760
0x402033e8: loop_wrapper() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_main.cpp line 117
0x40202a03: EspClass::getHeapStats(unsigned int*, unsigned short*, unsigned char*) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/Esp-frag.cpp line 27
0x40202512: loop() at /Users/adriandybwad/Google Drive/Arduino/DisconnectBug/DisconnectBug.ino line 25
0x40202d24: HardwareSerial::begin(unsigned long, SerialConfig, SerialMode, unsigned char) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/HardwareSerial.cpp line 51
0x40202b46: EspClass::eraseConfig() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/Esp.cpp line 482
0x40203408: loop_wrapper() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_main.cpp line 125

[adrian-dybwad]

Ok, here is the second.

Exception (28):
epc1=0x40228a2d epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000038 depc=0x00000000

>>>stack>>>

ctx: cont
sp: 3ffffd00 end: 3fffffc0 offset: 01a0
3ffffea0:  3ffeeac4 00010000 3fffdad0 40228a50  
3ffffeb0:  3fffdad0 00000000 3ffeeac4 4022ab00  
3ffffec0:  4022d6d2 3ffeeac4 3ffecdbc 00000000  
3ffffed0:  4020278c 3ffee2f8 2f343232 352f3533  
3ffffee0:  34363800 656e6f6e 74736978 70746e65  
3ffffef0:  77737361 0064726f 3ffeeac4 00000000  
3fffff00:  00000000 3ffee32c feefeffe feefeffe  
3fffff10:  3fffff50 3fffff20 00000008 00013578  
3fffff20:  401050f8 00452df1 3ffee494 00000000  
3fffff30:  3ffedcf0 3ffee494 3ffe8508 3ffee494  
3fffff40:  3fffdad0 3ffee438 4020335c 3fffefa0  
3fffff50:  402014ba 000007d0 000007d0 402014af  
3fffff60:  3fffdad0 00000000 3ffee2f8 40202520  
3fffff70:  00000ba0 001ff000 3ffee32c 40202d24  
3fffff80:  000030f7 0000001c 3ffee32c 40202b46  
3fffff90:  00000000 00000000 00000001 3ffee438  
3fffffa0:  3fffdad0 00000000 3ffee408 40203408  
3fffffb0:  feefeffe feefeffe 3ffe8508 40100a59  
<<<stack<<<

last failed alloc call: 40100244(32) 

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
PC: 0x40228a2d: netif_set_down at glue-esp/lwip-esp.c line 614
EXCVADDR: 0x00000038
Memory allocation of 32 bytes failed at 0x40100244: pvPortZalloc at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/heap.c line 174

Decoding stack results
0x40228a50: netif_remove at glue-esp/lwip-esp.c line 535
0x4020278c: ESP8266WiFiSTAClass::disconnect(bool) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/libraries/ESP8266WiFi/src/ESP8266WiFiSTA.cpp line 323
0x4020335c: esp_yield() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_main.cpp line 91
0x402014ba: delay at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_wiring.c line 54
0x402014af: delay at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_wiring.c line 51
0x40202520: loop() at /Users/adriandybwad/Google Drive/Arduino/DisconnectBug/DisconnectBug.ino line 36
0x40202d24: HardwareSerial::begin(unsigned long, SerialConfig, SerialMode, unsigned char) at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/HardwareSerial.cpp line 51
0x40202b46: EspClass::eraseConfig() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/Esp.cpp line 482
0x40203408: loop_wrapper() at /Users/adriandybwad/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.0/cores/esp8266/core_esp8266_main.cpp line 125

[adrian-dybwad]

Those traces should now be proper.

[earlephilhower]

Thanks, @adrionics . It's the same as the others, though, memory corruption causing either UMM crashes or OOM because the heap is in an inconsistent state. Not sure the best way to track it down, but as the two calls in question are very thin wrappers around SDK functions you might want to start talking on the espressif BBS and nonos-sdk sites and see if they're known...

[adrian-dybwad]

I posted on expressif BBS. We will see if they respond.