Native USB debugging on ESP32 in PlatformIO without ESP-IDF toolchain? #7879

papakpmartin · 2023-02-22T17:24:16Z

papakpmartin
Feb 22, 2023

I have a custom ESP32-S2-based circuit board with USB-C which does not have a USB-serial converter IC like a common dev board might. On this board, USB(-) is on GPIO19, and USB(+) is on GPIO20. USB works great for powering the board and for uploading firmware.

The board works well and I have access to good old-fashioned serial console via a USB-to-logic level serial interface, but it would be nice to be able to get serial out to USB along with the firmware upload (like a dev board).

I’m using PlatformIO in Visual Studio Code, and writing with arduino-esp32 rather than ESP-IDF.

I understand by these instructions from Espressif that when using ESP-IDF I can configure log output to go to USB CDC rather than UART, and this has been done successfully on my custom board. But I would like to be able to do this without having to switch over to ESP-IDF.

I’m presuming that one of the partitions that PlatformIO is building for me is this configuration with some nice common sense defaults, but I can’t see how I might alter those defaults to do what I’m looking for.

Any thoughts or pointers?

Answered by Jason2866

Feb 23, 2023

Use a custom boards.json for your DIY board. Like this assuming no PSRAM. Name it for example esp32s2-cdc.json Make a new folder boards in your root folder of the project. Place your created json file there

{
  "build": {
    "arduino":{
      "ldscript": "esp32s2_out.ld"
    },
    "core": "esp32",
    "extra_flags": [
      "-DARDUINO_USB_CDC_ON_BOOT=1"
    ],
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "mcu": "esp32s2",
    "variant": "esp32s2"
  },
  "connectivity": [
    "wifi"
  ],
  "debug": {
    "openocd_target": "esp32s2.cfg"
  },
  "frameworks": [
    "arduino",
    "espidf"
  ],
  "name": "ESP32-S2 CDC",
  "upload": {
    "flash_size": "…

View full answer

me-no-dev · 2023-02-23T11:40:25Z

me-no-dev
Feb 23, 2023
Maintainer

This functionality is already available. @Jason2866 could you please help with PIO instructions?

0 replies

Jason2866 · 2023-02-23T11:56:40Z

Jason2866
Feb 23, 2023
Collaborator

Use a custom boards.json for your DIY board. Like this assuming no PSRAM. Name it for example esp32s2-cdc.json Make a new folder boards in your root folder of the project. Place your created json file there

{
  "build": {
    "arduino":{
      "ldscript": "esp32s2_out.ld"
    },
    "core": "esp32",
    "extra_flags": [
      "-DARDUINO_USB_CDC_ON_BOOT=1"
    ],
    "f_cpu": "240000000L",
    "f_flash": "80000000L",
    "flash_mode": "qio",
    "mcu": "esp32s2",
    "variant": "esp32s2"
  },
  "connectivity": [
    "wifi"
  ],
  "debug": {
    "openocd_target": "esp32s2.cfg"
  },
  "frameworks": [
    "arduino",
    "espidf"
  ],
  "name": "ESP32-S2 CDC",
  "upload": {
    "flash_size": "4MB",
    "maximum_ram_size": 327680,
    "maximum_size": 4194304,
    "require_upload_port": true,
    "speed": 460800
  },
  "url": "https://www.myboard.com",
  "vendor": "DIY"
}

and create as usual an env in your platformio.ini and add as board your board

...
board  = esp32s2-cdc
....

Looking in the predefined boards.json will help to understand the different settings https://github.com/platformio/platform-espressif32/tree/develop/boards

If your board has PSRAM add in section extra_flags the entry "-DBOARD_HAS_PSRAM"

The needed options can be set in platformio.ini too. But i like it more to define a custom boards.json which exactly meets the boards specs. So one time done no more thinking about which settings in platformio.ini are needed when a new env is done with this board...

4 replies

papakpmartin Feb 23, 2023
Author

This is fantastic. Thank you @Jason2866 and @me-no-dev! I will try this shortly and mark as answer when i see it work. This makes perfect sense.

papakpmartin Feb 23, 2023
Author

I had been using esp32-s2-saola-1.json. What does -DARDUINO_ESP32S2_DEV do? Is there a reference for these flags? Should I ditch that one when adding -DARDUINO_USB_CDC_ON_BOOT=1?

papakpmartin Feb 23, 2023
Author

Do I also need to set Serial up differently in setup()? Currently I'm doing...

  Serial.begin(115200);
  while (!Serial);
  Serial.println("SERIAL: Setup complete");

Asking because it appears I'm getting "stuck" here.

papakpmartin Feb 23, 2023
Author

Ah... interesting (to me)...

I was able to upload via USB as always and eventually got...

WARNING: ESP32-S2FH4 (revision v0.0) chip was placed into download mode using GPIO0.
esptool.py can not exit the download mode over USB. To run the app, reset the chip manually.

If I do this with "Upload and Monitor", the monitor doesn't "catch" when I hit the reset button.

However, if I use "Monitor" and if I manually choose the "Set upload/monitor/test port", then Monitor does pick it up and the board can start normally and get me the logging I was looking for!

Thanks so much! :)

papakpmartin · 2023-02-24T00:15:36Z

papakpmartin
Feb 24, 2023
Author

Might using "-DARDUINO_USB_CDC_ON_BOOT=1" have unexpected consequences or implications?

My new config is essentially identical to the esp32-s2-saola-1.json that I was using before, with only that flag added. However, with the old device config everything works as expected, and with the new device config I'm unable to connect to an MQTT broker as expected... just stops there... my logging is...

Running task__handleReporting
WIFI: About to connect
FIRMWARE: Version 2
Available Ram: 103256 bytes
WIFI: Status: 3
WIFI: Connecting...
WIFI: CONNECTED
MQTT: About to connect

...and I get nothing more than that even with this in place:

build_flags = 
  -DCORE_DEBUG_LEVEL=5

Immediately after that last log I'm running...

  if (!m_mqttClient.connected()) {
    Serial.println("MQTT: Setting server");
    m_mqttClient.setServer(MQTT_BROKER, MQTT_PORT);
    Serial.println("MQTT: Setting callback");
    ...

I'm using knolleary/PubSubClient@^2.8 for MQTT, in case that's helpful.

Again, rolling back to a plain esp32-s2-saola-1 build works perfectly.

Any thoughts would be appreciated! :)

0 replies

Jason2866 · 2023-02-24T09:27:56Z

Jason2866
Feb 24, 2023
Collaborator

@papakpmartin Mhh, the mqtt issue is strange and cant explain.
You can try our modified mqtt pubsubclient version. Working for all esp

https://github.com/arendst/Tasmota/tree/development/lib/default/pubsubclient-2.8.13

Have seen this too:

WARNING: ESP32-S2FH4 (revision v0.0) chip was placed into download mode using GPIO0.
esptool.py can not exit the download mode over USB. To run the app, reset the chip manually.

Maybe adding this helps

    "use_1200bps_touch": true,
    "wait_for_upload_port": true,
    "require_upload_port": true,

Finally you can try:

Modified (forked) Platformio platform-espressif32 and changed to make this settings possible in boards.json

    "require_upload_port": true,
    "before_reset": "usb_reset",
    "after_reset": "no_reset",
    "speed": 460800

You can try if it works for you. If yes i will provide a PR currently we use this change only for project Tasmota

platform = https://github.com/tasmota/platform-espressif32/releases/download/2023.02.00/platform-espressif32.zip

0 replies

papakpmartin · 2023-02-28T01:52:30Z

papakpmartin
Feb 28, 2023
Author

Well, I was able to give this a go... it was weird. It mostly did not make a difference. Same symptom of starting nicely (as long as USB was plugged in and Monitor window was open) and running until approximately the same point, then pause for about 8-10 seconds, then crash/restart the ESP. No stacktrace.

My device file is basically...

{
    "build": {
      "arduino":{
        "ldscript": "esp32s2_out.ld"
      },
      "core": "esp32",
      "extra_flags": [
        "-DARDUINO_ESP32S2_DEV",
        "-DARDUINO_USB_CDC_ON_BOOT=1"
      ],
      "f_cpu": "240000000L",
      "f_flash": "80000000L",
      "flash_mode": "qio",
      "mcu": "esp32s2",
      "variant": "esp32s2"
    },
    "connectivity": [
      "wifi"
    ],
    "debug": {
      "openocd_target": "esp32s2.cfg"
    },
    "frameworks": [
      "arduino",
      "espidf"
    ],
    "name": "My Custom ESP32-S2",
    "upload": {
      "flash_size": "4MB",
      "maximum_ram_size": 327680,
      "maximum_size": 4194304,
      "speed": 460800,
      "use_1200bps_touch": true,
      "wait_for_upload_port": true,
      "require_upload_port": true
    },
    "url": "...",
    "vendor": "..."
  }

...and my platformio.ini contains basically...

[env:my-custom-esp32s2]
; platform = espressif32
platform = https://github.com/tasmota/platform-espressif32/releases/download/2023.02.00/platform-espressif32.zip
board  = my-custom-esp32s2
framework = arduino
monitor_speed = 460800
monitor_filters =
  esp32_exception_decoder
  log2file
lib_deps = 
  knolleary/PubSubClient@^2.8
  ; others
; build_flags = 
;   -DCORE_DEBUG_LEVEL=5

Oddly, with that set up once in a while it worked perfectly for a few minutes, but would eventually revert back to the same behavior.

At one point I thought it was maybe xTaskCreatePinnedToCore as when I replaced the single call to that with...

  xTaskCreate(
    task__handleReporting,
    "task__handleReporting",
    25000,
    NULL,                            // parameters
    10,                              // priority, higher number is higher priority
    &taskHandle__handleReporting
    // CONFIG_ARDUINO_RUNNING_CORE   // Wi-Fi tasks must run on same core and Arduino
  );

...it worked. But it did not reliably work.

Not sure what to say, but I wanted to at least share what happened when I tried.

2 replies

papakpmartin Feb 28, 2023
Author

Oh, but I can say that...

"use_1200bps_touch": true,
"wait_for_upload_port": true,
"require_upload_port": true

...did help me reliably get upload and monitor connections.

Even when using “normal” platform = espressif32 and not the tasmota/platform-espressif32. (I mean, it seemed to be better in both cases.)

papakpmartin Feb 28, 2023
Author

OK, so I at least have it down to:

Using the esp32-s2-saola-1.json that I have been using all along works perfectly
Using a custom board identical to that one also works perfectly
Adding "-DARDUINO_USB_CDC_ON_BOOT=1" to the custom board (and doing nothing else differently)...
- Requires monitor to be connected in order to get through setup()
- Only works maybe 5% of the time, but instead usually reboots (watchdog timeout?)
Keeping that flag and using the custom platform and extra attributes has no obvious effect on performance
Keeping that flag and moving the Wi-Fi/MQTT code's task to xTaskCreateCore rather than the original xTaskCreatePinnedToCore might have a slightly positive effect, perhaps working 10% of the time instead of 5% of the time; but still it usually simply reboots.

So in the end, it appears that adding "-DARDUINO_USB_CDC_ON_BOOT=1" is doing something unexpected (by me) that interferes with my code.

Maybe in a couple of weeks I could try to come up with a simplified test case.

I didn't have a chance to try swapping out the MQTT library either. If it fails, it does mostly fail there (rarely before that in the Wi-Fi connection).

Jason2866 · 2023-02-28T14:07:18Z

Jason2866
Feb 28, 2023
Collaborator

@papakpmartin Thinking that the issue you see is caused from your sketch. We had a "funny" bug in mqtt lib which caused weird issues.

0 replies

papakpmartin · 2023-03-23T17:01:29Z

papakpmartin
Mar 23, 2023
Author

Well, I'm not sure what to make of this, but commenting out all of the Serial.p... lines in the library doing the MQTT connection allowed everything to work as expected! But these debug lines from the library just before the MQTT stuff did work prior to me commenting everything out:

WIFI: Status: 3
WIFI: Connecting...
WIFI: CONNECTED

So essentially, all Serial lines in Comms.c here were commented out. However, unlike that example project, I have other libraries that do use Serial just fine... in setup() anyway.

Weird. But thought I'd share as it seemed interesting. :)

0 replies

papakpmartin · 2023-03-29T15:41:03Z

papakpmartin
Mar 29, 2023
Author

The issue was indeed my code. Having fixed that, this USB debug logging works wonderfully.

The problem was this while having no delay:

    while (!m_mqttClient.connect(mac_address.c_str(), MQTT_USER, MQTT_PASS)) {
      Serial.println("MQTT: Connecting...");
      Serial.print("MQTT: State: ");
      Serial.println(m_mqttClient.state());
    }

So in brief, @Jason2866's advice above worked perfectly.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native USB debugging on ESP32 in PlatformIO without ESP-IDF toolchain? #7879

{{title}}

Replies: 8 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Native USB debugging on ESP32 in PlatformIO without ESP-IDF toolchain? #7879

papakpmartin Feb 22, 2023

Replies: 8 comments · 6 replies

me-no-dev Feb 23, 2023 Maintainer

Jason2866 Feb 23, 2023 Collaborator

papakpmartin Feb 23, 2023 Author

papakpmartin Feb 23, 2023 Author

papakpmartin Feb 23, 2023 Author

papakpmartin Feb 23, 2023 Author

papakpmartin Feb 24, 2023 Author

Jason2866 Feb 24, 2023 Collaborator

papakpmartin Feb 28, 2023 Author

papakpmartin Feb 28, 2023 Author

papakpmartin Feb 28, 2023 Author

Jason2866 Feb 28, 2023 Collaborator

papakpmartin Mar 23, 2023 Author

papakpmartin Mar 29, 2023 Author

papakpmartin
Feb 22, 2023

Replies: 8 comments 6 replies

me-no-dev
Feb 23, 2023
Maintainer

Jason2866
Feb 23, 2023
Collaborator

papakpmartin Feb 23, 2023
Author

papakpmartin Feb 23, 2023
Author

papakpmartin Feb 23, 2023
Author

papakpmartin Feb 23, 2023
Author

papakpmartin
Feb 24, 2023
Author

Jason2866
Feb 24, 2023
Collaborator

papakpmartin
Feb 28, 2023
Author

papakpmartin Feb 28, 2023
Author

papakpmartin Feb 28, 2023
Author

Jason2866
Feb 28, 2023
Collaborator

papakpmartin
Mar 23, 2023
Author

papakpmartin
Mar 29, 2023
Author