Merge branch 'arduino' into fix_giga_config

Author: facchinm

.github/workflows/package_core.yml

@@ -10,27 +10,24 @@ jobs:
     name: Build and package core
     runs-on: ubuntu-latest
     env:
-      ZEPHYR_SDK_INSTALL_DIR: /opt/zephyr-sdk-0.16.8
       CCACHE_IGNOREOPTIONS: -specs=*
     outputs:
       CORE_TAG: ${{ env.CORE_TAG }}
       CORE_ARTIFACT: ${{ env.CORE_ARTIFACT }}
       BOARD_VARIANTS: ${{ env.BOARD_VARIANTS }}
     steps:
-      - name: Install toolchain
+      - name: Install OS dependencies
         working-directory: /opt
         run: |
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends git cmake wget python3-pip ninja-build ccache
-          wget -nv https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.8/zephyr-sdk-0.16.8_linux-x86_64_minimal.tar.xz
-          tar xf zephyr-sdk-0.16.8_linux-x86_64_minimal.tar.xz && cd zephyr-sdk-0.16.8 && ./setup.sh -t arm-zephyr-eabi -c
 
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           persist-credentials: false
 
-      - name: Initialize
+      - name: Initialize Zephyr environment
         run: |
           ./extra/bootstrap.sh -o=--filter=tree:0
           echo "CORE_TAG=$(git describe --always)" >> "$GITHUB_ENV"

README.md

@@ -80,23 +80,40 @@ Unlike traditional Arduino implementations, where the final output is a standalo
 
 The `loader` is responsible for managing the interaction between your sketches and the underlying Zephyr system. After the initial bootloader installation, the `loader` takes over the sketch loading process automatically.
 
-To ensure flexibility, the `loader` project is designed to be generic. Any necessary modifications for specific boards should be made in the corresponding `DTS overlay` or a special `fixup` file, using appropriate guards to maintain stability.
+To ensure flexibility, the `loader` project is designed to be generic. Any necessary modifications for specific boards should be made in the corresponding "DTS overlay" or a special "fixup" file, using appropriate guards to maintain compatibility.
 
-The behavior of the `loader` can be adjusted through the `Mode` menu:
+The behavior of the `loader` can be adjusted through the `Mode` menu of the IDE:
 - `Standard`: The sketch is loaded automatically.
 - `Debug`: The user must type `sketch` in Zephyr's shell, which is accessible via the default Serial.
 
 The most important components of this project are:
 
 * [Zephyr based loader](/loader)
 * [LLEXT](https://docs.zephyrproject.org/latest/services/llext/index.html)
-* [Actual core](/cores/arduino) with [variants](/variants) and the usual `{platform,boards}.txt`
+* [Actual core](/cores/arduino) with [variants](/variants) and the usual [platform](/platform.txt) and [boards](/boards) files
 * [ArduinoCore-API](https://github.com/arduino/ArduinoCore-API)
-* [post_build_tool](/extra/post_build_tool)
+* [zephyr-sketch-tool](/extra/zephyr-sketch-tool)
 
-## 🛠️ Setup the environment
+## 🏃 Shortcut: using the Core in Arduino IDE/CLI without installing Zephyr
 
-In this section, we’ll guide you through setting up your environment to work with the core.
+> [!TIP]
+>
+> If you are only interested in developing features in the [core](/cores/arduino)
+> or [libraries](/libraries), and do not want to set up a full Zephyr build
+> environment, you can use the [`sync-zephyr-artifacts`](/extra/sync-zephyr-artifacts) 
+> utility to download a pre-built version of the files needed to compile
+> sketches and flash the loader.
+>
+> To do so, after cloning this repo, compile the `sync-zephyr-artifacts`
+> utility via `go build` and run it as `sync-zephyr-artifacts .` to retrieve
+> the precompiled files for the current revision of the core. 
+>
+> Next, follow the instructions in [Using the Core in Arduino IDE/CLI](#using-the-core-in-arduino-idecli).
+> Remember to [update the loader on your board](#flash-the-loader) as well.
+
+## 🛠️ Setup a Zephyr build environment
+
+In this section, we’ll guide you through setting up your environment to work on and update the Zephyr core.
 
 Shell scripts are available to simplify the installation process (Windows is not supported at the moment 😔).
 
@@ -118,57 +135,85 @@ sudo apt install python3-pip python3-setuptools python3-venv build-essential git
 cd ArduinoCore-zephyr
 ./extra/bootstrap.sh
 ```
-### Install the Zephyr SDK
-Download and install the Zephyr SDK for your OS from [here](https://github.com/zephyrproject-rtos/sdk-ng/releases/tag/v0.16.8).
 
-> [!NOTE]  
-> This core is validated for version v0.16.8. Compatibility with later versions has not been tested yet.
+This will take care of installing `west`, the Zephyr build tool. It will then
+download all packages required for a Zephyr build in addition to the toolchains
+in the Zephyr SDK.
+
+> [!NOTE]
+> This core is validated with version v0.17.0 of the SDK. Compatibility with later versions has not been tested yet.
+
+## 🛠️ Regenerate the compiled core files
 
 ### Build the Loader
 
-To build the loader, run the following commands:
+The loader is compiled for each board by running the `./extra/build.sh` script.
+The target can be specified either with the Arduino board name (as defined in
+boards.txt), or with the Zephyr board name and any additional arguments that
+may be required by the Zephyr build system.
+
+For example, to build for the Arduino Portenta H7, you can use either the
+Arduino name:
 ```bash
-export ZEPHYR_SDK_INSTALL_DIR=$folder_where_you_installed_the_sdk
-./extra/build.sh $zephyr_board_name $arduino_variant_board_name
+./extra/build.sh portentah7
 ```
-Replace `$zephyr_board_name` and `$arduino_variant_board_name` with the appropriate names for your board.
 
-Example for Arduino Portenta H7:
+or the Zephyr board target:
+
 ```bash
-./extra/build.sh arduino_portenta_h7//m7 arduino_portenta_h7
+./extra/build.sh arduino_portenta_h7//m7
 ```
 
-The firmwares will be copied to [firmwares](/firmwares) folder.
+The firmwares will be copied to the [firmware](/firmware) folder, and the
+associated variant will be updated.
 
 ### Flash the Loader
 
 If the board is fully supported by Zephyr, you can flash the firmware directly onto the board using the following command:
 ```bash
 west flash
 ```
+This can also be performed via the "Burn bootloader" action in the IDE if the core is properly installed, as detailed below.
 
 ### Using the Core in Arduino IDE/CLI
 
-After running the `bootstrap` script, you can symlink the core to `$sketchbook/hardware/arduino-git/zephyr`. Once linked, it will appear in the IDE/CLI, and the board's Fully Qualified Board Name (FQBN) will be formatted as `arduino-git:zephyr:name_from_boards_txt`.
+After running the `bootstrap.sh` script, you can symlink the core to `$sketchbook/hardware/arduino-git/zephyr`. Once linked, it will appear in the IDE/CLI, and the board's Fully Qualified Board Name (FQBN) will be formatted as `arduino-git:zephyr:name_from_boards_txt`.
 
-### Using the Core in Arduino IDE/CLI (without installing Zephyr toolchain)
+## 🚀 Adding a new target
 
-To help core developers (who might not be interested at all in setting up a full Zephyr build system) we are providing [sync-zephyr-artifacts](/extra/sync-zephyr-artifacts) utility. After compiling it via `go build`, run as `sync-zephyr-artifacts .` to retrieve the files needed to compile sketches and flash the loader.
+> [!TIP]
+>
+> While Zephyr supports a lot of different hardware targets, only the few
+> currently used by the Arduino core are installed by default. To add the
+> support for every Zephyr target to your workspace, run the following
+> commands:
+>
+> ```bash
+> . venv/bin/activate
+> west config -d manifest.project-filter
+> west sdk install --version 0.17.0
+> west update
+> ```
 
-## 🚀 Adding a new target
+To add a new board that is already supported by mainline Zephyr with the target `$your_board`, follow these steps:
 
-To add a new board that is already supported by mainline Zephyr, follow these steps:
+* Get the variant name from your board by running `extra/get_variant_name.sh $your_board`.
+* Create a folder in the [`variants/`](/variants) directory with the same name as the variant for your new board.
+* Create the DTS `<variant>.overlay` and Kconfig `<variant>.conf` files in that directory.
 
-* Create the `DTS overlay` and `.conf` files in the [loader](/loader/boards) directory.
   The overlay must include:
-  * A flash partition called `user_sketch`, tipically located near the end of the flash.
+  * A flash partition called `user_sketch`, typically located near the end of the flash.
   * A `zephyr,user` section containing the description for GPIOs, Analog, UART, SPI and I2C devices. Feel free to leave some fields empty in case Zephyr support is missing. This will result in some APIs not being available at runtime (eg. `analogWrite` if PWM section is empty).
-* Build the Loader: run `./extra.build.sh $your_board $your_board` and start debugging the errors. :grin:
+
+  The Kconfig file must include any board-specific options required by this target.
+* Build the Loader: run `./extra/build.sh $your_board` (with any additional arguments as required) and start debugging the errors. :grin:
 * Update the `boards.txt`: add an entry for your board, manually filling the required fields.
-* Implement touch support: if your board supports the `1200bps touch` method, implement `_on_1200_bps` in a file located inside the `variant/your_board` folder.
-* ⏳ Temporary steps
-  * Create `includes.txt` based on `llext-edk/Makefile.cflags`, taking inspiration for other variants.
-  * Amend `your_board.compiler.zephyr.*` with information from `llext-edk/Makefile.cflags`.
+
+  Make sure to set:
+   * `build.zephyr_target` and `build.zephyr_args` to the arguments used in the `build.sh` call;
+   * `build.zephyr_hals` to the (space-separated list of) HAL modules required by the board;
+   * `build.variant` to the variant name identified above.
+* Implement touch support: if your board supports the "1200bps touch" method, implement `_on_1200_bps` in a file located inside the variant folder of your board.
 
 ## 🐛 Bug Reporting
 
@@ -183,14 +228,12 @@ Contributions are always welcome. The preferred way to receive code contribution
 
 ## 📌 Upcoming features
 
-- [x] Unify overlay in [loader](/loader/boards) with the one provided in [variant](/variant) for interoperability with GSoC project
-- [x] Autogenerate `defines.txt`, `includes.txt`, `cflags.txt` from `llext-edk` output
+- [ ] Remove binaries from this repo history (arduino/ArduinoCore-zephyr#102, :warning: will require a clean clone)
 - [x] Network: support UDP and TLS
-- [ ] USB: switch to USB_DEVICE_STACK_NEXT to support PluggableUSB
-- [ ] Relocate RODATA in flash to accomodate sketches with large assets
+- [ ] USB: switch to `USB_DEVICE_STACK_NEXT` to support PluggableUSB
+- [ ] Relocate RODATA in flash to accommodate sketches with large assets
 - [ ] Provide better error reporting for failed llext operations
-- [ ] Replace [llext_exports.c](/loader/llext_exports.c) with proper symbols generation (via includes)
-- [x] Provide better usability for `Debug` builds (eg. shell over USB)
+- [ ] Replace [`llext_exports.c`](/loader/llext_exports.c) with proper symbols generation (via includes)
 - [ ] Fix corner cases with `std::` includes (like `<iterator>`)
 - [ ] Get rid of all warnings
 

boards.txt

@@ -14,6 +14,7 @@ giga.menu.debug.true.postbuild_debug=-debug
 
 giga.build.zephyr_target=arduino_giga_r1//m7
 giga.build.zephyr_args=--shield arduino_giga_display_shield
+giga.build.zephyr_hals=hal_stm32
 giga.build.variant=arduino_giga_r1_stm32h747xx_m7
 giga.build.mcu=cortex-m7
 giga.build.fpu=-mfpu=fpv5-d16
@@ -79,6 +80,7 @@ nano33ble.menu.debug.true.postbuild_debug=-debug
 
 nano33ble.build.zephyr_target=arduino_nano_33_ble//sense
 nano33ble.build.zephyr_args=
+nano33ble.build.zephyr_hals=hal_nordic
 nano33ble.build.variant=arduino_nano_33_ble_nrf52840_sense
 nano33ble.build.mcu=cortex-m4
 nano33ble.build.fpu=-mfpu=fpv4-sp-d16
@@ -142,6 +144,7 @@ ek_ra8d1.menu.debug.true.postbuild_debug=-debug
 
 ek_ra8d1.build.zephyr_target=ek_ra8d1
 ek_ra8d1.build.zephyr_args=
+ek_ra8d1.build.zephyr_hals=hal_renesas
 ek_ra8d1.build.variant=ek_ra8d1_r7fa8d1bhecbd
 ek_ra8d1.build.mcu=cortex-m85+nomve
 ek_ra8d1.build.fpu=-mfpu=fpv5-d16
@@ -200,6 +203,7 @@ frdm_mcxn947.menu.debug.true.postbuild_debug=-debug
 
 frdm_mcxn947.build.zephyr_target=frdm_mcxn947//cpu0
 frdm_mcxn947.build.zephyr_args=
+frdm_mcxn947.build.zephyr_hals=hal_nxp
 frdm_mcxn947.build.variant=frdm_mcxn947_mcxn947_cpu0
 frdm_mcxn947.build.mcu=cortex-m33
 frdm_mcxn947.build.fpu=-mfpu=fpv5-sp-d16
@@ -253,6 +257,7 @@ portentah7.menu.debug.true.postbuild_debug=-debug
 
 [email protected]//m7
 portentah7.build.zephyr_args=
+portentah7.build.zephyr_hals=hal_stm32
 portentah7.build.variant=arduino_portenta_h7_stm32h747xx_m7
 portentah7.build.mcu=cortex-m7
 portentah7.build.fpu=-mfpu=fpv5-d16
@@ -318,6 +323,7 @@ frdm_rw612.menu.debug.true.postbuild_debug=-debug
 
 frdm_rw612.build.zephyr_target=frdm_rw612
 frdm_rw612.build.zephyr_args=
+frdm_rw612.build.zephyr_hals=hal_stm32
 frdm_rw612.build.variant=frdm_rw612_rw612
 frdm_rw612.build.mcu=cortex-m33+nodsp
 frdm_rw612.build.fpu=-mfpu=fpv5-sp-d16
@@ -370,6 +376,7 @@ niclasense.menu.debug.true.postbuild_debug=-debug
 
 niclasense.build.zephyr_target=arduino_nicla_sense_me
 niclasense.build.zephyr_args=
+niclasense.build.zephyr_hals=hal_nordic
 niclasense.build.variant=arduino_nicla_sense_me_nrf52832
 niclasense.build.mcu=cortex-m4
 niclasense.build.fpu=-mfpu=fpv4-sp-d16
@@ -449,6 +456,7 @@ portentac33.menu.mode.linked.postbuild_mode=-prelinked
 
 portentac33.build.zephyr_target=arduino_portenta_c33
 portentac33.build.zephyr_args=
+portentac33.build.zephyr_hals=hal_renesas
 portentac33.build.variant=arduino_portenta_c33_r7fa6m5bh3cfc
 portentac33.build.mcu=cortex-m33
 portentac33.build.fpu=-mfpu=fpv5-sp-d16
@@ -504,6 +512,7 @@ opta.menu.debug.true.postbuild_debug=-debug
 
 opta.build.zephyr_target=arduino_opta//m7
 opta.build.zephyr_args=
+opta.build.zephyr_hals=hal_stm32
 opta.build.variant=arduino_opta_stm32h747xx_m7
 opta.build.mcu=cortex-m7
 opta.build.fpu=-mfpu=fpv5-d16

cores/arduino/SerialUSB.h

@@ -18,6 +18,8 @@ class SerialUSB_ : public ZephyrSerial {
 	void begin(unsigned long baudrate) { begin(baudrate, SERIAL_8N1); }
 
 	operator bool() override;
+	size_t write(const uint8_t *buffer, size_t size) override;
+	void flush() override;
 
 protected:
 	uint32_t dtr = 0;

cores/arduino/USB.cpp

@@ -124,5 +124,16 @@ arduino::SerialUSB_::operator bool() {
     return dtr;
 }
 
+
+size_t  arduino::SerialUSB_::write(const uint8_t *buffer, size_t size) {
+    if (!Serial) return 0;
+    return arduino::ZephyrSerial::write(buffer, size);
+}
+
+void arduino::SerialUSB_::flush() {
+    if (!Serial) return;
+    arduino::ZephyrSerial::flush();
+}
+
 arduino::SerialUSB_ Serial(usb_dev);
 #endif

extra/bootstrap.sh

@@ -5,14 +5,23 @@ if [ ! -f platform.txt ]; then
   exit 2
 fi
 
+NEEDED_HALS=$(grep 'build.zephyr_hals=' boards.txt | cut -d '=' -f 2 | xargs -n 1 echo | sort -u)
+
+HAL_FILTER="-hal_.*"
+for hal in $NEEDED_HALS; do
+  HAL_FILTER="$HAL_FILTER,+$hal"
+done
+
 python3 -m venv venv
 source venv/bin/activate
 pip install west
 west init -l .
+west config manifest.project-filter -- "$HAL_FILTER"
 west update "$@"
 west zephyr-export
 pip install -r ../zephyr/scripts/requirements-base.txt
-# download slim toolchain from https://github.com/zephyrproject-rtos/sdk-ng/releases/tag/v0.16.8
+west sdk install --version 0.17.0 -t arm-zephyr-eabi
 
-# add here the required blobs based on supported platforms
-west blobs fetch hal_nxp
+for hal in $NEEDED_HALS; do
+	west blobs fetch $hal
+done