Missing versioning in Modules (may become a problem when one is trying to verify former Zephyr commits)

[rjmaris]

Actually, I'm experiencing a problem with actual version of Zephyr. However, when I checkout an older commit of Zephyr, suddenly, problems (may) then occur as well. The reason:
When I do west update, the modules folder outside the zephyr folder may be updated as well. But all of its content, e.g. HAL, is only versioned at its original location.
Practically, I might encounter an error like Malformed "build" section in file [...] modules/hal/expressif/zephyr/module.yml.

In order to allow such verification tests (e.g. with the goal to analyze problems in the newer version, I'd have to do git versionin on the modules folder as well, and manually commit when I did a west update.

[yashi]

Please provide us a minimal reproducible example.

When I do west update, the modules folder outside the zephyr folder may be updated as well. But all of its content, e.g. HAL, is only versioned at its original location.

Would you please also elaborate a bit on this? HAL for espressif seems to use git submodules but zephyr/west.yml pins down with a hash value, like:

    - name: hal_espressif
      revision: db4a5893e45db100497fbe5262847280fb749dac
      path: modules/hal/espressif
      west-commands: west/west-commands.yml
      groups:
        - hal

ERROR: Malformed "build" section in file:

This is what I found

$ cd zephyr
$ git checkout v3.2.0
$ west update
$ git checkout v3.1-branch
$ rm -rf build && west build -b esp32c3_devkitm samples/hello_world
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /home/yashi/src/zephyr-project/zephyr/samples/hello_world
-- Found Python3: /home/yashi/src/zephyr-project/.venv/bin/python3 (found suitable exact version "3.11.2") found components: Interpreter 
-- Cache files will be written to: /home/yashi/.cache/zephyr
-- Zephyr version: 3.1.0 (/home/yashi/src/zephyr-project/zephyr)
-- Found west (found suitable version "1.0.0", minimum required is "0.7.1")
CMake Error at /home/yashi/src/zephyr-project/zephyr/cmake/modules/zephyr_module.cmake:78 (message):
  validation.invalid

   --- All found errors ---

  ["Key 'blobs' was not defined.  Path: ''"]

  ERROR: Malformed "build" section in file:
  /home/yashi/src/zephyr-project/modules/hal/espressif/zephyr/module.yml

  <SchemaError: error code 2: Schema validation failed:

   - Key 'blobs' was not defined. Path: ''.: Path: '/'>

Call Stack (most recent call first):
  /home/yashi/src/zephyr-project/zephyr/cmake/modules/zephyr_default.cmake:121 (include)
  /home/yashi/src/zephyr-project/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:51 (include)
  /home/yashi/src/zephyr-project/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:76 (include_boilerplate)
  CMakeLists.txt:5 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /usr/bin/cmake -DWEST_PYTHON=/home/yashi/src/zephyr-project/.venv/bin/python -B/home/yashi/src/zephyr-project/zephyr/build -S/home/yashi/src/zephyr-project/zephyr/samples/hello_world -GNinja -DBOARD=esp32c3_devkitm

Note: To reproduce you don't west update before the build. Once you west update the error message is gone (but I still get other build errors with Zephyr SDK 0.16.0.

Failed to west update

I also found that you can't change v3.0.0 to v3.1.0 if you west build -b esp32c3_devkitm samples/hello_world between.

$ git checkout v3.0.0
$ west update
$ west build -b esp32c3_devkitm samples/hello_world
$ git checkout v3.1.0
$ west update
   :
=== updating hal_espressif (modules/hal/espressif):
error: The following untracked working tree files would be overwritten by checkout:
	components/bootloader/subproject/components/micro-ecc/micro-ecc/.gitignore
	components/bootloader/subproject/components/micro-ecc/micro-ecc/LICENSE.txt
	components/bootloader/subproject/components/micro-ecc/micro-ecc/README.md
           :
 ERROR: update failed for project hal_espressif

[rjmaris]

@yashi - I think you fairly (already) reproduced what I encounter actually. But tomorrow (in office) I'll check the espressif stuff. I however do not see git modules because outside of the zephyr folder there are no git repos.
I see that you 'simply' play around with west update where I'm hesitant now because of fear that if problems after an update occurs, a restore to previous version (git checkout) may yield build errors as well.

Regarding your final experiment: There are indeed circumstances where git takes care of not 'accidentally' overwriting untracked files. Normal behaviour is: when you add untracked files to commit, then after commit, these files are part of the versioning. If you then restore the previous commit, the files are not present as "untracked" (which I'd consider as exact step back - once I asked about this here) but these are simply no longer present (and as such, no overwrite error is reported upon checkout a later commit). I'd suggest that your git checkout v3.1.0 puts some 'new' versioned files, where west update treats the same files as unversioned?? during its process? There are a couple of questions on stackoverflow when you seek for the The following untracked working tree files would be overwritten by checkout search term. Perhaps one of them fits to this use case.

Just a sidenote: around 10 years ago I worked with a predecessor of buildroot, and never had such issues. both kernel and root file system (the latter also with externally originated code) had full (&consistent) git coverage.

[yashi]

I'm sure you need to learn a bit about git. The behaviour you asked in the question is exactly what git is designed to do. You ask to go back to the history where the file didn't exist, it deletes it. It replicates the state (files) in the working directory at the time you asked.

Good luck.

[rjmaris]

I regard myself as a quite experienced - years long - git user. That question I referred to is really old ;-)

[rjmaris]

Regarding espressif-stuff (as announced for today): I get another error as you, @yashi :

ERROR: malformed "build" section in file:
.../modules/hal/espressif/zephyr/module.ym
.../

This file has the date 23/11/2022. That was the date where I did west update. Hence I'd suggest that this file has been updated in the process of west update that does git pull (or git fetch & git merge) on zephyrproject/zephyr/ as well as unversioned updates in the modules folder.
Because of a few months project pause, I do not remember whether west update truly updates the local zephyr version to the upstream version (would today be 3.3.99 instead of 3.2.99).

Is there an option to do a sort of west update to a given zephyr version where the contents of the modules folder is updated matching to that version? Or is it your invocation of west update after doing git checkout that takes care of restoring modules/ stuff according to this version? In the latter case, there would be no reason to hesitate about invoking west update.

[rjmaris]

I already read these documents (partially). Still not easy for anyone who is still in the learning curve phase of Zephyr. Tomorrow, I'll make a plain backup of the zephyrproject folder and then do a range of actions regarding version changing in order to grasp the exact behaviour.

[rjmaris]

Well, meanwhile, I did the experiments, resulting in final trust that I can truly checkout desired (old or actual) git commits in zephyr folder while modules folder content is updated correspondingly to that zephyr commit (Multiple Repository Model). After doing west update from v3.1.0, build errors do not occur, also not yours related to espressif.

Part of my problem was the longer project pause, and due to the review of the getting started docs page, I concluded that west update also updates zephyr from upstream. Only trough reading Beyond the Getting Started Guide, section Keeping Zephyr updated. And also west update basics I got it.

I'd suggest to add a short hint to the Get Zephyr and install Python dependencies section, just after numbered list 5 (Get the Zephyr source code):

Note that west init also clones the git manifest repository. Refer to ... know about updating and checking out commits. This requires ordinary git commands, always followed by west update which takes care of synchronizing other dependencies.

[yashi]

west update also updates zephyr from upstream.

It does not. west update updates modules according to your manifest repository. But it doesn't touch the manifest repository. You must have git pull in it.

I'd suggest to add a short hint to the Get Zephyr and install Python dependencies section, just after numbered list 5 (Get the Zephyr source code):

Even if you run west init, it will error out if the specifed dir is already initialized.

$ west init zephyr-project
=== Initializing in /home/yashi/src/zephyr-project
FATAL ERROR: already initialized in /home/yashi/src/zephyr-project, aborting.

[rjmaris]

Sorry regarding the first citation. I meant that prior to start of this conversation. Just now I see that that assumption was not correct.
Of course would a new west init result in error (just like git init)

[rjmaris]

By the way: thanks for your help, yashi!