allow board variant to has its own custom partition.csv and bootloader.bin

[hathach]

This PR is essentially adding same function like #1, but targeted the v4.2 instead of esp32s2 branch

This PR allow variant to has its own custom patition.csv and 2nd stage bootloader.bin. If those don't exist, it will use default sdk partition.csv and sdk/bootloader. This is required for board that have pre-loaded tinyuf2 bootloader and allow to switching back and forth between circuitpython and arduino without overwrite the user fatfs and/or the bootloader + tinyuf2

It looks good however, but require modifying the platform.txt, maybe espressif would have other ideas on how to implement this.

[me-no-dev]

the reason why esptool is defined this way is because the path gets replaced when packaging for the board manager. Can not accept this in the main repo :) it will break the process.

[hathach]

ah, sorry, I didn't know about that. is trying to remove duplication code, maybe we could have an variable for upload e.g

tools.esptool_py.upload.pattern_args=--chip {build.mcu} --port "{serial.port}" --baud {upload.speed} {upload.flags} --before default_reset --after hard_reset write_flash -z --flash_mode {build.flash_mode} --flash_freq {build.flash_freq} --flash_size detect 0xe000 "{runtime.platform.path}/tools/partitions/boot_app0.bin" 0x1000 "{build.path}/{build.project_name}.bootloader.bin" 0x10000 "{build.path}/{build.project_name}.bin" 0x8000 "{build.path}/{build.project_name}.partitions.bin"

tools.esptool_py.upload.pattern="{path}/{cmd}" {pattern_args}
tools.esptool_py.upload.pattern.linux=python "{path}/{cmd}" {pattern_args}

Though it is totally up to you, I would revert it if that is your preference, since you are the maintainer.

[me-no-dev]

sure :) any optimisation and removing of duplication is great :) as long as that line stays the same so it will be properly replaced

[me-no-dev]

I would have implemented it the same way :) maybe not a bad idea to add the option to have a bootloader in the sketch folder as well.

I have one question though... who will take care of updating those bootloaders when the other ones get updated? Is all of their source public? If so, we should add them to the lib builder repo (4.2 branch) then they will get rebuilt every time that libs are updated. This happens mostly by cron action.

[hathach]

I would have implemented it the same way :) maybe not a bad idea to add the option to have a bootloader in the sketch folder as well.

I will update the PR to also include the command to check for the bootloader.bin in the sketch folder as well.

I have one question though... who will take care of updating those bootloaders when the other ones get updated? Is all of their source public? If so, we should add them to the lib builder repo (4.2 branch) then they will get rebuilt every time that libs are updated. This happens mostly by cron action.

I am glad you asked, Adafruit is working on an opensource cross-platfrom uf2 bootloader based on tinyusb called tinyuf2 which also feature esp32s2 https://github.com/adafruit/tinyuf2/tree/master/ports/esp32s2, it contains a slight modification of IDF bootloader to load a factory app (which can be used to update ota partition) and/or set an indicator to user.
https://github.com/adafruit/tinyuf2/blob/master/ports/esp32s2/components/bootloader/subproject/main/bootloader_start.c#L156
For most circuitpython-compatible board, they are very likely to come pre-flashed with that tinyuf2 project (bootloader + factory app). Since it is very specific to each board, I think it is best to leave it to its maker to test and make an PR to update the bootlaoder.bin and/or partion.cvs to avoid update bin that isn't go through hand-on testing. They are rarely changed anyway

Note: if you are new to uf2, the format is introduced by Microsoft for their makecode https://github.com/microsoft/uf2 and used as main method to update firmware for Adafruit Circuitpython https://learn.adafruit.com/adafruit-feather-m0-express-designed-for-circuit-python-circuitpython/uf2-bootloader-details . It shows up as MSC drive and user can just drag and drop the uf2 file to perform DFU. Although Espressif IDF is easy to install by developer standard, it is still a problem to install it on hundreds of computers in a school or institution which requires training, setup for teachers and students. For Circuitpython, they can just download the pre-built uf2 binary and drag &drop, actually the firmware update is rare, most of the modification is the python script :) . Though some of the user may want to switch back and forth between Arduino and Circuipython (many do !), therefore it is nice to have the custom partition and bootloader for board/variant to make the switch seamlessly and user doesn't need to re-flash tinyuf2 or loose any files on fatfs.

[hathach]

Updated esptool using the path and pattern_arg to remove duplication. and support sketch custom's bootloader.bin

PS: @me-no-dev I don't quite understand how the CI failed, it say the source list and cmakelist don't match, but I haven't changed either of those. Would you mind giving me some hints on how to fix it :)

[hathach]

@me-no-dev I have another question, it would be nice to also have a custom factory.bin as well for variant/sketch. Though I am not entirely sure how we should specify the factory.bin address. Are you willing merge such as PR. I guess the most difficult work probably is having bash/batch script to parse the partition.csv for the factory address :)

[ladyada]

this is tested and ready to merge!

[me-no-dev]

@hathach yes I also looked at the fail and did not get the reason. It's a script that makes sure that CMakeLists.txt includes all source files in the framework (so that it works as IDF component).

On the factory.bin I have to think about it a bit.

[me-no-dev]

For factory to work in any way, the following string will have to be added to the flash command, based on a board option: 0x2d0000 "{build.path}/factory.bin". That whole thing will have to be defined to a variable like {build.esptool_xtra} which to be added to the flash command and I am not sure that the second variable will be expanded ({build.path}).

[me-no-dev]

@hathach there is something wrong with this branch. Did you start it off of master? CI is downloading old toolchain and probably have other old stuff also. CMakeLists.txt and the parser have both changed. Seems you probably have only one changed here :) so the test fails.

[ladyada]

we are working off of 4.2 branch

[hathach]

@hathach there is something wrong with this branch. Did you start it off of master? CI is downloading old toolchain and probably have other old stuff also. CMakeLists.txt and the parser have both changed. Seems you probably have only one changed here :) so the test fails.

It is based on the release/v4.2 , maybe it has some hipcup, we will figure it out when submitting PR to official repo :)

For factory to work in any way, the following string will have to be added to the flash command, based on a board option: 0x2d0000 "{build.path}/factory.bin". That whole thing will have to be defined to a variable like {build.esptool_xtra} which to be added to the flash command and I am not sure that the second variable will be expanded ({build.path}).

ah thank you very much for the suggestion, {build.esptool_extra} is a great solution, though it probably needs to be added for all boards as empty much like {build.define} which is kind of annoying. I wish Arduino parser would be easier to work with :) Will test it out and submit an PR for this soon enough