Custom partition table

[beegee-tokyo]

See http://docs.platformio.org/en/latest/platforms/espressif32.html#partition-tables

---

Configuration

Operating system: Windows 7 Ultimate 64bit Service Pack 1

PlatformIO Version (platformio --version): 3.5.0rc10

Visual Studio Code Version: 1.18.0

Atom Version: 1.23.1 x64

Description of problem

Partition sizes doesn't change even if default.csv is changed.

Steps to Reproduce

Following this issue I tried to change the partition sizes on my ESP32 Dev Module.
The changed default.csv:

Name, Type, SubType, Offset, Size, Flags

nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x150000,
app1, app, ota_1, 0x160000,0x150000,
eeprom, data, 0x99, 0x310000,0x1000,
spiffs, data, spiffs, 0x311000,640K,

But after uploading the partition sizes are still the default ones:

App partition table:
SubType 10 Address 0x010000 Size 0x140000 Encryption 0 Label app0
SubType 11 Address 0x150000 Size 0x140000 Encryption 0 Label app1
Data partition table:
SubType 02 Address 0x009000 Size 0x005000 Encryption 0 Label nvs
SubType 00 Address 0x00e000 Size 0x002000 Encryption 0 Label otadata
SubType 99 Address 0x290000 Size 0x001000 Encryption 0 Label eeprom
SubType 82 Address 0x291000 Size 0x16f000 Encryption 0 Label spiffs

Then I tried the exact same procedure on Arduino IDE and the partition size changed to:

App partition table:
Type: 00 SubType 10 Address 0x010000 Size 0x150000 Encryption 0 Label app0
Type: 00 SubType 11 Address 0x160000 Size 0x150000 Encryption 0 Label app1
Data partition table:
Type: 01 SubType 02 Address 0x009000 Size 0x005000 Encryption 0 Label nvs
ype: 01 SubType 00 Address 0x00E000 Size 0x002000 Encryption 0 Label otadata
Type: 01 SubType 99 Address 0x310000 Size 0x001000 Encryption 0 Label eeprom
Type: 01 SubType 82 Address 0x311000 Size 0x0A0000 Encryption 0 Label spiffs

Going back to PlatformIO, I uploaded my code again from there and the changed partition sizes are still valid.

It seems that PlatformIO doesn't change the partition size at all. Or is my method to change the partition sizes wrong?

[beegee-tokyo]

@ivankravets. I moved the issue as requested. I saw that you labelled the old issue as wontfix? Why? Is it not possible to change partition sizes on PlatformIO? I remember on ESP8266 it was possible for spiffs with flags in platformio.ini.

If not possible at the moment it is ok , But then it should be mentioned somewhere.

[ivankravets]

See https://github.com/espressif/arduino-esp32/blob/master/tools/platformio-build.py#L185

Do you modify the correct file? ~/.platformio/packages/framework-arduinoespressif32/tools/partitions/default.csv

[beegee-tokyo]

Yes, that is the file I changed. Content of default.csv at C:\Users\beegee.platformio\packages\framework-arduinoespressif32\tools\partitions:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1F0000,
app1,     app,  ota_1,   0x200000,0x200000,
#eeprom,   data, 0x99,    0x3F0000,0x1000,
#spiffs,   data, spiffs,  0x3F1000,0xF000,

which should give me 2 app partitions with the size of 0x1F0000 and 0x200000. But using the code

// Get Partitionsizes
size_t ul;
esp_partition_iterator_t _mypartiterator;
const esp_partition_t *_mypart;
ul = spi_flash_get_chip_size(); Serial.print("Flash chip size: "); Serial.println(ul);
Serial.println("Partition table:");
_mypartiterator = esp_partition_find(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, NULL);
if (_mypartiterator) {
	Serial.println("App Partition table:");
	do {
		_mypart = esp_partition_get(_mypartiterator);
		printf("Type: %02x SubType %02x Address 0x%06X Size 0x%06X Encryption %i Label %s\r\n", _mypart->type, _mypart->subtype, _mypart->address, _mypart->size, _mypart->encrypted, _mypart->label);
	} while (_mypartiterator = esp_partition_next(_mypartiterator));
}
esp_partition_iterator_release(_mypartiterator);
_mypartiterator = esp_partition_find(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, NULL);
if (_mypartiterator) {
	Serial.println("Data Partition table:");
	do {
		_mypart = esp_partition_get(_mypartiterator);
		printf("Type: %02x SubType %02x Address 0x%06X Size 0x%06X Encryption %i Label %s\r\n", _mypart->type, _mypart->subtype, _mypart->address, _mypart->size, _mypart->encrypted, _mypart->label);
	} while (_mypartiterator = esp_partition_next(_mypartiterator));
}
esp_partition_iterator_release(_mypartiterator);

gives me

Flash chip size: 4194304
Partition table:
App Partition table:
Type: 00 SubType 10 Address 0x010000 Size 0x1F0000 Encryption 0 Label app0
Type: 00 SubType 11 Address 0x200000 Size 0x1F0000 Encryption 0 Label app1
Data Partition table:
Type: 01 SubType 02 Address 0x009000 Size 0x005000 Encryption 0 Label nvs
Type: 01 SubType 00 Address 0x00E000 Size 0x002000 Encryption 0 Label otadata
Type: 01 SubType 99 Address 0x3F0000 Size 0x001000 Encryption 0 Label eeprom
Type: 01 SubType 82 Address 0x3F1000 Size 0x00F000 Encryption 0 Label spiffs

Found (maybe) the problem!

Playing around a little bit I found out that if I change D:\Arduino\hardware\espressif\esp32\tools\partitions\default.csv (Which should be for Arduino IDE and not for PlatformIO), the changes are reflected even I flash only from PlatformIO and NOT from Arduino IDE. Looks like platformio-build.py is pulling the default.csv from Arduino IDE package and not from PlatformIO IDE package.
Now the output is:

Flash chip size: 4194304
Partition table:
App Partition table:
Type: 00 SubType 10 Address 0x010000 Size 0x1F0000 Encryption 0 Label app0
Type: 00 SubType 11 Address 0x200000 Size 0x200000 Encryption 0 Label app1
Data Partition table:
Type: 01 SubType 02 Address 0x009000 Size 0x005000 Encryption 0 Label nvs
Type: 01 SubType 00 Address 0x00E000 Size 0x002000 Encryption 0 Label otadata

Any other info you need, let me know.

[ivankravets]

PlatformIO does not depend on Arduino IDE. It seems that you didn't clean project after modification. Please run pio run -t clean or remove .pioenvs folder manually.

[beegee-tokyo]

Tried both pio run -t clean and removing .pioenvs, still partition size didn't change.

Then (getting desperate) I tried to see if there is a difference if I flash through USB/serial or OTA. And there I got finally lucky.

Result:

Flashing through OTA with a changed partition table DOESN'T change the partitions.
Flashing through USB/serial with a changed partition table DOES change the partitions.
Any idea why?

[beegee-tokyo]

Info

I added an entry to the ESP32 Wiki I am creating: Change partition size

[theproxy]

@beegee-tokyo your wiki link doesnt work. I'm trying to figure this out under platformio in Visual Studio Code on OSX. I changed the default and it still giving me the program size is greater than maximum allowed error. I'm only trying to program via cable, so no OTA here.

[beegee-tokyo]

@theproxy I know, I need to change the Wiki. I submitted this issue because I run into the problem. First it seem to work on PlatformIO but now I see the partition size never change when using PlatformIO. @ivankravets why is it not working? Why is PlatformIO ignoring the changes even when flashing over USB? Is there something I missed to change?
As a temporary solution, I use ArduinoIDE to change the partition size and update the ESP only with OTA.

[theproxy]

Thank you for your reply. I got it working correctly in the Arduino IDE, but not in the platformio IDE. So, I'm developing in PlatformIO and programming it through Arduino IDE. Not ideal, but not sure how to fix at the moment. If anyone has an pointers I can help test any fixes.

[beegee-tokyo]

@theproxy
If you choose method one for the partition resizing, it works in PlatformIO. I updated the Wiki.

@ivankravets question regarding the partition tables
My approach is to use new defined board to have a specific partitioning only for this board. I think I created all necessary parts, because the code is compiled and uploaded and works.
My platformio.ini looks like:

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git#feature/stage
board = esp32maxapp
framework = arduino

The board esp32maxapp is defined in boards.txt with the entry

esp32maxapp.name=ESP32 Dev Module MaxAppPart
esp32maxapp.build.partitions=partitions
esp32maxapp.upload.maximum_size=1966080
esp32maxapp.build.board=ESP32_MAXAPP

Other entries are the same as for board ESP32 Dev Module

I created the esp32maxapp.json with the correct "maximum_size": 1966080.

But the partition size never changes with this setup (Flashing over USB).

When I run pio run --target upload >log.txt I can see in the log that actually default.csv is pulled and not partitions.csv.
gen_esp32part.py is called with:

"c:\users\beegee\.platformio\penv\scripts\python.exe" "C:\Users\BeeGee\.platformio\packages\framework-arduinoespressif32@src-537c58760dafe7fcc8a1d9bbcf00b6f6\tools\gen_esp32part.py" -q C:\users\beegee\.platformio\packages\framework-arduinoespressif32@src-537c58760dafe7fcc8a1d9bbcf00b6f6\tools\partitions\default.csv .pioenvs\esp32dev\partitions.bin

So it looks like esp32maxapp.build.partitions=partitions from boards.txt is never used.

Any ideas why always default.csv is used? What is wrong with my board definition? From where is gen_esp32part.py called? Couldn't find it.

[beegee-tokyo]

@ivankravets

FOUND MY PROBLEM!

I needed to add a "partitions" entry in my esp32maxapp.json in the boards folder.
Now it looks like:

{
  "build": {
    "core": "esp32",
    "extra_flags": "-DARDUINO_ESP32_DEV",
    "f_cpu": "240000000L",
    "f_flash": "40000000L",
    "flash_mode": "dio",
    "ldscript": "esp32_out.ld",
    "mcu": "esp32",
    "variant": "esp32maxapp",
   "partitions": "partitions"
  },
  "connectivity": [
    "wifi",
    "bluetooth",
    "ethernet",
    "can"
  ],
  "frameworks":  [
    "arduino",
    "espidf"
  ],
  "name": "Espressif ESP32 Dev Module MaxAppPart",
  "upload": {
    "flash_size": "4MB",
    "maximum_ram_size": 294912,
    "maximum_size": 1966080,
    "require_upload_port": true,
    "speed": 115200,
    "wait_for_upload_port": true
  },
  "url": "https://en.wikipedia.org/wiki/ESP32",
  "vendor": "Espressif"
}

And now the correct partition definition is loaded. HURRAY!
I will update this in the Wiki.

@theproxy
Could you verify if this works for you?

[theproxy]

@beegee-tokyo looks good so far! Thanks for putting the work in. Wish there was an easier way in platformio to change the parititon table, looks like if we could have a platformio.ini variable that provides a path to partition csv file, and setting maximum_size that would make it enough to just generate the command lines dynamically for every project.

Thanks for your help :)

[beegee-tokyo]

@ivankravets
What do you think about following idea for making it easier to create custom partition tables for projects:

1. Custom partition definition is inside project folder, e.g. named custompart.csv
2. platformio-build.py checks if custompart.csv exists in project folder
3. if exists, use custompart.csv to create the partition.bin
4. if not exist, use the partition table from the board definition

I tested it with following changes and it works without problems:
customparts.csv in the project folder (same level as platformio.ini)

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xE000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1D0000,
app1,     app,  ota_1,   0x1E0000,0x1D0000,
eeprom,   data, 0x99,    0x3D0000,0x21000,
spiffs,   data, spiffs,  0x3F1000,0xF000,

Changes inside platformio-build.py:

...
from os.path import isdir, join, isfile
...
#
# Generate partition table
#
checkpath = join(env.get("PROJECT_DIR"), "custompart.csv")
if isfile(checkpath):
    custompart = checkpath
else:
    custompart = join(FRAMEWORK_DIR, "tools", "partitions",
         "%s.csv" % env.BoardConfig().get("build.partitions", "default"))

partition_table = env.Command(
    join("$BUILD_DIR", "partitions.bin"),
    custompart,
#    join(FRAMEWORK_DIR, "tools", "partitions",
#         "%s.csv" % env.BoardConfig().get("build.partitions", "default")),
    env.VerboseAction('"$PYTHONEXE" "%s" -q $SOURCE $TARGET' % join(
        FRAMEWORK_DIR, "tools", "gen_esp32part.py"),
                      "Generating partitions $TARGET"))
env.Depends("$BUILD_DIR/$PROGNAME$PROGSUFFIX", partition_table)

If you think this idea is good and could be implemented, I can create a pull request for it.
Of course it would be nicer if a custom partition table could be defined in platformio.ini, but that would mean more changes in many other parts.

[BorisFR]

Why not just call the file "partition.csv" instead of "custompart.csv"?
Your changes are near ok but still have problem that we must change all platform/espressif32/xxx.json files :(

[beegee-tokyo]

@BorisFR on ArduinoIDE there seems now to be a solution, but not for PlatformIO :(

[BorisFR]

Surely, but PlatformIO + Visual Studio Code is awesome! I could not come back to Arduino IDE ;)

[beegee-tokyo]

@BorisFR same here, sticking to PlatformIO + Visual Studio Code as well, that's why I wrote this

[ivankravets]

Please re-test:

1. Switch to upstream version http://docs.platformio.org/en/latest/platforms/espressif32.html#stable-and-upstream-versions
2. pio update
3. Create a custom post script http://docs.platformio.org/en/latest/projectconf/advanced_scripting.html

platformio.ini

[env:myenv]
extra_scripts = post:extra_script.py

extra_script.py

Import("env")
env.Replace(PARTITION_TABLE_CSV="/path/to/custom/partition.csv")

Does it work?

[beegee-tokyo]

@ivankravets I am using
platform = https://github.com/platformio/platform-espressif32.git#feature/stage
I guess thats what you mean.

I will try the custom post script tomorrow, played around with them before, but does that solve the problem with max app size error while flashing? Is there another env. variable that can be changed in the script to get around that problem as well?

[BorisFR]

My test:
platformio.ini

[env:featheresp32]
extra_scripts = post:partitions.py

partitions.py

Import("env")
partitionsfile = join(env.get("PROJECT_DIR"), "partitions.csv")
if isfile(partitionsfile):
    print("Using special partition: {0}".format(partitionsfile))
    env.Replace(
        PARTITION_TABLE_CSV=partitionsfile)

Still having the problem for max size

text       data     bss     dec     hex filename
1271086  446376   59248 1776710  1b1c46 .pioenvs/featheresp32/firmware.elf
Error: The program size (1717462 bytes) is greater than maximum allowed (1310720 bytes)

I change the json file... but finally: it does not work. Partitions are still as default.

I have found that in platformio-build.py, PARTITION_TABLE_CSV is replace by env.BoardConfig().get("build.partitions", "default") (line 186), just before generating the partitions.bin file.

[BorisFR]

For the featheresp32.json file:
Create a folder "boards" in the project, put a copy of featheresp32.json and change the value of maximum_size
:)

[beegee-tokyo]

@ivankravets
I tried your proposal
extra_script.py

Import("env")
env.Replace(PARTITION_TABLE_CSV="B:\Projects\ESP32\ESP32-Weatherstation\custompart.csv")

platformio.ini

[env:esp32dev]
; platform = espressif32
platform = https://github.com/platformio/platform-espressif32.git#feature/stage
board = esp32dev
framework = arduino
upload_port = ESP32-Test-2481CE9C.local
extra_scripts = post:extra_script.py

Result:

Error: The program size (1879798 bytes) is greater than maximum allowed (1310720 bytes)

Only if I create a folder boards and put a changed esp32dev.json in there the upload works.

The problem with using only extra_script.py is that this function in pioupload.py

def CheckUploadSize(_, target, source, env):  # pylint: disable=W0613,W0621
    if "BOARD" not in env:
        return
    max_size = int(env.BoardConfig().get("upload.maximum_size", 0))
    if max_size == 0 or "SIZETOOL" not in env:
        return

    sysenv = environ.copy()
    sysenv['PATH'] = str(env['ENV']['PATH'])
    cmd = [
        env.subst("$SIZETOOL"), "-B",
        str(source[0] if isinstance(target[0], Alias) else target[0])
    ]
    result = util.exec_command(cmd, env=sysenv)
    if result['returncode'] != 0:
        return
    print result['out'].strip()

    line = result['out'].strip().splitlines()[1]
    values = [v.strip() for v in line.split("\t")]
    used_size = int(values[0]) + int(values[1])

    if used_size > max_size:
        sys.stderr.write("Error: The program size (%d bytes) is greater "
                         "than maximum allowed (%s bytes)\n" % (used_size,
                                                                max_size))
        env.Exit(1)

The line max_size = int(env.BoardConfig().get("upload.maximum_size", 0)) loads the max size from the boards configuration file.
I am not good at all in Python. Is there any possibility to change the value of env.BoardConfig().get("upload.maximum_size) programmatically?

[beegee-tokyo]

Found a way to do it with just the extra_script.py and a custom partition table in the project folder.
Tricky part was to get around the CheckUploadSize() function and still check that the app will fit into the app partition. To achieve this I

1. put the code to check the size into the extra_script.py and test against the new app partition size
2. change the SIZETOOL variable to point to an app that returns an error, that way the original CheckUploadSize() will stop checking

To get the extra_script.py to work correct, two values have to be updated manually
newPartSize = 1966080 which is the new app partition size
customPartFile = "custompart.csv" which is the name of the partition table which must be placed in the project folder (same level as src folder)

Here is the extra_script.py:

import sys
from os import environ
from platformio import util
from os.path import isfile, join

Import("env")

################################################################
#
# put your new app partition size to newPartSize
# put your custom partition table file name to customPartFile
#
################################################################
newPartSize = 1966080
customPartFile = "custompart.csv"

print "Checking app size against custom partition table!"
targetfirm = env.subst("$BUILD_DIR") + "\\" + env.subst("$PROGNAME") + ".elf"

sysenv = environ.copy()
sysenv['PATH'] = str(env['ENV']['PATH'])
cmd = [
  env.subst("$SIZETOOL"), "-B",
  targetfirm
]
result = util.exec_command(cmd, env=sysenv)
if result['returncode'] == 0:
  print result['out'].strip()

  line = result['out'].strip().splitlines()[1]
  values = [v.strip() for v in line.split("\t")]
  used_size = int(values[0]) + int(values[1])

  if used_size > newPartSize:
    sys.stderr.write("Error: The program size (%d bytes) is greater "
        "than maximum allowed (%s bytes)\n" % (used_size, newPartSize))
    env.Exit(1)

env.Replace(SIZETOOL=env.subst("$PIOHOME_DIR") + "\\penv\\scripts\\wheel.exe")
env.Replace(PARTITION_TABLE_CSV=env.subst("$PROJECT_DIR") + "\\" + customPartFile)

Tested only on Windows, not sure if it works on Linux or MacOS because of the line
env.Replace(SIZETOOL=env.subst("$PIOHOME_DIR") + "\\penv\\scripts\\wheel.exe")
which might need to be changed on other OS

I think it should be possible to actually search for the custom partition table from within the extra_script.py, get the app partition size and fill the two variables automatically. But for now I have enough of this Python language nightmare (which programming language fails if you intent the codes sometimes with spaces and sometimes with tabs :( ).

[ivankravets]

Anyone, please provide me a custom partition table which raises an error. I'll fix. Also, if you provide complete project which should work, I would be thankful!

[BorisFR]

My "ultimate" solution ;)

1/ create folder "boards" in the project root folder and copy the json file in it.
Edit and add in section "build"
"partitions": "partitions"

edit value (corresponding to your partition) in section "upload"
"maximum_size": 4063232,

2/ create your "partitions.csv" file in the project root folder.

3/ edit platformio.ini
add
extra_scripts = post:partitions.py

4/ add file partitions.py in the project root folder.
view joined files

featheresp32.json.txt
platformio.ini.txt
partitions.csv.txt
partitions.py.txt

[ivankravets]

@BorisFR could you try in extra script this ?

env.BoardConfig().manifest['upload']['maximum_size'] = 4063232

?

[BorisFR]

Just test: it does not work.
Test 1: With json file in my 'boards' folder, with default maximum_size
Test 2: I remove the json file from my folder 'boards'

in both cases, I have

Checking program size
text data bss dec hex filename
1281886 453096 59752 1794734 1b62ae
/Users/sfardoux/temp/tempplatformio/heltec_wifi_kit_32/firmware.elf
Error: The program size (1734982 bytes) is greater than maximum allowed (1310720 bytes)

[ivankravets]

1. Please upgrade PIO Core to the latest 3.5.3rc1 via pio upgrade --dev
2. Please use the latest upstream version (NOT STAGING, we need https://github.com/espressif/arduino-esp32/pull/1440/files)
3. YOU DO NOT NEED ANY EXTRA SCRIPTS. See updated docs http://docs.platformio.org/en/latest/platforms/espressif32.html#partition-tables

Please re-test and confirm. The final release of dev/platform is planned for the next week.

Thanks!

[beegee-tokyo]

@ivankravets
Followed the instructions step by step and confirm it works now!
Thank you!