Feature request: Pre-encrypted OTA support from ESP-IDF.

[astorun]

Hi there.

Is there any possibility to implement this feature as it is already being a part of esp32?

https://github.com/espressif/esp-idf/tree/master/examples/system/ota/pre_encrypted_ota

That will be great addition to this library!

[tobozo]

hi,

since this is a library for Arduino C++ layer, and using C functions from esp-idf would create a versions nightmare along with a never ending tech debt, direct use of esp-idf internals is reduced to its strict minimum.

luckily arduino-esp32 core 3.x.x is bundled with support for AES OTA updates, so it's just a matter of calling Update.setupCrypt() from your setup().

• See the HTTP_Client_AES_OTA_Update example from the Update library

[plystdr]

Good idea!

Sadly, it is not part of the platformio version of esp-idf. I am using arduino framework as a component.

I am trying to import it to my project, however it is refusing to compile with bunch of idf version mismatch errors...

[tobozo]

platformio espressif support is (in a business sense) at a halt, but there's an ongoing community effort to support more recent versions of esp-idf and arduino framework:

https://github.com/pioarduino/platform-espressif32
https://github.com/pioarduino/esp-idf
https://github.com/pioarduino/registry

it is based on idf version 5.1.4 + arduino core 3.0.3 so you'll have to match these in your project, and obviously use a modified platformio.ini

[plystdr]

Thanks @tobozo . We tried to compile our code with pioarduino, however there is a bug that is needed to be fixed which causing build error when converting elf file to bin due to long file path names. So, until they figure out how to fix that, it is also not viable solution. I dont understand why espressif stopped supporting platformio. Stupid decision IMO.

[plystdr]

So I am able to run our code with pioarduino and trying to implement Update.setupCrypt(), however I am lost about implementation part since I am using ESP32-S3 and not sure how to set it up to work with the ESP32FOTA.

It stuck during firmware download progress if I try to download encrypted firmware bin file.

I do also opened an ticket regarding with issue in esp32-arduino repository.(espressif/arduino-esp32#10155)

Any suggestions will be helpful!

[ 52927][V][esp32fota.cpp:1086] getHTTPStream(): This server supports resume!
[ 52934][D][esp32fota.cpp:1151] getHTTPStream(): updateSize : 2377152, contentType: application/octet-stream
[ 52948][D][esp32fota.cpp:576] execOTA(): compression: disabled
[ 52955][D][Updater.cpp:140] begin(): OTA Partition: app1
[ 52961][I][esp32fota.cpp:625] execOTA(): Begin Firmware OTA. This may take 2 - 5 mins to complete. Things might be quiet for a while.. Patience!
[ 52983][D][Updater.cpp:348] _writeBuffer(): Decrypting OTA Image
[ 52989][W][Updater.cpp:292] _decryptBuffer(): AES key not set
[ 52995][E][esp32fota.cpp:640] execOTA(): Written only : 0/2377152 Premature end of stream?
[ 53003][D][HTTPClient.cpp:373] disconnect(): still data in buffer (17), clean up.

[ 53011][D][HTTPClient.cpp:378] disconnect(): tcp keep open for reuse

[tobozo]

[ 52989][W][Updater.cpp:292] _decryptBuffer(): AES key not set

looks like setupCrypt() is ignored, if possible, can you show the source of the main sketch?

I've double checked the espressif example, although they seem to call Update.setupCrypt() after Update.begin(), the values set by setupCrypt() shouldn't be reset.

I can't test anything before next monday but if you're ready to play with esp32fota.cpp, this is the line where Update.setupCrypt() should be inserted to match with espressif example:

    // BEGIN INSERT
    if (!Update.setupCrypt(OTA_KEY, OTA_ADDRESS, OTA_CFG, OTA_MODE)) {
      Serial.println("Update.setupCrypt failed!");
      return false;
    }
    // END INSERT

    // Some activity may appear in the Serial monitor during the update (depends on Update.onProgress)
    size_t written = F_writeStream();

please let me know if this gives results and I'll think about a way to integrate that in esp32fota more elegantly

[plystdr]

@tobozo Thanks for reopening the issue. I managed to get it working just a moment ago.

Only thing you need to be careful is it is required to encrypt the plain firmware binaries using the OTA key generated for Update.setupCrypt, not with the flash encryption keys that is burned to efuse of the ESP32. I used the following command to encrypt the binaries for OTA.

espsecure.py encrypt_flash_data --keyfile firmware_keys\sample_key.bin --flash_crypt_conf 0x0 --address 0x10000 --output firmware_encrypted/firmware_OTA_encrypted.bin firmware_bin/firmware.bin

Basically you need to create 2 different encryption keys.

• One for flash encryption to encrypt plaintext binaries on ESP32 itself to be burned in efuse.
• Another one to encrypt OTA files to be included in firmware for decrypting with Update.setupCrypt.

If understand it correctly, it works like this: When OTA update begins, the file downloaded will be decrypted using OTA key and then encrypted again with efuse key into ESP32 flash memory in realtime as it is downloaded from the server. So when ESP32 reboots, the firmware is already encrypted, so, no need to wait for encryption again, nice feature!

I suggest to test it further and create a separate example demo for the next release of this plugin, since a lot of people is asking for this feature for ages, or maybe including this feature into ESP32FOTA itself by calling it from config section.

/**
   esp32 firmware OTA

   Purpose: Perform an OTA update to both firmware and filesystem from binaries located
			on a webserver (HTTPS) while using progmem to check for certificate validity
*/

#include "FS.h"
#include <LittleFS.h>
#include <esp32fota.h>

#include "root_ca_cert.h"

const bool check_signature = false;
const bool disable_security = false;

esp32FOTA FOTA;

CryptoMemAsset *MyRootCA = new CryptoMemAsset("Root CA", root_ca_cert, strlen(root_ca_cert) + 1);
// CryptoMemAsset *MyRSAKey = new CryptoMemAsset("RSA Public Key", pub_key, strlen(pub_key)+1 );

int firmwareUpdateProgress = 0; // Global variable to store update progress

const uint8_t  OTA_KEY[32] = { 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, \
                               0x38, 0x39, 0x20, 0x74, 0x68, 0x69, 0x73, 0x20, \
                               0x61, 0x20, 0x73, 0x69, 0x6d, 0x70, 0x6c, 0x65, \
                               0x74, 0x65, 0x73, 0x74, 0x20, 0x6b, 0x65, 0x79 };

void my_progress_callback(size_t progress, size_t size)
{
	// Calculate the progress as a percentage
	firmwareUpdateProgress = static_cast<int>((static_cast<float>(progress) / size) * 100);

	if (progress == size || progress == 0)
	{
		// Firmware update is complete or hasn't started yet
	}
}

void fotaSetup()
{
	// Set up the progress callback
	FOTA.setProgressCb(my_progress_callback);
	FOTA.setExtraHTTPHeader("XXX", "XXX");

	Update.setupCrypt(OTA_KEY, 0, 0x0, U_AES_DECRYPT_AUTO);

	{
		auto cfg = FOTA.getConfig();
		cfg.name = (char *)FW_NAME;
		cfg.manifest_url = (char *)FW_ADDR;
		cfg.sem = SemverClass(firmware_version_major, firmware_version_minor, firmware_version_patch);
		cfg.check_sig = check_signature;
		cfg.unsafe = disable_security;
		cfg.root_ca = MyRootCA;
		// cfg.pub_key      = MyRSAKey;
		FOTA.setConfig(cfg);
	}
	// FOTA.printConfig();
}

void fwUpdateCheck()
{
	if (wifiConnected)
	{
		fwUpdatedNeeded = FOTA.execHTTPcheck(true);
		log_e("HTTP code returned: (httpCode=%i)", fwUpdatedNeeded);
		fwUpdateChecked = true;
		fwUpdateChecking = false;
	}
}

void fwUpdateAutoCheckInit()
{
	if (wifiConnected)
	{
		if (fwUpdateAutoCheck)
		{
			if (fwUpdateCheckBootTicker.active())
			{
				fwUpdateCheckBootTicker.detach();
			}

			fwUpdateChecking = true;
			fwUpdateChecked = false;

			fwUpdateStartCheckTicker.once(3, fwUpdateCheck);
		}
	}
}

void fwUpdateExecute()
{
	if (fwUpdatedNeeded == 200 || fwUpdatedNeeded == 301)
	{
		FOTA.execOTA();
	}
}

void OTAUpdateTask(void *pvParameters)
{
	// Perform the firmware update
	fwUpdateExecute();

	// Introduce a delay to yield the task
	vTaskDelay(pdMS_TO_TICKS(100)); // Adjust the delay duration as needed

	// The task can be deleted once the update is complete
	vTaskDelete(NULL);
}

void startOTAUpdateTask()
{
	// Create the OTA update task and start it
	xTaskCreate(OTAUpdateTask, "OTAUpdateTask", 8192, NULL, 0, NULL);
}

[plystdr]

BTW I modified the ESP32FOTA library in my favor to handle HTTP response codes better for my application. So you can ignore the following section in my code.

if (fwUpdatedNeeded == 200 || fwUpdatedNeeded == 301)
	{
		FOTA.execOTA();
	}

[tobozo]

Update.setupCrypt() working from outside the esp32FOTA class is good news 👍 I'll close the issue now