mbedTLS ITS is not thread safe

[alxelax]

Describe the bug
mbedTLS has PSA ITS (Internal Trusted Storage) implementation that is based on posix file API. It doesn't consider ability to run multiple processes\treads with mbedTLS instances. As a consequence, all instances compete for the single file that reproduces ITS functionality.
It impacts on BLE mesh tests behavior. All mesh bsim tests are run in parallel environment. Tests those check mesh functionality related to storing\restoring\usage keys with PSA_KEY_LIFETIME_PERSISTENT fail.

Tests pass successfully if disable parallel environment.
Finally, it causes the situation when it is not possible to run bsim tests for mesh with PSA in Zephyr CI.

• Platform: nrf52_bsim (this is posix based platform)
• Potential workaround: disable parallel environment for bsim tests (this will increase time of BabbleSim step in CI dramatically)

To Reproduce
Steps to reproduce the behavior:

1. config build ble mesh with mbedTLS PSA.

diff --git a/subsys/bluetooth/mesh/Kconfig b/subsys/bluetooth/mesh/Kconfig
index 4f2438b9bb..7813311f05 100644
--- a/subsys/bluetooth/mesh/Kconfig
+++ b/subsys/bluetooth/mesh/Kconfig
@@ -15,7 +15,7 @@ if BT_MESH
 
 choice BT_MESH_CRYPTO_LIB
        prompt "Crypto library selection for mesh security"
-       default BT_MESH_USES_TINYCRYPT
+       default BT_MESH_USES_MBEDTLS_PSA
 
 config BT_MESH_USES_TINYCRYPT
        bool "Use TinyCrypt"

2. build mesh bsim tests
   All steps and links how to deploy bsim environment is possible to find in Zephyr documentation

/zephyr/tests/bsim/bluetooth/mesh$ bash compile.sh

4. run bsim tests

/zephyr/tests/bsim$ bash run_parallel.sh

7. See error

If remove parallel environment, all tests pass

diff --git a/tests/bsim/run_parallel.sh b/tests/bsim/run_parallel.sh
index 395dc7f608..d8bebe3dbd 100755
--- a/tests/bsim/run_parallel.sh
+++ b/tests/bsim/run_parallel.sh
@@ -70,26 +70,26 @@ export CLEAN_XML="sed -E -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' \
 
 echo -n "" > $tmp_res_file
 
-if [ `command -v parallel` ]; then
-  parallel '
-  echo "<testcase name=\"{}\" time=\"0\">"
-  {} $@ &> {#}.log
-  if [ $? -ne 0 ]; then
-    (>&2 echo -e "\e[91m{} FAILED\e[39m")
-    (>&2 cat {#}.log)
-    echo "<failure message=\"failed\" type=\"failure\">"
-    cat {#}.log | eval $CLEAN_XML
-    echo "</failure>"
-    rm {#}.log
-    echo "</testcase>"
-    exit 1
-  else
-    (>&2 echo -e "{} PASSED")
-    rm {#}.log
-    echo "</testcase>"
-  fi
-  ' ::: $all_cases >> $tmp_res_file ; err=$?
-else #fallback in case parallel is not installed
+#if [ `command -v parallel` ]; then
+#  parallel '
+#  echo "<testcase name=\"{}\" time=\"0\">"
+#  {} $@ &> {#}.log
+#  if [ $? -ne 0 ]; then
+#    (>&2 echo -e "\e[91m{} FAILED\e[39m")
+#    (>&2 cat {#}.log)
+#    echo "<failure message=\"failed\" type=\"failure\">"
+#    cat {#}.log | eval $CLEAN_XML
+#    echo "</failure>"
+#    rm {#}.log
+#    echo "</testcase>"
+#    exit 1
+#  else
+#    (>&2 echo -e "{} PASSED")
+#    rm {#}.log
+#    echo "</testcase>"
+#  fi
+#  ' ::: $all_cases >> $tmp_res_file ; err=$?
+#else #fallback in case parallel is not installed
   for case in $all_cases; do
     echo "<testcase name=\"$case\" time=\"0\">" >> $tmp_res_file
     $case $@ &> $i.log
@@ -107,7 +107,7 @@ else #fallback in case parallel is not installed
     rm $i.log
     let i=i+1
   done
-fi
+#fi
 echo -e "</testsuite>\n</testsuites>\n" >> $tmp_res_file
 dur=$(($SECONDS - $start))
 echo -e "<testsuites>\n<testsuite errors=\"0\" failures=\"$err\"\

Expected behavior
All tests should pass without errors in parallel environment

Impact
Impossible to add ble mesh bsim tests with mbedTLS PSA crypto in Zephyr's CI

Environment (please complete the following information):

• OS: Linux
• Toolchain: zephyr sdk 0.16.1
• Commit SHA: 68589ca

Additional context
Alternative solution to fix the issue is implementation PSA ITS based on Zephyr's settings subsystem. If mbedTLS supports ITS based on settings, then each simulated device will include itself its own flash area. There is no competition for the common source.

[jgl-meta]

@PavelVPV have you seen this?

[PavelVPV]

@jgl-meta this is I guess issue with mbedTLS, not with Bluetooth mesh. As an mbedTLS user I can't use different filenames for ITS in mbedTLS.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[alxelax]

I removed Stale label since the issue is actual. mbedtls with built-in ITS is not possible to use in BabbleSim tests.
The temporary workaround based on --allow-multiple-definition linker option and ITS emulator was implemented: https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/bsim/bluetooth/mesh/CMakeLists.txt#L87-L97

If mbedtls PSA is going to be the main security library in Zephyr then it should be compatible with Zephyr's code base including tests approaches.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.