RFC: New Testing Approach for Power Management Subsystem. #80989
Comments
gbarkadiusz
added
area: Power Management
RFC
|
@erwango @bjarki-andreasen @ceolin please take a look |
|
I agree! We need to add power usage testing! Some notes from my side:
I do have ideas for solutions for this as well but that's for later :) |
|
Can't help it, I would tie expected power measurement in unit tests to output from probe like: device output: test runner listens for over UART, This is scalable to probing for other things like temperatures, voltage RMS, etc etc. |
Regarding these pointed things. Another question to consider is whether we should focus on measuring the relative changes in power consumption between different power modes/states, or if we should aim to measure the specific values for each application state. For example, should we also measure the power usage of every peripheral available on a specific target? |
Yeah, with some probes it may not be possible to measure joules, but in these cases, the test suites should still indicate a number of expected jules, and the user should provide a "fixed" voltage for twister to use to calculate joules from the current measurement :)
Looking into it, twister is the tool used to manage tests, so it would be the place to add probes, alongside harnesses IMO :)
I would strongly encourage specific measurements :) Test suites are about as controlled an environment we can get, changes somewhere which unexpectedly enables/disables some peripheral should be caught as an anomaly. |
|
I noticed you updated the comment—thank you for that! I have another question regarding the testing setup. I saw that you're using the Thank you! PS: I can imagine using Serial/Uart to correspond zephyr app with python script to trigger the measurement. |
To send back an answer, we would likely need to enable shell support, and add a shell backend for testing, something that would look like: in the shell, where we would listen for The code could look like: excluding the implementation of the However, test suites currently run with only console enabled, since its simple, and all we need if the entire test suite is run on target. Enabling the shell takes quite a bit of ROM, and may make some targets incompatible with the test suite. For that reason, I would compromise to something like: Its likely less stable then the shell approach, but its an option :) The last option would be to have something completely custom similar to your suggestion with directly interacting with the UART, also a fine solution, but we may be limiting ourselves a bit compared to adding shell support |
|
I see your point. A Zephyr application can request measurements at any time using methods like TC_PRINT or shell commands. The challenge lies in capturing these requests and performing the measurements. We need a mechanism to handle such commands. As you mentioned, we could use the shell backend (e.g., UART) to send commands, but we still need something on the other end to process these requests. Additionally, I’m considering whether we should use the same serial port for communication with the Zephyr application and for verifying tests via Twister. It could make the Twister log messy. Thinking out loud: The |
This looks pretty nice :) |
|
@bjarki-andreasen I thought about the unit of measured value. We have several options. RMS (Root Mean Square) is a better metric for fluctuating signals as it considers the variation in current over time. Total Energy Instead of Power. The question is. How to calculate the expected value? Maybe it is not a good question at this time but let me treat this RFC as a brainstorm and place to share all ideas and warries. :) |
|
I don't think we should limit ourselves at all :) We could add support for power over time, min/max voltages, average current, rms current etc. Any test can specify whatever it needs, and will get its way if the probe supports the requested measurement. |
Sounds great ;) |
|
One little update: power is in watts :) |
|
measure with on-board adc is an options, and I from NXP would like to propose to use our mcu-link-pro as a standalone measure device to do this. see https://www.nxp.com/docs/en/user-manual/UM11673.pdf |
|
@bjarki-andreasen FYI. pytest: Unfortunately, I found a limitation in the onboard power monitor's firmware. Each 'start' command resets the target and starts the measurement, so it can't be triggered during the application. I need to find another solution. |
Absolutely we need to support it :) flashing, voltage and current measurement in one device, really neat :) Also, we could write a zephyr sample application which uses an on-board ADC for power measurement, our own "in-house" power probe firmware :) |
In that case, could the current measurement not be started and running constantly in the "background"? We don't need to actually start it only when we care about the samples right? |
I will add same support when the initial pr merged #80692 |
it is difficult, for power measurement, we need some additional circuits, now only lpcxpresso55s36 has such circuits on board |
That was my first thought. WIP. :) |
|
I performed tests using the Zephyr app and triggered commands by passing them to pytest. The integration worked quite well, but I encountered an issue— I didn’t know how to pass the result of the pytest test back to the Zephyr app. As a workaround, I used a method where Zephyr triggers a command, such as From my perspective, this approach seems valid, but I would appreciate your opinion on whether it's correct. |
I think the solution of simply terminating the test from pytest, not caring what the DUT does after the current measurement has failed, is a valid solution. What we loose doing it this way though is that we have to terminate the entire test suite if any single test fails. Fixing issues is often easier if one can see the set of tests that failed: with this output, its clearly a UART issue, not a larger issue affecting ADC. Compare this to followed by pytest failing, one has to chase the actual issue, is it with UART only, or more peripherals? not clear :) |
|
Forgot to add a solution here :) I found this https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/testsuite/ztest/src/ztest_shell.c so there is already a shell for test execution, surely we can extend it with test feedback :) |
Ztest Shell supports executing a specific test via the shell. However, there are still valid questions about how to trigger measurements within the test, collect the results, and determine whether the test passes or fails. The main challenge lies in how to communicate between Ztest and the measurement environment. Using Pytest Using Shell Using Console Subsystem |
I looked into using the shell for control, and came up with: creating a simple low overhead shell which is only used for feedback (no running test suites etc. like the full ztest shell). With this, the DUT prints "PROBE POWER ...", pytest does the measurements, and writes back Definately all for doing something way simpler using the console directly, primarily to save on ROM/RAM, but we would need to check what the actual difference in ROM/RAM is comparing the shell to the console solution to choose (the shell one seems optimal to me if ROM/RAM is OK) |
|
Have you tested this approach? but: |
No, I just wanted to show how to set up the shell, are you in a test case? to test it, do: The functions only work while a test case is actively running |
|
The problem lies in enabled shell. If it is enabled, the tests won't be executed. After boot zephyr, the app goes to shell immediately. The valid question is, Is it so important to trigger the measurement by zephyr app? |
|
Are you selecting |
|
Please, take a look. main.c prj.conf log: There is no output with TC_PRINT parts. I need to dive deeper into the shell implementation to gather knowledge why the output is moved somewhere to background. |
|
Quick update, It seems to work. pytest the log: |
|
This is a bit more graceful (though not perfect) for waiting on the shell: |
Would be nice to support multiple types since they address different things. In general total energy is probably what people want because it is directly related to battery life, but it is not super helpful for debugging power states consumption, for this RMS and pure average of current are better. I am wondering if RMS is the best measurement unit for testing sleep states, it seems that pure average of current is better for this. |
|
@gbarkadiusz thanks ! really appreciate it. |
It's my pleasure ;) |
|
@bjarki-andreasen @ceolin FYI. |
Its more likely PM is suspending the UART device in some low power state as defined by the SoC, hence the shell becomes unresponsive :) In any case, could the solution be to trigger the measurement while in a higher power state, set some timer to exit low power state, then enter low power state? measurement is done by pytest, then system exits low power state and UART is resumed, allowing the result to be passed to the shell :) |
Did you try to use the following configuration: If not working then this is a regression |
I tried to use this sample with the STM32L562E-DK, but it didn't work, or I might have done something wrong. The shell was not responding, |
Ok. Would you mind raising a dedicated issue ? |
|
Before that, I'd like to double check that sample with another board. But OFC I'll raise the issue with appropriate description. |
|
@erwango |
|
@erwango @bjarki-andreasen @ceolin, can we drive it to some initial version merged? We can rework/improve/extend this later. |
|
closing this as done |
Introduction
This proposal suggests a new method for testing the Power Management (PM) subsystem in Zephyr. The approach enhances the testing environment by adding an external power monitor to verify that the system reaches expected power states accurately. This improvement aims to make power management tests more reliable and to provide precise measurements for validating power state transitions.
Problem description
The Zephyr Power Management subsystem provides multiple sleep modes for efficient power usage. However, reliably confirming that a device enters each specific power state is challenging. For example, the
stm32l562e_dkboard supports multiple power states (k_cpu_idle,stop_0,stop_1,stop_2), but current test methods are limited in confirming that each state is reached as expected.Proposed change
This proposal involves integrating a
Power Monitorto track current consumption during testing. Using thestm32l562e_dkboard’s onboardPower Monitor, the test setup can measure current in each power state. Pytest will then calculate the Root Mean Square (RMS) value of each detected power state, making it easier to verify whether each state has been achieved. This allows for accurate state detection and validation based on measured current consumption.Detailed RFC
An external
Power Monitorwill be added to measure the current of the target device during tests. Thestm32l562e_dkboard, which has an onboardPower Monitor, will be used for testing. A script namedpwsh_stm32l562.pywill facilitate communication with the board, handling the data retrieval and current measurement.Proposed change (Detailed)
The
Power_Monitorabstract class serves as a base for implementing this functionality. It requires the following methods to be implemented:init_power_monitor(device_id: str) -> bool:Initializes the power monitor for a specific device ID.
measure_current(duration: int):Measures current over a given duration.
get_data(duration: int) -> List[float]:Retrieves an array of current measurements over the specified duration, returning values in amperes.
Setup and Configuration
Documentation:
STM32L562E-DK Evaluation Board: STMicroelectronics STM32L562E-DK
STM32L562E-DK User Manual: User Manual DM0063555
Power Monitor Firmware Documentation: Getting Started with PowerShield Firmware
Configuration: Based on the documentation provided, the
stm32l562e_dkboard should be configured as follows:Three tests demonstrate this approach, each designed to validate different aspects of power management:
pm.states: Verifies the transition to all available power states on
stm32l562e_dk.pm.residency_time: Tests the residency time of each power state.
pm.wakeup_timer: Tests the RTC alarm’s ability to wake the CPU from the lowest power state.
Each test calculates RMS current values for each detected state and compares them to expected values, which were manually set based on experimental data.
Test: pm.states
This test is designed to verify all available power states on the
stm32l562e_dkboard. The results from themeasure_currentfunction can be found in thepm_states.csvfile.The chart above illustrates the measured current consumption during the execution of the
pm_statestest. Key elements of the chart are highlighted as follows:Startup: The current peak observed at the beginning of the test.
Boot Delay: The delay associated with the additional configuration set to
CONFIG_BOOT_DELAY=500.k_cpu_idle: The low power state achieved using
k_msleep(Tms), whereTmsis shorter than the minimum residency time for thestop_0power state defined in the device tree source(.dts)file.Wake Up and Print Log: The target board wakes up between each power state and logs the corresponding messages.
stop_0, stop_1, stop_2, active_state: The measured current consumption for each respective power state.
During the test, pytest detects each step and calculates the RMS (Root Mean Square) current for every detected state. The test then compares the calculated RMS values against the expected values, which have been determined manually through experimental observation
To Reproduce:
stm32l562e_dkboard according to the instructions provided above.Power Monitor: e.g.,/dev/ttyACM0or/dev/serial/by-id/usb-STMicroelectronics_PowerShield__Virtual_ComPort_in_FS_Mode__FFFFFFFEFFFF-if00Target: e.g.,/dev/ttyACM1or/dev/serial/by-id/usb-STMicroelectronics_STLINK-V3_004##############39-if02Power Monitorand update the<path_to_power_monitor>in thetests/subsys/pm/power_states/testcase.yamlfile under thepytest_argssection as follows:Pytest log:
Test: pm.residency_time
This test evaluates the residency time of different power states on the
stm32l562e_dkboard. The sequence of states is as follows:The chart above illustrates the measured current consumption during the execution of the
pm_residency_timetest. Key elements of the chart are highlighted as follows:Startup: The current peak observed at the beginning of the test.
Boot Delay: The delay associated with the additional configuration set to
CONFIG_BOOT_DELAY=500.k_cpu_idle: This represents the kernel idle state, where the sleep time is shorter than the residency time required for the
stop_0state.stop_0: In this state, there is sufficient sleep time for the system to enter
stop_0.A_stop_1: Here, the sleep time is again shorter than the residency time required to transition to the
stop_1state.B_stop_1: This state allows for enough sleep time to transition into
stop_1.A_stop_2: In this iteration, the sleep time is shorter than the residency time required to transition to the
stop_2state.B_stop_2: The system has enough sleep time to enter the
stop_2state.active_state: This state indicates that the application is running in an infinite loop, representing the
active stateof the system.Wake Up and Print Log: The target board wakes up between each power state and logs the corresponding messages.
To Reproduce:
(1-2) As previously.
3. Copy the path of the
Power Monitorand update the<path_to_power_monitor>in thetests/subsys/pm/power_residency_time/testcase.yamlfile under thepytest_argssection as follows:Pytest Log:
Test: pm.wakeup_timer
In this test, the
stm32l562e_dktarget board is configured to set the RTC Alarm for 1 second before entering the lowest power state by calling thek_msleep()function. The sequence of operations is as follows:-The RTC Alarm is set for 1 second.
-The board enters a low-power state, effectively minimizing power consumption.
-After the alarm triggers, the RTC Alarm callback wakes up the CPU.
-The CPU then enters an anti-sleeping loop to prevent returning to sleep mode.
-Throughout this process, the power monitor continuously measures the power consumption of the board.
This test helps verify the functionality of the wake-up timer and assesses the power consumption during low-power operation.
The chart above illustrates the measured current consumption during the execution of the
pm_wakeup_timertest. The following key elements are highlighted in the chart:Startup: The current peak observed at the beginning of the test.
Boot Delay: The delay associated with the additional configuration set to
CONFIG_BOOT_DELAY=500.Go Sleep: This phase represents when the alarm is set, and the board enters a low-power state by calling the
k_msleep()function.Stop 2: The measured current consumption during the
stop_2power state.Alarm Interrupt: This indicates the moment the RTC alarm interrupt wakes the CPU, transitioning into an infinite loop.
Active State: This state signifies that the application is running in an infinite loop, representing the active operational state of the system.
To Reproduce:
(1-2) As previously.
3. Copy the path of the
Power Monitorand update the<path_to_power_monitor>in thetests/subsys/pm/power_wakeup_timer/testcase.yamlfile under thepytest_argssection as follows:Pytest_log:
All the tests mentioned above are available in the following pull request: #80692. This pull request is currently in draft status and will remain so until the RFC is reviewed and applied.
Dependencies
This change requires access to the
stm32l562e_dkboard and an external (onboarded)Power Monitor. It may also necessitate modification of test scripts if additionalpower monitorsare used.The text was updated successfully, but these errors were encountered: