CivetWeb HTTP Sample - reduce RAM usage #28376

KozhinovAlexander · 2020-09-14T21:47:29Z

Reduce RAM memory usage by the CivetWeb HTTP sample

This PR solves #21179

KozhinovAlexander · 2020-09-14T21:47:53Z

@erwango Please review and test it.

tbursztyka · 2020-09-15T06:28:15Z

add "Fixes #21179" into your commit message

erwango · 2020-09-15T09:40:19Z

@Nukersson I'm not able to build samples/net/sockets/civetweb:

/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c: In function 'websocket_client_thread':
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17311:19: error: storage size of 'sa' isn't known
17311 |  struct sigaction sa;
      |                   ^~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17315:18: error: 'SIG_IGN' undeclared (first use in this function)
17315 |  sa.sa_handler = SIG_IGN;
      |                  ^~~~~~~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17315:18: note: each undeclared identifier is reported only once for each function it appears in
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17316:2: warning: implicit declaration of function 'sigaction' [-Wimplicit-function-declaration]
17316 |  sigaction(SIGPIPE, &sa, NULL);
      |  ^~~~~~~~~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17316:12: error: 'SIGPIPE' undeclared (first use in this function); did you mean 'EPIPE'?
17316 |  sigaction(SIGPIPE, &sa, NULL);
      |            ^~~~~~~
      |            EPIPE
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17311:19: warning: unused variable 'sa' [-Wunused-variable]
17311 |  struct sigaction sa;
      |                   ^~

This isn't linked with your PR, nor the board. I'm seeing this also on master and on other board.
I'm using v2.4.0-rc1-118-gdc79b37167 (latest master), and run west update.

Do I miss something ?

KozhinovAlexander · 2020-09-15T09:50:51Z

@Nukersson I'm not able to build samples/net/sockets/civetweb:

/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c: In function 'websocket_client_thread':
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17311:19: error: storage size of 'sa' isn't known
17311 |  struct sigaction sa;
      |                   ^~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17315:18: error: 'SIG_IGN' undeclared (first use in this function)
17315 |  sa.sa_handler = SIG_IGN;
      |                  ^~~~~~~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17315:18: note: each undeclared identifier is reported only once for each function it appears in
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17316:2: warning: implicit declaration of function 'sigaction' [-Wimplicit-function-declaration]
17316 |  sigaction(SIGPIPE, &sa, NULL);
      |  ^~~~~~~~~
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17316:12: error: 'SIGPIPE' undeclared (first use in this function); did you mean 'EPIPE'?
17316 |  sigaction(SIGPIPE, &sa, NULL);
      |            ^~~~~~~
      |            EPIPE
/local/mcu/zephyrproject/modules/lib/civetweb/src/civetweb.c:17311:19: warning: unused variable 'sa' [-Wunused-variable]
17311 |  struct sigaction sa;
      |                   ^~

This isn't linked with your PR, nor the board. I'm seeing this also on master and on other board.
I'm using v2.4.0-rc1-118-gdc79b37167 (latest master), and run west update.

Do I miss something ?

Thanks for the testing. Yes, you're missing latest civetweb. Just add it to west.yaml file and provide west update. This change comes with my other PR (see #28323). Sorry, I did not expected this error in this case. If you want we can skype - then I will give you more explanations. Please let me know your test result of this PR.

KozhinovAlexander · 2020-09-15T10:30:44Z

@erwango I've just added updated west.yml file and corresponding PR for it. Thus, compiling this PR should work.

KozhinovAlexander · 2020-09-15T10:38:09Z

add "Fixes #21179" into your commit message

Done

samples/net/sockets/civetweb/src/main.c

erwango · 2020-09-15T13:33:21Z

I'm not able to test, but I can confirm the reduction of memory consumption.
Before:

Memory region         Used Size  Region Size  %age Used
           FLASH:      241096 B         2 MB     11.50%
             CCM:          0 GB        64 KB      0.00%
            SRAM:      236159 B       192 KB    120.12%
        IDT_LIST:         216 B         2 KB     10.55

After:

Memory region         Used Size  Region Size  %age Used
           FLASH:      125880 B         2 MB      6.00%
             CCM:          0 GB        64 KB      0.00%
            SRAM:       93295 B       192 KB     47.45%
        IDT_LIST:         216 B         2 KB     10.55%

KozhinovAlexander · 2020-09-15T13:51:59Z

I'm not able to test, but I can confirm the reduction of memory consumption.
Before:

Memory region         Used Size  Region Size  %age Used
           FLASH:      241096 B         2 MB     11.50%
             CCM:          0 GB        64 KB      0.00%
            SRAM:      236159 B       192 KB    120.12%
        IDT_LIST:         216 B         2 KB     10.55

After:

Memory region         Used Size  Region Size  %age Used
           FLASH:      125880 B         2 MB      6.00%
             CCM:          0 GB        64 KB      0.00%
            SRAM:       93295 B       192 KB     47.45%
        IDT_LIST:         216 B         2 KB     10.55%

Under not able do you mean - the server does not starts or you have some other problems?

erwango · 2020-09-15T13:54:16Z

Under not able do you mean - the server does not starts or you have some other problems?

Server doesn't start indeed, but I'm at home with limited test possibilities, so I haven't checked much more.
I should be able to test more at the office tomorrow.

KozhinovAlexander · 2020-09-15T18:00:38Z

Under not able do you mean - the server does not starts or you have some other problems?

Server doesn't start indeed, but I'm at home with limited test possibilities, so I haven't checked much more.
I should be able to test more at the office tomorrow.

Probably following advice would be helpful for you:

My observations shows, that civetweb server samples needed too much time to start up (sometimes about > 30 sec.). This behavior can be seen on current master's branch civetweb implementation and also on implementation of this PR.
Following thing can help: Ping the IP of your server. And as soon as the response comes - go to the server over your web browser.

P.S.: The fastest browser on my MacBook was Chrome. The Safari needed much more time to load pages from embedded CivetWeb server on my NUCLEO-H745ZI-Q board running @480MHz. Mozilla were not properly tested by me.

erwango · 2020-09-16T15:09:21Z

Under not able do you mean - the server does not starts or you have some other problems?

Server doesn't start indeed, but I'm at home with limited test possibilities, so I haven't checked much more.
I should be able to test more at the office tomorrow.

Probably following advice would be helpful for you:

My observations shows, that civetweb server samples needed too much time to start up (sometimes about > 30 sec.). This behavior can be seen on current master's branch civetweb implementation and also on implementation of this PR.
Following thing can help: Ping the IP of your server. And as soon as the response comes - go to the server over your web browser.

P.S.: The fastest browser on my MacBook was Chrome. The Safari needed much more time to load pages from embedded CivetWeb server on my NUCLEO-H745ZI-Q board running @480MHz. Mozilla were not properly tested by me.

Had some more tries today, but can't connect to the board anyhow. I don't blame the samples though, as heavily controlled corporate machines might now help.

KozhinovAlexander · 2020-09-16T15:19:29Z

Under not able do you mean - the server does not starts or you have some other problems?

Server doesn't start indeed, but I'm at home with limited test possibilities, so I haven't checked much more.
I should be able to test more at the office tomorrow.

Probably following advice would be helpful for you:
My observations shows, that civetweb server samples needed too much time to start up (sometimes about > 30 sec.). This behavior can be seen on current master's branch civetweb implementation and also on implementation of this PR.
Following thing can help: Ping the IP of your server. And as soon as the response comes - go to the server over your web browser.
P.S.: The fastest browser on my MacBook was Chrome. The Safari needed much more time to load pages from embedded CivetWeb server on my NUCLEO-H745ZI-Q board running @480MHz. Mozilla were not properly tested by me.

Had some more tries today, but can't connect to the board anyhow. I don't blame the samples though, as heavily controlled corporate machines might now help.

Have you tested this PR on dev board connected to your local machine's Eth port, s.t. you can provide proper IPv4 to your board? If you want I can give you support over TeamViewer or so. Or we can Skype anytime tomorrow or today evening.

P.S.: My board runs connected to my local network router.

KozhinovAlexander · 2020-09-16T15:20:42Z

@lochej Can you test the sample also please?

lochej · 2020-09-16T16:22:11Z

@lochej Can you test the sample also please?

@Nukersson I performed a west update then fetched your branch locally to test.
Connected my board directly to my PC: Nucleo_H743ZI (no st-link/custom) + Killer ethernet E4200

I changed the IP config to accomodate for my static IP local ethernet link settings:

CONFIG_NET_CONFIG_MY_IPV4_ADDR="192.0.2.1"
CONFIG_NET_CONFIG_MY_IPV4_NETMASK="255.255.255.0"
CONFIG_NET_CONFIG_MY_IPV4_GW="192.0.2.2"
CONFIG_NET_CONFIG_PEER_IPV4_ADDR="192.0.2.2"

So far so good, the sample runs fine and displays the CivetWeb sample for each routes correctly:

Also network performance is pretty good:

ping -i 0.2 192.0.2.1 -c 30
--- 192.0.2.1 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 5877ms
rtt min/avg/max/mdev = 0.535/0.894/1.145/0.128 ms

Edit:
CivetServer is up and accessible 5ish seconds after releasing reset to the board. In fact, I press the refresh button of Chrome and release the reset at the same time. Note that I'm connected directly to the board

erwango · 2020-09-17T07:04:44Z

@Nukersson, I finally made it on nucleo_f429zi.
Runs successfully! And perf are not that bad neither (for a lower end platform vs H7)

--- 10.0.0.111 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29382ms
rtt min/avg/max/mdev = 0.402/0.563/0.729/0.085 ms

lochej

LGTM, Works on Nucleo-H743ZI

erwango · 2020-09-17T07:30:13Z

@Nukersson Thanks to this PR. I'm adding the V2.5.0 milestone as this doesn't exactly fit the criteria to be merged in V2.4.0 release stabilization period (not strictly a "bug" fix).
It will be integrated at V2.5.0 merge window opening

KozhinovAlexander · 2020-09-17T07:55:29Z

@Nukersson Thanks to this PR. I'm adding the V2.5.0 milestone as this doesn't exactly fit the criteria to be merged in V2.4.0 release stabilization period (not strictly a "bug" fix).
It will be integrated at V2.5.0 merge window opening

I am very happy. Thank you for testing it also. Yes, I am agree - it should be merged into v2.5.

KozhinovAlexander · 2020-09-18T07:38:49Z

@Nukersson, I finally made it on nucleo_f429zi.
Runs successfully! And perf are not that bad neither (for a lower end platform vs H7)
--- 10.0.0.111 ping statistics ---
30 packets transmitted, 30 received, 0% packet loss, time 29382ms
rtt min/avg/max/mdev = 0.402/0.563/0.729/0.085 ms

��@erwango Have you observed ping between system startup and ping answer on your F4 board? How much time does it took, if yes?

erwango · 2020-09-18T07:40:49Z

@erwango Have you observed ping between system startup and ping answer on your F4 board? How much time does it took, if yes?

No, I haven't. If finding spare time today, I'll have a new try, but do not rely on it.

samples/net/sockets/civetweb/prj.conf

samples/net/sockets/civetweb/src/main.c

KozhinovAlexander · 2020-09-28T17:20:44Z

@pfalcon @tbursztyka Please provide review.

adjust configuration so that civetweb uses less memory fixes zephyrproject-rtos#21179 Signed-off-by: Alexander Kozhinov <[email protected]>

KozhinovAlexander requested review from jukkar, pfalcon and tbursztyka as code owners September 14, 2020 21:47

github-actions bot added area: Networking area: Samples Samples labels Sep 14, 2020

KozhinovAlexander requested review from alexanderwachter and erwango September 14, 2020 21:48

KozhinovAlexander self-assigned this Sep 14, 2020

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from 59656b0 to 9aadfbd Compare September 15, 2020 10:27

github-actions bot added the area: Modules label Sep 15, 2020

KozhinovAlexander mentioned this pull request Sep 15, 2020

Enable Civetweb WebSocket Support #28395

Merged

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from 9aadfbd to 6295ab2 Compare September 15, 2020 10:37

jukkar reviewed Sep 15, 2020

View reviewed changes

samples/net/sockets/civetweb/src/main.c Outdated Show resolved Hide resolved

KozhinovAlexander requested a review from jukkar September 15, 2020 17:46

KozhinovAlexander requested a review from lochej September 16, 2020 12:09

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from 6295ab2 to e353a82 Compare September 16, 2020 13:24

KozhinovAlexander requested a review from lochej September 16, 2020 19:09

erwango approved these changes Sep 17, 2020

View reviewed changes

lochej approved these changes Sep 17, 2020

View reviewed changes

erwango added this to the v2.5.0 milestone Sep 17, 2020

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from a39a1aa to 0549fcc Compare September 17, 2020 20:54

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch 4 times, most recently from 19264aa to 13cce24 Compare September 24, 2020 11:51

jukkar requested changes Sep 28, 2020

View reviewed changes

samples/net/sockets/civetweb/prj.conf Show resolved Hide resolved

samples/net/sockets/civetweb/src/main.c Outdated Show resolved Hide resolved

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from 13cce24 to 906d7ed Compare September 29, 2020 07:53

KozhinovAlexander requested a review from jukkar September 29, 2020 07:53

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch 2 times, most recently from 996a9eb to fedc4c2 Compare September 29, 2020 07:59

jukkar approved these changes Sep 29, 2020

View reviewed changes

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch 2 times, most recently from abef31b to 5fa3152 Compare September 29, 2020 11:44

KozhinovAlexander requested a review from MaureenHelm September 29, 2020 20:37

samples: net: civetweb: reduce RAM usage

2092095

adjust configuration so that civetweb uses less memory fixes zephyrproject-rtos#21179 Signed-off-by: Alexander Kozhinov <[email protected]>

KozhinovAlexander force-pushed the civetweb_reduce_ram_usage branch from 5fa3152 to 2092095 Compare September 29, 2020 20:47

KozhinovAlexander added the area: civetweb label Sep 30, 2020

jukkar merged commit c28ffbb into zephyrproject-rtos:master Sep 30, 2020

KozhinovAlexander deleted the civetweb_reduce_ram_usage branch September 30, 2020 11:35

abrodkin mentioned this pull request Dec 26, 2021

Boards: ARC: QEMU: Enable networking SLIP networking #41458

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CivetWeb HTTP Sample - reduce RAM usage #28376

CivetWeb HTTP Sample - reduce RAM usage #28376

KozhinovAlexander commented Sep 14, 2020

KozhinovAlexander commented Sep 14, 2020

tbursztyka commented Sep 15, 2020

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

KozhinovAlexander commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

erwango commented Sep 16, 2020

KozhinovAlexander commented Sep 16, 2020

KozhinovAlexander commented Sep 16, 2020

lochej commented Sep 16, 2020 •

edited

Loading

erwango commented Sep 17, 2020

lochej left a comment

erwango commented Sep 17, 2020 •

edited

Loading

KozhinovAlexander commented Sep 17, 2020

KozhinovAlexander commented Sep 18, 2020

erwango commented Sep 18, 2020

KozhinovAlexander commented Sep 28, 2020

CivetWeb HTTP Sample - reduce RAM usage #28376

CivetWeb HTTP Sample - reduce RAM usage #28376

Conversation

KozhinovAlexander commented Sep 14, 2020

KozhinovAlexander commented Sep 14, 2020

tbursztyka commented Sep 15, 2020

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 • edited Loading

KozhinovAlexander commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 • edited Loading

erwango commented Sep 15, 2020

KozhinovAlexander commented Sep 15, 2020 • edited Loading

erwango commented Sep 16, 2020

KozhinovAlexander commented Sep 16, 2020

KozhinovAlexander commented Sep 16, 2020

lochej commented Sep 16, 2020 • edited Loading

erwango commented Sep 17, 2020

lochej left a comment

Choose a reason for hiding this comment

erwango commented Sep 17, 2020 • edited Loading

KozhinovAlexander commented Sep 17, 2020

KozhinovAlexander commented Sep 18, 2020

erwango commented Sep 18, 2020

KozhinovAlexander commented Sep 28, 2020

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

KozhinovAlexander commented Sep 15, 2020 •

edited

Loading

lochej commented Sep 16, 2020 •

edited

Loading

erwango commented Sep 17, 2020 •

edited

Loading