Skip to content

Glibc test suite for a target without native GNU toolchain

Jump to bottom
Pavel Kozlov edited this page Oct 13, 2022 · 38 revisions

Table of Contents

Introduction

Glibc test-suite was created in assumption that it will be run natively (i.e. on the same host where it was built) and even though now there's a way to run tests on remote target we still need to mimic "native" execution. For that we need:

  • Use the same user on both host and target. I.e. whichever user runs cross-test-ssh.sh on host must be specified for SSH connection in the wrapper. Otherwise some tests will fail, for example see posix/globtest.

  • Host's file-system must be accessible from target. Moreover paths must match exactly on both devices (i.e. /home/username/glibc-testsuite in both cases must be available and contain exactly the same data) because tests are being built on the fly and immediately get executed via SSH. Thus we need to mount host's filesystem with glibc testsuite on target.

    We have at least 2 options here: NFS or SSHFS.

    SSHFS may sporadically fail with symbol lookup error: ./csu/test-as-const-tcb-offsets: undefined symbol: __libc_start_main, version GLIBC_2.32. That said NFS is a recommended way.

Additionally since we're are talking about remote execution (host drives the whole process) we need to have SSH server on target so that host may login onto target and execute tests or special scripts.

Also note that some tests require much more memory than ~700 Mb we currently may use in DDR on HSDK. Fortunately this could be easily worked-around with use of SWAP on either micro SD-card or USB flash-drive.

Setting things up

We use Buildroot for building toolchain, Linux kernel and root file-system because it allows to build everything except Glibc tests itself in one run.

Buildroot configuration

To change default settings use Buildroot's configuration utility such as menuconfig, xconfig, nconfig or gconfig.

The following options must be enabled:

  • BR2_INSTALL_LIBSTDCPP - Some Glibc tests require C++ cross-compiler and in its absence won't be built and executed.
  • BR2_PACKAGE_OPENSSH - Host uses SSH connection for execution of tests on the target.
  • BR2_PACKAGE_NTP and BR2_PACKAGE_NTP_NTPDATE - Used to setup correct date and time on target so that user passwords are not treated as expired.
  • BR2_TOOLCHAIN_GLIBC_GCONV_LIBS_COPY - Copies gconv libraries on target, that's required for some tests (see iconv/test-iconvconfig)
  • BR2_PACKAGE_NFS_UTILS - To be able to mount NFS.
  • BR2_PACKAGE_SSHFS - [Optional] If SSHFS will be used for mounting host's file-system corresponding utilities must be built and installed on target.

For QEMU please use snps_archs38_haps_defconfig with options above and also:

  • BR2_LINUX_KERNEL_DEFCONFIG - set to haps_hs (disable SMP).
  • BR2_TARGET_ROOTFS_INITRAMFS - build-in initial ramfs to kernel.
  • BR2_SYSTEM_DHCP="eth0" - Because we are going to run QEMU user network with DHCP enabled.
  • BR2_PACKAGE_HAVEGED - [Optional] Used to collect lower-quality entropy for linux kernel random generator which speeds up rng initialization and ssh daemon startup. It is also recommended to pre-generate sshd keys and provide it to buildroot via overlay.

Linux kernel configuration

If Linux kernel gets built by Buildroot it's possible to change kernel's configuration as usual but starting configuration utility by a special Buildroot's target this way:

make linux-menuconfig

The following options should be set:

  • CONFIG_NFS_FS - Required to mount NFS share on target
  • CONFIG_NAMESPACES, CONFIG_UTS_NS, CONFIG_IPC_NS, CONFIG_USER_NS, CONFIG_PID_NS, CONFIG_NET_NS - Required for tests which are executed in container
  • CONFIG_CGROUPS, CONFIG_MEMCG - Recommended to be set with CONFIG_USER_NS
  • CONFIG_IPV6 - needed for some networking tests
  • CONFIG_FUSE_FS - [Optional] required if host's file-system will be mounted via SSHFS
  • CONFIG_SWAP - [Optional] required if SWAP partition will be used to extend available memory

Additional configuration for QEMU:

  • Apply following kernel patch to restrict instruction set.

    CONFIG_ARC_TUNE_MCPU="-mcpu=archs" - Since commit it is possible to specify required mcpu with kernel configs in order to restrict instruction set

Glibc rebuild

Even though that looks pretty strange but there's a need to rebuild Glibc from scratch after everything was already built by Buildroot. That's because Glibc's configuration scripts check availability of C++ compiler and if it is not found some tests later-on won't be built and run. And essentially at the time of initial Glibc compilation C++ compiler is not yet built even if it is "enabled" in Buildroot with mentioned above BR2_INSTALL_LIBSTDCPP.

Anyways this is as simple as: make glibc-dirclean && make

NFS server setup on host

  1. Configure exports (in /etc/exports):

    sudo echo "/home/$USERNAME *(rw,all_squash,anonuid=`id -u`,anongid=`id -g`)" > /etc/exports

    Note all_squash and anonuid/anongid are required for mapping target's user to you user on host.

    Otherwise use of no_root_squash is very dangerous as target's root will be treated as a root on your host and it:

    • Might be really harmful to your system
    • Target will generate files on host's filesystem owned by host's root user so if glibc's testsuite is executed by normal user test results won't be accessible for it. And idea to run testsuite by local root is very dangerous so please don't do that!

    For QEMU user network support it is also required to have insecure oiption set. QEMU user network is NATed so NFS should allow any source ports.

  2. Start NFS server

    sudo systemctl start nfs

  3. Open NFS and RPC-BIND ports in firewall

    firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --reload

Target board setup

Now when everything is built on host and ready for execution we need to start built Linux kernel on target and do some preparatory steps:

QEMU configuration

If you want to use QEMU as a target for glibc testing use the following command:

qemu-system-arc -cpu archs -display none -kernel "${BUILDROOT_TOP}/output/image/vmlinux" \
                -netdev "user,id=net0,hostfwd=tcp:127.0.0.1:${GUEST_SSH_PORT}-:22" \
                -device virtio-net-device,netdev=net0 --global cpu.freq_hz=50000000 -nographic

Where:

  • BUILDROOT_TOP - path to buildroot top directory.
  • GUEST_SSH_PORT - port to be used on host to connect to guest ssh. So QEMU guest can be reached via ssh -p "$GUEST_SSH_PORT" 127.0.0.1.
  • --global cpu.freq_hz=50000000 - needed for haps_hs config to have properly worked clock.

If DHCP is configured on Buildroot configuration step host can be reached from guest via 10.0.2.2 (see QEMU user network).

QEMU configuration with TAP interface

For some tests bidirectional ethernet interface is essential. Following steps will allow to run QEMU with the virtual TAP layer with the host:

# Create and setup interface
sudo ip tuntap add tap1 mode tap
sudo ip addr add 10.42.0.1/24 dev tap1
sudo ip link set tap1 up

The tun kernel module should be loaded. Then you can run QEMU with the tap1 interface, created on previous step:

qemu-system-arc -cpu archs -M virt -display none -kernel ${BUILDROOT_TOP}/output/image/vmlinux \
                -netdev tap,id=net0,ifname=tap1,script=no,downscript=no \
                -device virtio-net-device,netdev=net0 \
                --global cpu.freq_hz=50000000 -nographic

If DHCP is configured on Buildroot configuration you can run DHCP server for tap1 interface or set IP address for your target manually:

ifconfig eth0 10.42.0.100
ping 10.42.0.1

Generic preparations

  1. Make sure Ethernet interface is up and running:

    udhcpc

  2. Set correct date and time (for that we need to have working internet connection already):

    ntpdate

  3. Create home folder, otherwise on user creation its home folder won't appear:

    mkdir /home

  4. Create a user. Note it's required to have on target the same user as on host so let's create one:

    adduser username

  5. Install public key on target Now to let host to connect to the target's SSH server without asking user for password (otherwise user will need to enter password for all 5000+ tests) it's required to install host user's public key. We'll do it with help of ssh-copy-id on host:

    ssh-copy-id -i ~/.ssh/id_rsa.pub 10.42.0.100

    where 10.42.0.100 is target's IP-address. For QEMU use port GUEST_SSH_PORT and IP 127.0.0.1.

Mount host's filesystem on target

Create mounting point

Keep in mind that all paths must be the same on both host and target. The most obvious mounting point would be Glibc's source folder.

Note if Glibc was built as a part of Buildroot then it will be an output of realpath output/build/glibc-xxx command:

mkdir -p /__full_path_to_glibc_source__

Mount host's file-system

  • Via NFS (recommended)

    mount -t nfs 10.42.0.1:/__full_path_to_glibc_source__ /__full_path_to_glibc_source__ -o nolock

  • Via SSHFS (if NFS-share is not available)

    sshfs -o idmap=user,allow_other [email protected]:/__full_path_to_glibc_source__ /__full_path_to_glibc_source__

Note 10.42.0.1 here is host's IP-address. For QEMU user network it will be 10.0.2.2.

Glibc test suite execution with QEMU system emulation

As already mentioned the whole process is controlled from host. It's important to note that tests are being built right before execution.

Before execution of tests we need to cd into build folder of glibc. Note if Glibc was built as a part of Buildroot then we need to cd into output/build/glibc-xxx/build/.

Then start execution of entire test-suite:

make test-wrapper='/__full_path_to_glibc_source__/scripts/cross-test-ssh.sh 10.42.0.100' check ; \ # This target builds test env and runs main testpack
make test-wrapper='/__full_path_to_glibc_source__/scripts/cross-test-ssh.sh 10.42.0.100' xcheck # This target runs extra tests

To pass specific SSH port to cross-test-ssh.sh use --ssh ${SSH_CMD}. Where SSH_CMD should not contain spaces because it causes some tests to fail. Wrapper can be used to bypass this behavior, for example for port 5522:

$ cat ssh-p5522
#!/bin/sh

exec ssh -p 5522 "$@"

Glibc test suite execution with QEMU user mode emulation

QEMU allows to run executable compiled for one CPU on another CPU. You can run program compiled for ARCv2 CPU by following command:

qemu-arc -R 3G -L $ARC_SYSROOT_PATH -cpu archs arc-program-name

where ARC_SYSROOT_PATH is a path to the ARC target environment.

To use QEMU user mode emulation in purpose of glibc testing we will utilize the binfmt Linux kernel capability. The arc-gnu-toolchain repository contains scripts/qemu-binfmt-conf.sh script which register new binary types (arc64, arc32, arc) in binfmt_misc file-system interface. Run script and check new binfmt records:

sudo sh ./scripts/qemu-binfmt-conf.sh --qemu-suffix -load
ls /proc/sys/fs/binfmt_misc
cat /proc/sys/fs/binfmt_misc/qemu-arc

enabled
interpreter /usr/local/bin/qemu-arc-load
flags:
offset 0
magic 7f454c460201010000000000000000000200fd00
mask fffffffffffefe00fffffffffffffffffeffffff

As a result, every time you run a program compiled for ARCv2 CPU on your host Linux the /usr/local/bin/qemu-arc-load interpreter will be called. Now we have to prepare the qemu-arc-load wrapper to run QEMU user mode emulation. It will look like:

#!/bin/sh

qemu-arc -R 3G -L $ARC_SYSROOT_PATH -cpu archs $@

where ARC_SYSROOT_PATH should point to the sysroot directory with ARC target components. Put the qemu-arc-load wrapper to the /usr/local/bin directory. Before execution of tests, we need to cd to glibc build folder, then it will be enough to print make check to run tests. Unfortunately, some tests may get stuck, or unexpected test timeout can occur, so it will be a good idea to use the timeout program as the test wrapper and set the TIMEOUTFACTOR environment variable:

TIMEOUTFACTOR=10 make check test-wrapper='timeout -k 15m 15m'

Using QEMU user mode is another way to run glibc tests. It's less accurate and can cause more test failures than in system emulation mode. But this method is easy to use, and it is free from restrictions of full emulated ARC system.

Useful tips

On target

Use Google DNS server on target

Some proxy DNS servers might not resolve names to IPv6 addresses, see posix/tst-getaddrinfo4.

For that execute: echo "nameserver 8.8.8.8" > /etc/resolv.conf

USe swap partition on micro SD-card or USB flash-drive

Since some tests require more than 1Gb of memory on platforms with limited amount of RAM swap partition might be a viable option.

  1. Make sure CONFIG_SWAP is enabled in Linux kernel configuration
  2. Create "Linux swap" partition on micro SD-card or USB flash-drive (for simplicity do it on host system)
  3. Insert SD-card or USB flash-drive with swap partition into HSDK and enable swap: 
    mkswap /dev/mmcblk0p3
swapon -a -e /dev/mmcblk0p3
    where /dev/mmcblk0p3 is swap partition. In that example MMC's 3-rd partition was used. If one wants to use USB for that use /dev/sda3 for its third partition.

On host

Add timeout for frozen tests in the executor script

Change the last line of the file scripts/cross-test-ssh.sh to:

timeout --foreground 20m $ssh "$host" /bin/sh -c "$full_command"

Information about tests being executed

These scripts should be executed in Glibc's build folder.

  • Get all currently available results: 
    find . -type f -name '*.test-result' -exec cat {} \; > results.txt
  • Count all passed tests: 
    find . -type f -name '*.test-result' -exec cat {} \; | grep PASS | wc -l
  • Count all failed tests: 
    find . -type f -name '*.test-result' -exec cat {} \; | grep FAIL | wc -l
Clone this wiki locally