How to install debug packages to mcr.microsoft.com/mssql/server:2017-latest containers?

[hlovdal]

While the whole problem has a lot to it, this issue is only asking about adding debug packages (as a start), so this should be very easy to answer.

Background

Currently I am experiencing a problem where /opt/mssql/bin/sqlservr receives SIGABRT on startup in a container based on mcr.microsoft.com/mssql/server:2017-latest (a03c94c3147d) and fills up the /var/opt/mssql/log directory with core dump files.

I am using a custom CMD override, and while sqlservr dies immediately there are a few other commands started by the script and by running

pstree -p | grep sqlcmd | tr -d '|^(){}`[a-z]' | tr - ' ' | xargs echo kill -STOP

when the container starts up I will eventually be lucky and make the container stop its startup procedure so that I can then use docker exec -ti to log into it and start debugging.

Debug results

Running gdb /opt/mssql/bin/sqlservr gives the following result.

(gdb) run
...
[New Thread 0x7fc1ad619700 (LWP 7540)]
[New Thread 0x7fc1ad5f8700 (LWP 7541)]
[New Thread 0x7fc1ad5d7700 (LWP 7542)]
[New Thread 0x7fc1ad5b6700 (LWP 7543)]
[New Thread 0x7fc1ad595700 (LWP 7544)]
[New Thread 0x7fc1ad574700 (LWP 7545)]
[New Thread 0x7fc1ad553700 (LWP 7546)]
[New Thread 0x7fc1ad532700 (LWP 7547)]
[New Thread 0x7fc1ad511700 (LWP 7548)]
[New Thread 0x7fc1ad4f0700 (LWP 7549)]
[New Thread 0x7fc1ad4cf700 (LWP 7550)]
[New Thread 0x7fc1ad4ae700 (LWP 7551)]
[Thread 0x7fc1ad4ae700 (LWP 7551) exited]

Thread 33 "Wt-132" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fc1ad5d7700 (LWP 7542)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fc1b817d7f1 in __GI_abort () at abort.c:79
#2  0x0000559245659aa1 in ?? ()
#3  0x000055924565758c in ?? ()
#4  0x0000559245646920 in ?? ()
#5  0x00005592455e59ce in ?? ()
#6  0x00005592455e6309 in ?? ()
#7  0x000055924565df13 in ?? ()
#8  0x00000004002ce6f0 in ?? ()
#9  0x00000004002ce6c0 in ?? ()
#10 0x00003fff00000000 in ?? ()
#11 0x0000000000000202 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) quit

This __GI_abort function is some system library (glibc?) related function, and the corresponding debug symbols are not installed. For the normal case this is perfectly fine to make the containers small, but just now for debugging I want them, hence this issue.

Additional info

Running strace -s 9000 -f -ff -o output -v /opt/mssql/bin/sqlservr did not produce anything that stood out as an obvious error, all files just had a line with reference to SIGABRT at the end.

Running the program manually gives the following:

root@fc89ebb883e1:/tmp# /opt/mssql/bin/sqlservr
This program has encountered a fatal error and cannot continue running at Wed Dec 21 14:33:46 2022
The following diagnostic information is available:

         Reason: 0x00000002
     Parameters: 0x4802cc3d0
                  [0] 0x4802cfd78
                  [1] 0xffffffffc0000001
    Stack Trace:
                 file://package4/windows/system32/sqlpal.dll+0x000000000000E16F
                 file://package4/windows/system32/sqlpal.dll+0x000000000000CCB1
                 file://package4/windows/system32/sqlpal.dll+0x000000000004EBF5
                 file://package4/windows/system32/sqlpal.dll+0x000000000004ED9B
                 file://package4/windows/system32/sqlpal.dll+0x0000000000003D1F
                 file://package4/windows/system32/sqlpal.dll+0x00000000002023A8
                 file:///Windows/SYSTEM32/samsrv.dll+0x000000000008063A
                 file:///Windows/SYSTEM32/samsrv.dll+0x00000000000654DF
                 file:///Windows/SYSTEM32/lsasrv.dll+0x000000000005867E
                 file:///Windows/SYSTEM32/lsasrv.dll+0x0000000000059379
                 file:///windows/System32/lsass.exe+0x0000000000001607
                 file:///windows/System32/lsass.exe+0x0000000000001591
                 file:///windows/System32/lsass.exe+0x0000000000004108
                 file:///Windows/SYSTEM32/KERNEL32.DLL+0x0000000000014414
                 file:///windows/system32/ntdll.dll+0x0000000000075541
        Modules:
                 file://package4/windows/system32/sqlpal.dll=ECC222CFEB7E222B20132F798A3B66E91
                 file:///Windows/SYSTEM32/samsrv.dll=F3107457F0FA2561E69E1E553283AB261
                 file:///Windows/SYSTEM32/lsasrv.dll=7CAAD248E49A41C78196EBC37187D01A2
                 file:///windows/System32/lsass.exe=391101F2646434821AEA86EBD9C9ECFF1
                 file:///Windows/SYSTEM32/KERNEL32.DLL=C715300FB2664729A6126A3F591E6F302
                 file:///windows/system32/ntdll.dll=45137AA3F9814512B3123991067EEE6E2
        Process: 14039 - sqlservr
         Thread: 14072 (application thread 0x84)
    Instance Id: d4e730f9-cea2-489d-bcac-ed9a19d01ea1
       Crash Id: 4839a3e8-60a3-475c-a472-a17642643a5b
    Build stamp: 3c2e95f7e74e98f89adf9d81b95f7db186a2c023cefbbbf35f46d814ac714d72
   Distribution: Ubuntu 18.04.6 LTS
     Processors: 2
   Total Memory: 16482447360 bytes
      Timestamp: Wed Dec 21 14:33:46 2022
     Last errno: 2
Last errno text: No such file or directory
Capturing a dump of 14039
Successfully captured dump: /var/opt/mssql/log/core.sqlservr.12_21_2022_14_33_46.14039
Executing: /opt/mssql/bin/handle-crash.sh with parameters
     handle-crash.sh
     /opt/mssql/bin/sqlservr
     14039
     /opt/mssql/bin
     /var/opt/mssql/log/

     d4e730f9-cea2-489d-bcac-ed9a19d01ea1
     4839a3e8-60a3-475c-a472-a17642643a5b

     /var/opt/mssql/log/core.sqlservr.12_21_2022_14_33_46.14039

Ubuntu 18.04.6 LTS
Capturing core dump and information to /var/opt/mssql/log...
dmesg: read kernel buffer failed: Operation not permitted
/usr/bin/timeout: failed to run command '/bin/journalctl': No such file or directory
/usr/bin/timeout: failed to run command '/bin/journalctl': No such file or directory
Wed Dec 21 14:33:49 Europe 2022 Capturing program information
Dump already generated: /var/opt/mssql/log/core.sqlservr.12_21_2022_14_33_46.14039, moving to /var/opt/mssql/log/core.sqlservr.14039.temp/core.sqlservr.14039.gdmp
Moving logs to /var/opt/mssql/log/core.sqlservr.14039.temp/log/paldumper-debug.log
Wed Dec 21 14:33:50 Europe 2022 Capturing program binaries
Wed Dec 21 14:33:51 Europe 2022 Not compressing the dump files, moving instead to: /var/opt/mssql/log/core.sqlservr.12_21_2022_14_33_48.14039.d
root@fc89ebb883e1:/tmp#

Question

How to install debug packages so that gdb can produce a more useful stacktrace for the system libraries part?

I followed the instructions a https://wiki.ubuntu.com/Debug%20Symbol%20Packages (only manually replacing lsb_release -cs with bionic since lsb_release was not available in the container) without any luck (I also uncommented the deb-src lines in /etc/apt/sources.list, although I think that does not help with out the debug symbol packages?).

Running find-dbgsym-packages on all libraries that it uses does produce a list:

root@fc89ebb883e1:/tmp# ldd /opt/mssql/bin/sqlservr | awk '/=> \//{ print $3 }' | xargs find-dbgsym-packages
Cannot find debug package for [dso] (1f06001733b9be9478b105faf0dac6bdf36c85de)
Cannot find debug package for librt.so.1 (536945c2f46fc84dbd99d38c377cdae9d1d4fdab)
Cannot find debug package for libdl.so.2 (4c0610fa7b0706d9abc26de5e7940d2e2578eec5)
Cannot find debug package for [dso] (4938b15a667a41cc98755418556ea492500a927a)
Cannot find debug package for [dso] (fca7141592f5b96078ac7d83841e32beae06ca1a)
Cannot find debug package for [dso] (365483d22aefb38cd989ebc15bf6b135138f6c64)
No build-ID for [dso]
No build-ID for [dso]
Cannot find debug package for libc.so.6 (f7307432a8b162377e77a182b6cc2e53d771ec4b)
Cannot find debug package for [dso] (0f375714358902a9cc97b6255860e804e2027f0c)
Cannot find debug package for [dso] (8cff559d91e3376842a1ba41ccf74eb9decd159b)
Cannot find debug package for [dso] (ee6e9462ba2491f4ee8c4e52c3323274a9366614)
Cannot find debug package for [dso] (0e2784298e7d3f4d894fe130acefa77c3e624f72)
Cannot find debug package for [dso] (f0914c84fffd32a3def419bb08477e505de6d4a8)
Cannot find debug package for libsqlite3.so.0 (445b3b29f78ece9ee350c03278ea9c8fb4a7308d)
Cannot find debug package for libcrypt.so.1 (66b341634eb705798917083dc167aa926b0527ab)
libasn1-8-heimdal-dbgsym libatomic1-dbg libaudit1-dbgsym libcap-ng0-dbgsym libcom-err2-dbgsym libffi6-dbg libgcc1-dbg libgmp10-dbgsym libgnutls30-dbgsym libgssapi3-heimdal-dbgsym libhcrypto4-heimdal-dbgsym libheimbase1-heimdal-dbgsym libheimntlm0-heimdal-dbgsym libhogweed4-dbgsym libhx509-5-heimdal-dbgsym libkeyutils1-dbgsym libkrb5-26-heimdal-dbgsym libkrb5-dbg libldap-2.4-2-dbgsym libnettle6-dbgsym libnuma1-dbgsym libroken18-heimdal-dbgsym libsasl2-2-dbgsym libsss-nss-idmap0-dbgsym libtasn1-6-dbgsym libudev1-dbgsym zlib1g-dbg
root@fc89ebb883e1:/tmp#

however trying to install them fails:

root@fc89ebb883e1:/tmp# apt install libasn1-8-heimdal-dbgsym libatomic1-dbg libaudit1-dbgsym libcap-ng0-dbgsym libcom-err2-dbgsym libffi6-dbg libgcc1-dbg libgmp10-dbgsym libgnutls30-dbgsym libgssapi3-heimdal-dbgsym libhcrypto4-heimdal-dbgsym libheimbase1-heimdal-dbgsym libheimntlm0-heimdal-dbgsym libhogweed4-dbgsym libhx509-5-heimdal-dbgsym libkeyutils1-dbgsym libkrb5-26-heimdal-dbgsym libkrb5-dbg libldap-2.4-2-dbgsym libnettle6-dbgsym libnuma1-dbgsym libroken18-heimdal-dbgsym libsasl2-2-dbgsym libsss-nss-idmap0-dbgsym libtasn1-6-dbgsym libudev1-dbgsym zlib1g-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
libaudit1-dbgsym is already the newest version (1:2.8.2-1ubuntu1.1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libasn1-8-heimdal-dbgsym : Depends: libasn1-8-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libgssapi3-heimdal-dbgsym : Depends: libgssapi3-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libhcrypto4-heimdal-dbgsym : Depends: libhcrypto4-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libheimbase1-heimdal-dbgsym : Depends: libheimbase1-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libheimntlm0-heimdal-dbgsym : Depends: libheimntlm0-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libhx509-5-heimdal-dbgsym : Depends: libhx509-5-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libkrb5-26-heimdal-dbgsym : Depends: libkrb5-26-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
 libroken18-heimdal-dbgsym : Depends: libroken18-heimdal (= 7.5.0+dfsg-1ubuntu0.1) but 7.5.0+dfsg-1ubuntu0.2 is to be installed
E: Unable to correct problems, you have held broken packages.
root@fc89ebb883e1:/tmp#

PS

Can you please, please, please use ISO 8601 for core dump file names, e.g. core.sqlservr.2022_12_21_14_33_46.14039.