MDEV-25497 now you can run "docker-entrypoint.sh mysqld --user=root" and it will work

[briveramelo]

-fix for scenario when running the container in a multi-tenant, HPC environment with singularity where container root does not have full privileges ie: commands like "chown" will halt the script with permission denied errors

[briveramelo]

this allows for running with singularity's --fakeroot option. Before, running docker-entrypoint.sh would ignore the --user=root option needed for this to work

[tianon]

Thanks for the contribution! ✨

There's a lot of "how" here, but I feel like we're missing the underlying "why" -- can you elaborate on your use case? Why do you need mysqld to believe it is running as root, especially if it isn't actually running as root?

[briveramelo]

Thank you for your attention to this. I'm glad you're asking, as we may find a better path.

In my case, I have a requirement to run the container as singularity's fakeroot. As I do not own the machine, I cannot run as a true root. Fakeroot, however, allows me to establish the fakeroot network for inter-container communication*; mariadb has a natural place in this pod. *Additional admin configuration necessary

Running as root has the undesired consequence of triggering these two snippets in the docker-entrypoint.sh

https://github.com/docker-library/mariadb/blob/master/10.5/docker-entrypoint.sh#L153

if [ "$user" = "0" ]; then
	# this will cause less disk access than `chown -R`
	find "$DATADIR" \! -user mysql -exec chown mysql '{}' +
fi

https://github.com/docker-library/mariadb/blob/master/10.5/docker-entrypoint.sh#L325

# If container is started as root user, restart as dedicated mysql user
if [ "$(id -u)" = "0" ]; then
	mysql_note "Switching to dedicated user 'mysql'"
	exec gosu mysql "$BASH_SOURCE" "$@"
fi

The unexpected outcome is that although root is calling this script, root does not have the permissions you might typically associate with root because it is a fakeroot with only the permissions of the user launching the container (me). chown fails.

I believe when I ran mysqld from a shell inside the container (as fakeroot) so as to mimic the final line of the _main() function
https://github.com/docker-library/mariadb/blob/master/10.5/docker-entrypoint.sh#L356, I received an error indicating that I visit the mariadb documentation about how to run as root. I came across the option --user=root and found that it worked.

However, when I would run docker-entrypoint.sh mysqld --user=root the option would be ignored and the chown command would still run, fail, and halt the script. So, I looked at why, changed the script, and submitted the request.

I do realize that this request is a rather patchy, inelegant approach, but I could not see a clearer path forward. Perhaps you see a better alternative

[tianon]

Thanks for providing more details!

In my case, I have a requirement to run the container as singularity's fakeroot. As I do not own the machine, I cannot run as a true root. Fakeroot, however, allows me to establish the fakeroot network for inter-container communication*; mariadb has a natural place in this pod. *Additional admin configuration necessary

I think this is the part I need more elaboration on -- why do you need to run as (fake)root? Why not run directly as a non-root user instead? If you (or singularity) can fix the permissions of your data directory / volume, you should be able to run as a non-root user natively (see #59 for the initial PR which provided that ability).

[briveramelo]

Interesting question. Singularity containers by default are read-only with no rwX permission modification, though both are configurable in the build step. My original container allows writing, but not read-write permission modification, so just now I rebuilt the container to allow for rwX permission mods with the "--fix-perms" flag.

However, I still get an error because I believe the mariadb container itself does not give permissions to non-root users to change directory permissions. The error occurs on /var/lib/mysql/, which I am not mounting as a data volume, so this issue cannot be related to my user permissions or to singularity's permissions.

Furthermore, if I did mount this volume, I would especially not have the permission to modify the permissions as it would then be located in the host machine space.

Scenarios:

1. I run mysqld --user=root with the changes in this PR, it works
2. I run mysqld --user=mysql w/ and w/o the changes in this PR, and I get this error

chown: changing ownership of '/var/lib/mysql/': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log_control': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log.00000001': Operation not permitted
This list of errors goes on for a while for every file recursively located here...

3. I run mysqld --user=mysql w/ and w/o the changes in this PR also with the rebuilt container to allow file permission modifications in the container, and I get this error only, nothing more

chown: changing ownership of '/var/lib/mysql/': Operation not permitted

So I'm still a fan of scenario 1, though I'm glad I tried the others.

Can you explain the aversion to running as root in the scenario?

[tianon]

I do not mean using --user= on mysqld, but rather whatever functionality your container orchestrator has to run the whole container as a different UID/GID (if you want the simplest experience, I'd suggest using 1000 / 1000 which matches the mysql user and group inside the image, but any should do). In Docker, that's something like docker run --user mysql:mysql ... mariadb:xxx, in docker-compose.yml that's something like user: mysql:mysql, etc. Surely Singularity has an analogous feature? 😕

Doing so successfully will cause the script to no longer attempt to adjust permissions at all.

The aversion to running as root is because it's frankly a really bad idea in the general case (even if in the case of fakeroot it's arguably ~safe), and in order to support it for the uncommon use case of fakeroot, we have to add support for it generally (because we can't really differentiate). What we do not want to do is encourage non-fakeroot users to use this functionality.

[tianon]

It actually sounds like it might not run as root by default? Can you share the error message you get without --fakeroot? What output do you get if you change the container "command" to id?

[briveramelo]

Thanks for the explanation, that makes sense.

The output with command id is this

uid=0(root) gid=0(root) groups=0(root)

So fakeroot really does pose as root, but the singularity runner still restricts its permissions

I ran into the errors without running as fakeroot some time ago, but it's essentially required for my application. I think it fails to create the portmappings I request on the command line. Singularity does not create networks by default, you cannot create networks as a non-root user, but you can create a fakeroot network. This is the only network non-root users can create. From here, you still need the admin to customize the fakeroot network configuration file to allow for inter-container communication.

It is a bit odd that creating this network and running the container as a fakeroot are tied together, but they are. I do not believe I can split the two apart.

In other words, I can only run mariadb in singularity as fakeroot, which precludes me from running as anything else. I may be wrong on this point, but this has been my experience and understanding from reading/working through their docs/forums.

[briveramelo]

Without fakeroot
FATAL: container creation failed: network requires root or --fakeroot, users need to specify --network=none with --net

[grooverdan]

By using a volume for the datadir, and using --scratch /var/run/mysqld, prevents [ERROR] Can't start server : Bind on unix socket: Read-only file system. The scratch space isn't enough for a full datadir. Alternately instead of scratch, you can also specify --socket=/var/lib/mysql/mariadb.sock --pid-file=/var/lib/mysql/mariadb.pid

Starting as non-root.

$ sudo rm -rf mydatadir && mkdir mydatadir

$ singularity run --no-home --bind $HOME/mydatadir:/var/lib/mysql --env MARIADB_RANDOM_ROOT_PASSWORD=1 --net --network-args "portmap=3308:3306/tcp" --fakeroot --scratch=/run/mysqld  docker://mariadb:10.5
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
ERROR:   Network fakeroot is not permitted for unprivileged users.

2022-02-03 19:18:22+11:00 [Note] [Entrypoint]: Temporary server started.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
2022-02-03 19:18:24+11:00 [Note] [Entrypoint]: GENERATED ROOT PASSWORD: NF(((fevo7z!;}@p@[R8:7t]vhCP]?Im
...
2022-02-03 19:18:25 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-02-03 19:18:25 0 [Note] InnoDB: Buffer pool(s) load completed at 220203 19:18:25
2022-02-03 19:18:25 0 [Note] Server socket created on IP: '::'.
2022-02-03 19:18:25 0 [Warning] 'proxies_priv' entry '@% [email protected]' ignored in --skip-name-resolve mode.
2022-02-03 19:18:25 0 [Note] Reading of all Master_info entries succeeded
2022-02-03 19:18:25 0 [Note] Added new Master_info '' to hash table
2022-02-03 19:18:25 0 [Note] mysqld: ready for connections.
Version: '10.5.13-MariaDB-1:10.5.13+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Client connection from the host:

$  mysql --protocol tcp --host 127.0.0.1 --port 3308 -u root -p$'NF(((fevo7z!;}@p@[R8:7t]vhCP]?Im'
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.5.13-MariaDB-1:10.5.13+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> \s
--------------
mysql  Ver 15.1 Distrib 10.8.0-MariaDB, for Linux (x86_64) using  EditLine wrapper

Connection id:		3
Current database:	
Current user:		[email protected]

Datadir looks like:

$ sudo find ~/mydatadir/ -ls
  1573002      4 drwxrwxr-x   4 100998   dan          4096 Feb  3 19:18 /home/dan/mydatadir/
  1573309      4 -rw-rw----   1 100998   100998        976 Feb  3 19:18 /home/dan/mydatadir/ib_buffer_pool
  1573176      4 -rw-rw----   1 100998   100998         52 Feb  3 19:18 /home/dan/mydatadir/aria_log_control
  1573227  12292 -rw-rw----   1 100998   100998   12582912 Feb  3 19:18 /home/dan/mydatadir/ibdata1
  1573178     32 -rw-rw----   1 100998   100998      32768 Feb  3 19:18 /home/dan/mydatadir/aria_log.00000001
  1573268      4 drwx------   2 100998   100998       4096 Feb  3 19:18 /home/dan/mydatadir/mysql
  1573285      8 -rw-rw----   1 100998   100998       8192 Feb  3 19:18 /home/dan/mydatadir/mysql/plugin.MAI
...

$ singularity --version
singularity version 3.8.5-2.fc35

$ ps -ef | grep mysqld
100998   1794807 1794737  0 19:18 pts/0    00:00:00 mysqld
dan      1795390  275369  0 19:23 pts/8    00:00:00 grep --color=auto mysqld

$ cat /proc/1794807/uid_map 
         0       1000          1
         1     100000      65536

Sorry for delayed response @briveramelo, I missed that singularity was available as a package. I haven't done anything changing the standard install. I hope this works for you or whoever comes across this next.