Skip to content

Commit c775e1e

Browse files
hhorakhhorak
committed
Upgrading and data directory version checking
MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23). For version changes in Z part, the server's binary data format stays compatible and thus no special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual steps as described at https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/ Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported; the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0. **Important**: Upgrading to a new version is always risky and users are expected to make a full back-up of all data before. A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database. Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally faster for large data directory, but only possible if upgrading from the very previous version, so skipping versions is not supported. This container detects whether the data needs to be upgraded using `mysql_upgrade` and we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values: * `warn` -- If the data version can be determined and the data come from a different version of the daemon, a warning is printed but the container starts. This is the default value. Since historically the version file `mysql_upgrade_info` was not created, when using this option, the version file is created if not exist, but no `mysql_upgrade` will be called. However, this automatic creation will be removed after few months, since the version should be created on most deployments at that point. * `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version can be determined and the data come with the very previous version. A warning is printed if the data come from even older or newer version. This value effectively enables automatic upgrades, but it is always risky and users should still back-up all the data before starting the newer container. Set this option only if you have very good back-ups at any moment and you are fine to fail-over from the back-up. * `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present in the root of the data directory; this file holds information about the version of the data. * `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version of the data is detected. It optimizes all the tables. * `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version of the data is detected. It analyzes all the tables. * `disable` -- nothing is done regarding data directory version. Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
1 parent 987e5c5 commit c775e1e

File tree

7 files changed

+445
-87
lines changed

7 files changed

+445
-87
lines changed

‎5.6/root/usr/share/container-scripts/mysql/README.md

+50
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,56 @@ Such a directory `sslapp` can then be mounted into the container with -v,
260260
or a new container image can be built using s2i.
261261

262262

263+
Upgrading and data directory version checking
264+
---------------------------------------------
265+
266+
MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
267+
For version changes in Z part, the server's binary data format stays compatible and thus no
268+
special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
269+
steps as described at
270+
https://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html.
271+
272+
Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
273+
the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.
274+
275+
**Important**: Upgrading to a new version is always risky and users are expected to make a full
276+
back-up of all data before.
277+
278+
A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
279+
load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.
280+
281+
Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
282+
and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
283+
faster for large data directory, but only possible if upgrading from the very previous version,
284+
so skipping versions is not supported.
285+
286+
This container detects whether the data needs to be upgraded using `mysql_upgrade` and
287+
we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:
288+
289+
* `warn` -- If the data version can be determined and the data come from a different version
290+
of the daemon, a warning is printed but the container starts. This is the default value.
291+
Since historically the version file `mysql_upgrade_info` was not created, when using this option,
292+
the version file is created if not exist, but no `mysql_upgrade` will be called.
293+
However, this automatic creation will be removed after few months, since the version should be
294+
created on most deployments at that point.
295+
* `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
296+
can be determined and the data come with the very previous version. A warning is printed if the data
297+
come from even older or newer version. This value effectively enables automatic upgrades,
298+
but it is always risky and users should still back-up all the data before starting the newer container.
299+
Set this option only if you have very good back-ups at any moment and you are fine to fail-over
300+
from the back-up.
301+
* `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
302+
come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
303+
in the root of the data directory; this file holds information about the version of the data.
304+
* `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
305+
of the data is detected. It optimizes all the tables.
306+
* `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
307+
of the data is detected. It analyzes all the tables.
308+
* `disable` -- nothing is done regarding data directory version.
309+
310+
Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
311+
312+
263313
Changing the replication binlog_format
264314
--------------------------------------
265315
Some applications may wish to use `row` binlog_formats (for example, those built

‎5.7/root/usr/share/container-scripts/mysql/README.md

+50
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,56 @@ Such a directory `sslapp` can then be mounted into the container with -v,
260260
or a new container image can be built using s2i.
261261

262262

263+
Upgrading and data directory version checking
264+
---------------------------------------------
265+
266+
MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
267+
For version changes in Z part, the server's binary data format stays compatible and thus no
268+
special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
269+
steps as described at
270+
https://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html.
271+
272+
Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
273+
the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.
274+
275+
**Important**: Upgrading to a new version is always risky and users are expected to make a full
276+
back-up of all data before.
277+
278+
A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
279+
load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.
280+
281+
Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
282+
and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
283+
faster for large data directory, but only possible if upgrading from the very previous version,
284+
so skipping versions is not supported.
285+
286+
This container detects whether the data needs to be upgraded using `mysql_upgrade` and
287+
we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:
288+
289+
* `warn` -- If the data version can be determined and the data come from a different version
290+
of the daemon, a warning is printed but the container starts. This is the default value.
291+
Since historically the version file `mysql_upgrade_info` was not created, when using this option,
292+
the version file is created if not exist, but no `mysql_upgrade` will be called.
293+
However, this automatic creation will be removed after few months, since the version should be
294+
created on most deployments at that point.
295+
* `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
296+
can be determined and the data come with the very previous version. A warning is printed if the data
297+
come from even older or newer version. This value effectively enables automatic upgrades,
298+
but it is always risky and users should still back-up all the data before starting the newer container.
299+
Set this option only if you have very good back-ups at any moment and you are fine to fail-over
300+
from the back-up.
301+
* `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
302+
come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
303+
in the root of the data directory; this file holds information about the version of the data.
304+
* `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
305+
of the data is detected. It optimizes all the tables.
306+
* `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
307+
of the data is detected. It analyzes all the tables.
308+
* `disable` -- nothing is done regarding data directory version.
309+
310+
Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
311+
312+
263313
Changing the replication binlog_format
264314
--------------------------------------
265315
Some applications may wish to use `row` binlog_formats (for example, those built

‎root-common/usr/share/container-scripts/mysql/common.sh

+58
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ function export_setting_variables() {
3939
export MYSQL_INNODB_LOG_FILE_SIZE=${MYSQL_INNODB_LOG_FILE_SIZE:-$((MEMORY_LIMIT_IN_BYTES*15/1024/1024/100))M}
4040
export MYSQL_INNODB_LOG_BUFFER_SIZE=${MYSQL_INNODB_LOG_BUFFER_SIZE:-$((MEMORY_LIMIT_IN_BYTES*15/1024/1024/100))M}
4141
fi
42+
export MYSQL_UPGRADE=${MYSQL_UPGRADE:-warn}
4243
}
4344

4445
# this stores whether the database was initialized from empty datadir
@@ -103,6 +104,11 @@ function initialize_database() {
103104
fi
104105
start_local_mysql "$@"
105106

107+
# Running mysql_upgrade creates the mysql_upgrade_info file in the data dir,
108+
# which is necessary to detect which version of the mysqld daemon created the data.
109+
# Checking empty file should not take longer than a second and one extra check should not harm.
110+
mysql_upgrade --socket=/tmp/mysql.sock
111+
106112
if [ -v MYSQL_RUNNING_AS_SLAVE ]; then
107113
log_info 'Initialization finished'
108114
return 0
@@ -213,3 +219,55 @@ function process_extending_config_files() {
213219
fi
214220
done <<<"$(get_matched_files "$custom_dir" "$default_dir" '*.cnf' | sort -u)"
215221
}
222+
223+
# Converts string version to the integer format (5.5.33 is converted to 505,
224+
# 10.1.23-MariaDB is converted into 1001, etc.
225+
function version2number() {
226+
local version_major=$(echo "$1" | grep -o -e '^[0-9]*\.[0-9]*')
227+
printf %d%02d ${version_major%%.*} ${version_major##*.}
228+
}
229+
230+
# Converts the version in format of an integer into major.minor
231+
function number2version() {
232+
local numver=${1}
233+
echo $((numver / 100)).$((numver % 100))
234+
}
235+
236+
# Prints version of the mysqld that is currently available (string)
237+
function mysqld_version() {
238+
${MYSQL_PREFIX}/libexec/mysqld -V | awk '{print $3}'
239+
}
240+
241+
# Returns version from the daemon in integer format
242+
function mysqld_compat_version() {
243+
version2number $(mysqld_version)
244+
}
245+
246+
# Returns version from the datadir
247+
function get_datadir_version() {
248+
local datadir="$1"
249+
local upgrade_info_file=$(get_mysql_upgrade_info_file "$datadir")
250+
[ -r "$upgrade_info_file" ] || return
251+
local version_text=$(cat "$upgrade_info_file" | head -n 1)
252+
version2number "${version_text}"
253+
}
254+
255+
# Returns name of the file in the datadir that holds version information about the data
256+
function get_mysql_upgrade_info_file() {
257+
local datadir="$1"
258+
echo "$datadir/mysql_upgrade_info"
259+
}
260+
261+
# Writes version string of the daemon into mysql_upgrade_info file (should be only used when the file is missing and only when
262+
function write_mysql_upgrade_info_file() {
263+
local datadir="$1"
264+
local version=$(mysqld_version)
265+
local upgrade_info_file=$(get_mysql_upgrade_info_file "$datadir")
266+
if [ -f "$datadir/mysql_upgrade_info" ] ; then
267+
echo "File ${upgrade_info_file} exists, nothing is done."
268+
else
269+
log_info "Storing version '${version}' information into the data dir '${upgrade_info_file}'"
270+
echo "${version}" > "${upgrade_info_file}"
271+
mysqld_version >"$datadir/mysql_upgrade_info"
272+
fi
273+
}

‎root-common/usr/share/container-scripts/mysql/helpers.sh

+4
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@ function log_info {
22
echo "---> `date +%T` $@"
33
}
44

5+
function log_warn {
6+
echo "---> `date +%T` Warning: $@"
7+
}
8+
59
function log_and_run {
610
log_info "Running $@"
711
"$@"

‎root-common/usr/share/container-scripts/mysql/init/40-check-upgrade.sh

+111
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
upstream_upgrade_info() {
2+
echo -n "For upstream documentation about upgrading, see: "
3+
case ${MYSQL_VERSION} in
4+
10.0) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-55-to-mariadb-100/" ;;
5+
10.1) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-100-to-mariadb-101/" ;;
6+
10.2) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/" ;;
7+
5.6) echo "https://dev.mysql.com/doc/refman/5.6/en/upgrading-from-previous-series.html" ;;
8+
5.7) echo "https://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html" ;;
9+
*) echo "Non expected version '${MYSQL_VERSION}'" ; return 1 ;;
10+
esac
11+
}
12+
13+
check_datadir_version() {
14+
local datadir="$1"
15+
local datadir_version=$(get_datadir_version "$datadir")
16+
local mysqld_version=$(mysqld_compat_version)
17+
local datadir_version_dot=$(number2version "${datadir_version}")
18+
local mysqld_version_dot=$(number2version "${mysqld_version}")
19+
20+
for upgrade_action in ${MYSQL_UPGRADE//,/ } ; do
21+
log_info "Running upgrade action: ${upgrade_action}"
22+
case ${upgrade_action} in
23+
auto|warn)
24+
if [ -z "${datadir_version}" ] || [ "${datadir_version}" -eq 0 ] ; then
25+
# Writing the info file, since historically it was not written
26+
log_warn "Version of the data could not be determined."\
27+
"It is because the file mysql_upgrade_info is missing in the data directory, which"\
28+
"is most probably because it was not created when initialization of data directory."\
29+
"In order to allow seamless updates to the next higher version in the future,"\
30+
"the file mysql_upgrade_info will be created."\
31+
"If the data directory was created with a different version than ${mysqld_version_dot},"\
32+
"it is required to run this container with the MYSQL_UPGRADE environment variable"\
33+
"set to 'force', or run 'mysql_upgrade' utility manually; the mysql_upgrade tool"\
34+
"checks the tables and creates such a file as well. $(upstream_upgrade_info)"
35+
write_mysql_upgrade_info_file "${MYSQL_DATADIR}"
36+
continue
37+
# This is currently a dead-code, but should be enabled after the mysql_upgrade_info
38+
# file gets to the deployments (after few monts most of the deployments should already have the file)
39+
log_warn "Version of the data could not be determined."\
40+
"Running such a container is risky."\
41+
"The current daemon version is ${mysqld_version_dot}."\
42+
"If you are not sure whether the data directory is compatible with the current"\
43+
"version ${mysqld_version_dot}, restore the data from a back-up."\
44+
"If restoring from a back-up is not possible, create a file 'mysql_upgrade_info'"\
45+
"that includes version information (${mysqld_version_dot} in this case) in the root"\
46+
"of the data directory."\
47+
"In order to create the 'mysql_upgrade_info' file, either run this container with"\
48+
"the MYSQL_UPGRADE environment variable set to 'force', or run 'mysql_upgrade' utility"\
49+
"manually; the mysql_upgrade tool checks the tables and creates such a file as well."\
50+
"That will enable correct upgrade check in the future. $(upstream_upgrade_info)"
51+
fi
52+
53+
if [ "${datadir_version}" -eq "${mysqld_version}" ] ; then
54+
log_info "MySQL server version check passed, both server and data directory"\
55+
"are version ${mysqld_version_dot}."
56+
continue
57+
fi
58+
59+
if [ $(( ${datadir_version} + 1 )) -eq "${mysqld_version}" -o "${datadir_version}" -eq 505 -a "${mysqld_version}" -eq 1000 ] ; then
60+
log_warn "MySQL server is version ${mysqld_version_dot} and datadir is version"\
61+
"${datadir_version_dot}, which is a compatible combination."
62+
if [ "${MYSQL_UPGRADE}" == 'auto' ] ; then
63+
log_info "The data directory will be upgraded automatically from ${datadir_version_dot}"\
64+
"to version ${mysqld_version_dot}. $(upstream_upgrade_info)"
65+
log_and_run mysql_upgrade --socket=/tmp/mysql.sock
66+
else
67+
log_warn "Automatic upgrade is not turned on, proceed with the upgrade."\
68+
"In order to upgrade the data directory, run this container with the MYSQL_UPGRADE"\
69+
"environment variable set to 'auto' or run running mysql_upgrade manually. $(upstream_upgrade_info)"
70+
fi
71+
else
72+
log_warn "MySQL server is version ${mysqld_version_dot} and datadir is version"\
73+
"${datadir_version_dot}, which are incompatible. Remember, that upgrade is only supported"\
74+
"by upstream from previous version and it is not allowed to skip versions. $(upstream_upgrade_info)"
75+
if [ "${datadir_version}" -gt "${mysqld_version}" ] ; then
76+
log_warn "Downgrading to the lower version is not supported. Consider"\
77+
"dumping data and load them again into a fresh instance. $(upstream_upgrade_info)"
78+
fi
79+
log_warn "Consider restoring the database from a back-up. To ignore this"\
80+
"warning, set 'MYSQL_UPGRADE' variable to 'force', but this may result in data corruption. $(upstream_upgrade_info)"
81+
return 1
82+
fi
83+
;;
84+
85+
force)
86+
log_and_run mysql_upgrade --socket=/tmp/mysql.sock --force
87+
;;
88+
89+
optimize)
90+
log_and_run mysqlcheck --socket=/tmp/mysql.sock --optimize --all-databases --force
91+
;;
92+
93+
analyze)
94+
log_and_run mysqlcheck --socket=/tmp/mysql.sock --analyze --all-databases --force
95+
;;
96+
97+
disable)
98+
log_info "Nothing is done about the data directory version."
99+
;;
100+
*)
101+
log_warn "Unknown value of MYSQL_UPGRADE variable: '${MYSQL_UPGRADE}', ignoring."
102+
;;
103+
esac
104+
done
105+
}
106+
107+
check_datadir_version "${MYSQL_DATADIR}"
108+
109+
unset -f check_datadir_version upstream_upgrade_info
110+
111+

0 commit comments

Comments
 (0)