Improve test runner performance by running from CRaC checkpoint

The test runner suffers from the slow startup time of the JVM.
My goal with this PR was to significantly improve the performance
of the Java test runner.

There are at the time of writing several options to improve JVM
startup time:
1. Graal native image (AOT)
2. Shared AppCDS
3. Project Leyden
4. CRaC

Native image can't be used for the test runner as it has to
dynamically load classes at runtime and Graal AOT depends on a closed
world assumption.

Shared AppCDS improves performance. In my testing a test run did go
down from ~4s to ~3s.

Project Leyden is very promising, as it tries to do as much work
as possible ahead of time without making a closed world assumption.
At the time of this writing the project is only in early access,
so it's probably going to take a while before it lands in a LTS
release.

CRaC brings the best performance improvements, but with some caveats:
- build gets more complicated
- the CRIU based engine needs to run with two additional capabilities:
    - CHECKPOINT_RESTORE
    - SYS_PTRACE
- performance speedup depends on how well the JVM gets warmed up
  before the checkpoint is taken

bin/run-tests-in-docker.sh had to be adjusted to start a new
container for each test. The restored JVM needs to be run as a
specific PID, so it can only be restored once per container life
cycle.
The test run still finishes faster than before.

This commit uses the CRIU engine as I had some issues getting the warp
engine to work properly. The warp engine is also only supported by
Azul right now and isn't compatible with musl / Alpine yet.
In my tests the runtime of my example test did go down from ~4s to
>1s.
By switching to an Alpine based image this change also reduces to size
of the container (exercism/java-test-runner-crac-checkpoint) to 271MB,
down from previously 464MB.

CRaC documentation:
- https://crac.org/
- https://docs.azul.com/core/crac/crac-introduction
- https://openjdk.org/projects/crac/

Author: fapdash

.dockerignore

@@ -1,15 +1,12 @@
 .appends
 .git
 .github
-.gradle
 .idea
-gradle
 tests
 .dockerignore
 .gitattributes
 .gitignore
 bin/run-in-docker.sh
 bin/run-tests.sh
 bin/run-tests-in-docker.sh
-gradlew
 gradlew.bat

Dockerfile

@@ -1,13 +1,14 @@
-FROM gradle:8.12-jdk21 AS build
+FROM bellsoft/liberica-runtime-container:jdk-21-crac-musl AS build
 
 WORKDIR /app
-COPY --chown=gradle:gradle . /app
-RUN gradle -i --stacktrace clean build
+COPY . /app
+RUN /app/gradlew -i --stacktrace clean build
 
-FROM eclipse-temurin:21
+FROM bellsoft/liberica-runtime-container:jdk-21-crac-musl
 
 WORKDIR /opt/test-runner
-COPY bin/run.sh bin/run.sh
+COPY bin/run-to-create-crac-checkpoint.sh bin/run-to-create-crac-checkpoint.sh
+COPY bin/run-restore-from-checkpoint.sh bin/run-restore-from-checkpoint.sh
 COPY --from=build /app/build/libs/java-test-runner.jar .
 
-ENTRYPOINT ["sh", "/opt/test-runner/bin/run.sh"]
+ENTRYPOINT ["sh", "/opt/test-runner/bin/run-to-create-crac-checkpoint.sh"]

bin/build-crac-checkpoint-image.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env sh
+
+# Synopsis:
+# Build a Docker image containing a CraC checkpoint to restart from.
+# An initial image is built. A container is created from that image
+# and tests are run to warm up the JVM. Then a checkpoint is created.
+# The final image is created by committing the containiner
+# containing the checkpoint.
+
+docker build -t exercism/java-test-runner-crac-checkpoint .
+
+# Copy all tests into one merged project, so we can warm up the JVM
+# TODO(FAP): this is missing some tests as most tests use the same filenames
+mkdir -p tests/merged
+for dir in tests/*; do
+    if [ -d "$dir" ] && [ "$dir" != "tests/merged/" ]; then
+        rsync -a "$dir"/ tests/merged/
+    fi
+done
+
+slug="merged"
+solution_dir=$(realpath "tests/merged/")
+output_dir=$(realpath "tests/merged/")
+
+docker run --cap-add CHECKPOINT_RESTORE \
+           --cap-add SYS_PTRACE \
+           --name java-test-runner-crac \
+           --network none \
+           --mount type=bind,src="${solution_dir}",dst=/solution \
+           --mount type=bind,src="${output_dir}",dst=/output \
+           --mount type=tmpfs,dst=/tmp \
+           --tmpfs /openjfx:exec,rw \
+           exercism/java-test-runner-crac-checkpoint "${slug}" /solution /output
+
+docker commit --change='ENTRYPOINT ["sh", "/opt/test-runner/bin/run-restore-from-checkpoint.sh"]' java-test-runner-crac exercism/java-test-runner-crac-restore
+
+docker rm -f java-test-runner-crac
+rm -rf tests/merged/

bin/run-in-docker-without-build.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env sh
+
+# Synopsis:
+# Run the test runner on a solution using the test runner Docker image.
+# The container image is assumed to be already available.
+
+# Arguments:
+# $1: exercise slug
+# $2: path to solution folder
+# $3: path to output directory
+
+# Output:
+# Writes the test results to a results.json file in the passed-in output directory.
+# The test results are formatted according to the specifications at https://github.com/exercism/docs/blob/main/building/tooling/test-runners/interface.md
+
+# Example:
+# ./bin/run-in-docker.sh two-fer path/to/solution/folder/ path/to/output/directory/
+
+# If any required arguments is missing, print the usage and exit
+if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
+    echo "usage: ./bin/run-in-docker.sh exercise-slug path/to/solution/folder/ path/to/output/directory/"
+    exit 1
+fi
+
+slug="$1"
+solution_dir=$(realpath "${2%/}")
+output_dir=$(realpath "${3%/}")
+
+# Create the output directory if it doesn't exist
+mkdir -p "${output_dir}"
+
+# Run the Docker image using the settings mimicking the production environment
+docker run \
+    --rm \
+    --network none \
+    --read-only \
+    --mount type=bind,src="${solution_dir}",dst=/solution \
+    --mount type=bind,src="${output_dir}",dst=/output \
+    --mount type=tmpfs,dst=/tmp \
+    exercism/java-test-runner-crac-restore "${slug}" /solution /output

bin/run-in-docker.sh

@@ -30,7 +30,7 @@ output_dir=$(realpath "${3%/}")
 mkdir -p "${output_dir}"
 
 # Build the Docker image
-docker build --rm -t exercism/java-test-runner .
+bin/build-crac-checkpoint-image.sh
 
 # Run the Docker image using the settings mimicking the production environment
 docker run \
@@ -40,4 +40,4 @@ docker run \
     --mount type=bind,src="${solution_dir}",dst=/solution \
     --mount type=bind,src="${output_dir}",dst=/output \
     --mount type=tmpfs,dst=/tmp \
-    exercism/java-test-runner "${slug}" /solution /output 
+    exercism/java-test-runner-crac-restore "${slug}" /solution /output

bin/run-restore-from-checkpoint.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env sh
+
+# Synopsis:
+# Run the test runner on a solution using the test runner Docker image.
+# The test runner Docker image is built automatically.
+
+# Arguments:
+# $1: exercise slug
+# $2: path to solution folder
+# $3: path to output directory
+
+# Output:
+# Writes the test results to a results.json file in the passed-in output directory.
+# The test results are formatted according to the specifications at https://github.com/exercism/docs/blob/main/building/tooling/test-runners/interface.md
+
+# Example:
+# ./bin/run-restore-from-checkpoint.sh two-fer path/to/solution/folder/ path/to/output/directory/
+
+if [ $# -lt 3 ]
+then
+    echo "Usage:"
+    echo "./bin/run-restore-from-checkpoint.sh two-fer ~/input/ ~/output/"
+    exit 1
+fi
+
+problem_slug="$1"
+input_folder="$2"
+output_folder="$3"
+tmp_folder="/tmp/solution"
+
+mkdir -p $output_folder
+
+rm -rf $tmp_folder
+mkdir -p $tmp_folder
+
+cd $tmp_folder
+cp -R $input_folder/* .
+
+find . -mindepth 1 -type f | grep 'Test.java' | xargs -I file sed -i "s/@Ignore(.*)//g;s/@Ignore//g;s/@Disabled(.*)//g;s/@Disabled//g;" file
+
+java -XX:CRaCRestoreFrom=/opt/test-runner/crac-checkpoint com.exercism.Restore $problem_slug . $output_folder

bin/run-tests-in-docker.sh

@@ -13,16 +13,28 @@
 # ./bin/run-tests-in-docker.sh
 
 # Build the Docker image
-docker build --rm -t exercism/java-test-runner .
-
-# Run the Docker image using the settings mimicking the production environment
-docker run \
-    --rm \
-    --network none \
-    --read-only \
-    --mount type=bind,src="${PWD}/tests",dst=/opt/test-runner/tests \
-    --mount type=tmpfs,dst=/tmp \
-    --volume "${PWD}/bin/run-tests.sh:/opt/test-runner/bin/run-tests.sh" \
-    --workdir /opt/test-runner \
-    --entrypoint /opt/test-runner/bin/run-tests.sh \
-    exercism/java-test-runner
+bin/build-crac-checkpoint-image.sh
+
+exit_code=0
+
+# Iterate over all test directories
+for test_dir in tests/*; do
+    test_dir_name=$(basename "${test_dir}")
+    test_dir_path=$(realpath "${test_dir}")
+    results_file_path="${test_dir_path}/results.json"
+    expected_results_file_path="${test_dir_path}/expected_results.json"
+
+    bin/run-in-docker-without-build.sh "${test_dir_name}" "${test_dir_path}" "${test_dir_path}"
+
+    # Normalize the results file
+    sed -i "s~${test_dir_path}~/solution~g" "${results_file_path}"
+
+    echo "${test_dir_name}: comparing results.json to expected_results.json"
+    diff "${results_file_path}" "${expected_results_file_path}"
+
+    if [ $? -ne 0 ]; then
+        exit_code=1
+    fi
+done
+
+exit ${exit_code}

bin/run-to-create-crac-checkpoint.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+# Synopsis:
+# Run the test runner on a solution using the test runner Docker image.
+# The test runner Docker image is built automatically.
+
+# Arguments:
+# $1: exercise slug
+# $2: path to solution folder
+# $3: path to output directory
+
+# Output:
+# Writes the test results to a results.json file in the passed-in output directory.
+# The test results are formatted according to the specifications at https://github.com/exercism/docs/blob/main/building/tooling/test-runners/interface.md
+
+# Example:
+# ./bin/run-create-crac-checkpoint.sh two-fer path/to/solution/folder/ path/to/output/directory/
+
+if [ $# -lt 3 ]
+then
+    echo "Usage:"
+    echo "./bin/run-create-crac-checkpoint.sh two-fer ~/input/ ~/output/"
+    exit 1
+fi
+
+problem_slug="$1"
+input_folder="$2"
+output_folder="$3"
+tmp_folder="/tmp/solution"
+
+mkdir -p $output_folder
+
+rm -rf $tmp_folder
+mkdir -p $tmp_folder
+
+cd $tmp_folder
+cp -R $input_folder/* .
+
+find . -mindepth 1 -type f | grep 'Test.java' | xargs -I file sed -i "s/@Ignore(.*)//g;s/@Ignore//g;s/@Disabled(.*)//g;s/@Disabled//g;" file
+
+# -XX:-UsePerfData option worked outside of Docker, but inside of Docker the restore would fail
+# See https://docs.azul.com/core/crac/crac-debugging#restore-conflict-of-pids and https://docs.azul.com/core/crac/crac-debugging#using-cracminpid-option
+# for info about -XX:CRaCMinPid
+java -XX:CRaCMinPid=128 -XX:CRaCCheckpointTo=/opt/test-runner/crac-checkpoint -Xshare:off -jar /opt/test-runner/java-test-runner.jar $problem_slug . $output_folder

build.gradle

@@ -24,6 +24,8 @@ dependencies {
     implementation "com.fasterxml.jackson.datatype:jackson-datatype-jdk8:$jacksonVersion"
     implementation 'com.github.javaparser:javaparser-core:3.26.2'
 
+    implementation 'org.crac:crac:1.5.0'
+
     implementation 'org.assertj:assertj-core:3.25.3'
     implementation 'org.apiguardian:apiguardian-api:1.1.2' // https://github.com/exercism/java-test-runner/issues/79
     implementation platform('org.junit:junit-bom:5.11.3')

src/main/java/com/exercism/Restore.java

@@ -0,0 +1,11 @@
+package com.exercism;
+
+import java.io.IOException;
+
+public class Restore {
+
+	public static void main(String[] args) throws IOException {
+		new TestRunner(args[0], args[1], args[2]).run();
+	}
+
+}

src/main/java/com/exercism/TestRunner.java

@@ -17,6 +17,10 @@
 import java.util.List;
 import java.util.stream.Stream;
 
+import org.crac.CheckpointException;
+import org.crac.Core;
+import org.crac.RestoreException;
+
 public final class TestRunner {
 
     private final JUnitTestParser testParser;
@@ -33,14 +37,16 @@ public TestRunner(String slug, String inputDirectory, String outputDirectory) {
         this.inputDirectory = inputDirectory;
     }
 
-    public static void main(String[] args) throws IOException {
+    public static void main(String[] args) throws IOException, CheckpointException, RestoreException {
         if (args.length < 3) {
             throw new IllegalArgumentException("Not enough arguments, need <slug> <inputDirectory> <outputDirectory>");
         }
         new TestRunner(args[0], args[1], args[2]).run();
+
+        Core.checkpointRestore();
     }
 
-    private void run() throws IOException {
+    void run() throws IOException {
         var sourceFiles = resolveSourceFiles();
         var testFiles = resolveTestFiles();