gradlew bootBuildImage fails with Podman on macOS Sonoma

[acisternino]

Problem

gradlew buildBootImage fails with permission denied when used on macOS Sonoma (14) with podman as a container engine.

Steps to reproduce

1. Create a simple project on start.spring.io with the following characteristics:
   • Project: Gradle-kotlin
   • Language: Java
   • Spring Boot: 3.2.3
   • Packaging: Jar
   • Java: 21
   • Dependencies: Spring Web
2. Unzip the project somewhere and cd into it
3. Upgrade the Gradle Wrapper to v8.6
4. Run the ./gradlew bootBuildImage command

Error

$ cd spring-podman

Using java version 21.0.2-librca in this shell.
$ ./gradlew bootBuildImage

> Task :bootBuildImage
Building image 'docker.io/library/spring-podman:0.0.1-SNAPSHOT'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base:latest' ..................................................
 > Pulled builder image 'docker.io/paketobuildpacks/builder-jammy-base@sha256:94593b2ad54362e1256da43631661d3dad936a177609a3d297fc638e87c6ee91'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-base:latest' ..................................................
 > Pulled run image 'docker.io/paketobuildpacks/run-jammy-base@sha256:c2bc9ac828642aee2376a649ef5affd6912507c739247a40086bcae6d62a8b41'
 > Executing lifecycle version v0.18.5
 > Using build cache volume 'pack-cache-f19174dfe539.build'

 > Running creator
    [creator]     ===> ANALYZING
    [creator]     ERROR: failed to initialize analyzer: getting previous image: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Builder lifecycle 'creator' failed with status code 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 5s
5 actionable tasks: 5 executed

More info

Podman is configured to have docker compatibility activated. This is visible in Podman Desktop's dashboard.

These are the sockets and their properties:

$ ls -l /var/run/docker.sock
lrwxr-xr-x  1 root  daemon  75 Mar  4 09:05 /var/run/docker.sock@ -> /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock

$ ls -l /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock
lrwxr-xr-x@ 1 andrea.cisternino  staff  80 Feb  2 09:23 /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock -> /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock

$ ls -l /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock
srw-------  1 andrea.cisternino  staff  0 Mar  4 09:05 /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock

Versions:

• Spring 3.2.3
• macOS: Sonoma 14.3.1
• Java 21.0.2 (BellSoft 21.0.2+14-LTS)
• Gradle 8.6
• Podman Desktop 1.7.1

$ podman version
Client:       Podman Engine
Version:      4.9.2
API Version:  4.9.2
Go Version:   go1.21.6
Git Commit:   f9a48ebcfa9a39144be0f86f4ba842752835f945
Built:        Sat Feb  3 00:31:39 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.8.3
API Version:  4.8.3
Go Version:   go1.21.5
Built:        Wed Jan  3 15:10:40 2024
OS/Arch:      linux/arm64

[scottfrederick]

The Spring Boot Gradle plugin documentation includes an example of configuring the plugin for use with podman.

The podman compatibility mode that creates a symlink for /var/run/docker.sock is enough for the Gradle plugin code to access the podman daemon, as demonstrated by the fact that the builder images are being pulled successfully. Processes inside the builder container also need to access the daemon, and this is the part that is not working for you (the error is coming from the creator process in the builder lifecycle).

Please try configuring the plugin as documented and let us know if that works.

[ThomasVitale]

@scottfrederick I experienced the same issue. If I downgrade Spring Boot from 3.2.3 to 3.1.9, the bootBuildImage task completes successfully.

Also, I tried out the same test using Maven instead of Gradle, and the spring-boot:build-image command completes successfully for both 3.2.3 and 3.1.9.

So, it seems to be a problem specific to Spring Boot 3.2.x with Gradle.

[acisternino]

Apologies, I completely forgot that section of the manual.
However, even with those changes the result is still an error.

Before providing more details I'd like to note that the changes suggested by the manual are a bit misleading for people on macOS.

First, the following line is valid only for Linux. On macOS /run doesn't exist and this could lead to misunderstandings.

host.set("unix:///run/user/1000/podman/podman.sock")

The podman command for finding the socket is also wrong on macOS:

$ podman info --format='{{.Host.RemoteSocket.Path}}'
/run/podman/podman.sock
$ ls -l /run/podman/podman.sock
ls: /run/podman/podman.sock: No such file or directory

In the following comments I will highlight the specific problems I am still having.

[acisternino]

With host set to:

host.set("unix:///var/run/docker.sock")

I get (abbreviated):

$ ./gradlew clean bootBuildImage

> Task :bootBuildImage
Building image 'docker.io/library/spring-podman:0.0.1-SNAPSHOT'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base:latest' ..................................................
 > Pulled builder image 'docker.io/paketobuildpacks/builder-jammy-base@sha256:94593b2ad54362e1256da43631661d3dad936a177609a3d297fc638e87c6ee91'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-base:latest' ..................................................
 > Pulled run image 'docker.io/paketobuildpacks/run-jammy-base@sha256:c2bc9ac828642aee2376a649ef5affd6912507c739247a40086bcae6d62a8b41'
 > Executing lifecycle version v0.18.5
 > Using build cache volume 'pack-cache-f19174dfe539.build'

 > Running creator
    [creator]     ===> ANALYZING
    [creator]     ERROR: failed to initialize analyzer: getting previous image: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Builder lifecycle 'creator' failed with status code 1

BUILD FAILED in 6s
6 actionable tasks: 6 executed

$ ls -l /var/run/docker.sock
lrwxr-xr-x  1 root  daemon  75 Mar  6 08:52 /var/run/docker.sock -> /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock

[acisternino]

This is interesting.

With host set to:

host.set("unix:///Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock")

I get:

$ ./gradlew clean bootBuildImage

> Task :bootBuildImage
Building image 'docker.io/library/spring-podman:0.0.1-SNAPSHOT'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base:latest' ..................................................
 > Pulled builder image 'docker.io/paketobuildpacks/builder-jammy-base@sha256:94593b2ad54362e1256da43631661d3dad936a177609a3d297fc638e87c6ee91'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-base:latest' ..................................................
 > Pulled run image 'docker.io/paketobuildpacks/run-jammy-base@sha256:c2bc9ac828642aee2376a649ef5affd6912507c739247a40086bcae6d62a8b41'
 > Executing lifecycle version v0.18.5
 > Using build cache volume 'pack-cache-f19174dfe539.build'

 > Running creator

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Docker API call to '/Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock/v1.24/containers/create' failed with status code 500 "Internal Server Error" and message "container create: statfs /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock: operation not supported"

BUILD FAILED in 4s

$ ls -l /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock
lrwxr-xr-x@ 1 andrea.cisternino  staff  80 Feb  2 09:23 /Users/andrea.cisternino/.local/share/containers/podman/machine/podman.sock -> /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock

[acisternino]

Finally, with host set to:

host.set("unix:///Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock")

I get the exact same result as above.

For info:

$ ls -l /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock
srw-------  1 andrea.cisternino  staff  0 Mar  6 08:52 /Users/andrea.cisternino/.local/share/containers/podman/machine/qemu/podman.sock

[awsp]

I also happened to get the exact same error when using 3.2.2 using Gradle Kotlin.
OS: macOS Ventura 13.6
Podman: 1.7.1

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Docker API call to '/Users/awsp/.local/share/containers/podman/machine/qemu/podman.sock/v1.24/containers/create' failed with status code 500 "Internal Server Error" and message "container create: statfs /Users/awsp/.local/share/containers/podman/machine/qemu/podman.sock: operation not supported"

[scottfrederick]

There appears to be a few different things going on here.

First, there is the issue of Podman Desktop's Docker compatibility mode.

Podman is configured to have docker compatibility activated. This is visible in Podman Desktop's dashboard.
These are the sockets and their properties:

As stated above, this compatibility mode socket symlink works for the code in the Spring Boot plugins that connect to the daemon using the Docker socket. However, the /var/run/docker.sock socket is not set up correctly for the CNB lifecycle code that runs inside the builder image, leading to the permission denied errors. I suspect this is not a configuration that can be easily supported with CNB buidpacks, and we should recommend using the explicit Podman remote configuration instead.

Second, there is the issue of the operation not supported errors from the /containers/create API shown by @acisternino and @awsp when the Podman socket is explicitly configured. We'll have to debug more into this on a macOS machine to figure out what's going on there.

The podman command for finding the socket is also wrong on macOS:

The only other guidance I've seen in the Podman documentation is to set the socket address to $XDG_RUNTIME_DIR/podman/podman.sock, but I don't think this is portable to macOS either. We might just need to qualify that documentation tip as applicable to Linux only if Podman does not provide a reliable way to get the socket path on macOS or Windows.

Also, I tried out the same test using Maven instead of Gradle, and the spring-boot:build-image command completes successfully for both 3.2.3 and 3.1.9.
So, it seems to be a problem specific to Spring Boot 3.2.x with Gradle.

@ThomasVitale This is very surprising, since the code that is specific to the Maven and Gradle plugins is very thin and the vast majority of the code that builds images is the same between the two plugins. We'll have to do more testing to see if we can reproduce this difference.

If anyone is able to test their Podman on macOS configuration using the pack CLI along with the Boot plugins, I'd be curious to know if pack shows the same behavior.

[sdeleuze]

Seems to work for me when enabling Docker compatibility and installing podman-mac-helper.

[acisternino]

Interesting. @sdeleuze I have checked my environment and I have podman-mac-helper installed and running.

I went through the documentation and I can confirm that all the steps in the Verification paragraph are indeed OK on my machine but the build still fails with the same errors as above.

@scottfrederick I am no pack expert but I tried to use it to build an image for my super simple demo project but both my attempts failed for unclear reasons after downloading the Gradle distribution.. This is what I tried:

$ pack build spring-podman --builder=gcr.io/buildpacks/builder --env GOOGLE_RUNTIME_VERSION=21
$ pack build spring-podman --builder=paketobuildpacks/builder-jammy-full --env BP_JVM_VERSION=21

[scottfrederick]

We will need to debug this further to see why users are having different results with Podman Desktop and podman-mac-helper, but we've created a wiki page in the Spring Boot GitHub repo to capture some of the items here that can be covered with documentation. See https://github.com/spring-projects/spring-boot/wiki/Building-OCI-Images-with-Spring-Boot#podman-support. Feel free to comment with feedback on this documentation.

[scottfrederick]

I am no pack expert but I tried to use it to build an image for my super simple demo project but both my attempts failed for unclear reasons after downloading the Gradle distribution.. This is what I tried:

@acisternino There is some information here about testing with the pack CLI. Building from a jar file as shown there should eliminate the problems with downloading a Gradle distribution. If you can try this again with your Podman setup that would be helpful.

[acisternino]

Hi @scottfrederick,

I have tried using pack following the instructions in the link above and it worked.

This is what I tried from inside the project (named spring-podman) directory:

$ ./gradlew clean bootJar

$ pack build --builder paketobuildpacks/builder-jammy-base:latest \
             --path build/libs/spring-podman-0.0.1-SNAPSHOT.jar \
             --env BP_JVM_VERSION=21 \
             spring-podman

[ThomasVitale]

@scottfrederick is there anything I can do to help with this issue? I created a repo with two samples showing how to reproduce the error: everything works well with Spring Boot 3.1, but it fails on 3.2 and 3.3.

https://github.com/ThomasVitale/spring-boot-buildpacks-test

When using the pack CLI directly, the build is successful for Spring Boot 3.1, 3.2, and 3.3. So it would seem to be a bug/regression in the Spring Boot plugin introduced in 3.2.

[der-ambi]

FWIW: For me gradle clean bootBuildImage worked with Spring Boot 3.3.2 and Podman 5.2.0 on Sonoma 14.4 after slightly adjusting the settings given on the Gradle plugin page

tasks.named("bootBuildImage") {
	docker {
		host = "/Users/ambi/.local/share/containers/podman/machine/podman.sock"
	}
}

I used the local socket name and left out the bindHostToBuilder piece.