Allow provided and optional dependencies to be exclude when building a uber jar with Maven

[koscejev]

I stumbled upon #413 when trying to resolve exactly the same problem. We're using some provided artifacts and don't want them to end up in the final deliverable. This is exactly 100% according to maven definition of provided scope (i.e., available on the compile and test classpaths, but not at runtime).

Case in point, recommended usage of Lombok annotations is via provided scope. But that's just one example. We're using many other dependencies that are essentially build-time dependencies, not runtime ones. (And not just annotations that can be wired via annotationProcessorPaths property mentioned in #10539.) Trying to switch to spring-boot-maven plugin, but this results in huge repackaged jar containing a lot of unnecessary junk due to transitive dependencies of provided artifacts.

TLDR: I'd really appreciate an includeProvidedScope=false option (similar to includeSystemScope) for people who know what they are doing, so that I can configure this in just one place for all projects and don't need to duplicate a list of dependencies as excludes in the plugin config.

[snicoll]

This is exactly 100% according to maven definition of provided scope (i.e., available on the compile and test classpaths, but not at runtime).

That's the Maven textbook definition but it does not apply to what the repackage goal does, see the documentation. I am not against providing more flexibility in that area but I'd like we take a step back and we look at your use cases first. Given that provided isn't transitive and that you add them in a project that generates "an app" (since the repackage goal is involved), I'd like to understand what is flagged as provided and why.

Can you share more details please?

[koscejev]

One other example (besides annotations) is we have dependencies with provided scope, but unpack and process the contents via some custom build step, generating some code based on the contents, or using some parts of the dependency statically, etc.

Although some parts could probably be streamlined to not use provided dependencies, it would definitely take a lot of time to make so many changes to the existing build. So even in cases where it's possible to change this, we would still prefer to do gradual changes instead of one big rewrite of the whole build.

[snicoll]

but unpack and process the contents via some custom build step

You can unpack and process dependencies with the maven-dependencies-plugin (they don't need to be added to your dependencies). Perhaps they shouldn't be managed there at all and the result of the post-processing should be made available as a dedicated artifact.

The reason why I insist so much is because it's part of the user experience you get with Spring Boot. Anything that is required that way means that it's harder for you to run the application from your IDE.

[koscejev]

Perhaps they shouldn't. However, this is reality of the build we have right now and it works in maven with every other plugin and worked like this before for every project we have. Perhaps we could change that, but that would take some time and would not be done in a day.

Since you're mentioning IDE, the current build works perfectly in IntelliJ IDEA and we aren't really using other IDEs, so I don't know how other IDEs handle this.

[snicoll]

We've discussed at the team call and given that provided dependencies are clearly identified as being included in a repackaged app, we're not keen to add this option. If you want to pursue with your build setup, you can easily create a custom layout that excludes the dependencies you don't want. There is a sample that shows you how to get a custom layout and Layout.getLibraryDestination is the method you should override.

[morvenhuang]

pretty silly I'd say

[dantran]

we are running into the same issue where spring-boot plugin pulling in the provided scope 'javax.ws.rs-api' which requires jersey-common at runtime

At the same time we str also using the thin boot, but need the option to switch to fat jar at build time, this means we need 2 layout factories. Spring boot plugin bails with java.lang.IllegalStateException: No unique LayoutFactory found

The layout same sample is not trivial ( sorry about my ignorance), not able to get it working yet. if anyone has already implemented the customer factory, I would like to have a look

Also note: thin-boot does not include provided scope, so I am convinced that spring-boot-plugin violates maven spec/convention and best to provide an option for user to disable provided scope inclusion

[dantran]

i am able to get custom layout working and use 2 maven profiles, one build by default with thin boot and another one with activation

<profile>
  <id>thinjar</id>
  <activation>
    <property>
      <name>!fatjar</name>
    </property>
  </activation>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-maven-plugin</artifactId>
          <version>${spring.boot.version}</version>
          <dependencies>
            <dependency>
              <groupId>org.springframework.boot.experimental</groupId>
              <artifactId>spring-boot-thin-layout</artifactId>
              <version>${thin.boot.version}</version>
            </dependency>
          </dependencies>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</profile>
<profile>
  <id>fatjar</id>
  <activation>
    <property>
      <name>fatjar</name>
    </property>
  </activation>
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-maven-plugin</artifactId>
          <version>${spring.boot.version}</version>
          <dependencies>
            <dependency>
              <groupId>com.xxxx.yyy</groupId>
              <artifactId>mylayout</artifactId>
              <version>${mylayout.version}</version>
            </dependency>
          </dependencies>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</profile>

quite complex. But it works

[brianwhu]

Using configuration exclusion seems to be a much simpler solution. Not sure whether this fits all your other use cases, though.

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                  <excludes>
                    <exclude>
                      <groupId>org.projectlombok</groupId>
                      <artifactId>lombok</artifactId>
                    </exclude>
                  </excludes>
                </configuration>
            </plugin>

[michaelboyles]

I wound up here because I was wondering why Spring Boot packages lombok into my fat jar. Excluding artifacts individually may be okay in some cases, but configuration to opt-out of packaging all provided-scope artifacts sounds like a good idea to me. Explicit excludes are simply not maintainable.

[OrangeDog]

Found a far more painful annotation-processing example: querydsl-apt. If you're using the standard <scope>provided</scope> method you have to exclude all this from the package:

<exclude>
  <groupId>com.querydsl</groupId>
  <artifactId>querydsl-apt</artifactId>
</exclude>
<exclude>
  <groupId>com.querydsl</groupId>
  <artifactId>querydsl-codegen</artifactId>
</exclude>
<exclude>
  <groupId>com.mysema.codegen</groupId>
  <artifactId>codegen</artifactId>
</exclude>
<exclude>
  <groupId>org.eclipse.jdt.core.compiler</groupId>
  <artifactId>ecj</artifactId>
</exclude>
<exclude>
  <groupId>org.reflections</groupId>
  <artifactId>reflections</artifactId>
</exclude>
<exclude>
  <groupId>com.google.code.findbugs</groupId>
  <artifactId>annotations</artifactId>
</exclude>

[snicoll]

@chanseokoh the issue tracker is not the right place to have such conversation. There is a section with some examples that shows you what the plugin does at the moment and how to configure it to do so. If those are not giving you enough information, feel free to raise a separate issue or, better yet, create a PR to improve the documentation.

[elangoravi]

Is someone working on this issue or any update ? Unfortunately, I still am facing problems with exclusion (with provided scope) or wildcards.

@pgerhard @wilkinsona

[wilkinsona]

It has been several months since we've heard from @pgerhard so I doubt that anyone's working on it. I'll update the issue's assignment to reflect that.

[Gyanaranjan1993]

Is there any latest update to this issue?

[wilkinsona]

@Gyanaranjan1993 Nothing beyond what you can see in the issue.

[mnisius]

Oh my god! I just fund this issue...

I was debugging for hours and and was questioning my sanity.

This is really bad, I cannot think of any valid reason why an optional dependency should be packed in the spring boot fat jar. I can see a point for provided scope because of the fat jar thing but it still is a violation of the maven spec/convention so there should be a big, prominent warning in the README regarding the current default behavior.

[OrangeDog]

a violation of the maven spec

No it isn't. An optional dependency is one that this project uses for build/deploy, but if you depend on this project then it's not included by default.

Building a fat jar is for deploying this project. Nothing else should be using the fat jar as a dependency.

If optional dependencies weren't included in the build, there would be no point adding them to the pom in the first place, So why are you if you don't want them?

[mnisius]

Yes it is a violation. please read the official maven docs:

Optional Dependencies

Optional dependencies are used when it's not possible (for whatever reason) to split a project into sub-modules. The idea is that some of the dependencies are only used for certain features in the project and will not be needed if that feature isn't used. Ideally, such a feature would be split into a sub-module that depends on the core functionality project. This new subproject would have only non-optional dependencies, since you'd need them all if you decided to use the subproject's functionality.

However, since the project cannot be split up (again, for whatever reason), these dependencies are declared optional. If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project. This is not the clearest way to handle this situation, but both optional dependencies and dependency exclusions are stop-gap solutions.

https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html

I would agree with you if we would be talking about the provided scope

[wilkinsona]

We intend to change the default behavior with optional dependencies. That change is being tracked by #25403. In the meantime, I think we could update this section of the documentation. I've opened #34636 for that.

[OrangeDog]

If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project.

This is how they behave. It has nothing to do with the fat jars.

[mnisius]

If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project.

This is how they behave. It has nothing to do with the fat jars.

No that is currently not the case. Let me give you an example:

Let's assume you write a library to provide a database specific feature. To implement the feature you need to access die classes in the jdbc driver. However you want support multiple databases. So instead of creating a additional module for each driver you only implement one module with all drivers you want to support as an optional dependency.

In such a case you absolutely don't want the final application to contain all optional jdbc drivers. Instead you expect the developer of the final application to add the driver he/she is actually using as a dependency.

[wilkinsona]

This should already work exactly as you want. The fat jar will not include optional transitive dependencies, only optional dependencies that are declared directly in its own pom.

[OrangeDog]

you write a library to provide a database specific feature

That library will not be building a Spring Boot fat jar, so there's no problem.

[mnisius]

Yes you're right I checked again and was totally wrong in that regard.

I was mislead by a dependency that was marked as optional in my library but was actually also a transitive dependency from another (not optional) dependency. That's why it was packaged in the fat jar. 🙃 Sorry for that.

However I still think it would be a good to have a solution to mark a direct dependency as "this should not land in the fat jar" without the need to exclude it in the spring boot plugin. (for annotation processors and stuff like that)

[mfriedenhagen]

I am really curious why dependencies with scope provided are included. We have a division pom where we included the following in dependencyManagement:

            <dependency>            
                <groupId>org.slf4j</groupId>
                <artifactId>jcl-over-slf4j</artifactId>
                <version>${org.slf4j.version}</version>
                <!--                
                    For spring based application this is provided by spring-jcl
                    which is a compile dependency of spring-core since 5.0.0
                    People using older versions of spring need to overrride
                    the scope in their poms to runtime.
                -->             
                <scope>provided</scope>
            </dependency>   
            <dependency>        
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
                <version>1.2</version>
                <!-- by spring-jcl -->
                <scope>provided</scope>
            </dependency>   
            <dependency>        
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
                <version>1.1</version>
                <!-- by spring-jcl -->
                <scope>provided</scope>
            </dependency>

We use provided e.g. for the javax/jakarta servlet API as well because the implementations are provided by tomcat or jetty. With the WAR plugin this worked like a charm. I only by chance now saw that lombok and commons-logging were included in our fat jars (spring-boot 3.2.x).

[wilkinsona]

When you're building a fat jar, there's no runtime environment like Tomcat or Jetty to provide the dependencies. The jar itself needs to contain all of the dependencies that aren't part of the JDK. For this reason, provided dependencies are included as there's nothing else to provide them.

[mfriedenhagen]

For us there is a runtime, we use spring-boot starters and these then include stripped-down versions of tomcat and jetty.

[wilkinsona]

In that case, it's not clear to me how or why you're using Spring Boot's fat jar support which is intended to produce a standalone artifact that does not require a runtime other than the JVM.

[mfriedenhagen]

Maybe the wording does confuse me here?

We build spring-boot applications as executable fat jar and package these as Debian DPKG (depending on a JRE) resp. put the same "fat" jar into a Docker image which only contains a JRE.

              <plugin>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${spring-boot.version}</version>
                    <configuration>
                        <classifier>exec</classifier>
                        <executable>true</executable>
                    </configuration>
                </plugin>

Alternatively these may be started with java -jar EXEC.jar as well.
We depend on org.springframework.boot:spring-boot-starter-web and that one does include an embedded TomCat by default.
Logback does come as a dependency from org.springframework.boot:spring-boot-starter-actuator.

[mstahv]

Dropping in my insight why it would be great to get this (or technically the optional part discussed in #25403) in.

In Vaadin projects we have these certain developement time extensions, which are essentially brought in by a dependency vaadin-dev. As we don't want those to end up in production artifacts, there is curently this ugly production Maven profile in all projects using Vaadin, which exclude that dependency. That should be infamously known by @snicoll & @wilkinsona at least, kind of pain in the ass it is. I was recently investigating how we could get rid of completely and thought I got a great idea that "the way how spring-boot-devtools does it" should work for us as well. But during my tests I found out that it appears to work with a special rule in the spring-boot-maven-plugin and found this ticket as well.

So, pretty please, consider taking this (or at least the optional part) to your TODO list. It would pave way to make Vaadin builds look cleaner and we could remove some code from Spring Initializr 🤓. I guess there might be some rare regressions, but I guess those apps are today working "accidentally" as there is no good reason today to have optional dependency in Spring Boot project 🤷‍♂️

Gradle users are lucky with their developmentOnly "scope". I guess it would be best in the long term for all (including Spring Boot Maven Plugin) if Maven would get similar scope in the future.

The related Vaadin issue: vaadin/flow#17737

[mstahv]

I was trying to make a (somewhat ugly) workaround by making another plugin (that would finally be vaadin plugin) adjust the excludes, but I didn't come up with any other way but properties to defined excludes from another plugin. But then I couldn't figure out how to configure excludes via property.

I guess this tiny enhancment/fix is missing (and probably can be handy for others for less ugly hacks) 🤔 #39837