Classpath too long… with Spring Boot and Gradle

Java applications get more and more complex and we rely on more libraries than before. But command lines have some length limits and eventually you can get into troubles if your classpath gets too long. There are ways how to dodge the problem for a while – like having your libraries on shorter paths. Neither Gradle nor Maven are helping it with their repository formats. But this is still just a pseudo-solution.

Edit (Oct 29): I added section about the plugin that makes the whole solution automatic would work if it treated paths as URLs.

When you suddenly can’t run the application

On Windows 10 we hit the wall with the command line of our Spring Boot application going over 32 KB. On Linux this limit is configurable and in general much higher, but there is often some hard limit for a single argument – and classpath is just that, a single argument. The question is whether we really want to blow our command line with all those JARs or whether we can do better.

Before we get there, though, let’s propose some other solutions:

• Shorten the common path (as mentioned before). E.g. copy all your dependencies into something like c:\lib and make those JARs your classpath.
• With all the JARs in a single place, you may actually use Java 6+ feature -cp “lib/*”. That is, wildcard classpath using * (not *.jar!) and quotes. This is not a shell wildcard (that would just expand it into a long command line again) but actual feature of java command (here docs from version 7). This is actually quite usable and it also scales – but you have to copy the JARs.
• Perhaps you want to use environment variable CLASSPATH instead? This does not work, the limit is 32 KB as well. So no-solution.
• You can also extract all the JARs into a single tree and then repackage as a single JAR. This also scales well, but involves a lot of disk operations. Also, because first class appearance is used, you have to extract the JARs in classpath order without overrides (or in reverse with overrides).

From all these options I like the second one the best. But there must be some better one, or not?

JAR with Class-Path in its manifest

I’m sure you know the old guy META-INF/MANIFEST.MF that contains meta-information about the JAR. It can also contain the classpath which will be added to the initial one on the command line. Let’s say the MANIFEST.MF in some my-cp.jar contains a line like this:

Class-Path: other1.jar other2.jar

If you run the application with java -cp my-cp.jar MainClass it will search for that MainClass (and other needed ones) in both “other” JARs mentioned in the manifest. Now I recommend you to experiment with this feature a bit and perhaps Google around it because it seems easy, but it has couple of catches:

• The paths can be relative. Typically, you have your app.jar with classpath declared in manifest and deliver some ZIP with all the dependencies with known relative paths from your app.jar. You can still run your application with java -cp app.jar MainClass, or, even better, java -jar app.jar with Main-Class declared in the manifest as well.
• The paths can also be absolute, but then you need to start with a slash (natural on Linux, not so on Windows). On Windows it can be any slash actually, I guess it works the same on Linux (compile once, run anywhere?).
• If the path is a directory (like exploded JAR) it has to end with a slash too.
• And with spaces you get into some escaping troubles… but by that time you’d probably figure out that the paths are not paths (as in -classpath argument) but in fact URLs.
• Now throw in the specifics of the format of MANIFEST.MF like max line length of 72 chars, continuation lines with leading space, CRLF, … Oh, and if you try to do your Manifest manually, don’t forget to add one empty line – or, in other words, don’t forget to terminate the last line with CRLF as well. (Talking about line separators and line terminators can get very confusing.)

Quickly you wish you had a tool that does this all for you. And luckily you do.

Gradle to the rescue

We actually had also specific needs for our classpath. We ran bootRun task with the test classes as well for development reasons. In the end, bootRun is not used for anything but for development, right?

Adding test runtime classpath to the total classpath “helped” us to go over that command-line limit too. But we still needed it. So instead of just having classpath = sourceSets.test.runtimeClasspath in the bootRun section we needed to prepare the classpath JAR first. For that I created classpathJar task like so:

task classpathJar(type: Jar) {
  inputs.files sourceSets.test.runtimeClasspath

  archiveName = "runboot-classpath.jar"
  doFirst {
    // If run in configuration phase, some artifacts may not exist yet (after clean)
    // and File.toURI can’t figure out what is directory to add the critical trailing slash.
    manifest {
      def classpath = sourceSets.test.runtimeClasspath.files
      attributes "Class-Path": classpath.collect {f -> f.toURI().toString()}.join(" ")
    }
  }
}

This code requires couple of notes, although some of it is already in the comments:

• We need to treat the files (components of the classpath) as URLs and join them with spaces.
• To do that properly, all the components of the classpath must exist at the time of processing.
• Because after clean at the time of the task configuration (see Gradle’s Build Lifecycle) some components don’t exist yet, we need to set the classpath in the task execution phase. What may not exist yet? JARs for other projects/modules of our application or classes dirs for the current project. Important stuff, obviously. (If you run into seemingly illogical class not found problems, this may be the culprit.)
• Another reason why these artifacts may not exist is missing proper dependencies. That’s why I mention all three concatenated components of the classpath in the inputs.files declaration.

EDIT: For the first day of this post I’ve got dependsOn instead of inputs.files. It was a mistake causing unreliable task execution when something upstream was changed. Sorry for that. (I am, I suffered.)

And that’s it

Now we need to just mention this JAR in the bootRun section:

bootRun {
  classpath = classpathJar.outputs.files
  //…other settings, like...
  main = appMainClass // used to specify alternative "devel" main from test classpath
}

I’m pretty sure we can do this in other build tools and we can make some plugin for it too. It would probably be possible with some doFirst directly in the bootRun, but I didn’t want to mix it there.

But again, this nicely shows that Gradle lets you do what you need to do without much fuss. It constantly shouts: “Yes, you can!” And I like that.

But wait, there’s a plugin for it!

EDIT: October 28th, 2018

Weeks after I resolved the problem myself I decided to re-search the internet about it with fresh head again. Through this issue I found gradle-utils-plugin and its recently updated clone. I decided to use the latter and when I removed my original solution (bootRun task in my case has classpath = sourceSets.test.runtimeClasspath) all I need to do is add plugin declaration at the beginning of the build script:

plugins {
    // solves the problem with long classpath using JAR instead of classpath on command line
    id "ua.eshepelyuk.ManifestClasspath" version "1.0.0"
}

(EDIT: November 2nd) However – there’s a twist. This plugin does not work with spaces – that is, it’s not tested and it’s broken. It does not treat paths as URLs, which is critical in case of class-path in Manifest. Repository does not offer “Issues”, so we’re back to our own solution.