Unable to set CAPSULE_CACHE_DIR env var in capsule manifest about capsule HOT 22 CLOSED

You can't set CAPSULE_CACHE_DIR in the manifest. The environment variables you set in the manifest are "outgoing" variables: they will be set by the capsule in the environment used to run your application. CAPSULE_CACHE_DIR however, is an "incoming" variable, which Capsule reads from its own environment. Also, it makes sense: a capsule has to be portable, and it's the decision of the capsule user -- not its author where to place the cache.

So, in order to set the capsule cache to a different directory, set CAPSULE_CACHE_DIR in the shell used to run the capsule or in your .bashrc or similar initialization file (or, on Windows, in your global environment variables).

from capsule.

Actually, you know what? Let me fix the cache-dir creation when Extract-Capsule is set to false. Also, can you send me the stack trace of the error you've mentioned?

from capsule.

That should work, because OPS will generally use the bash wrapper I have that sets the env var if it's not there, since they could care less about one library's caching, unless there are implications to the overall product's running.

And as a fallback, if a dev or other user from other departments use the capsule jar directly, they will operate on the fat jar, just without caching, which doesn't stop the correctness of the program.

I will get you the relevant portions of the error / stacktrack from the Extract-Capsule option.

Thanks for the follow-up!

from capsule.

Also, before getting to that error/stacktrace, a couple of things that would've personally helped me in my first workings with capsule:

• When am I configuring / interacting with the capsule loader process vs. the app process? I think that is one good partitioning of organization in the documentation, or at least an opportunity for a separate section.
• Is there a way to remotely debug the capsule loader process? Thank you for pointing out how to pass the debug jvm args to the app process, but I found that I wanted to download Capsule.class and step through it, but I couldn't think of a good way to attach a debugger to that process, since the port collision issue will come up with the app process.

Thanks!

from capsule.

Hi @pron,

I noticed that Capsule logging happens against System.err, so that's where I looked for the stack trace. Unfortunately, there is no stack trace, see below my output from running my capsule jar:

$ java -jar ./my-capsule.jar -Dcapsule.log=verbose myAppSpecificOptionsAndArgsList
Error: Could not find or load main class #####

from capsule.

You can't pass JVM arguments to java after specifying the jar. -Dcapsule.log=verbose must appear before -jar my-capsule (all following arguments are passed to your program). If you're still not getting a stack trace, the problem may be that your capsule hasn't been built correctly (specifically, the manifest's Main-Class hasn't been set, or it has been set to a class that isn't in the jar.

from capsule.

Ah, good catch!

Here's the output of my manifest and updated bash command, respectively (I'm replacing all specific class ref's with X). Note that it looks like I was lying. My VM is running JRE 7.045, not 7.025 ;)

Manifest-Version: 1.0
Main-Class: Capsule
Application-Class: X
Extract-Capsule: false

$ java -jar -Dcapsule.log=verbose ./my-capsule.jar myAppSpecificOptionsAndArgsList
CAPSULE: Launching app X
CAPSULE: Using JVM: /usr/java/jdk1.7.0_45/jre
CAPSULE: /usr/java/jdk1.7.0_45/jre/bin/java -Dcapsule.app=X -Djava.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib -Dcapsule.jar=/path/to/clt-project-SNAPSHOT/my-capsule.jar -classpath /path/to/clt-project-SNAPSHOT/my-capsule.jar X myAppSpecificOptionsAndArgsList
Error: Could not find or load main class X

Regarding the Main-Class prop of the manifest, this is set automatically by capsule-gradle-plugin. Is it incorrect the way it's formatted (to set Main-Class as Capsule and Application-Class as my app's class)?

from capsule.

Yeah, the way the gradle plugin is correct.
What's happening here, I'm guessing, is that class X isn't directly in your JAR, but in an internal JAR. If that's the case, you must have the Extract-Capsule attribute set to true (the default), or put your app's classes directly in the capsule (I don't know how to tell the gradle plugin to do that. @danthegoodman ?).

Extract-Capsule can only be set to false if you have no embedded dependencies and your application's classes are placed directly in the capsule jar. But if you have only external dependencies you still need the cache because that's where there downloaded.

So the only way you can do without the cache directory at all is if your application has absolutely no dependencies. Is that the case?

from capsule.

The un-jar'ing (jar xvf ) of my capsule jar shows that the jar containing the application main class is present at the root of the classpath.

from capsule.

Right, so it's an internal jar inside the capsule jar, rather than in the capsule jar directly. But before you try to do that, consider that if you have any dependencies -- embedded or external -- then Capsule must make use of the cache directory (which you can set with the env var).

from capsule.

Alright, that makes sense. From a conceptual standpoint, what is a good scenario when to not extract jars, then?

Let me step back and revisit how I was thinking about the overall flow, and maybe you can help clear up any confusion.

I came over from OneJar (specifically the "executable-jar" gradle plugin), and after debugging issues with its classloading (specifically, the way it interacted with Spring's ClassPathResource plumbing), I couldn't continue with it to make my project work. It appeared that it would construct "jar url's", and use that to allow you to dig out resources from any nested jar's within the main jar. One advantage I thought this gained was the ability to be able to "dig into nested jars" in a simple way and also to scope dependency versions to the individual nested jars. However, I may have been mis-conceptualizing that, as I didn't dig too deep into it. The "executable-jar" plugin has stated that it is now abandoned and to use the gradle capsule plugin, instead, but to be honest, I did not give full thought to the implications of moving over, since it was recommended by the executable-jar author that capsule would be a good replacement.

You did a good job of basically explaining what capsule is trying to do in your documentation, but it would be good to better understand, based off what was gone over in this ticket, what capsule is doing conceptually, and how it is a good replacement for OneJar, and what are some differences with OneJar?

I was surprised by the need to cache local copies of dependencies, so maybe explaining why that is needed would be a good place to start for me to better understand how to use Capsule.

from capsule.

Alright, that makes sense. From a conceptual standpoint, what is a good scenario when to not extract jars, then?

When you don't have embedded dependencies (only external ones).

... what capsule is doing conceptually, and how it is a good replacement for OneJar, and what are some differences with OneJar?

Certainly. Capsule work very differently from OneJar, and has many advantages over it, as well as one disadvantage. Unlike OneJar, Capsule does not employ its own clever class loader to "dig into nested jars" at all. Instead, the capsule JAR is extracted (well, most of it; class files aren't extracted but embedded JARs are) into a cache directory, and a new JVM instance is spawned with a command line that Capsule constructs. Capsule's advantages are:

• No interference with custom class loaders
• Support for native libraries -- not just Java libraries.
• Support for setting JVM flags
• Support for setting the boot classpath.
• Support for Java and native instrumentation agents.
• It can choose an appropriate JRE to launch the application, possibly with a different version from the one used to launch the capsule.
• Support for external Maven dependencies that are declared in the manifest rather than embedded (the dependencies can be automatically upgraded) -- they will be downloaded the first time a capsule is launched and cache for future use by the capsule or other capsules depending on the same libraries.

OneJar can do none of those things. Capsule's disadvantage is that it needs write access to some library in the filesystem.

So the only case when Capsule does not need to write to the local filesystem is if the application has no dependencies, or if its dependencies are merged with the application files using something like gradle-shadow-plugin.

from capsule.

Hi @pron,

Thanks for the great explanation! That cleared up a lot of confusion for me. While I believe that needing write access to the local file system, as well as needing an env var set to specify where the cache is placed, is not ideal (imo), the point you made about "no interference with custom class loaders" is a big win. From what I've read about doing custom class loading, it is very easy to get wrong, with many subtle bugs. I also experienced that more intimately by having to step through 80+ stack frames with OneJar's classloader ;)

Also, I now understand the basic motivation behind "extract-capsule" better, so if I ever need to use it, I will know if it's appropriate for that or not. In my current case, it is not appropriate.

I will go with the bash wrapper approach of setting the env var, if it's not already set, and update my team about the env var if they interact with the capsule jar directly.

I will mark this issue as closed, and one final request would be to see if there's any tweaks to documentations are in order based off this ticket?

Thanks for the patience and help!

from capsule.

any tweaks to documentations are in order based off this ticket?

What would you suggest?

from capsule.

Also, can you say which directories the process does have access to? Perhaps we can let Capsule fall back to some temp subdirectory under /tmp if it can't write to the home dir and no alternative cache dir is specified.

from capsule.

A couple of things would help, in my mind:

• A section that describes what can be done with the capsule loader process vs. the app process. Most specifically related to which configuration applies to which process. I don't know if this can be fixed, but if there's any tools/guidance that can be given for interacting with each process. For example, you explained in the -Dcapsule.jvm.args piece that remote debugging setup would naturally go there. This would tell me that I'm configuring the args for the second jvm being setup. Also, the explanation of Environment-Variables in the manifest referencing outgoing vars to the spawned process would help clear things up. Both of those things, and potentially others, would nicely fall under the heading of "how to interact with the app process".
• A personal request would be to provide guidance (if there is any to be given) on how to interact with the bootstrapping process. If there isn't a way, just a blurb explaining that in this proposed new section of the documentation would be helpful.
• Also, a separate section outlining the main advantages / disadvantages of each competing library in detail would be super helpful. If I was coming from a different library, and I saw a section like this in TOC, I would go there first. I know you do give a general outline of some key points that are differences in your document, but maybe having dedicated comparison sections would be more helpful.

from capsule.

Due to the network infrastructure in our setup, the running process won't be able to write to the process' user home dir. We were also planning on going to /tmp, which seems like a natural place for this cache, as it can be deleted at any time, and regenerated.

from capsule.

Thank you. I will make some changes to the docs in the coming weeks. But what do you mean by "interacting with the bootstrapping process"?

from capsule.

My main interaction would be attaching a remote debugger to the bootstrapping process itself, if I wanted to debug it. If I try to attach one now, through normal setup, the second process will already have tried to bind to the same port.

from capsule.

Oh, there's a trick for that (intentionally undocumented): just add -Dcapsule.trampoline. The app won't be launched at all; instead, Capsule will just print to STDOUT the command line to launch the application (after extracting the JAR and/or downloading external dependencies). I.e., Capsule will do everything it does except for actually launching the application.

from capsule.

Oh, that's great to know! That may be good to include in this new proposed section of "interaction with the different processes", but I wouldn't argue with keeping it undocumented.

from capsule.

@pron, in response to:

If that's the case, you must have the Extract-Capsule attribute set to true (the default), or put your app's classes directly in the capsule. (I don't know how to tell the gradle plugin to do that.)

If you want just the app's classes directly in the capsule, you would use a ThinCapsule type instead of a FatCapsule. The default thin/fat capsule configuration should line up with the definitions in your README under Usage Examples.

And, like you mentioned later, if you're wanting to also decompose a dependency jar and include that in a thin capsule as well, you would need to use the gradle-shadow-plugin.

Also, another advantage to Capsule is the ability to work with signed jars, which is necessary for at least the BouncyCastle cryptography library. The signed code in there will not function when they've been copied to a different jar. </lessons-learned-the-hard-way>

from capsule.