slimjar / slimjar Goto Github PK

View Code? Open in Web Editor NEW

57.0 5.0 16.0 656 KB

JVM Runtime Dependency Management.

License: MIT License

Java 88.97% Kotlin 11.03%

java gradle java16 java8 dependency-manager

slimjar's Introduction

Slim Jar

Runtime Dependency Management

What is SlimJar?

SlimJar allows you to download and load up dependencies at runtime as an alternative to shading your dependencies. This helps you reduce build output size and share downloaded dependencies between projects at client side. It is built mainly with the gradle eco-system in mind and is easily configurable being an almost a drop-in replacement/add-on to gradle projects.

Why use SlimJar?

SlimJar makes the process of switching out jars easier by providing jars that are much lesser in size, all "slimmed" dependencies are already available and do not need to be explicitly moved back to your working directory during an update or change. This can be extremely useful for users who have lower bandwidth connections to push large updates to production or testing environments. It also provides vital features such as package relocation, module isolation, auto configuration generation...etc with the simplicity of minor tweaks in your build file.

Usage Example

Note: Use the shadowJar task to compile your project

// this needs to be ran before you reference your dependencies
ApplicationBuilder.appending("MyApplicationName").build()

(NOTE: If you have specified relocations and are running in a IDE or any environment that does not use the shadowjar-ed build file, use the ignoreRelocation flag while running by using -DignoreRelocation in your runner arguments) build.gradle GROOVY DSL

plugins {
  id 'com.github.johnrengelman.shadow' version '6.0.0'
  id 'io.github.slimjar' version '1.3.0'
}
dependencies {
  implementation slimjar("1.2.6")
  slim 'group.id:artifact.id:version'
}

slimJar {
  relocate 'a.b.c', 'm.n.o'
}

(For Kotlin DSL, to use the slimjar extension in dependencies block, you will need the following import - import io.github.slimjar.func.slimjar)

Development setup

git clone https://github.com/SlimJar/slimjar.git
gradlew test

Releases

Distributed under the MIT license. See LICENSE for more information.

Contributing

Fork it (https://github.com/SlimJar/slimjar/fork)
Create your feature branch (git checkout -b feature/abcd)
Commit your changes (git commit -am 'Added some feature abcd')
Push to the branch (git push origin feature/fooBar)
Create a new Pull Request

slimjar's People

Contributors

Stargazers

Watchers

Forkers

conclube lichthund emilyy-dev alessiodp mhugi-hxyz forkstuffs lorenzo0111 powercasgamer joshuasing daracci cyberflamego robothanzo

slimjar's Issues

Choose download location for SlimJar's own dependencies

Already talked about this in #36, but decided to make a new issue for tracking purposes. It would be great if I could choose the location where SlimJar drops its own dependencies, out of the $HOME directory. We're waiting to implement SlimJar into our plugin, but currently this feature being missing is blocking our implementation.

sometimes slimjar task stucks

14:24:16: Executing task 'publishToMavenLocal'...

> Task :publishToMavenLocal UP-TO-DATE
> Task :common:generatePomFileForShadowPublication

> Task :common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :common:processResources NO-SOURCE
> Task :common:slimJar

its not passing the slimjar task for about 4~ minutes. but sometime it does immediately.

Cannot resolve GSON

SlimJar always gives errors about gson download.

Stacktrace

Accidental PR

Could not determine the dependencies of task ':publishMavenJavaPublicationToMavenRepository'.

I'm quite sure that it is a slimjar problem.

When I try to run the task gradlew clean build publish it gives me the error in the title.
You can view a build log here.

build.gradle: https://github.com/Lorenzo0111/MultiLang/blob/82fe6c6f77be4cdc135d8e41d09d6e7ab15fadad/build.gradle

Unexpected checksum behavior.

Imagine someone edits the sha1 file, the slimjar library downloading the file again, that's good but, it's not deleting and updating the corrupted sha1 file so, it's printing like that:

[23:48:38 INFO]: [InfumiaLibraryPlugin] Checksum match failed for morphia-core
[23:48:43 INFO]: [InfumiaLibraryPlugin] Resolved morphia-core @ https://papermc.io/repo/repository/maven-public/dev/morphia/morphia/morphia-core/2.2.1/morphia-core-2.2.1.jar
[23:48:43 INFO]: [InfumiaLibraryPlugin] Downloading morphia-core...
[23:48:43 INFO]: [InfumiaLibraryPlugin] Verifying checksum for morphia-core
[23:48:43 INFO]: [InfumiaLibraryPlugin] Checksum match failed for morphia-core
[23:48:43 INFO]: [InfumiaLibraryPlugin] Downloaded morphia-core successfully!

So, probably the solution would be if the dep. is corrupted it should delete all files in the dep's version directory, then download again.

Skip download if already exists

Some users are reporting me this error:

[07:38:39] [Server thread/WARN]: UnresolvedDependencyException{dependency=Dependency{groupId='mysql', artifactId='mysql-connector-java', version='8.0.25', snapshotId='null', transitive=[]}}
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.downloader.URLDependencyDownloader.lambda$download$0(URLDependencyDownloader.java:64)
[07:38:39] [Server thread/WARN]: 	at java.util.Optional.orElseThrow(Optional.java:290)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.downloader.URLDependencyDownloader.download(URLDependencyDownloader.java:64)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.injector.helper.InjectionHelper.fetch(InjectionHelper.java:45)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.injector.SimpleDependencyInjector.injectDependencies(SimpleDependencyInjector.java:56)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.injector.SimpleDependencyInjector.inject(SimpleDependencyInjector.java:50)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.addons.external.slimjar.app.builder.InjectingApplicationBuilder.build(InjectingApplicationBuilder.java:51)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.common.ADPPlugin.loading(ADPPlugin.java:64)
[07:38:39] [Server thread/WARN]: 	at com.alessiodp.parties.core.bukkit.bootstrap.ADPBukkitBootstrap.onLoad(ADPBukkitBootstrap.java:29)
[07:38:39] [Server thread/WARN]: 	at org.bukkit.craftbukkit.v1_8_R3.CraftServer.loadPlugins(CraftServer.java:297)
[07:38:39] [Server thread/WARN]: 	at net.minecraft.server.v1_8_R3.DedicatedServer.init(DedicatedServer.java:198)
[07:38:39] [Server thread/WARN]: 	at net.minecraft.server.v1_8_R3.MinecraftServer.run(MinecraftServer.java:525)
[07:38:39] [Server thread/WARN]: 	at java.lang.Thread.run(Thread.java:748)

I think its happening because Maven central was down for some time.
Is it possible to do not contact Maven central (download) if the file has already been downloaded? This will allow servers to work offline if libraries are present.

Project fails to build with Gradle plugin v1.2.2

For some reason, package io.github.slimjar.app.builder does not exist when using Slimjar Gradle plugin version 1.2.2. With 1.2.1 it builds fine.

See gradle log here: https://github.com/plan-player-analytics/Plan/runs/3269302665

Auto-apply slimjar plugin on isolated modules not working

Plugin does not seem to apply itself to child isolated modules

Add the ability to provide own logger instance

Also from #36, for tracking purposes.

It would be great to be able to provide my own Logger instance to SlimJar, since many platforms don't log fine level logs and it's sometimes a hassle to change the log level of the parent logger.

it would be good if you push the sources with javadocs of slimjar artifact to the repository

[Question/Suggestion] Redundant relocated jars

I've noticed slimjar generates a copy of every dependency for every application using it under a relocated folder regardless of whether anything was relocated. I'm curious as to whether there's a reason for this, seeing as the files aren't exactly identical in size. If not, is there currently a way to avoid this behaviour?

Improve Constructor Resolution for Isolated Applications

Constructor resolution in IsolatedApplicationBuilder is currently quite poor. Needs to be improved to find most suitable and expected constructor given types

Add exclude feature to "slim"

I tried to exclude un-needed dependencies from a dependency like that

slim("org.jetbrains.kotlin:kotlin-stdlib") {
    exclude group: "org.jetbrains", module: "annotations"
}

It's not working, or there is no any feature such like that, idk. But this should be good to customize the repo/deps json.
The result json is like that

{
  "groupId": "org.jetbrains.kotlin",
  "artifactId": "kotlin-stdlib",
  "version": "1.5.21",
  "transitive": [
	{
	  "groupId": "org.jetbrains",
	  "artifactId": "annotations",
	  "version": "21.0.1",
	  "transitive": []
	},
	{
	  "groupId": "org.jetbrains.kotlin",
	  "artifactId": "kotlin-stdlib-common",
	  "version": "1.5.21",
	  "transitive": []
	}
  ]
}

This case should be gone

{
  "groupId": "org.jetbrains",
  "artifactId": "annotations",
  "version": "21.0.1",
  "transitive": []
}

Wrong exception caught in VerifyingRelocationHelper

Wrong exception was caught in VerifyingRelocationHelper which is supposed to throw NoSuchFileException if attribute isnt found while FileNotFoundException is caught mistakenly.

IllegalArgumentException: URI is not hierarchical

The following exception occurred during relocation when trying using application builder

java.lang.IllegalArgumentException: URI is not hierarchical
        at java.io.File.<init>(File.java:418) ~[?:1.8.0_241]
        at io.github.slimjar.relocation.helper.VerifyingRelocationHelperFactory.<init>(VerifyingRelocationHelperFactory.java:53) ~[VerifyingRelocationHelperFactory.class:?]
        at io.github.slimjar.app.builder.ApplicationBuilder.getRelocationHelperFactory(ApplicationBuilder.java:370) ~[ApplicationBuilder.class:?]
        at io.github.slimjar.app.builder.ApplicationBuilder.createInjector(ApplicationBuilder.java:447) ~[ApplicationBuilder.class:?]
        at io.github.slimjar.app.builder.InjectingApplicationBuilder.buildApplication(InjectingApplicationBuilder.java:58) ~[InjectingApplicationBuilder.class:?]
        at io.github.slimjar.app.builder.ApplicationBuilder.build(ApplicationBuilder.java:462) ~[ApplicationBuilder.class:?]
        at com.djrapitops.plan.PlanSponge.onLoad(PlanSponge.java:134) ~[PlanSponge.class:?]
        at com.djrapitops.plan.PlanSponge.onServerLoad(PlanSponge.java:109) ~[PlanSponge.class:?]
        at org.spongepowered.common.event.listener.GamePreInitializationEventListener_PlanSponge_onServerLoad7.handle(Unknown Source) ~[?:?]
        at org.spongepowered.common.event.RegisteredListener.handle(RegisteredListener.java:97) ~[RegisteredListener.class:1.12.2-7.3.1-RC391]
        at org.spongepowered.common.event.SpongeEventManager.post(SpongeEventManager.java:435) ~[SpongeEventManager.class:1.12.2-7.3.1-RC391]
        at org.spongepowered.common.event.SpongeEventManager.post(SpongeEventManager.java:480) ~[SpongeEventManager.class:1.12.2-7.3.1-RC391]
        at org.spongepowered.common.SpongeImpl.postEvent(SpongeImpl.java:255) ~[SpongeImpl.class:1.12.2-7.3.1-RC391]
        at org.spongepowered.common.SpongeImpl.postState(SpongeImpl.java:263) ~[SpongeImpl.class:1.12.2-7.3.1-RC391]
        at org.spongepowered.server.SpongeVanilla.preInitialize(SpongeVanilla.java:135) ~[SpongeVanilla.class:1.12.2-7.3.1-RC391]
        at net.minecraft.server.dedicated.DedicatedServer.handler$zpc000$vanilla$onServerLoad(SourceFile:2235) ~[nz.class:?]
        at net.minecraft.server.dedicated.DedicatedServer.func_71197_b(SourceFile:117) ~[nz.class:?]
        at net.minecraft.server.MinecraftServer.run(SourceFile:434) ~[MinecraftServer.class:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

Java version

java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

Slimjar version

Gradle plugin 1.3.0
Library 1.2.6

Another developer (@Vankka) gave some extra info:

VerifyingRelocationHelperFactory.class.getProtectionDomain().getCodeSource().getLocation(); gives different URI:s when executing from IDE vs actual application
file:/home/vankka/IdeaProjects/Plan/Plan/common/build/libs/common-5.4-SNAPSHOT-all.jar

jar:file:///home/vankka/mctest/spongevanilla8/./mods/sponge-5.4-SNAPSHOT-all.jar!/io/github/slimjar/relocation/helper/VerifyingRelocationHelperFactory.class
top is via jshell in ide, bottom is from when the plugin is initialized.

seems to me like it doesn't like the File constructor doesn't like there being two schemes (jar and file)

Occurs on both Linux and Windows

Here's the application builder in use

ApplicationBuilder.appending("Plan")
        .logger((message, args) -> java.util.logging.Logger.getGlobal().log(Level.INFO, message, args))
        // Use paper repository for downloading slimjar dependencies
        .internalRepositories(Collections.singletonList(new Repository(new URL("https://papermc.io/repo/repository/maven-public/"))))
        .downloadDirectoryPath(Paths.get(getDataFolder().getAbsolutePath()).resolve("libraries"))
        .build();

Download directory gets removed after dependencies are loaded

When using ApplicationBuilder.appending("Plan").downloadDirectoryPath(Paths.get(getDataFolder().getAbsolutePath()).resolve("libraries")).build(); to load deps with SlimJar, after a while, the whole libraries folder disappears from the plugin's data folder. This happens in a Pterodactyl Docker container running Java 16.

Doesn't support downloading without the Content-Length header

When downloading from

slimjar/slimjar/src/main/java/io/github/slimjar/downloader/output/ChanneledFileOutputWriter.java

Line 49 in b11e575

output.getChannel().transferFrom(channel, 0, length);

an error occurs if the Content-Length header of the content/file being read is undefined: https://pastes.dev/hoxs2QmtoA

Example of Kotlin stdlib from Maven Central: https://reqbin.com/n5ujzrp4
There's a valid Content-Length header

Trove4J from a maven repository: https://reqbin.com/npv13ydb
There's no Content-Length header

To fix this issue, you could just put "unknown" if there's no Content-Length header in

slimjar/slimjar/src/main/java/io/github/slimjar/downloader/output/ChanneledFileOutputWriter.java

Line 46 in b11e575

LOGGER.log(Level.FINE, "Writing {0} bytes...", length);

Then to fix the real issue, you can do:

LOGGER.log(Level.FINE, "Attempting to write from inputStream...");
// check if outputFile doesn't exist
LOGGER.log(Level.FINE, "Writing {0} bytes...", length == -1 ? "unknown" : length);
if (length == -1) { // or just don't use the channel stuff, but I'm keeping it since I don't know what the diff between the channel is
    Files.copy(inputStream, outputFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
    inputStream.close();
    return outputFile;
}
try (final ReadableByteChannel channel = Channels.newChannel(inputStream)) {
    try (final FileOutputStream output = new FileOutputStream(outputFile)) {
        output.getChannel().transferFrom(channel, 0, length);
    }
}
inputStream.close();
return outputFile;

or something similar to that

Working Fork
https://github.com/dkim19375/slimjar

Commit that fixes this issue (tested)
https://github.com/dkim19375/slimjar/commit/07fa6d1ccfce1421271a84abbd835e5e1860730e

VerifyingRelocationHelper - No data available

Hello,
an user reported me this error in his MC server console:

[13:07:40 WARN]: com.alessiodp.parties.core.common.addons.external.slimjar.injector.InjectionFailedException: SlimJar failed to inject dependency: name -> Dependency{groupId='com.h2database', artifactId='h2', version='1.4.200', snapshotId='null', transitive=[]}
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.injector.SimpleDependencyInjector.injectDependencies(SimpleDependencyInjector.java:63)
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.injector.SimpleDependencyInjector.inject(SimpleDependencyInjector.java:50)
...
...
[13:07:40 WARN]: Caused by: java.nio.file.FileSystemException: /hlserver/27897/plugins/Parties/libraries/com/h2database/h2/1.4.200/relocated/Parties/h2-1.4.200.jar: Unable to get size of extended attribute 'slimjar.owner': No data available
[13:07:40 WARN]: at java.base/sun.nio.fs.LinuxUserDefinedFileAttributeView.size(LinuxUserDefinedFileAttributeView.java:153)
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.relocation.meta.AttributeMetaMediator.readAttribute(AttributeMetaMediator.java:43)
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.relocation.helper.VerifyingRelocationHelper.relocate(VerifyingRelocationHelper.java:68)
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.injector.helper.InjectionHelper.fetch(InjectionHelper.java:49)
[13:07:40 WARN]: at com.alessiodp.parties.core.common.addons.external.slimjar.injector.SimpleDependencyInjector.injectDependencies(SimpleDependencyInjector.java:56)
[13:07:40 WARN]: ... 19 more

It seems it started to do that after some time that the libraries folder has been created, deleting that folder and restarting the server resolved the issue.
What can be the root cause?

Dependency URL Path with Space causes injection to fail.

URL#getPath() uses URL formatting and replaces spaces by %20 which breaks injection by Instrumentation.

Library requiring loading into PluginClassLoader

On spigot 1.17 it gives this error with all libraries:

IllegalArgumentException: class <class> is not provided by class org.bukkit.plugin.java.PluginClassLoader

is it a bug? is there a way to fix?

Package io.github.slimjar.app.builder does not exist

Hey. Been trying to use SlimJar in my project but I get this error:

error: package io.github.slimjar.app.builder does not exist
import io.github.slimjar.app.builder.ApplicationBuilder;
                                     ^

Relevant lines in my build.gradle.kts

id("com.github.johnrengelman.shadow") version "7.0.0"
id("io.github.slimjar") version "1.2.2"

SlimJar repository

I can't find the new slimjar repository. Where can I download it?

Snapshot dependencies not updating

Reproduce:

Upload snapshot dependency
Run mc server using that dependency
Upload another snapshot
Run mc server
Notice the server using the first snapshot

Note that I've only tested this once so gradle might not have gotton the latest snapshot, but uh, i doubt that since all the other times I've reloaded gradle dependencies it got it

Problem when downloading a dependency

I don't know if it is a slimjar problem.
When I try to load the jar it gives an error.

Files:

Change default download repo

Hey, is there a way to change the repository list for downloading at runtime? Slimjar currently uses your own repo (repo.vshnv.tech) to download many dependencies for us, however we'd like to rely on Maven central by default since they use a CDN & are more reliable in general.

Depending on a project that uses slimjar requires adding a repository

Say project A uses slimjar, and B is a project that depends on A.

compileOnly(A) works fine, but

implementation(A) will result in an error (ex. Could not resolve io.github.slimjar:slimjar:1.2.0).

To fix this B must add https://repo.vshnv.tech/ to its list of repositories, which is a little annoying. Is this intended?

Can't inject dependencies

The injector works for most users but there is an user who reported the injector is not working with him.
I tried lots of things but nothing..

Full log here.

If you need some other file just ask. Thanks in advance

Internal dependency String Literals Subject to relocation

Dependency package/artifact literals are present in raw form, for eg (https://github.com/SlimJar/slimjar/blob/master/slimjar/src/main/java/io/github/slimjar/relocation/facade/ReflectiveJarRelocatorFacadeFactory.java#L86), this might cause user relocations to affect loading.

Gradle platform support

Gradle lets you create platform projects that only contain dependencies or dependency constraints.

Currently, I think slimjar attempts to get the jar for the project itself which fails due to the platform not actually having any jar published. I don't know how it works under the hood but I think they're basically just some transitive dependencies.

I would like to be able to define a slim(platform("group:name:version")) in Gradle and have all the dependencies from that platform be downloaded.

Thanks for your time!

Multi-module issue.

I have A and B modules.
A is like a common module.
B depends A.

I'm using slim 'xx:yy:zz' on A module but dependencies don't come to B module so, I have to write all dependencies which are in the A module to B module's dependencies again.

I think we should able to use the A module's dependencies in the B module without the duplication.

Real example:

Self-Dependency downloading

Need to set-up downloading dependencies of slimjar itself at runtime to further reduce final jar-size