ClassLoader leak prevention / protection
Home Page: http://java.jiderhamn.se/2012/03/04/classloader-leaks-vi-this-means-war-leak-prevention-library/
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Do the static methods in the class need to be static since it is already a singleton and the only callers to those methods would be inside the class or a sub-class? I would like to override the various logging methods in a subclass but I don't think I can do that if they are static.
Thanks.
Can this leak preventer work around Timer instance leaks, like this one?
Take transitive and circular dependencies into account
I'm using version 1.5.1, and see this in my logs a lot:
[4/19/13 16:21:33:241 EDT] 00000007 SystemErr R java.lang.NullPointerException
[4/19/13 16:21:33:291 EDT] 00000007 SystemErr R at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.getFieldValue(ClassLoaderLeakPreventor.java:974)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.stopThreads(ClassLoaderLeakPreventor.java:639)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.contextDestroyed(ClassLoaderLeakPreventor.java:382)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextDestroyed(WebApp.java:1739)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.webapp.WebApp.destroy(WebApp.java:3011)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.webapp.WebAppImpl.destroy(WebAppImpl.java:1287)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.container.AbstractContainer.destroy(AbstractContainer.java:75)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.webapp.WebGroup.destroy(WebGroup.java:228)
[4/19/13 16:21:33:292 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.webapp.WebGroup.removeWebApplication(WebGroup.java:269)
[4/19/13 16:21:33:301 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.VirtualHost.removeWebApplication(VirtualHost.java:297)
[4/19/13 16:21:33:301 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.VirtualHostImpl.removeWebApplication(VirtualHostImpl.java:211)
[4/19/13 16:21:33:301 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.WSWebContainer.removeWebApplication(WSWebContainer.java:820)
[4/19/13 16:21:33:301 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.component.WebContainerImpl.uninstall(WebContainerImpl.java:454)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.webcontainer.component.WebContainerImpl.stop(WebContainerImpl.java:725)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:1196)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStop(DeployedApplicationImpl.java:1374)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.DeployedModuleImpl.stop(DeployedModuleImpl.java:671)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.DeployedApplicationImpl.stop(DeployedApplicationImpl.java:1148)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:947)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:900)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerHelper.stopComponent(ContainerHelper.java:476)
[4/19/13 16:21:33:302 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerHelper.stopComponents(ContainerHelper.java:460)
[4/19/13 16:21:33:311 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.stopComponents(ContainerImpl.java:650)
[4/19/13 16:21:33:311 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.stop(ContainerImpl.java:638)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ApplicationServerImpl.stop(ApplicationServerImpl.java:263)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerHelper.stopComponent(ContainerHelper.java:476)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerHelper.stopComponents(ContainerHelper.java:460)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.stopComponents(ContainerImpl.java:650)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.stop(ContainerImpl.java:638)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ServerImpl.stop(ServerImpl.java:615)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook$1.run(ServerCollaborator.java:860)
[4/19/13 16:21:33:312 EDT] 00000007 SystemErr R at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5453)
[4/19/13 16:21:33:313 EDT] 00000007 SystemErr R at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5579)
[4/19/13 16:21:33:321 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook.run(ServerCollaborator.java:850)
[4/19/13 16:21:33:321 EDT] 00000007 SystemErr R at com.ibm.ws.runtime.component.ServerCollaborator$StopAction.alarm(ServerCollaborator.java:809)
[4/19/13 16:21:33:321 EDT] 00000007 SystemErr R at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133)
[4/19/13 16:21:33:322 EDT] 00000007 SystemErr R at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
I believe a null check is in order. Here's the offending line of code for reference:
https://github.com/mjiderhamn/classloader-leak-prevention/blob/classloader-leak-prevention-1.5.2/src/main/java/se/jiderhamn/classloader/leak/prevention/ClassLoaderLeakPreventor.java#L974
Hello @mjiderhamn !
What do you think about a diagnosis system that could show some alerts on console about the use of a library version which is known for create a classloader leak?
Thanks in advance.
First of all thanks a lot for your hard work creating this library. This has been a life-safer. We can properly deploy our stuff using this :)
To make sure everything works as expected we have created an integration test that checks whether our web applications can be properly undeployed: https://github.com/evosec/tomcat-classloader-leak-test
I have now found a weird reproducible issue on Windows 8 64 bit with Java 1.8.0_144 64 bit. The tests for the issue were added with the latest commit: evosec/tomcat-classloader-leak-test@96ba544 The corresponding build on Linux works fine: https://travis-ci.org/evosec/tomcat-classloader-leak-test/builds/269231898
Sadly it does not work on Windows. I have attached the build log. The webapps used for testing were created using http://start.spring.io, change Packaging to war and Generate. I then added the se.jiderhamn.classloader-leak-prevention:classloader-leak-prevention-servlet3:2.4.0 dependency.
Compare this with the plain Spring Boot application (w/o classloader-leak-prevention-servlet3) that runs just fine.
$ mvnw clean test -Dtest=de.evosec.leaktest.WebAppTestTest#testSpringBootClassLoaderLeakPreventor3_2_4_0
...
[INFO] Running de.evosec.leaktest.WebAppTestTest
Aug 28, 2017 5:22:13 PM org.apache.coyote.AbstractProtocol init
INFORMATION: Initializing ProtocolHandler ["http-nio-auto-1"]
Aug 28, 2017 5:22:13 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFORMATION: Using a shared selector for servlet write/read
Aug 28, 2017 5:22:13 PM org.apache.catalina.core.StandardService startInternal
INFORMATION: Starting service [Tomcat]
Aug 28, 2017 5:22:13 PM org.apache.catalina.core.StandardEngine startInternal
INFORMATION: Starting Servlet Engine: Apache Tomcat/8.5.20
Aug 28, 2017 5:22:13 PM org.apache.coyote.AbstractProtocol start
INFORMATION: Starting ProtocolHandler ["http-nio-auto-1-56972"]
Aug 28, 2017 5:22:14 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFORMATION: No global web.xml found
Aug 28, 2017 5:22:16 PM org.apache.jasper.servlet.TldScanner scanJars
INFORMATION: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
ClassLoaderLeakPreventorListener: Settings for se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener (CL: 0x4f638935):
ClassLoaderLeakPreventorListener: stopThreads = true
ClassLoaderLeakPreventorListener: stopTimerThreads = true
ClassLoaderLeakPreventorListener: executeShutdownHooks = true
ClassLoaderLeakPreventorListener: threadWaitMs = 5000 ms
ClassLoaderLeakPreventorListener: shutdownHookWaitMs = 10000 ms
Aug 28, 2017 5:22:16 PM se.jiderhamn.classloader.leak.prevention.JULLogger info
INFORMATION: Initializing by loading some known offenders with leak safe classloader
Aug 28, 2017 5:22:16 PM org.apache.catalina.core.ApplicationContext log
INFORMATION: 2 Spring WebApplicationInitializers detected on classpath
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.6.RELEASE)
ClassLoaderLeakPreventorListener: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener shutting down context by removing known leaks (CL: 0x4f638935)
javassist.CannotCompileException: by java.lang.OutOfMemoryError: Metaspace
at javassist.ClassPool.toClass(ClassPool.java:1170)
at javassist.CtClass.toClass(CtClass.java:1316)
at de.evosec.leaktest.WebAppTest$4.run(WebAppTest.java:280)
Caused by: java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javassist.ClassPool.toClass2(ClassPool.java:1183)
at javassist.ClassPool.toClass(ClassPool.java:1164)
... 2 more
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 129.388 s <<< FAILURE! - in de.evosec.leaktest.WebAppTestTest
[ERROR] testSpringBootClassLoaderLeakPreventor3_2_4_0(de.evosec.leaktest.WebAppTestTest) Time elapsed: 129.341 s <<< ERROR!
de.evosec.leaktest.WebAppTestException: ClassLoader not GC'ed
at de.evosec.leaktest.WebAppTestTest.testSpringBootClassLoaderLeakPreventor3_2_4_0(WebAppTestTest.java:72)
Caused by: org.awaitility.core.ConditionTimeoutException: Condition returned by method "testLeak" in class de.evosec.leaktest.WebAppTest was not fulfilled within 120 seconds.
at de.evosec.leaktest.WebAppTestTest.testSpringBootClassLoaderLeakPreventor3_2_4_0(WebAppTestTest.java:72)
...
$ mvnw clean test -Dtest=de.evosec.leaktest.WebAppTestTest#testSpringBoot
...
[INFO] Running de.evosec.leaktest.WebAppTestTest
Aug 28, 2017 5:24:39 PM org.apache.coyote.AbstractProtocol init
INFORMATION: Initializing ProtocolHandler ["http-nio-auto-1"]
Aug 28, 2017 5:24:39 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFORMATION: Using a shared selector for servlet write/read
Aug 28, 2017 5:24:39 PM org.apache.catalina.core.StandardService startInternal
INFORMATION: Starting service [Tomcat]
Aug 28, 2017 5:24:39 PM org.apache.catalina.core.StandardEngine startInternal
INFORMATION: Starting Servlet Engine: Apache Tomcat/8.5.20
Aug 28, 2017 5:24:39 PM org.apache.coyote.AbstractProtocol start
INFORMATION: Starting ProtocolHandler ["http-nio-auto-1-57092"]
Aug 28, 2017 5:24:40 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFORMATION: No global web.xml found
Aug 28, 2017 5:24:42 PM org.apache.jasper.servlet.TldScanner scanJars
INFORMATION: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Aug 28, 2017 5:24:43 PM org.apache.catalina.core.ApplicationContext log
INFORMATION: 2 Spring WebApplicationInitializers detected on classpath
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.6.RELEASE)
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.008 s - in de.evosec.leaktest.WebAppTestTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
...
Would it be possible to get the solution realised with JUnit rules instead of the runner?
Hi,
It seems that tests jar is not available in maven central, and because of that it is not possible to use classes for testing classloader leaks in other projects
<dependency>
<groupId>se.jiderhamn</groupId>
<artifactId>classloader-leak-prevention</artifactId>
<version>1.14.1</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
To fix that you need to add test-jar goal to jar-plugin. Detail on that can be found here: https://maven.apache.org/guides/mini/guide-attached-tests.html
P.S. maybe it's worth simply introducing new maven module with this helpful junit code?
If I dont create a new class that extends ClassLoaderLeakPreventor an exception is thrown because of log4j2 and lmax disruptor library.
Feb 13, 2016 5:00:28 PM com.lmax.disruptor.FatalExceptionHandler handleEventException SEVERE: Exception processing: 762 org.apache.logging.log4j.core.async. RingBufferLogEvent@dd477f3 java.lang.InterruptedException at java.util. concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util. concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:2048) at com.lmax.disruptor. BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) at com.lmax. disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java: 55) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java: 123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor .java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) <13-Feb-2016 17:00:28 o'clock EET> <Error> <com.lmax.disruptor.FatalExceptionHandler> <BEA-000000> <Exception processing: 762 org.apache.logging.log4j.core.async.RingBufferLogEvent@dd477f3 java.lang. InterruptedException at java.util.concurrent.locks. AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait( AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks. AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:2048) at com.lmax.disruptor. BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) at com.lmax. disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java :55) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor. java:123) Truncated. see log file for complete stacktrace>
Hi,
I try to fix a metaspace memory leak when I redeploy a web application in Tomcat (8.5.15).
Before using the classloader-leak-prevention library, the Path To GC Roots gives me
After using the library, I got
The library succeed to release some resources, but it creates new threads that are no garbage collected.
Any idea, what goes wrong?
Thanks.
Vincent.
JEditorPane registers different editorKits in sun.awt.AppContext
(see "SwingUtilities.appContextPut" strings in JEditorPane source)
something like
new JEditorPane("text/plain", "dummy");will be enough to prevent this memory leak
I have an application deployed to weblogic 12c R2(12.2.1). jdk is 1.8. I've placed ojdbc6(11.2.0.4) and than also ojdbc7(12.1.0.2). OutOfMemoryError error is thrown after a few redeployments. jconsole shows that "oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" is not killing until the server is restarted. I have created a new class that extends ClassLoaderLeakPreventor to solve my memory leak problem but still having leak problem. What can be the problem?
I am using classloader-leak-prevention-1.15.4.jar with Postgresql JDBC (postgresql-9.4-1201-jdbc41.jar) in the tomcat ${catalilna.base}/lib folder (and not in the webapp) but the listener is stopping a thread that probably shouldn't be tied to the webapp classloader (but apparently it is).
Stopping Timer thread 'PostgreSQL-JDBC-SharedTimer-1' running in classloader.
When the new version of the app tries to deploy and use Postgresql jdbc driver it gets an error like this:
java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397) ~[na:1.8.0_102]
at java.util.Timer.schedule(Timer.java:193) ~[na:1.8.0_102]
at org.postgresql.jdbc2.AbstractJdbc2Connection.addTimerTask(AbstractJdbc2Connection.java:1319) ~[postgresql-9.4-1201-jdbc41.jar:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.startTimer(AbstractJdbc2Statement.java:3494) ~[postgresql-9.4-1201-jdbc41.jar:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:569) ~[postgresql-9.4-1201-jdbc41.jar:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:420) ~[postgresql-9.4-1201-jdbc41.jar:9.4]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:305) ~[postgresql-9.4-1201-jdbc41.jar:9.4]
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:688) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:629) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:680) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:712) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:777) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
I haven't looked into this but I guessing that whichever app uses Postgresql first will get that thread tied to its webapp classloader and postgresql will stop working if that app is ever undeployed.
On line 969 there is a cast to Objects but the handback variable is only used in methods that expect an Object. In Weblogic I get a:
java.lang.ClassCastException: javax.management.ObjectName cannot be cast to java.util.Objects.
line 969 -
final Objects handback = getFieldValue(handbackField, listenerInfo);
should it be:
final Object handback = getFieldValue(handbackField, listenerInfo);
Hello,
The leak preventor contains a method gc()which ensures that System.gc() effectively performs a garbage collection.
This code can however potentionally result in an endless loop. In the field we have multiple installations where we disable the explict garbage collection with the -XX:+DisableExplicitGC JVM option, this causes that a System.gc() call will never do anything, resulting in an endless loop upon reload of a web application (except if by accident a full GC occurs).
Workaround code which we applied:
import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;
/**
* Unlike <code>{@link System#gc()}</code> this method guarantees that garbage collection has been performed before returning.
*/
protected static void gc() {
if (isDisableExplicitGCEnabled()) {
System.err.println(ClassLoaderLeakPreventor.class.getSimpleName() + ": "
+ "Skipping GC call since -XX:+DisableExplicitGC is supplied as VM option.");
return;
}
Object obj = new Object();
WeakReference ref = new WeakReference<Object>(obj);
// noinspection UnusedAssignment
obj = null;
while (ref.get() != null) {
System.gc();
}
}
/**
* Check is "-XX:+DisableExplicitGC" enabled.
*
* @return true is "-XX:+DisableExplicitGC" is set als vm argument, false otherwise.
*/
private static boolean isDisableExplicitGCEnabled() {
RuntimeMXBean bean = ManagementFactory.getRuntimeMXBean();
List<String> aList = bean.getInputArguments();
return aList.contains("-XX:+DisableExplicitGC");
}
Kind regards,
Kenny Moens
CIPAL IT Solutions
Keep getting IllegalArgumentException when shutting down the server, inside the JettyJMXRemover class constructor.
19:59:51,847 INFO [org.springframework.jmx.export.MBeanExporter] - Unregistering JMX-exposed beans on shutdown
19:59:51,857 INFO [org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean] - Closing Hibernate SessionFactory
19:59:51,857 INFO SessionFactoryImpl:927 - closing
19:59:51,857 INFO [org.hibernate.impl.SessionFactoryImpl] - closing
Oct 17, 2014 7:59:51 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Oct 17, 2014 7:59:51 PM org.apache.catalina.core.ApplicationContext log
INFO: Shutting down log4j
ClassLoaderLeakPreventor: nz.co.doltech.framework.libraries.clleakprev.ClassLoaderLeakPreventor shutting down context by removing known leaks (CL: 0x573cd86d)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.IllegalArgumentException: object is not an instance of declaring class
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at nz.co.doltech.framework.libraries.clleakprev.ClassLoaderLeakPreventor$JettyJMXRemover.<init>(ClassLoaderLeakPreventor.java:1735)
at nz.co.doltech.framework.libraries.clleakprev.ClassLoaderLeakPreventor.unregisterMBeans(ClassLoaderLeakPreventor.java:522)
at nz.co.doltech.framework.libraries.clleakprev.ClassLoaderLeakPreventor.contextDestroyed(ClassLoaderLeakPreventor.java:409)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4980)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5626)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
ClassLoaderLeakPreventor: Internal registry of java.beans.PropertyEditorManager not found
ClassLoaderLeakPreventor: Looping 4 RMI Targets to find leaks
ClassLoaderLeakPreventor: Looping 4 RMI Targets to find leaks
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transaction synchronizations with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: ThreadLocal of type java.lang.ThreadLocal: java.lang.ThreadLocal@782c7dfc with value [Lorg.springframework.cglib.proxy.Callback;@31036907 of type [Lorg.springframework.cglib.proxy.Callback; that is loaded by web app will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Current transaction read-only status with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Prototype beans currently in creation with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Actual transaction active with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: ThreadLocal of type java.lang.ThreadLocal: java.lang.ThreadLocal@56e463b4 with value [Lorg.springframework.cglib.proxy.Callback;@2c4deafd of type [Lorg.springframework.cglib.proxy.Callback; that is loaded by web app will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transactional resources with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Current transaction name with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Current transaction isolation level with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Hibernate Sessions registered for deferred close with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Current aspect-driven transaction with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Prototype beans currently in creation with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Since Java 1.6+ is used, we can call public static final void java.util.ResourceBundle.clearCache(java.lang.ClassLoader)
ClassLoaderLeakPreventor: Releasing web app classloader from Apache Commons Logging
Oct 17, 2014 7:59:51 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Oct 17, 2014 7:59:51 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Oct 17, 2014 7:59:51 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8080"]
Oct 17, 2014 7:59:51 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
Create separate module containing web-fragment.xml for auto configuration of ServletContextListener as of #45. Investigate how to make sure this listener is "outermost".
Create separate module with ServletContextListener that invokes the new POJO orchestrator as of #42.
Since classloader-leak-prevention version 1.15, there was introduced a JRE 1.7 dependency.
Following constructor was only introduced since 1.7 and this will give an error during startup when using an older JRE.
final Constructor constructor =
RuntimeException.class.getDeclaredConstructor(String.class, Throwable.class, Boolean.TYPE, Boolean.TYPE);
In the pom files it still says:
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
I would suggest to keep the minimum required version to 1.6 since Oracle still supports JAVA 1.6 with their "Long Term Support" program.
Turn ClassLoaderLeakPreventor into a non-servlet POJO, invoking preventative mechanisms as per #41
Tests failing with JDK 9 jigsaw:
The ClassLoaderLeakPreventor library does not handle classloader leaks caused
by the RuntimeExceptions in the static final abort fields in the JRE classes
com.sun.org.apache.xerces.internal.dom.DOMNormalizer and
com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl.
When each of these classes is initialized, its abort exception is created. If a web app class happens to be in the call stack at that point, the web app class cannot be garbage-collected when the web app is undeployed because a static field somewhere in a JRE class has a reference to it.
To reproduce the problem, use the following servlet:
package com.example;
import javax.servlet.ServletException;
import javax.servlet.http.*;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
public class DOMNormalizerLeakServlet extends HttpServlet {
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException {
try {
Document document = DocumentBuilderFactory.newInstance().newDocumentBuilder().newDocument();
document.normalizeDocument();
response.getWriter().write("done");
}
catch (Exception e) {
throw new ServletException(e);
}
}
}
and the following web.xml:
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
version="3.1">
<listener>
<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor</listener-class>
</listener>
<servlet>
<servlet-name>test</servlet-name>
<servlet-class>com.example.DOMNormalizerLeakServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>test</servlet-name>
<url-pattern>/index.html</url-pattern>
</servlet-mapping>
</web-app>
Here's a path from a destroyed Tomcat web app classloader to DOMNormalizer.abort, courtesy of JVisualVM:
this - value: org.apache.catalina.loader.WebappClassLoader #2
<- <classLoader> - class: com.example.DOMNormalizerLeakServlet, value: org.apache.catalina.loader.WebappClassLoader #2
<- [2] - class: java.lang.Object[], value: com.example.DOMNormalizerLeakServlet class DOMNormalizerLeakServlet
<- [2] - class: java.lang.Object[], value: java.lang.Object[] #4167
<- backtrace - class: java.lang.RuntimeException, value: java.lang.Object[] #4166
<- abort (sticky class) - class: com.sun.org.apache.xerces.internal.dom.DOMNormalizer, value: java.lang.RuntimeException #1
Similarly, if you replace
document.normalizeDocument();
in the servlet above with
DOMImplementationLS implementation = (DOMImplementationLS)document.getImplementation();
implementation.createLSSerializer().writeToString(document);
and add an import for org.w3c.dom.ls.DOMImplementationLS, then you get another memory leak. Here's a path from another destroyed Tomcat web app classloader to DOMSerializerImpl.abort:
this - value: org.apache.catalina.loader.WebappClassLoader #1
<- <classLoader> - class: com.example.DOMNormalizerLeakServlet, value: org.apache.catalina.loader.WebappClassLoader #1
<- [2] - class: java.lang.Object[], value: com.example.DOMNormalizerLeakServlet class DOMNormalizerLeakServlet
<- [2] - class: java.lang.Object[], value: java.lang.Object[] #3416
<- backtrace - class: java.lang.RuntimeException, value: java.lang.Object[] #3415
<- abort (sticky class) - class: com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl, value: java.lang.RuntimeException #1
If I dont create a new class that extends ClassLoaderLeakPreventor an exception is thrown because of log4j2 and lmax disruptor library. So I extended and added below code not to get exception:
if (LogManager.getContext() != null) { try { AsyncLoggerContext logContext = (AsyncLoggerContext) LogManager.getContext(); logContext.stop(); } catch (Exception e) { }log4j2 version is 2.3. If ı use log4j 2.5 Log4jThread classes stay alive after undeployment.
We found another possible leak by means of the ProxySelector Java class. To clear this one, we had to create a new method like this one:
protected void clearDefaultProxySelector() {
AccessController.doPrivileged(new PrivilegedAction<Void>() {
@Override
public Void run() {
ProxySelector selector = ProxySelector.getDefault();
if (isLoadedInWebApplication(selector)) {
ProxySelector.setDefault(null);
warn("Removing default java.net.ProxySelector");
}
return null;
}
});
}
Additionally the clearDefaultProxySelector() method is called from the contextDestroyed() method.
Kind regards,
Kenny Moens
CIPAL
According to the comment above it, the org.apache.cxf:cxf-rt-transports-http dependency in the POM is used only in the tests. Unlike other similar dependencies, it is missing <scope>test</scope>.
(Great work compiling all this into a library. It has saved me a lot of time fixing one problem just to find another.)
We had a case where a ThreadLocal contained a collection. This collection contained entries which were loaded by the WebAppClassLoader, but the methods provided in the library did not properly find them.
We made the following extensions to the code for this:
/** Test if provided object is loaded with web application classloader. */
@Override
protected boolean isLoadedInWebApplication(final Object o) {
if (o instanceof Map) {
Map m = (Map) o;
return isLoadedByWebApplication(o.getClass()) || isLoadedInWebApplication(m.keySet()) || isLoadedInWebApplication(m.values());
}
if (o instanceof Collection) {
Collection c = (Collection) o;
return isLoadedByWebApplication(o.getClass()) || isLoadedInWebApplication(c);
}
return o != null && isLoadedByWebApplication(o.getClass());
}
/** Test if an object of the collection is loaded by the web application classloader. */
protected boolean isLoadedInWebApplication(final Collection o) {
if (o == null) {
return false;
}
for (Object value : o) {
if ((value instanceof Class) && isLoadedByWebApplication((Class) value)) {
return true;
} else if (isLoadedByWebApplication(value.getClass())) {
return true;
}
}
return false;
}
Kind regards,
Kenny Moens
CIPAL
We are experiencing some strange things when using this library on our application deployed to Tomcat7.
ClassLoaderLeakPreventor: Stopping Thread 'Thread[catalina-exec-720,5,main]' of type org.apache.tomcat.util.threads.TaskThread running in web app after 5000 ms
This happens randomly during application deployment, but eventually halts all Tomcat executor threads, which then makes Tomcat quite unresponsive as all executors are in this kind of state:
"catalina-exec-919" #32896 daemon prio=5 os_prio=0 tid=0x00007f780c0f7000 nid=0x46d5 waiting on condition [0x00007f77e4c1b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e08c9478> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2083)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
After a while, all threads in Tomcat threadpool (maxThreadPoolSize) will be in this state, and Tomcat will be stuck until restart
Has anyone else experienced anything like this ?
Any ideas to resolve this ?
I would not want to skip shutting down threads as we do have some JDBC related threads that we do need to close..
About five weeks ago I updated classloader leak prevention to the latest 1.x version from a version that was about a year old. I am running with Java 8 and Tomcat 8.30 or Tomcat 8.35 depending on the environment and we deploy apps using the parallel deployment (e.g. app##1.0.1.war). I started to see lots of strange/bad behaviour and issues with hot deploying new versions of apps, forcing me to restart servers to get things working. Today when I was deploying one app I noticed the listener stopping a quartz thread and the app I was re-deploying doesn't use quartz (but another application on the server does). Then I noticed that it was stopping Tomcat threads like http worker threads which explains why I was having to restart tomcat to fix problems. (This is all in dev by the way). I have switched back to the listener from a year ago and things seem to be working better.
This is what isWebAppClassLoaderOrChild looks like in the version that seems to work for me:
/** Test if provided ClassLoader is the classloader of the web application, or a child thereof */
protected boolean isWebAppClassLoaderOrChild(ClassLoader cl) {
final ClassLoader webAppCL = getWebApplicationClassLoader();
// final ClassLoader webAppCL = Thread.currentThread().getContextClassLoader();
while(cl != null) {
if(cl == webAppCL)
return true;
cl = cl.getParent();
}
return false;
}
The new version that was causing me problems uses isAncestor(). I am just guessing that a problem lies with that method ( at least in my setup). I haven't narrowed it down to that one change in the listener, but it seems the most likely suspect.
I feel the problem I was seeing must be unique to my setup unless nobody is using recent versions of this library (1.x) because the problems would be hard not to notice. Or maybe they only happen on certain recent versions of Tomcat 8/Java 8 or with the classloaders for parallel deployment? Just guessing...
Hi @mjiderhamn,
This is excellent library! I found leak in Xalan library (https://issues.apache.org/jira/browse/XALANJ-2600). It would be great if your library support prevention for this leak.
Thanks.
Might it be worth logging a warning for still-registered JDBC drivers, MBeans, ThreadLocals, etc, at least for the ones which can be detected? (Shutdown hooks seem to be covered already.)
There's not much an application can do about leaks in API classes, but leaking registrations are probably something that can be cleaned up by a well-behaved application.
As reported by @hdeadman in #60:
In the 1.x version of this listener I could extend the listener and override the logging methods. I am currently using SLF4J/Logback and I was going to provide an implementation of Logger that used that but I didn't see an easy way to extend the listener and pass in my instance of the Logger so it could be set on ClassLoaderLeakPreventorFactory. Should "void contextInitialized(final ServletContext servletContext)" be overloaded with a version that takes a Logger?
It's not a big deal for me because I am using the org.slf4/jul-to-slf4j bridge so the default JULLogger happens to work for me but it isn't clear to me how one is supposed to specify their own logger (or use StdLogger).
Another option would be to have more implementations of Logger and have a context parameter for choosing between them. If the context parameter specified the Logger impl class name then you wouldn't have to bundle all the impl classes.
After using your excellent library we (unfortunately) had a new Oracle thread start running. This thread did not get shutdown on reload, thus causing a classloader leak.
We have the Oracle JDBC driver in our <TOMCAT_HOME>/lib (though we're currently not using the Oracle JDBC driver). I'm not sure if this is relevant.
I must confess I don't understand why the oracle thread's context classloader should be the System classloader. Does that prevent stopThreads() from stopping it?
Work around
I extended ClassLoaderLeakPreventor and overrode contextInitialized(), skipping the code that spins up the Oracle thread.
Just a suggestion: I think I have found another classloader leak.
You can find the description of the Problem on Stackoverflow: http://stackoverflow.com/q/39741147/4864870
A demonstration project with a working preventor can be found here: https://github.com/CptS/solrj-classloader-leak
(This preventor is not yet implemented as "classloader-leak-prevention-Plugin", it's just a simple ServletContextListener and there is potential for improvement but maybe it could be interesting for you)
When I run the latest 1.x release of this listener (I have listener that extends it and overrides logger methods) I get a stack overflow exception if I am running tomcat with a security manager policy (catalina.sh start -security).
-security adds these options to java:
-Djava.security.manager -Djava.security.policy==/var/lib/tomcats/clustera/conf/catalina.policy
I will look into changing the policy file to let this listener do whatever it wants but is there any way the listener could avoid getting stuck in this loop?
Aug 29, 2016 9:42:56 AM org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/xyz##1.2.3] listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.xyz.web.memleak.ClassloaderMemoryLeakAvoidanceListener
java.lang.StackOverflowError
at java.security.AccessControlContext.optimize(AccessControlContext.java:608)
at java.security.AccessController.checkPermission(AccessController.java:883)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1528)
at java.lang.ClassLoader.getParent(ClassLoader.java:1374)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.isWebAppClassLoaderOrChild(ClassLoaderLeakPreventor.java:1803)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor$1.combine(ClassLoaderLeakPreventor.java:212)
at java.security.AccessControlContext.optimize(AccessControlContext.java:608)
at java.security.AccessController.checkPermission(AccessController.java:883)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1528)
at java.lang.ClassLoader.getParent(ClassLoader.java:1374)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.isWebAppClassLoaderOrChild(ClassLoaderLeakPreventor.java:1803)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor$1.combine(ClassLoaderLeakPreventor.java:212)
at java.security.AccessControlContext.optimize(AccessControlContext.java:608)
at java.security.AccessController.checkPermission(AccessController.java:883)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1528)
at java.lang.ClassLoader.getParent(ClassLoader.java:1374)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.isWebAppClassLoaderOrChild(ClassLoaderLeakPreventor.java:1803)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor$1.combine(ClassLoaderLeakPreventor.java:212)
at java.security.AccessControlContext.optimize(AccessControlContext.java:608)
at java.security.AccessController.checkPermission(AccessController.java:883)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1528)
at java.lang.ClassLoader.getParent(ClassLoader.java:1374)
etc, etc, etc,...
Hi,
I had some problems with the unsetCachedKeepAliveTimer() method on JBOSS 7.2. The 'sun.net.www.http.HttpClient' class is not available in the webapp classloader, which means this method cannot get the 'kac' static field from it (calling getStaticFieldValue("sun.net.www.http.HttpClient", "kac") will return null).
I was able to solve this problem by loading this HttpClient class with the system classloader instead of the webapp classloader. Could you fix this in your codebase?
Thanks,
Maarten
just an FYI, somebody pointed me at this project because I've given it some thought too https://github.com/fommil/class-monkey/ (by no means a silver bullet)
Maybe wrap the awt-call with an appropriate try-catch?
java.lang.InternalError: Can't connect to X11 window server using ':0.0 xterm' as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:74)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:102)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:81)
at sun.awt.X11.XToolkit.(XToolkit.java:119)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at java.awt.Toolkit$2.run(Toolkit.java:868)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:860)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.contextInitialized(ClassLoaderLeakPreventor.java:228)
We have an application that runs on Weblogic 10 and uses an inverted classloader heirarchy so it can leverage newer versions of libraries (including xerces and xml apis). This works but sometimes during deployment if certain XML operations occur we have problems, possibly due to the deployer having already loaded a different version of JAXP related classes as part of its parsing of the XML deployment descriptors. This application uses Spring which probably uses JAXP to parse config files during deployment and I am not sure why that works. When I added the ClassLoaderLeakPreventor listener to this application the deployment fails due to:
Target state: deploy failed on Cluster Cluster1
java.lang.ClassCastException: weblogic.xml.jaxp.RegistryDocumentBuilderFactory
at javax.xml.parsers.DocumentBuilderFactory.newInstance(Unknown Source)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.contextInitialized(ClassLoaderLeakPreventor.java:252)
at org.xxxxxxx.web.memleak.ClassloaderMemoryLeakAvoidanceListener.contextInitialized(ClassloaderMemoryLeakAvoidanceListener.java:18)
My request for change would be to either catch the ClassCastException and ignore it/log it, or move that particular check into it's own protected method so I could override it easier in my sub-class. If you can think of a way to acheive the goal of that block of code without it bombing out on Weblogic 10.0 (in this scenario), that would be great too, but I am working on getting us upgraded to Weblogic 12 where the classloader control for WAR files is improved and the XML versions available from system classloader are probably new enough that we won't need them in the WAR.
Create new test framework that simplifies testing single preventative mechanisms as of #41. Allow testing both preventative/startup and cleanup/shutdown mechanisms.
I think the thread stopping is a bit too aggressive because it doesn't wait before interrupting threads that are shutting down on their own. It seems like the waiting time is only applied after interrupting running threads, which in the case of Quartz triggers an "ERROR" message.
2015-09-20 18:00:06.699:INFO:/test:Scanner-0: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called
[INFO] 2015-09-20 18:00:06,701 org.example.services - Deregistering JDBC drivers
[INFO] 2015-09-20 18:00:06,702 org.example.services - Deregistering JDBC driver org.postgresql.Driver@3cd16578
[INFO] 2015-09-20 18:00:06,702 org.quartz.core.QuartzScheduler - Scheduler MyScheduler_$_NON_CLUSTERED shutting down.
[INFO] 2015-09-20 18:00:06,702 org.quartz.core.QuartzScheduler - Scheduler MyScheduler_$_NON_CLUSTERED paused.
[INFO] 2015-09-20 18:00:06,703 org.quartz.core.QuartzScheduler - Scheduler MyScheduler_$_NON_CLUSTERED shutdown complete.
[INFO] 2015-09-20 18:00:06,708 org.quartz.ee.servlet.QuartzInitializerListener - Quartz Scheduler successful shutdown.
ClassLoaderLeakPreventor: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor shutting down context by removing known leaks (CL: 0x60d6d6d4)
ClassLoaderLeakPreventor: Internal registry of java.beans.PropertyEditorManager not found
ClassLoaderLeakPreventor: Looping 0 RMI Targets to find leaks
ClassLoaderLeakPreventor: Looping 0 RMI Targets to find leaks
ClassLoaderLeakPreventor: Stopping Thread 'Thread[MyScheduler_Worker-1,5,main]' of type org.quartz.simpl.SimpleThreadPool$WorkerThread running in web app after 5000 ms
[ERROR] 2015-09-20 18:00:06,742 org.quartz.simpl.SimpleThreadPool - Worker thread was interrupt()'ed.
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
ClassLoaderLeakPreventor: Since Java 1.6+ is used, we can call public static final void java.util.ResourceBundle.clearCache(java.lang.ClassLoader)
ClassLoaderLeakPreventor: Releasing web app classloader from Apache Commons Logging
2015-09-20 18:00:06.746:INFO:oejsh.ContextHandler:Scanner-0: Stopped o.e.j.w.WebAppContext@59e0b2fa{/test,file:/tmp/jetty-0.0.0.0-8080-test.war-_test-any-2539080646960400183.dir/webapp/,UNAVAILABLE}{/tmp/test.war}
Looking at the source of SimpleThreadPool.java I see that there's a shutdown mechanism for the worker thread. Without Classloader Leak Prevention, the worker thread shuts down cleanly.
Wouldn't it be better to wait for some seconds, then interrupt running threads, and finally wait another few seconds for them to finish before calling stop()?
I have a NPE in fixGeoToolsLeak because exit method in org.geotools.util.WeakCollectionCleaner is not static, so invoke(null) cannot be used.
I found this method: https://navatintarev.wordpress.com/2012/05/18/geotools-memory-leak/
Could you check pls? If you think that could be a new solution I can prepare a patch.
Thank you.
When compiling with Java 1.5.0_29-b02 I am getting the following error:
ClassLoaderLeakPreventor.java:[962,24] type parameters of T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,java.lang.Object
It works with newer compiler but this is the code it is complaining about:
protected T getFieldValue(Object obj, String fieldName) {
final Field field = findField(obj.getClass(), fieldName);
return getFieldValue(field, obj);
}
The fix for me is to add a cast:
protected T getFieldValue(Object obj, String fieldName) {
final Field field = findField(obj.getClass(), fieldName);
return (T) getFieldValue(field, obj);
}
It looks like this issue:
http://stackoverflow.com/questions/314572/bug-in-eclipse-compiler-or-in-javac
Anyway, not a big deal but might be worth fixing (even if it is a compiler bug) next time you are in there. Someday I will get off of Java 5!
Thanks for the updates.
It's not a big deal since we don't care about cleaning up memory leaks when we are shutting down the server, but it would be nice not to see stack traces when killing a server (ctrl-c) since that usually triggers application undeployments.
Map<Thread, Thread> shutdownHooks = (Map<Thread, Thread>) getStaticFieldValue("java.lang.ApplicationShutdownHooks", "hooks");
// Iterate copy to avoid ConcurrentModificationException
for(Thread shutdownHook : new ArrayList<Thread>(shutdownHooks.keySet())) {
I saw an NPE on the line with shudownHooks.keySet() so apparently during shutdown that hooks map must have been nulled out. I am running on Weblogic 12 with Java 7.
Not sure if there is a way to tell if a VM is in process of being shutdown because you could probably skip all the cleanup you are doing on shutdown.
This is from the version as of 3/20 which is not quite the latest.
java.lang.NullPointerException
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.deregisterShutdownHooks(ClassLoaderLeakPreventor.java:427)
at se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.contextDestroyed(ClassLoaderLeakPreventor.java:334)
at org.adnet.web.memleak.ClassloaderMemoryLeakAvoidanceListener.contextDestroyed ClassloaderMemoryLeakAvoidanceListener.java:21)
at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run
(EventsManager.java:583)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
Any thoughts on leaks related to oracle.jdbc.driver.OracleTimeoutPollingThread having a reference to a particular webapp's classloader? I am seeing it in one of my OOM dumps as the reason for one classloader hanging around.
It's discussed here:
http://www.mail-archive.com/[email protected]/msg106186.html
http://www.mail-archive.com/[email protected]/msg106377.html
There is some discussion on that thread of a bug report being filed with Oracle but from what I could tell on my support account the issue was suspended.
Bug 16841748 : FIXED SIZE MEMORY LEAK DUE TO "ORACLETIMEOUTPOLLINGTHREAD" THREAD
Status 33 - Suspended, Req'd Info not Avail
It seems like the technique of changing the context classloader to the system classloader and then having that thread startup (probably by running a query) would work here but it would require telling Oracle some connection info. Might an alternative be to find the thread on shutdown and change the classloader it was referring to after the fact (in the even that the thread's classloader was equal to that of the webapp)?
Hi @mjiderhamn ,
I'd like to leverage your excellent library though I'm not confortable with the System.gc that will be triggered if the -XX:+DisableExplicitGC jvm switch as not been turned on.
This is done in these 2 classes:
Here's the use case:
I think, solely relying on the -XX:+DisableExplicitGC is a bit problematic. Sure I could most likely extends your library to change the behavior though, I feel it would be a great addition if we could instead leverage a configuration to explicitly enable or not the System.gc.
Maybe a bit like the other "ClassLoaderLeakPreventor. ..." configuration you already provide (through the context param).
I also think, it could be worthwile to enable/disable any individual leaks prevention feature one by one. So when hunting down for leaks and fixing them, it could be possible to still leverage your library to solve one issue at the time.
WDYT?
Again a big thank you for your excellent library.
It appears that 16fc362 introduced a runtime dependency on CXF. I don't think such a dependency was intended, and it certainly can be problematic. It breaks my application, as it uses CXF 2.3, and because of the classloader-leak-prevention introduced dependency on CXF 2.6, the application now resolves the version conflict by using 2.6 which doesn't work for my application.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.