Unreliable tests

[Nikos410]

Hi @mjiderhamn ,

while working on this great library for making it compatible with Java 11 (#87), I've noticed that some tests fail unpredictably.
For example, when I run the ShutdownHookCleanUpTest it works, but if I set a breakpoint and step through it, the cleanup fails.

First I thought this was an issue with my local Eclipse, but I've noticed similar problems on the TravisCI builds as well:
Compare this build and this one. Both were run after the same commit, but the former one fails, because triggering the leak in StopThreadsCleanUp_TimerTest.createTimer does not work, while the latter one succeeds.

This should also not be caused by any configuration differences between your TravisCI and my TravisCI:
To test if this is the problem, I added a commit which just added a missing newline to a test (which should not change the test results).
Compare Job 366.2 and Job 367.2. As you can see, the test results are different: On the first run, GeoToolsCleanUpTest.cleanUpAfterTriggeringLeak fails because the cleanup does not work, on the second run, it succeeds.

366.2:

Failed tests: 
   BeanELResolverCleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.BeanELResolverCleanUpTest.cleanUpAfterTriggeringLeak]@2a798d51
=> GeoToolsCleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.GeoToolsCleanUpTest.cleanUpAfterTriggeringLeak]@7c417213
   JavaServerFaces2746CleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.JavaServerFaces2746CleanUpTest.cleanUpAfterTriggeringLeak]@38d8f54a
   JavaUtilLoggingLevelCleanUpTest.triggerLeakWithoutCleanup ClassLoader has been garbage collected, while test is expected to leak
   ReplaceDOMNormalizerSerializerAbortExceptionCleanUpTest.triggerLeakWithoutCleanup ClassLoader has been garbage collected, while test is expected to leak
   ShutdownHookCleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.ShutdownHookCleanUpTest.cleanUpAfterTriggeringLeak]@7c417213
   LdapPoolManagerInitiatorTest.firstShouldLeak ClassLoader has been garbage collected, while test is expected to leak
   SunGCInitiatorTest.firstShouldLeak ClassLoader has been garbage collected, while test is expected to leak

367.2:

Failed tests: 
   BeanELResolverCleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.BeanELResolverCleanUpTest.cleanUpAfterTriggeringLeak]@6d763516
   JavaServerFaces2746CleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.JavaServerFaces2746CleanUpTest.cleanUpAfterTriggeringLeak]@38d8f54a
   JavaUtilLoggingLevelCleanUpTest.triggerLeakWithoutCleanup ClassLoader has been garbage collected, while test is expected to leak
   ReplaceDOMNormalizerSerializerAbortExceptionCleanUpTest.triggerLeakWithoutCleanup ClassLoader has been garbage collected, while test is expected to leak
   ShutdownHookCleanUpTest.cleanUpAfterTriggeringLeak ClassLoader has not been garbage collected se.jiderhamn.classloader.RedefiningClassLoader[se.jiderhamn.classloader.leak.prevention.cleanup.ShutdownHookCleanUpTest.cleanUpAfterTriggeringLeak]@b62fe6d
   LdapPoolManagerInitiatorTest.firstShouldLeak ClassLoader has been garbage collected, while test is expected to leak
   SunGCInitiatorTest.firstShouldLeak ClassLoader has been garbage collected, while test is expected to leak

Of course, dealing with internal stuff like this library does is always prone to errors, so I don't know if there can really be done anything to improve this. But it is kind of frustrating to try and fix something, when you can not be sure if the test results are correct.

[mjiderhamn]

You are very much welcome to try to find a solution to the brittleness of the tests!
There is a commented out setting in the Travis YAML file that will upload heap dumps to S3, so that the memory can be analyzed after running on Travis.