Upgrade to Jackson 2.7.1

[joschi]

This PR upgrades dropwizard-jackson to Jackson 2.7.1.

Release notes: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.7

[joschi]

Error in https://travis-ci.org/dropwizard/dropwizard/builds/107434037#L4784-L4872:

Running io.dropwizard.testing.junit.DropwizardAppRuleConfigOverrideTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.005 sec <<< FAILURE! - in io.dropwizard.testing.junit.DropwizardAppRuleConfigOverrideTest
io.dropwizard.testing.junit.DropwizardAppRuleConfigOverrideTest  Time elapsed: 1.005 sec  <<< ERROR!
java.lang.RuntimeException: io.dropwizard.configuration.ConfigurationParsingException: /home/travis/build/dropwizard/dropwizard/dropwizard-testing/target/test-classes/test-config.yaml has an error:
  * Unrecognized field at: server.requestLog
    Did you mean?:
      - adminConnectors
      - adminContextPath
      - adminMaxThreads
      - adminMinThreads
      - applicationConnectors
        [1 more]

    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at io.dropwizard.testing.DropwizardTestSupport.startIfRequired(DropwizardTestSupport.java:199)
    at io.dropwizard.testing.DropwizardTestSupport.before(DropwizardTestSupport.java:115)
    at io.dropwizard.testing.junit.DropwizardAppRule.before(DropwizardAppRule.java:128)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: io.dropwizard.configuration.ConfigurationParsingException: /home/travis/build/dropwizard/dropwizard/dropwizard-testing/target/test-classes/test-config.yaml has an error:
  * Unrecognized field at: server.requestLog
    Did you mean?:
      - adminConnectors
      - adminContextPath
      - adminMaxThreads
      - adminMinThreads
      - applicationConnectors
        [1 more]

    at io.dropwizard.configuration.ConfigurationParsingException$Builder.build(ConfigurationParsingException.java:276)
    at io.dropwizard.configuration.ConfigurationFactory.build(ConfigurationFactory.java:166)
    at io.dropwizard.configuration.ConfigurationFactory.build(ConfigurationFactory.java:99)
    at io.dropwizard.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:115)
    at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:63)
    at io.dropwizard.testing.DropwizardTestSupport.startIfRequired(DropwizardTestSupport.java:197)
    at io.dropwizard.testing.DropwizardTestSupport.before(DropwizardTestSupport.java:115)
    at io.dropwizard.testing.junit.DropwizardAppRule.before(DropwizardAppRule.java:128)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestLog" (class io.dropwizard.server.DefaultServerFactory), not marked as ignorable (6 known properties: "adminMaxThreads", "adminConnectors", "applicationConnectors", "applicationContextPath", "adminMinThreads", "adminContextPath"])
 at [Source: N/A; line: -1, column: -1] (through reference chain: io.dropwizard.testing.junit.TestConfiguration["server"]->io.dropwizard.server.DefaultServerFactory["requestLog"])
    at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:62)
    at com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:855)
    at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1083)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1389)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1367)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:266)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:163)
    at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:121)
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedUsingDefaultImpl(AsPropertyTypeDeserializer.java:136)
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:99)
    at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142)
    at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:488)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:224)
    at com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:152)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3761)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2042)
    at io.dropwizard.configuration.ConfigurationFactory.build(ConfigurationFactory.java:153)
    at io.dropwizard.configuration.ConfigurationFactory.build(ConfigurationFactory.java:99)
    at io.dropwizard.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:115)
    at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:63)
    at io.dropwizard.testing.DropwizardTestSupport.startIfRequired(DropwizardTestSupport.java:197)
    at io.dropwizard.testing.DropwizardTestSupport.before(DropwizardTestSupport.java:115)
    at io.dropwizard.testing.junit.DropwizardAppRule.before(DropwizardAppRule.java:128)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

[arteam]

@joschi, could you rebase this change against master? Let's see if b7c8721 helps.

[joschi]

@arteam Unfortunately that didn't help.

[evnm]

This line in the exception chain indicates that none of the config fields inherited from AbstractServerFactory are being picked up:

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestLog" (class io.dropwizard.server.DefaultServerFactory), not marked as ignorable (6 known properties: "adminMaxThreads", "adminConnectors", "applicationConnectors", "applicationContextPath", "adminMinThreads", "adminContextPath"])

Those six fields are defined in DefaultServerFactory. Could something have changed in the way jackson-databind traverses class hierarchies resulting in it not resolving a comprehensive set of properties? Nothing stands out as such from a quick scan of the Jackson 2.7 changelog, but I'm admittedly not familiar with the codebase.

[cowtowncoder]

@evnm One suggestion is to try whether problem triggers without Afterburner registration.

If it still fails, my guess would be that the type resolution (construction of JavaType from generic types, TypeReference and so on) changes could be causing whatever issue there is. There were some problems with direct construction of SimpleType (something not recommended, but possible) with 2.7.0, one of which would have similar symptoms (method introspection not being done recursively unless invoked via TypeFactory), but those were supposed to be resolved.
Still, it sounds eerily similar. So reproduction would be great.

[joschi]

@cowtowncoder Removing the Jackson Afterburner module unfortunately didn't fix the issue.

[cowtowncoder]

@joschi Ok. Then I would need to trimmed down unit test; given that I would expect it to likely be easy to find and fix the problem. I would think that creating local Configuration subtype, stripping out things other than specific allegedly unknown property, should work.

I'll try to reproduce this locally from DW trunk as well, in parallel

[cowtowncoder]

I started trying to tease out whatever is causing this: I can reproduce the issue locally wrt DW tests.
My first guess was that this is related to handling of defaultImpl, and incomplete type resolution: there was an earlier bug, but that was fixed in 2.7.1. So far have not been able to replicate the failure.
Fail is still present in head of Jackson 2.7 branch, although changes since 2.7.1 are very small.

[cowtowncoder]

Hoping that this:

FasterXML/jackson-databind#1128

is related. Have been able to fix part of the problem so far, but it does not seem to resolve this problem.

What would be useful from Jackson perspective would be to have an isolated test case I could play with.

[nickbabcock]

Looks like the same error occurs with Jackson 2.7.2.

[cowtowncoder]

@nickbabcock Unfortunately fix to a seemingly related problem was not a fix to whatever is causing this failure. At this point I do not have a simple reproduction of the problem, unfortunately, and I am beginning to wonder if this could be either:

1. Problem due to test harness using methods (or generating scaffolding) that should be changed (that is, failure of test code), or
2. Related to custom Type Id resolver that DropWizard uses to (sort of) dynamically register subtype mapping for configuration values

I suspect it is most likely to be (2), but have forgotten where to look. There was some magic in DW code to look for specific annotations, found via class-scanning, to sort of auto-register subtypes of various polymorphic types that standard configuration has.
It would be great if anyone could help distill a test case from those, if possible. I'll update this issue if I find anything else relevant.

[nickbabcock]

DiscoverableSubtypeResolver is what is currently being used. I'll have to look up what it means for jackson to resolve sub-types before creating a test-case

[cowtowncoder]

@nickbabcock ah yes, that's it. I am suspecting that perhaps some aspects of subtype-refinement might have issues, if and when polymorphic types are being used. This is not a widely used thing, but I think DW configs are quite elaborate.

[cowtowncoder]

Looking at code on Jackson side I suspect this is indeed where the problem arises. Also, haven't seen anything particularly suspicious on DW side (at least wrt this problem). Hoping to cobble up a stand-alone test case from that.

[cowtowncoder]

Still no luck. Simple subtype/supertype via annotations does not cause problems it seems.
And my earlier suspicion wrt use of AnnotatedClass.constructWithoutSuperTypes(...) might not be problematic.

[cowtowncoder]

With some more work, debugging, was able to finally reproduce the issue. It is a combination of things, but basically:

1. If defaultImpl type extends base type (on which @JsonTypeInfo is defined)
2. and there is at least one intermediate class that declares properties not inferred by base type or default implementation
3. then such intermediate properties are not visible due to one gnarly part of type refinement

in this case, that would be ServerFactory -> AbstractServerFactory -> DefaultServerFactory, where middle one defines many of properties.

An earlier fix alleviated the problem for other cases, but this one was not covered; full fix for problem that makes clean resolution harder will go in Jackson 2.8 (since it needs further changes to internal APIs), but I'll figure out a way to make it work at very least for this particular type of problem.

Issue on jackson-databind for the problem is: FasterXML/jackson-databind#1125

[arteam]

Thanks, @cowtowncoder, for tackling this issue!

[cowtowncoder]

Fixed for Jackson 2.7.3; will try to see what else could be fixed, then issue 2.7.3 hopefully by this weekend.

[joschi]

Superseded by #1479.