Jolt

JSON to JSON transformation library written in Java where the "specification" for the transform is itself a JSON document.

Useful For

1. Transforming JSON data from ElasticSearch, MongoDb, Cassandra, etc before sending it off to the world
2. Extracting data from a large JSON documents for your own consumption

Table of Contents

• 1 Overview
• 2 Documentation
• 3 Shiftr Transform DSL
• 4 Getting Started
• 5 Getting Transform Help
• 6 Alternatives
• 7 Performance
• 8 CLI
• 9 Code Coverage
• 10 Release Notes

Overview

Jolt provides a set transform components, that can be "chained" together to form the overall JSON to JSON transform.

Jolt consumes and produces "hydrated" JSON : in-memory tree of Maps, Lists, Strings, etc.

Jolt lets Jackson serialize and deserialize the JSON text which offloads the work of handling commas and closing brackets.

Each transform component covers a specific transform task with a unique JSON format DSL for that task.

The provided transforms are:

shift   : copy data from the input tree and put it the output tree
default : apply default values to the tree
remove  : remove data from the tree
sort    : sort the Map key values alphabetically ( for debugging and human readability )
java    : run any Java class that implements the `Transform` interface

The out-of-the-box Jolt transforms should be able to do 90% of what you need, with the java component giving you a way to implement that last 10%.

Documentation

Jolt Slide Deck : covers motivation, development, and transforms.

Javadoc explaining each transform DSL :

• shift
• default
• remove
• sort
• java : Implement the Transform or SpecDriven interfaces
  • Transform interface
  • SpecDriven

Running a Jolt transform means creating an instance of Chainr with a list of transforms.

The JSON spec for Chainr looks like : unit test.

The Java side looks like :

Chainr chainr = new Chainr( ...getResourceAsStream( "/path/to/chainr/spec.json" ) );

Object input = elasticSearchHit.getSource(); // ElasticSearch already returns hydrated JSon

Object output = chainr.transform( input );

return output;

Shiftr Transform DSL

The Shiftr transform generally does most of the "heavy lifting" in the transform chain. To see the Shiftr DSL in action, please look at our unit tests (shiftr tests) for nice bite sized transform examples, and read the Shiftr docs.

Our unit tests follow the pattern :

{
    "input": {
        // sample input
    },

    "spec": {
        // transform spec
    },

    "expected": {
        // what the output of the transform looks like
    }
}

We read in "input", apply the "spec", and Diffy it against the "expected".

To learn the Shiftr DSL, examine "input" and "output" json, get an understanding of how data is moving, and then look at the transform spec to see how it facilitates the transform.

For reference, this was the very first test we wrote.

Getting Started

Has it's own doc.

Getting Transform Help

If you can't get a transform working and you need help, create and Issue in Jolt (for now).

Make sure you include what your "input" is, and what you want your "output" to be.

Alternatives

Aside from writing your own custom code to do a transform, there are two general approaches to doing Json to Json transforms in Java.

1. JSON -> XML -> XSLT or STX -> XML -> JSON

Aside from being a Rube Goldberg approach, XSLT is more complicated than Jolt because it is trying to do the whole transform with a single DSL.

2. Write a Template (Velocity, FreeMarker, etc) that take hydrated JSON input and write textual JSON output

With this approach you are working from the output format backwards to the input, which is complex for any non-trivial transform. Eg, the structure of your template will be dictated by the output JSON format, and you will end up coding a parallel tree walk of the input data and the output format in your template. Jolt works forward from the input data to the output format which is simpler, and it does the parallel tree walk for you.

Performance

The primary goal of Jolt was to improve "developer speed" by providing the ability to have a declarative rather than imperative transforms. That said, Jolt should have a better runtime than the alternatives listed above.

Work has been done to make the stock Jolt transforms fast:

1. Transforms can be initialized once with their spec, and re-used many times in a multi-threaded environment.
   • We reuse initialized Jolt transforms to service multiple web requests from a DropWizard service.
2. "*" wildcard logic was redone to reduce the use of Regex in the common case, which was a dramatic speed improvement.
3. The parallel tree walk performed by Shiftr was optimized.

Two things to be aware of :

1. Jolt is not "stream" based, so if you have a very large Json document to transform you need to have enough memory to hold it.
2. The transform process will create and discard a lot of objects, so the garbage collector will have work to do.

Jolt CLI

Jolt Transforms and tools can be run from the command line. Command line interface doc here.

Code Coverage

For the moment we have Cobertura configured in our poms. When we move to a proper open source CI build, this can go away.

mvn cobertura:cobertura
open jolt-core/target/site/cobertura/index.html

Currently code coverage is at 89% line, and 81% branch.

Release Notes

On the Github Jolt Wiki.