Hootenanny:
- A gathering at which folksingers entertain often with the audience joining in
Conflation:
- Fancy word for merge
Hootenanny is an open source map data conflation tool developed with machine learning techniques to facilitate automated and semi-automated conflation of critical Foundation GEOINT features in the topographic domain. In short, it merges multiple maps into a single seamless map.
- Automatically combine geospatial features for decision making
- Allow for reviewing and manually resolving features which cannot be automatically matched with sufficient certainty
- Maintain geometry and attribute provenance for combined features
- Create up-to-date routable transportation networks from multiple sources
Hootenanny conflation occurs at the dataset level, where the user’s workflow determines the best reference dataset, source content, geometry, and attributes to transfer to the output map. Hootenanny's internal processing leverages the key/value pair structure of OpenStreetMap (OSM) for improved utility and applicability to broader user groups. Normalized attributes can be used to aid in feature matching, and OSM’s free tagging system allows the map to include an unlimited number of attributes describing each feature.
No automated map conflation technology is perfect. If you are conflating a relatively small number of features, you may be best served to combine them manually yourself, given you are looking for a perfect result and want to avoid any potential time spent configuring software options to get the best conflated output. For larger datasets, however, Hootenanny is a valuable tool that can be utilized standalone or as an inital step in conjunction with a crowd sourced mapping campaign to add new data into your dataset. You will find that the conflation automation provided by Hootenanny saves effort overall, and that most inaccuracies in the conflated output are a small subset of the input data which often end up being flagged for human review so that they may be manually corrected.
Documentation is included as PDF files with each release. Download
the Hootenanny .tar.gz installation file and locate the documentation within the "docs" directory.
Hootenanny has specifically tailored conflation algorithms available for the following data types:
- Areas
- Buildings
- Points of Interest (POIs)
- Power Lines
- Railways
- Rivers
- Roads
Any feature whose type does not fit into the list of types above, or has no type at all, will be conflated with Generic Geometry Conflation, which uses a simpler rules based approach than the type specific conflation algorithms.
You can create your own custom conflation algorithms for additional feature types via Javascript or C++.
Conflation Workflows
A conflation workflow defines the manner in which two maps are merged together. Hootenanny has the following workflows:
- Reference Conflation (default) - Keep the best of both maps while favoring the first
- Use this type of conflation when you want map output based on the best state of two maps while favoring the first one.
- Average Conflation - Keep an average of both maps
- Use this type of conflation when you consider both input maps equal in quality and want a result that is an average of the two.
- Currently, geometry averaging only applies to linear features but could be extended to point and polygon geometries. Point and polygon geometries are merged the same as in Reference Conflation.
- Average Conflation is currently not available from iD Editor.
- Horizontal Conflation (aka Cookie Cutter Conflation) - Completely replace a section
- Use this type of conflation if you have a specific region of your map that you would like to completely replace with a region from another map.
- Differential Conflation - Add new features that do not conflict
- Use this type of conflation when you want to fill holes in your map with data from another source without actually modifying any of the data in your map.
- There is an option available to additionally transfer tags to existing features in your map from matching features in another map where overlap occurs.
- Attribute Conflation - Transfer attributes over to existing geometries
- Use this type of conflation when one map's geometry is superior to that of a second map, but the attributes of the second map are superior to that of the first map.
Hootenanny leverages the OSM key value pair tag concept to support translation between various data schemas and supports the following schemas:
- Topographic Data Store (TDS)
- Multi-National Geospatial Co-Production Program (MGCP)
- Geonames
- OSM
- others
Users can define their own custom schema translations via Javascript or Python.
Hootenanny has the capability to let you selectively pick the features that are conflated from your data to save you from pre-conflation data wrangling.
Some examples:
- Conflate only buildings
- Conflate only restaurant buildings
- Conflate only restaurant buildings with "Subway" in the name
Hootenanny has several available filters that may be used to perform feature filtering during conflation.
Hootenanny performs no data validation reporting of its own, but JOSM validation routines may be invoked via Hootenanny during batch processing of data.
Hootenanny has a variety of map cleaning capabilities to automatically correct erroneous data. Some of them are invoked automatically during a conflation job, however, you may also run them separately. In addition to its own built-in cleaning operations, Hootenanny is also integrated with the auto-fix capabilities of JOSM.
NOTE: As of 3/19/20 the cleaning/validation integration with JOSM is only available in the Vagrant version of Hootenanny. It will be added to the RPM version of Hootenanny at a later date.
Although Hootenanny is configured by default to provide the best conflation results, at times you will need to tweak its wide range of configuration options in order to get the best conflated result for your input data.
Hootenanny's web user interface is built upon the open source Mapbox iD Editor, which provides an intuitive and user-friendly conflation experience.
Access to Hootenanny core capabilities are exposed through a web services API for those wishing to develop their own conflation clients. The web services use OAuth authentication.
Command line access is available to aid in custom scripting of conflation capabilities.
Example:
# conflate two datasets together
hoot conflate input1.osm input2.osm output.osm
Hootenanny has nodejs bindings available which expose core conflation capabilities for creating custom workflows.
Example:
// conflate two datasets together
var hoot = require(process.env.HOOT_HOME + '/lib/HootJs');
var map = new hoot.OsmMap();
hoot.loadMap(map, "input1.osm", false, 1);
hoot.loadMap(map, "input2.osm", false, 2);
new hoot.UnifyingConflator().apply(map)
hoot.saveMap(map, "output.osm");
In addition to conflating map data, Hootenanny also provides these supporting capabilities:
- Add missing type tags to features implicitly based on their name (experimental)
- Align the geometries of two maps more closely with each other
- Apply data transformation operations to a map
- Calculate the geospatial extent of a map
- Clean map data
- Compare maps with each other
- Convert maps between different geodata formats
- Create a covering shape that minimally covers all features in a map
- Crop a map to a geospatial extent
- De-duplicate features within a single map or across two maps or simply find duplicate features in a map (experimental)
- Derive changesets between maps and push through the OpenStreetMap API
- Detect spoken languages in a map's tag data (experimental)
- Explore tag data
- Filter a subset of features before conflation based on tag content and schema relationships
- Find street intersections by name
- Generate a task grid
- Generate pre-configured or custom map statistics
- Grab a random section of data from a map for testing against
- Identify street intersection locations in a map
- Locate phone numbers geographically
- Replace or remove data completely within a bounding box via changeset and push the changes through the OpenStreetMap API
- Perturb map data for testing purposes
- Plot node density
- Process map postal address data
- Simplify ways by removing unnecessary nodes
- Snap unconnected ways to neighboring ways
- Sort map data
- Split a map into tiles
- Translate feature tags using standardized or user defined schemas
- Translate feature tags to English (experimental)
- Validate features
Hootenanny currently does not strive to conflate data at the global level. An earlier implementation of Hootenanny supported a map-reduce architecture that was capable of global conflation for some data types but was shelved due to general lack of interest and the maintenance costs to support the seldomly used capability. So some of the conflation algorithms are actually capable of supporting distributed computing, if you want to try and go that route and revive the capability. However, such capabilities are likely out of date with the rest of the codebase and may have limitations as far as the feature types they can conflate.
Hootenanny generally can scale well running on a single machine from the larger city level up to the smaller country level, depending on the density of the data being conflated and the RAM available on the machine. Beyond that, new conflation algorithms and/or parallelization of existing algorithms would need to be developed to handle very large quantities of map data.
Don't hesitate to ask for help if features aren't conflating how you expect them to or if you're experiencing difficulty while installing the software. We officially support installing RPMs to CentOS and launching CentOS virtual machines via Vagrant/VirtualBox. For anything else you need to do we'll do our best to help you out, but there are no guarantees. If you have any support questions, please create an issue in this repository.
As there are lot of different data scenarios out in the wild, there is no one-size fits all conflation workflow or algorithm. Hootenanny attempts to provide a useful set of conflation workflows with pre-configured options that capture most conflation requirements. However, you may require a new conflation scenario for your purposes (let us know), or you may need to custom modify configuration options for an existing scenario in order to get the best results.
Also, the availability of new software features to the user interface may lag their initial availability from the command line interface (CLI) by one or more development cycles. If you find a conflation feature you wish to use that is mentioned in the CLI documentation but is not present within the UI, let us know.