Skip to content

Commit

Permalink
- add placeholder shp parse for missing federal states, as a check re…
Browse files Browse the repository at this point in the history 
…veiled that all of their GIS systems support shp export with various projections

- add support for more projections with additional query parameter
- update readme
- update version
  • Loading branch information
@chrispahm
chrispahm committed Jun 16, 2020
1 parent 9ca1d23 commit c823eed
Show file tree
Hide file tree
Showing 9 changed files with 228 additions and 39 deletions.
68 changes: 39 additions & 29 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,42 @@
# harmonie :seedling:
*harmonie* is a software package that harmonizes IACS (ZID) files of all German federal states. With *harmonie*, you can outsource parsing of the different direct payment application files, so you can rely on a given input data structure.
*harmonie* is a software package that harmonizes farm subsidy application files of all German federal states.
It takes the various xml, gml, shp, and dbf files issued by the federal farm subsidy application softwares, and converts them into **WGS84** projected
GeoJSON geometries suitable for any web mapping service.

With *harmonie*, you can outsource parsing of the different direct payment application files, so you can rely on a given input data structure.

*harmonie*
- ✅ works in the browser/Node.js
- ✅ adheres to the [agroJSON](https://github.com/fruchtfolge/agroJSON) specification
- ✅ is open source (MIT license)

This repository is **WIP**, APIs and data structures are likely to change.
Currently, **9/16** federal states are supported.

Converting the geometries (fields) contained in the application files is supported for **all** federal states in Germany.
For most federal states, additional information as the primary crop cultivated on a field, as well as the [fieldblock number (FLIK)](https://de.wikipedia.org/wiki/Feldblock) are obtained as well.

The following overview table displays the current state of the project:

| Federal state | ISO 3166-2 code | Test data available? | Supported by 'harmonie' | ToDos |
|:-----------------------|:----------------|:---------------------|:------------------------|:-----------------------------------------------|
| Brandenburg | DE-BB | ✅ | ✅ | - |
| Berlin | DE-BE | ✅ | ✅ | - |
| Baden-Württemberg | DE-BW | ✅ | ✅ | - |
| Bayern | DE-BY | ✅ | ✅ | - |
| Bremen | DE-HB | ⬜️ | ⬜️ | Waiting on test data (ANDI NDS) |
| Hessen | DE-HE | ✅ | ✅ | - |
| Hamburg | DE-HH | ⬜️ | ⬜️ | Waiting on test data (ANDI NDS) |
| Mecklenburg-Vorpommern | DE-MV | ✅ | ✅ | - |
| Niedersachsen | DE-NI | ⬜️ | ⬜️ | Waiting on test data (ANDI NDS) |
| Nordrhein-Westfalen | DE-NW | ✅ | ✅ | - |
| Rheinland-Pfalz | DE-RP | ✅ | ⬜️ | Create property mapping |
| Schleswig-Holstein | DE-SH | ⬜️ | ⬜️ | Waiting on test data (ELSA Schleswig-Holstein) |
| Saarland | DE-SL | ✅ | ✅ | - |
| Sachsen | DE-SN | ⬜️ | ⬜️ | Waiting on test data (DIANAweb Sachsen-Anhalt) |
| Sachsen-Anhalt | DE-ST | ⬜️ | ⬜️ | Waiting on test data (ELAISA Sachsen) |
| Thüringen | DE-TH | ✅ | ✅ | - |
✅ Federal states fully supported by 'harmonie': Plot geometries, including crop cultivation code and FLIK are obtained.
☑️ Federal states partially supported by 'harmonie': Only plot geometries are obtained.

| Federal state | ISO 3166-2 code | Subsidy application program | Test data available? | Supported by 'harmonie' | ToDos |
|:-----------------------|:----------------|:-----------------------------------------------------------------------------------------------------------------------------------------|:---------------------|:------------------------|:-----------------------------------------------|
| Brandenburg | DE-BB | [WebClient Agrarantrag BB](https://www.agrarantrag-bb.de/webClient_BB_P/) | ✅ | ✅ | - |
| Berlin | DE-BE | [WebClient Agrarantrag BB](https://www.agrarantrag-bb.de/webClient_BB_P/) | ✅ | ✅ | - |
| Baden-Württemberg | DE-BW | [FIONA - Flächeninformation und Online-Antrag](https://fiona.landbw.de/fiona/pages/login.xhtml) | ✅ | ✅ | - |
| Bayern | DE-BY | [iBalis Bayern](https://www.stmelf.bayern.de/ibalis/hRPSYCJ9iai73RtXboYewXCBR-_cYW-D/hRPf5) | ✅ | ✅ | - |
| Bremen | DE-HB | [ANDI - Agrarförderung Niedersachsen Digital](https://sla.niedersachsen.de/andi-web/) | ⬜️ | ☑️ | Waiting on test data (ANDI NDS) |
| Hessen | DE-HE | [Antragsmappe der WI-Bank](https://www.wibank.de/wibank/direktzahlungen) | ✅ | ✅ | - |
| Hamburg | DE-HH | [ANDI - Agrarförderung Niedersachsen Digital](https://sla.niedersachsen.de/andi-web/) | ⬜️ | ☑️ | Waiting on test data (ANDI NDS) |
| Mecklenburg-Vorpommern | DE-MV | [WebClient Agrarantrag MV](https://online.agrarantrag-mv.de/webClient_MV_P/) | ✅ | ✅ | - |
| Niedersachsen | DE-NI | [ANDI - Agrarförderung Niedersachsen Digital](https://sla.niedersachsen.de/andi-web/) | ⬜️ | ☑️ | Waiting on test data (ANDI NDS) |
| Nordrhein-Westfalen | DE-NW | [ELAN-NRW WebClient](https://www.elan-nrw.de/webClient_NW/#docs) | ✅ | ✅ | - |
| Rheinland-Pfalz | DE-RP | [eAntrag](https://www.dlr.rlp.de/Internet/global/inetcntr.nsf/dlr_web_full.xsp?src=6F7G9TYH1A&p1=IT5HUS52Z8&p3=SXP6I7GS55&p4=JM94D5V1SK) | ✅ | ☑️ | Create property mapping |
| Schleswig-Holstein | DE-SH | [WebClient Agrarantrag SH](https://www.sammelantrag-sh.dataport.de/webClient_SH_P/SHWebClient.html) | ⬜️ | ☑️ | Waiting on test data (ELSA Schleswig-Holstein) |
| Saarland | DE-SL | [ASdigital Saarland](https://www.saarland.de/126854.htm) | ✅ | ✅ | - |
| Sachsen | DE-SN | [DianaWeb WebClient SN](https://www.diana.sachsen.de/webClient_SN_P/#login) | ⬜️ | ☑️ | Waiting on test data (DIANAweb Sachsen) |
| Sachsen-Anhalt | DE-ST | [ELAISA WebClient ST](https://www.inet17.sachsen-anhalt.de/webClient_ST_P/) | ⬜️ | ☑️ | Waiting on test data (ELAISA Sachsen-Anhalt) |
| Thüringen | DE-TH | [Verona Thüringen](https://verona.thueringen.de/#) | ✅ | ✅ | - |

## Installation

Expand Down Expand Up @@ -80,8 +89,9 @@ Returns an array of objects containing individual parts of fields (German: *Teil
- `gml` *\<string, UTF-8\>*
- `shp` *\<blob\>*
- `dbf` *\<blob\>*
- `projection` *\<string, UTF-8\> optional for shp and dbf request, can be specified when the input files are not EPSG:25832*

Sample minimum return value:
Sample return value:
```js
[{
id: 'harmonie_runningIndex_FieldBlockNumber', // e.g. 'harmonie_36_DEBBLI0261009129'
Expand Down Expand Up @@ -112,17 +122,17 @@ the federal state:
| Berlin | DE-BE | state, xml |
| Baden-Württemberg | DE-BW | state, xml, shp, dbf |
| Bayern | DE-BY | state, xml |
| Bremen | DE-HB | state, - |
| Bremen | DE-HB | state, shp, dbf |
| Hessen | DE-HE | state, shp, dbf |
| Hamburg | DE-HH | state, - |
| Hamburg | DE-HH | state, shp, dbf |
| Mecklenburg-Vorpommern | DE-MV | state, xml |
| Niedersachsen | DE-NI | state, - |
| Niedersachsen | DE-NI | state, shp, dbf |
| Nordrhein-Westfalen | DE-NW | state, xml, gml |
| Rheinland-Pfalz | DE-RP | state, - |
| Schleswig-Holstein | DE-SH | state, - |
| Rheinland-Pfalz | DE-RP | state, shp, dbf |
| Schleswig-Holstein | DE-SH | state, shp, dbf |
| Saarland | DE-SL | state, shp, dbf |
| Sachsen | DE-SN | state, - |
| Sachsen-Anhalt | DE-ST | state, - |
| Sachsen | DE-SN | state, shp, dbf |
| Sachsen-Anhalt | DE-ST | state, shp, dbf |
| Thüringen | DE-TH | state, shp, dbf |

## Specifics for certain federal states
Expand Down
64 changes: 61 additions & 3 deletions dist/harmonie.cjs.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ proj4.defs('EPSG:5650', '+proj=tmerc +lat_0=0 +lon_0=15 +k=0.9996 +x_0=33500000
proj4.defs('EPSG:31467', '+proj=tmerc +lat_0=0 +lon_0=9 +k=1 +x_0=3500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
// DE-SL
proj4.defs('EPSG:31462', '+proj=tmerc +lat_0=0 +lon_0=6 +k=1 +x_0=2500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
// others
proj4.defs('EPSG:31468', '+proj=tmerc +lat_0=0 +lon_0=12 +k=1 +x_0=4500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
proj4.defs('EPSG:25833', '+proj=utm +zone=33 +ellps=GRS80 +units=m +no_defs');
proj4.defs('EPSG:4647', '+proj=tmerc +lat_0=0 +lon_0=9 +k=0.9996 +x_0=32500000 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs');

const fromETRS89 = new proj4.Proj('EPSG:25832');
const toWGS84 = new proj4.Proj('WGS84');
Expand Down Expand Up @@ -438,6 +442,44 @@ async function bb$1 (query) {
return helpers.groupByFLIK(plots)
}

async function sl (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// if a projection was passed, check if it is supported
const supportedProjs = ['EPSG:25832', 'EPSG:5650', 'EPSG:31467', 'EPSG:31462', 'EPSG:31468', 'EPSG:25833', 'EPSG:4647'];
if (query.projection && supportedProjs.indexOf(query.projection) === -1) {
throw new Error(`Projection ${query.projection} is not supported by harmonie. The supported projections are: ${supportedProjs}`)
}
// parse the shape file information
const geometries = await parse.shape(query.shp, query.dbf);
// reproject coordinates into web mercator
geometries.features = geometries.features.map(f => helpers.reprojectFeature(f, query.projection));

// as we don't know anything about the structure of the shape files,
// we just make some assumptions based on the following information
//
const subplots = geometries.features.map((plot, count) => new Field({
id: `harmonie_${count}_${plot.properties.FLIK}`,
referenceDate: plot.properties.ANTJAHR,
NameOfField: '',
NumberOfField: count,
Area: plot.properties.AKT_FL,
FieldBlockNumber: plot.properties.FLIK,
PartOfField: '',
SpatialData: plot,
Cultivation: {
PrimaryCrop: {
CropSpeciesCode: plot.properties.KC_GEM,
Name: undefined
}
}
}));

// finally, group the parts of fields by their FLIK and check whether they are
// actually seperate parts of fields
return helpers.groupByFLIK(subplots)
}

async function nw (query) {
const incomplete = queryComplete(query, ['xml', 'gml']);
if (incomplete) throw new Error(incomplete)
Expand Down Expand Up @@ -472,7 +514,7 @@ async function nw (query) {
return helpers.groupByFLIK(plots)
}

async function sl (query) {
async function sl$1 (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// parse the shape file information
Expand Down Expand Up @@ -502,7 +544,7 @@ async function sl (query) {
return helpers.groupByFLIK(subplots)
}

async function sl$1 (query) {
async function sl$2 (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// parse the shape file information
Expand Down Expand Up @@ -541,20 +583,36 @@ function harmonie (query) {
switch (state) {
case 'DE-BB':
return bb(query)
case 'DE-BE':
return bb(query)
case 'DE-BW':
return bw(query)
case 'DE-BY':
return bw$1(query)
case 'DE-HB':
return sl(query)
case 'DE-HE':
return he(query)
case 'DE-HH':
return sl(query)
case 'DE-MV':
return bb$1(query)
case 'DE-NI':
return sl(query)
case 'DE-NW':
return nw(query)
case 'DE-RP':
return sl(query)
case 'DE-SH':
return sl(query)
case 'DE-SL':
return sl$1(query)
case 'DE-SN':
return sl(query)
case 'DE-ST':
return sl(query)
case 'DE-TH':
return sl$1(query)
return sl$2(query)
default:
throw new Error(`No such state as "${state}" according to ISO 3166-2 in Germany."`)
}
Expand Down
64 changes: 61 additions & 3 deletions dist/harmonie.esm.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ proj4.defs('EPSG:5650', '+proj=tmerc +lat_0=0 +lon_0=15 +k=0.9996 +x_0=33500000
proj4.defs('EPSG:31467', '+proj=tmerc +lat_0=0 +lon_0=9 +k=1 +x_0=3500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
// DE-SL
proj4.defs('EPSG:31462', '+proj=tmerc +lat_0=0 +lon_0=6 +k=1 +x_0=2500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
// others
proj4.defs('EPSG:31468', '+proj=tmerc +lat_0=0 +lon_0=12 +k=1 +x_0=4500000 +y_0=0 +ellps=bessel +datum=potsdam +units=m +no_defs');
proj4.defs('EPSG:25833', '+proj=utm +zone=33 +ellps=GRS80 +units=m +no_defs');
proj4.defs('EPSG:4647', '+proj=tmerc +lat_0=0 +lon_0=9 +k=0.9996 +x_0=32500000 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs');

const fromETRS89 = new proj4.Proj('EPSG:25832');
const toWGS84 = new proj4.Proj('WGS84');
Expand Down Expand Up @@ -434,6 +438,44 @@ async function bb$1 (query) {
return helpers.groupByFLIK(plots)
}

async function sl (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// if a projection was passed, check if it is supported
const supportedProjs = ['EPSG:25832', 'EPSG:5650', 'EPSG:31467', 'EPSG:31462', 'EPSG:31468', 'EPSG:25833', 'EPSG:4647'];
if (query.projection && supportedProjs.indexOf(query.projection) === -1) {
throw new Error(`Projection ${query.projection} is not supported by harmonie. The supported projections are: ${supportedProjs}`)
}
// parse the shape file information
const geometries = await parse.shape(query.shp, query.dbf);
// reproject coordinates into web mercator
geometries.features = geometries.features.map(f => helpers.reprojectFeature(f, query.projection));

// as we don't know anything about the structure of the shape files,
// we just make some assumptions based on the following information
//
const subplots = geometries.features.map((plot, count) => new Field({
id: `harmonie_${count}_${plot.properties.FLIK}`,
referenceDate: plot.properties.ANTJAHR,
NameOfField: '',
NumberOfField: count,
Area: plot.properties.AKT_FL,
FieldBlockNumber: plot.properties.FLIK,
PartOfField: '',
SpatialData: plot,
Cultivation: {
PrimaryCrop: {
CropSpeciesCode: plot.properties.KC_GEM,
Name: undefined
}
}
}));

// finally, group the parts of fields by their FLIK and check whether they are
// actually seperate parts of fields
return helpers.groupByFLIK(subplots)
}

async function nw (query) {
const incomplete = queryComplete(query, ['xml', 'gml']);
if (incomplete) throw new Error(incomplete)
Expand Down Expand Up @@ -468,7 +510,7 @@ async function nw (query) {
return helpers.groupByFLIK(plots)
}

async function sl (query) {
async function sl$1 (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// parse the shape file information
Expand Down Expand Up @@ -498,7 +540,7 @@ async function sl (query) {
return helpers.groupByFLIK(subplots)
}

async function sl$1 (query) {
async function sl$2 (query) {
const incomplete = queryComplete(query, ['shp', 'dbf']);
if (incomplete) throw new Error(incomplete)
// parse the shape file information
Expand Down Expand Up @@ -537,20 +579,36 @@ function harmonie (query) {
switch (state) {
case 'DE-BB':
return bb(query)
case 'DE-BE':
return bb(query)
case 'DE-BW':
return bw(query)
case 'DE-BY':
return bw$1(query)
case 'DE-HB':
return sl(query)
case 'DE-HE':
return he(query)
case 'DE-HH':
return sl(query)
case 'DE-MV':
return bb$1(query)
case 'DE-NI':
return sl(query)
case 'DE-NW':
return nw(query)
case 'DE-RP':
return sl(query)
case 'DE-SH':
return sl(query)
case 'DE-SL':
return sl$1(query)
case 'DE-SN':
return sl(query)
case 'DE-ST':
return sl(query)
case 'DE-TH':
return sl$1(query)
return sl$2(query)
default:
throw new Error(`No such state as "${state}" according to ISO 3166-2 in Germany."`)
}
Expand Down
4 changes: 2 additions & 2 deletions dist/harmonie.min.js
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"main": "dist/harmonie.cjs.js",
"module": "dist/harmonie.esm.js",
"browser": "dist/harmonie.min.js",
"description": "Harmonizes IACS (ZID) files of all German Federal States",
"description": "Farm subsidy application files of all German federal states",
"directories": {
"test": "test"
},
Expand Down
Loading

0 comments on commit c823eed

Please sign in to comment.