TG2-AMENDMENT_COUNTRYCODE_FROM_COORDINATES

[iDigBioBot]

TestField	Value
GUID	8c5fe9c9-4ba9-49ef-b15a-9ccd0424e6ae
Label	AMENDMENT_COUNTRYCODE_FROM_COORDINATES
Description	Proposes an amendment to the value of dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code.
TestType	Amendment
Darwin Core Class	dcterms:Location
Information Elements ActedUpon	dwc:countryCode
Information Elements Consulted	dwc:decimalLatitude
	dwc:decimalLongitude
Expected Response	EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is bdq:Empty, or if dwc:countryCode is bdq:NotEmpty; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED.
Data Quality Dimension	Completeness
Term-Actions	COUNTRYCODE_FROM_COORDINATES
Parameter(s)	bdq:sourceAuthority
Source Authority	bdq:sourceAuthority default = "10m-admin-1 boundaries UNION with Exclusive Economic Zones" {[https://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-admin-1-states-provinces/] spatial UNION [https://www.marineregions.org/downloads.php#marbound]}
Specification Last Updated	2024-08-18
Examples	[dwc:decimalLatitude="-25.23", dwc:decimalLongitude="135.43", dwc:countryCode="": Response.status=FILLED_IN, Response.result=dwc:countryCode="AU", Response.comment="dwc:decimalLatitude and dwc:decimalLongitude contain interpretable values"]
	[dwc:decimalLatitude="-38.280937", dwc:decimalLongitude="72.047790", dwc:countryCode="": Response.status=NOT_AMENDED, Response.result="", Response.comment="Coordinates do not fall in the boundary of any country"]
Source	ALA, GBIF, iDigBio
References	Chapman AD and Wieczorek JR (2020) Georeferencing Best Practices. Copenhagen: GBIF Secretariat. https://doi.org/10.15468/doc-gg7h-s853 VLIZ (2023). Marineregions.org. https://www.marineregions.org/downloads.php#marbound Dooley, JF Jnr. (2005) An inventory and comparison of globally consistent geospatial databases and libraries. Rome: FAO. http://www.fao.org/3/a0118e/a0118e00.htm#Contents Wikipedia (2020) ISO 3166-1 alpha-2. https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 DataHub (2018) List of all countries with their two digit codes (ISO 3166-1). https://datahub.io/core/country-list Kelso NV and Patterson T (2010) Introducing Natural Earth data—Naturalearthdata.com. Geographica Technica. Special issue, 2010 pp 82–89. https://technicalgeography.org/pdf/sp_i_2010/12_introducing_natural_earth_data__naturaleart.pdf Natural Earth (2022) Admin 1 – States, provinces. v5.1.1 2022-05-12. https://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-admin-1-states-provinces/ Natural Earth (2022) Natural Earth Free vector and raster map data at 1:10m, 1:50m, and 1:110m scales. v5.1.2. https://www.naturalearthdata.com/, https://github.com/nvkelso/natural-earth-vector/releases/tag/v5.1.2.
Example Implementations (Mechanisms)
Link to Specification Source Code
Notes	This amendment simply fills dwc:countryCode from a lookup of dwc:decimalLatitude and dwc:decimalLongitude. dwc:coordinateUncertaintyInMeters and dwc:coordinatePrecicision (if present) imply a buffer around the provided coordinates. Likewise, country polygons cannot be 100% accurate at all scales (Dooley 2005), so a spatial buffer of the country boundaries is also justified. Taking spatial buffers into account does however greatly complicate the logic and the implementation of this and related tests. In this test, a detection of multiple country codes by sampling within the buffer while possible, is not considered.

[iDigBioBot]

Comment by Paula Zermoglio (@pzermoglio) migrated from spreadsheet:
Only useful if performed AFTER decimalLat and decimalLong interpretation.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
@pz amendment sequence is indeed important. Tianhong Song has a paper on this in the context of prerequisites for workflow actors.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Implementation requires guidance on how to handle marine material inside a country's exclusive economic zone.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Should include coordinateUncertainty in meters. Also see Lee's note in Principles about some geographic tests needing buffers

[ArthurChapman]

Change Country to Country Codes - name and elsewhere

[tucotuco]

Agreed at TDWG 2018 DQIG meeting that the name TG2-AMENDMENT_COUNTRYCODE_FROM_COORDINATES is satisfactory.

[tucotuco]

Similar to the problem raised in Issue #185, this tests mentions a source authority that can not deliver the AMENDMENT. It is also silent on the authority for the geometries for the country codes. To me, the bdq:sourceAuthority should be the GBIF reverse geocoding API (https://github.com/gbif/geocode), coming in 2020. It will be based on Natural Earth, GADM, Open Street Maps, EEZones and more. The documentation says it will be for internal GBIF use, but @timrobertson100 says that he expects the API to be exposed.

[timrobertson100]

The geocode service is available today, and provides a lookup based on the given coordinate.

For example latitude 51.0 and longitude 1.0 yields this response:

[
  {
    "id": "80",
    "type": "Political",
    "source": "http://www.naturalearthdata.com",
    "title": "United Kingdom",
    "isoCountryCode2Digit": "GB"
},
{
    "id": "212",
    "type": "EEZ",
    "source": "http://vliz.be/vmdcdata/marbound/",
    "title": "United Kingdom",
    "isoCountryCode2Digit": "GB"
    }
]

Because boundaries don't align (resolution of the polygons), we buffer the search which is why multiple results can be returned.

To reduce WS traffic, we also encode the database into an image with dictionary encoded colors (i.e. by seeing a non-black or white colour, you can refer to the dictionary to know the country).

Today the service only has EEZ and NaturalEarth files, but can (will) be extended.

When a record has a stated country and coordinates we verify that seems reasonable, and if not flips coordinates around and negates them "hunting" for a match. This is because in many cases the negative sign is omitted, or coordinates swapped. All of this happens after a reprojection to WGS84 if necessary.

[tucotuco]

@timrobertson100 This is ideal. May we cite it as our bdq:sourceAuthority default?

[chicoreus]

We are conflating authority with service with thesaurus in bdq:sourceAuthority. The authority for country codes is the ISO two letter country code list. The GBIF service is a service that wraps natural earth plus (source?) EEZ layers (and will change overtime as layers are added (with versioning?), the thesaurus is the natural earth and EEZ layers. For implementation, I'd much rather use a local GIS data store containing natural earth data and some appropriate EEZ layer than consulting a remote service - I want to use the same thesaurus, but not the service.

[tucotuco]

OK, what is the consensus then on what to cite and how? I think it is more useful to point people to the thesaurus. If that can be done in a general way instead of to a particular service built around it, so much the better. For example, is https://github.com/gbif/geocode the right thing to cite for the source authority for this test? What does not seem useful is to just give the controlled vocabulary that the amendment should comply with as the source authority, and to not have the thesaurus in the References. The controlled vocabulary is incomplete as an authority for amendments of this type.

…

On Tue, Apr 14, 2020 at 11:51 AM Paul J. Morris ***@***.***> wrote: We are conflating authority with service with thesaurus in bdq:sourceAuthority. The authority for country codes is the ISO two letter country code list. The GBIF service is a service that wraps natural earth plus (source?) EEZ layers (and will change overtime as layers are added (with versioning?), the thesaurus is the natural earth and EEZ layers. For implementation, I'd much rather use a local GIS data store containing natural earth data and some appropriate EEZ layer than consulting a remote service - I want to use the same thesaurus, but not the service. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADQ72YD26BR3XCMM6AMOULRMRZ63ANCNFSM4EKSNA7Q> .

[Tasilee]

Thanks @timrobertson100 , @tucotuco and @chicoreus. I am the least able person in the group to provide wisdom but figure I would put my thoughts down, anyway.

As we have discussed, GBIF is likely for the near future, to be a service location that aggregates thesauri and what we currently call 'bdq:sourceAuthority' (authorities).

Are the thesauri themselves 'source authorities'? If we are dependent on them as the reference, then in our context, I'd say yes. I agree with @chicoreus that the services associated with any source authority are a separate issue. That's why we use " if the bdq:sourceAuthority service ..."

To address @tucotuco 's question about how we reference, we either reference to the GBIF namespace end point (?) which references the relevant external source authority (e.g., ISO) in a standard way or we reference both the GBIF and the 'external' source authority. The former would be nice. Then there is a separate reference to the service associated with the thesaurus.

In the Expected response, we have agreed to use bdq:sourceAuthority [and service.] where there are implementation-dependent options or directly use the name of the source authority where there is no choice.

What we place in References and Notes, I defer to more appropriate authorities.

[ArthurChapman]

My view is same as @tuco and his question to @timrobertson100
"@timrobertson100 This is ideal. May we cite it as our bdq:sourceAuthority default"

If can just add the GBIF Geocode Authority as our bdq:sourceAuthority as we have done elsewhere, then I think this is our best option.

[chicoreus]

The more I think about this, the less I like the idea of specifying a query endpoint as a sourceAuthority. This leaves the resolution of the shape files, buffering near borders, and changes over time as opaque to the consumer, and also forces implementations to use a service for a test that should not be implemented with remote service calls but with a local spatial data store.

We should specify three parameters: (1) A shape file for country boundaries. (2) A shape file for exclusive economic zones. (3) An explicit buffer for points that fall near country boundaries that takes into account the resolution of the shapefiles. In addition, we should explicilty state in the specification how points that fall into the buffer are to be handled (preferably by not asserting an amendment, probably with INTERNAL_PREREQUISITES_NOT_MET, point falls too near boundary of shape to determine placement). In addition, we need to consider coordinatePrecision, and how that as an uncertainty on the coordinate intersects with countries or buffers.

We need to be much, much more explicit in how edge cases are to be handled in any test that involves GIS data.

This amendment also needs to cover the case of FILLED_IN, AMENDED would only cover an existing case of countryCode being altered based on the coordinates. We should consider if this test should ever assert AMENDED, or should restrict itself to FILLED_IN. I would generally be much more comfortable with asserting only FILLED_IN, as AMENDED could fix the wrong value (the coordinates, or their precision, or their error radius could be in error, but the country and countryCode be correct). This is particularly true near boundaries, where the error may lie in the resolution of the shape rather than either the coordinate or the countryCode.

[chicoreus]

I would suggest:

EXTERNAL_PREREQUISITES_NOT_MET if an external source authority service or local spatial data store was not available; INTERNAL_PREREQUISITES_NOT_MET if the fields dwc:decimalLatitude, dwc:decimalLongitude are EMPTY or the dwc:decimalLatitude and dwc:decimalLongitude cannot be converted to the SRS used for queries to the spatial service or data store, or if the area represented by the dwc:decimalLatitude, dwc:decimalLongitude, and dwc:coordinatePrecision overlaps with a 3km buffer zone on any country or EEZ shape in the service or spatial data store; FILLED_IN if the value of dwc:countryCode was EMPTY and was unambiguously inferred from supplied dwc:decimalLatitude, dwc:decimalLongitude and dwc:coordinatePrecision falling within a single boundary defined by the combination of terrestrial and exclusive economic zone and not overlapping a 3km buffer around such boundary; otherwise NOT_CHANGED

[ArthurChapman]

Another issue with this test (and the others where we have bdq:sourceAuthority[xxxx]). If these are separated into different terms in the Vocabulary as suggested as a possibility in #152 (comment) - then we would need to change Parameter(s) to bdq:sourceAuthority to "bdq:sourceAuthority[countryshapes]" and Source Authority to "bdq:sourceAuthority default = "ADM1 boundaries" [https://gadm.org] UNION with "EEZs" [https://marineregions.org]"

ALSO - we have in bdq:souceAuthority "bdq:sourceAuthority[countryCode]" but this term doesn't occur anywhere in the test (or any other test) and is not in the Vocabulary - should this be bdq:sourceAuthority[countryshapes]???

[Tasilee]

I have updated the Source authority and description

[ArthurChapman]

I don't think there is anything outstanding wrt NEEDS WORK - other than a decision on how we treat bdq:sourceAuthority[countryshapes]

[ArthurChapman]

Updated Expected Response, bdq:sourceAuthority and Description to replace bdq:sourceAuthority:[countryShapes] with bdq:countryShapes and Specification Last Updated

Changed Parameter(s) from bdq:sourceAuthority to bdq:countryShapes

[Tasilee]

I thought @tucotuco's suggestion was bdq:sourceAuthority_countryshapes (or maybe "countryShapes") etc? That way, we don't loose the source authority link.

[ArthurChapman]

We don't use that for any other bdq: terms But if all agree would could change it - but see the Vocab or #205 for all the terms we would have to change

| bdq:annotationAlertIf | [https://github.com//issues/152] | |
| bdq:countryShapes | [https://github.com//issues/152] | |
| bdq:annotationAnnotationSystem | [https://github.com//issues/152] | |
| bdq:defaultGeodeticDatum | [https://github.com//issues/152] | |
| bdq:earliestValidDate | [https://github.com//issues/152] | |
| bdq:geospatialLand | [https://github.com//issues/152] | |
| bdq:includeEventDate | [https://github.com//issues/152] | |
| bdq:latestValidDate | [https://github.com//issues/152] | |
| bdq:maximumValidDepthInMeters | [https://github.com//issues/152] | |
| bdq:maximumValidElevationInMeters | [https://github.com//issues/152] | |
| bdq:minimumValidDepthInMeters | [https://github.com//issues/152] | |
| bdq:minimumValidElevationInMeters | [https://github.com//issues/152] | |
| bdq:sourceAuthority | [https://github.com//issues/152] | |
| bdq:spatialBufferInMeters | [https://github.com//issues/152] | |
| bdq:targetCRS | [https://github.com//issues/152] | |
| bdq:taxonomyIsMarine | [https://github.com//issues/152] | |

[Tasilee]

Thanks @ArthurChapman. The list includes terms that do relate to a bdq:sourceAuthority (annotations, country shapes/land, geodetic datum) but the rest don't really, unless you want to stretch the concept.

I have therefore changed my mind (backflip?) and think that the syntax like "bdq:countryShapes" seems expedient as all bdqs are listed under the 'heading' "Source Authority" anyway.

[Tasilee]

Aligned the Source Authority entries between #50 and this test

[Tasilee]

Post Zoom 11/7/2023, I have aligned the Source Authority with the suggested syntax:

bdq:sourceAuthority default = "ADM1 boundaries" {[https://gadm.org] spatial UNION with "Exclusive Economic Zones" [https://marineregions.org]}

[Tasilee]

In the light of #50 suggestion, changed Source Authority to

| Source Authority | bdq:sourceAuthority default = "ADM1 boundaries spatial UNION with Exclusive Economic Zones" {[https://gadm.org] spatial UNION [https://marineregions.org]} |

[Tasilee]

Changed Parameter(s) from bdq:countryShapes to bdq:sourceAuthority and presume we will leave it Parameterized.

[Tasilee]

Splitting bdqffdq:Information Elements into "Information Elements ActedUpon" and "Information Elements Consulted".

Also changed "Field" to "TestField", "Output Type" to "TestType" and updated "Specification Last Updated"

[Tasilee]

Standardized reference to bdq:sourceAuthority in Expected Response to "EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available"

[Tasilee]

Changed Expected Response from

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY or uninterpretable, or if dwc:countryCode is NOT_EMPTY; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.

to

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is EMPTY, or if dwc:countryCode is NOT_EMPTY; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.

[Tasilee]

The current Expected Response/Specification doesn't make sense as it uses bdq:countryShapes but this term is not defined, yet bdq:sourceAuthority is (and it already in the first part (EXTERNAL_PREREQUISITES....) that seems crazy given all the comments on this in this issue. Given the inconsistencies here could I suggest we change

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is bdq:Empty, or if dwc:countryCode is bdq:NotEmpty; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary from the bdq:countryShapes that is attributable to a single valid country code; otherwise NOT_AMENDED.

to

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is bdq:Empty, or if dwc:countryCode is bdq:NotEmpty; FILLED_IN dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority that is attributable to a single valid country code; otherwise NOT_AMENDED.

??

[chicoreus]

Our current practice is to only give distinct names to sourceAuthority parameters when there is more than one in an issue. Thus, yes, it makes sense to change bdq:countryShapes here to the generic term bdq:sourceAuthority.

[Tasilee]

Done

[ArthurChapman]

As with the discussion under #98 and the issue of "High Seas" with dwc:countryCode-'XZ' See also #62.

If thought appropriate, we could change this test to something like

EXTERNAL_PREREQUISITES_NOT_MET if the bdq:sourceAuthority is not available; INTERNAL_PREREQUISITES_NOT_MET if either dwc:decimalLatitude or dwc:decimalLongitude is bdq:Empty, or if dwc:countryCode is bdq:NotEmpty; FILLED_IN if 1) dwc:countryCode if dwc:decimalLatitude and dwc:decimalLongitude fall within a boundary in the bdq:sourceAuthority that is attributable to a single valid country code or 2) dwc:countryCode with 'XZ' if the coordinates fall with the boundary of "High Seas" as defined in [xxxxx]; otherwise NOT_AMENDED.

Or we can leave as it is now and just have "NOT_AMENDED. Just a suggestion.

[Tasilee]

As this test is an AMENDMENT, we are not implying lack of quality by a lack of an amendment. However, given the discussion, we could be proactive if we could set dwc:countryCode="XZ" using coordinates and a 'high seas' reference map.

@ArthurChapman: Your Expected Response is appealing. How say others?

[tucotuco]

We would also have to change the description if we change the expected response. I'm not fond of including the exception and complicating the test. Multipe bdq:sourceAuthorities would be required, and could each of those be parametrized?

I would not change this test. Rather, I would create set of tests around something like TG2-AMENDMENT-COUNTRYCODE_FROM_HIGH_SEAS_COORDINATES.

[Tasilee]

I have changed dwc:decimalLatitude and dwc:decimalLongitude from Information Elements ActedUpon to Consulted. OK? This came from checking AMENDMENT relationships.

[chicoreus]

@Tasilee good catch.