Example LAZ API USGS

[karelvancamp]

Collection of USGS LAZ files can be accessed using API exposed in https://apps.nationalmap.gov/downloader/

Endpoint accepts polygon selection.
Returns list of valid LAZ files.

example fetch

https://tnmaccess.nationalmap.gov/api/v1/products?prodFormats=LAS,LAZ&datasets=Lidar%20Point%20Cloud%20(LPC)&polygon=-104.94262695312236%2041.52867510196275,-102.83325195312291%2041.52867510196275,-102.83325195312291%2040.45065268246805,-104.94262695312236%2040.45065268246805,%20-104.94262695312236%2041.52867510196275&

example list item

{
"title": "USGS Lidar Point Cloud CO SoPlatteRiver-Lot1 2013 13TFE627477 LAS 2015",
"moreInfo": "Lidar (Light detection and ranging) discrete-return point cloud data are available in the American Society for Photogrammetry and Remote Sensing (ASPRS) LAS format. The LAS format is a standardized binary format for storing 3-dimensional point cloud data and point attributes along with header information and variable length records specific to the data. Millions of data points are stored as a 3-dimensional data cloud as a series of x (longitude), y (latitude) and z (elevation) points. A few older projects in this collection are in ASCII format. Please refer to http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html for additional information.",
"sourceId": "5a801f46e4b00f54eb2a10dc",
"sourceName": "ScienceBase",
"sourceOriginId": null,
"sourceOriginName": "gda",
"metaUrl": "https://www.sciencebase.gov/catalog/item/5a801f46e4b00f54eb2a10dc",
"vendorMetaUrl": "https://thor-f5.er.usgs.gov/ngtoc/metadata/waf/elevation/lidar_point_cloud/laz/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_LAS_2015/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_13TFE627477_LAS_2015_meta.xml",
"publicationDate": "2015-06-22",
"lastUpdated": "2022-02-04T06:37:01.725-07:00",
"dateCreated": "2018-02-11T03:47:34.444-07:00",
"sizeInBytes": null,
"extent": "Varies",
"format": "LAZ",
"downloadURL": "https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_LAS_2015/laz/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_13TFE627477_LAS_2015.laz",
"downloadURLRaster": null,
"previewGraphicURL": "https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/LPC/browse/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_13TFE627477_LAS_2015_thumb.jpg",
"downloadLazURL": "https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_LAS_2015/laz/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_13TFE627477_LAS_2015.laz",
"urls": {
"LAZ": "https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_LAS_2015/laz/USGS_LPC_CO_SoPlatteRiver_Lot1_2013_13TFE627477_LAS_2015.laz"
},
"datasets": [

  ],
  "boundingBox": {
    "minX": -103.496705935362,
    "maxX": -103.484712074143,
    "minY": 40.4382258600637,
    "maxY": 40.4469402549868
  },
  "bestFitIndex": 0.0,
  "body": "Lidar (Light detection and ranging) discrete-return point cloud data are available in the American Society for Photogrammetry and Remote Sensing (ASPRS) LAS format. The LAS format is a standardized binary format for storing 3-dimensional point cloud data and point attributes along with header information and variable length records specific to the data. Millions of data points are stored as a 3-dimensional data cloud as a series of x (longitude), y (latitude) and z (elevation) points. A few older projects in this collection are in ASCII format.  Please refer to http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html for additional information. ",
  "processingUrl": "processingUrl",
  "modificationInfo": "2022-02-04"
}

[giswqs]

This is great! Does it work for querying LiDAR DEM as well?

[karelvancamp]

Looks like it.

https://tnmaccess.nationalmap.gov/api/v1/products?prodFormats=GeoTIFF&prodExtents=1%20x%201%20degree&datasets=National%20Elevation%20Dataset%20(NED)%201/3%20arc-second%20Current&polygon=-102.6025390624978%2040.27066783395388,-102.55859374999781%2040.27066783395388,-102.55859374999781%2040.241322506327634,-102.6025390624978%2040.241322506327634,%20-102.6025390624978%2040.27066783395388&

[karelvancamp]

It look like all datasets offered inside this interface use same high level API.

[giswqs]

Great! I will look into that and implement some functions for easier access to the data.

[karelvancamp]

Let me know if you need some help ;)

[karelvancamp]

One can simplify get

https://tnmaccess.nationalmap.gov/api/v1/products?prodFormats=LAS,LAZ&polygon=-98.63235473632734%2043.233743566291835,-98.30963134765535%2043.233743566291835,-98.30963134765535%2043.091502392267635,-98.63235473632734%2043.091502392267635,%20-98.63235473632734%2043.233743566291835&

Catalogue docs

https://apps.nationalmap.gov/tnmaccess/#/

[karelvancamp]

https://apps.nationalmap.gov/help/documents/TNMAccessAPIDocumentation/TNMAccessAPIDocumentation.pdf

[karelvancamp]

https://tnmaccess.nationalmap.gov/api/v1/datasets?

Describes datasets & possible formats in detail

{
"title": "Elevation Source Data (3DEP) - Lidar, IfSAR",
"sbDatasetTag": "National Elevation Dataset (NED)",
...,
"formats": [
{
"displayName": "LAS,LAZ",
"value": "LAS,LAZ",
"isDefault": true,
"activeTags": [
"Lidar Point Cloud (LPC)"
]
},

[karelvancamp]

US - USGS

• 2D viewer USGS
  https://earthexplorer.usgs.gov/
  Search across datasets
  tile selector, downloader

• 2D viewer USGS vN
  https://apps.nationalmap.gov/downloader/
  Search across datasets, data types (jpeg, geotiff, laz, …), ...
  tile selector, downloader
  Partial branch?

• API
  https://apps.nationalmap.gov/tnmaccess/#/
  Open GET
  Select polygon, dataformat, types, …
  Across all types of data (LAZ, geoJSON, TIFF, jpeg, …), datasets, ...
  Mostly self describing API
  https://tnmaccess.nationalmap.gov/api/v1/datasets?

[karelvancamp]

TNM wraps Sciencebase (2016)

https://tnmaccess.nationalmap.gov/api/v1/products?q=7.5&datasets=Digital%20Elevation%20Model%20(DEM)%201%20meter
https://my.usgs.gov/confluence/display/sciencebase/ScienceBase+Catalog+Item+REST+Services

[karelvancamp]

class USGS_tnm():

    api_endpoint = r'https://tnmaccess.nationalmap.gov/api/v1/'
    datasets = f'{api_endpoint}datasets?'
    dataset_full = None

    def dataset_full():
        return 'json'

    def datasets():
        return list

    def datatypes():
        return list

    def find_tiles(dataset_keyword='', datatypes=[], region=[]):
        return list


def download_usgs(region, keyword, datatypes, out_dir, return_url, download_args):
    return


def download_ned(region, out_dir=None, return_url=False, download_args={}, **kwargs):
    """Download the US National Elevation Datasets (NED) for a region.

[karelvancamp]

https://github.com/giswqs/leafmap/blob/106a46f0b7785a5ad979d177cdc410011fcd1e52/examples/notebooks/The%20national%20map%20API.ipynb

[giswqs]

This looks great. Would you like to submit a PR?

[karelvancamp]

Will make time for it this week or weekend.

[giswqs]

Just found some interesting packages that are relevant.

https://pypi.org/project/py3dep

[karelvancamp]

Yes, also looked at it. Really liked the WMS features. Considred proposing to add as depency, but simple requests in notebook example was less bloated/no new depency.

[karelvancamp]

I kinda used that repo for inspiration ;)

[giswqs]

I am trying to figure out how to download the NED data using the API. The following link is expected to return the current DEM tiles. However, it only returns the historical DEM tiles. Do you know why?

https://tnmaccess.nationalmap.gov/api/v1/products?datasets=National%20Elevation%20Dataset%20(NED)%201/3%20arc-second%20Current&polyCode=01010002&polyType=huc8

The returned result is expected is contained the following links, but none of them is there.

['https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n48w070/USGS_13_n48w070.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n48w069/USGS_13_n48w069.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n47w070/USGS_13_n47w070.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n47w069/USGS_13_n47w069.tif']

[karelvancamp]

Would have to investigate.
Assume listing refer to older snapshot of datastorage pointers.

[karelvancamp]

I do get a fetch

[karelvancamp]

[karelvancamp]

[giswqs]

That means you are getting the same results. Those are historical DEM tiles. Note the word 'historical' in the URLs. It should return the current ones.

[karelvancamp]

I'm not certain this applies. I don't add keyword for 'current' to my fetch.

Do you have an alternative API/datalist pointer which show there is more recent data available/published?

[giswqs]

See this direct URL https://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/Elevation/13/TIFF/

[giswqs]

You can see that these historical tiles have duplicates.

I would expect it the return the URLs like this:

['https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n48w070/USGS_13_n48w070.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n48w069/USGS_13_n48w069.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n47w070/USGS_13_n47w070.tif',
 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n47w069/USGS_13_n47w069.tif']
``

[giswqs]

Because they have different dates. The current one should not have dates in the file name.

[karelvancamp]

Why is this a problem? Could be current one is kept on higher availabilty/disk types.

These 2 are the same.

['https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/current/n47w069/USGS_13_n47w069.tif',
'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/historical/n47w069/USGS_13_n47w069_20210611.tif']

[giswqs]

I just tested the two files. It appears that they are exactly the same. I guess the 'current' link will always stay the same, so no changes on the users' end. The more recent copy of the historical files are just a copy of the "recent" version.

[karelvancamp]

Yes indeed.

[karelvancamp]

Last year update was only in november. Could be some more months before 2022 is available.

[karelvancamp]

I'm currently looking at possible PR & comparing to functionality

def download_ned(region, out_dir=None, return_url=False, download_args={}, **kwargs)

Few questions:

• Region now accepts vector datasets based on URL or filepath (ignoring bounding box). What are these datatypes/structures for these? I can feed polygons to api.
• I propose just copy pattern download_args as is used in download_ned()
• For full API exposure i don't have full parsing availble (f.e. for parameters "Current" or need clean_list() for 'polygon'). I could introduce a paritially working API, or restrict to working one. What do you prefer (use kwargs or leave out)?

def find_tiles(collection=None, datatype=None, keyword=None, region=[], **kwargs):
    TNM = The_national_map_USGS()
    
    if collection:
        assert collection in TNM.collections, f'Unknown collection, must be element of {TNM.collections}'
    if datatype:
        assert datatype in TNM.datatypes, f'Unknown datatypes, must be element of {TNM.datatypes}'
 
    def clean_list(x):
        return ','.join(' '.join(map(str,point)) for point in x)
    
    parameters = {'prodFormats': datatype,
                  'datasets': collection,
                  'q':keyword,
                  'polygon': clean_list(region),
                  **kwargs}
    
    response = requests.get(f'{TNM.api_endpoint}products?', params=parameters)
    if response.status_code//100 == 2:
        return response.json()
    return {}

[karelvancamp]

usage

paras = {
         'collection':'National Elevation Dataset (NED) 1/3 arc-second',
         'polyCode': '01010002',
         'polyType': 'huc8'
        }

[giswqs]

I have just added two functions #289
The previous download_ned() can probably be refactored for some general purposes. PR is welcome.

[karelvancamp]

Ok, will

Region now accepts vector datasets based on URL or filepath (ignoring bounding box). What are these datatypes/structures for these? I can feed polygons to api.

I want to understand common formats used for url/filepaths. This way i can translate format as needed. Can you provide exhaustive list of datatypes (+ optionally example on request) inputs. Or partial list of most common supported.

[giswqs]

You can allow region to accept a URL for filepath. Use geopandas to read it and retrieve coordinates, then pass it to the polygon parameter for the API.

[karelvancamp]

Code review welcome