feat(cogify): Update the cogify create cog to support topo raster. BM-1116 #3388

Wentao-Kuang · 2025-01-08T01:41:03Z

Motivation

As a Basemaps user, I want to consume the NZTopo 50 & 250 Maps as a tile service.

Modifications

This work defines a new CLI command. The command's purpose is to generate STAC files for an NZTopo Map Series imagery collection. We have designed the command to process the following collections stored in the AWS S3 TopoReleaseArchive directory:

s3://topographic-upload/TopoReleaseArchive/NZTopo50_GeoTif_Gridless/
s3://topographic-upload/TopoReleaseArchive/NZTopo250_GeoTif_Gridless/

There is also a new Argo workflow in development that depends on this work. The workflow's purpose is to automate the standardisation of an NZTopo Map Series imagery collection. The workflow executes this command as a step in its process flow.

CLI Command: `cog` Updates

New Arguments

topo

Add --topo for cog cli to create cogs for topo raster maps.

Argument topo

Type boolean

Required no

Options true or false

CLI Command: `topo`

Arguments

The arguments that can be passed to the command are as follows:

Title

The name/title of the Map Series imagery collection.

Argument title

Type string

Required yes

Example Raster Topographic Maps 50k
Source

The source directory URL of the Map Series imagery collection.

Argument source

Type string

Required yes

Example s3://topographic-upload/TopoReleaseArchive/NZTopo50_GeoTif_Gridless/
Target

The target directory URL into which to save the generated directory structure of StacItem and StacCollection files.

Argument target

Type string

Required yes

Example s3://linz-workflows-scratch/<date>/<hash>/
Scale

The scale of the Map Series imagery collection's map sheets.

Argument scale

Type string

Required yes

Options topo25, topo50, or topo250
Resolution

The resolution of the Map Series imagery collection's map sheets.

Argument resolution

Type string

Required yes

Options gridded_600dpi or gridless_600dpi
Latest Only

A flag used to indicate whether all of the generated files should be saved to the target location, or only that of each map sheet's latest version.

Argument latestOnly

Type boolean

Required no

Options true or false

Process

The command processes a collection as follows:

Loads the collection of GeoTiff images into memory
Extracts the map code, version, and metadata (e.g. EPSG) from each image
Identifies the latest version of each map sheet by code
Groups the images by EPSG
Generates a StacItem file for each image, and a StacCollection file for each EPSG grouping
Structures the groups of StacItem and StacCollection files into a directory tree
Saves the files as they are structured to a target location

Outputs

The command groups the images by EPSG and then structures the generated StacItem and StacCollection files, as illustrated:

Schema	Example

The command then saves the generated tree of folders and files into the target location directory.

Limitations

Tile Matrices

For each image in a collection, the command extracts the EPSG of the image and converts it to a runtime Epsg Enum value. This value is then mapped to the EPSG's corresponding Tile Matrix definition. At this time, there is no such Tile Matrix definition for the Chatham Islands EPSG code, 3793. The task will need to be updated once Basemaps supports this definition.

packages/cogify/src/cogify/cli/cli.cover.ts

Co-authored-by: Blayne Chard <[email protected]>

packages/cogify/src/cogify/gdal.command.ts

blacha · 2025-01-28T03:07:59Z

packages/cogify/src/cogify/cli/cli.topo.ts

+ *
+ * @returns an array of StacItem objects
+ */
+async function loadTiffsToCreateStacs(


this is quite a huge function and would be very easy to mix the ordering of args it would be best to use a context object here.

blacha · 2025-01-28T03:08:33Z

packages/cogify/src/cogify/cli/cli.topo.ts

+
+/**
+ * @param source: Source directory URL from which to load tiff files
+ * @example TODO


TODO? would be best to just remove this comment?

blacha · 2025-01-28T03:09:43Z

packages/cogify/src/cogify/cli/cli.topo.ts

+
+    // write stac items into an JSON array
+    if (args.forceOutput || isArgo()) {
+      const targetURL = isArgo() ? fsa.toUrl('/tmp/topo-stac-creation/') : args.target;


generally its best to not upper case variables targetURL -> targetUrl

As you get into weird screaming words when you join two or more together HTMLURL

blacha · 2025-01-28T03:10:22Z

packages/cogify/src/cogify/cli/cli.topo.ts

+
+      // tiles.json makes the tiff files
+      await fsa.write(new URL('tiles.json', targetURL), JSON.stringify(stacItemPaths, null, 2));
+      await fsa.write(new URL('brokenTiffs.json', targetURL), JSON.stringify(brokenTiffs, null, 2));


do any of the other outputs use camelCasing.json ?

blacha · 2025-01-28T03:11:20Z

packages/cogify/src/cogify/gdal/gdal.command.ts


 const isPowerOfTwo = (x: number): boolean => (x & (x - 1)) === 0;
+const DEFAULT_TRIM_PIXEL_RIGHT = 1.7; // 1.7 pixels to trim from the right side of the topo raster imagery


we generally use TitleCase here, I wonder why the linter is not picking this up?

blacha · 2025-01-28T03:14:50Z

packages/cogify/src/cogify/gdal/gdal.command.ts

  if (source.length === 0) throw new Error('No source files given for :' + targetVrt.href);
  return {
    output: targetVrt,
    command: 'gdalbuildvrt',
-    args: [urlToString(targetVrt), ...source.map(urlToString)],
+    args: [addalpha ? ['-addalpha'] : undefined, urlToString(targetVrt), ...source.map(urlToString)]
+      .filter((f) => f != null)


I wonder if flatMap would work here

.flatMap(f => f != null ? String(f): []) [['foo'], undefined, ['bar'], [undefined, 'baz']].flatMap(f => f != null ? String(f): []) [ 'foo', 'bar', ',baz' ]

This only works with one value in the nested array, and not working with others with multiple values like [ ['-of', 'vrt'], ]

blacha · 2025-01-28T03:16:47Z

packages/cogify/src/cogify/stac.ts

+    /**
+     * The width of a map sheet in pixels.
+     */
+    'source.width': number;


. inside json objects property names can cause some tools to have lots of problems, some tools use it to expect object nesting { source: { width: 100 } } could these be moved or renamed?

We should also be prefixing keys that we are adding eg linz_basemaps:map_code

blacha · 2025-01-28T03:20:58Z

packages/cogify/src/cogify/topo/extract.ts

+    const code = Epsg.tryGet(epsg);
+
+    if (code != null) {
+      logger?.info({ found: true, method: 'direct' }, 'extractEpsgFromTiff()');


would be good to log which tiff, tiff.source.url.href in these log messages.

blacha · 2025-01-28T03:21:23Z

packages/cogify/src/cogify/topo/extract.ts

+  }
+
+  if (img.valueGeo(TiffTagGeo.GTRasterTypeGeoKey) === RasterTypeKey.PixelIsPoint) {
+    throw new Error("'Pixel is Point' raster grid spacing is not supported");


it is a good idea to throw with the tiff location so its easier to track broken tiffs.

blacha · 2025-01-28T03:21:58Z

packages/cogify/src/cogify/topo/extract.ts

+  try {
+    const size = tiff.images[0]?.size ?? null;
+
+    logger?.info({ found: size }, 'extractSizeFromTiff()');


mixing booleans and numbers will cause issues with logging.

blacha · 2025-01-28T03:22:29Z

packages/cogify/src/cogify/topo/mapper.ts

+} from './extract.js';
+import { brokenTiffs, ByDirectory, TiffItem } from './types.js';
+
+const slugs: { [key in EpsgCode]?: string } = {


Suggested change

const slugs: { [key in EpsgCode]?: string } = {

const slugs: Partial<Record<EpsgCode, string>> = {

would this work?

blacha · 2025-01-28T03:23:35Z

packages/cogify/src/cogify/topo/mapper.ts

+    const source = tiff.source.url;
+    const { mapCode, version } = extractMapCodeAndVersion(source.href, logger);
+
+    const bounds = extractBoundsFromTiff(tiff);


if one of these throws its going to break the loop early, we should process all items and collect all errors before exiting.

blacha · 2025-01-28T03:24:37Z

packages/cogify/src/cogify/topo/stac.creation.ts

+  const latestURL = new URL(`${latest.mapCode}_${latest.version}.json`, allTargetURL);
+
+  // add link to all items pointing to the latest version
+  allStacItems.forEach((stacItem) => {


NIT: we generally perffer for const (item of allStacItems) over .forEach

blacha · 2025-01-28T03:25:23Z

packages/cogify/src/cogify/topo/stac.creation.ts

+  const byDirectory = new ByDirectory<TiffItem>();
+
+  // create items for each tiff and store them into 'all' by {epsg} and {map code}
+  for (const tiff of tiffs) {


this appears to be duplicated above and should be simplified.

blacha · 2025-01-28T03:26:22Z

packages/cogify/src/cogify/topo/types.ts

+
+export const brokenTiffs = { noBounds: [] as string[], noEpsg: [] as string[], noSize: [] as string[] };
+
+export class TiffItem {


this should just be a object, there is not much point in making a class that only contains global state.

The original motive for TiffItem existing as a class rather than an object was to allow the use of JSON.stringify on the ByDirectory<TiffItem> instance. With this, we output an itemsByDirectory.json file that looks something like this:

{ "all": { "2193": { "250-01": [ "250-01_v1-01", ... ], "250-02": [ "250-02_v1-01", ... ], ... }, "3793": { "250-31": [ "250-31_v1-01", ... ] } }, "latest": { "2193": { "250-01": "250-01_v1-03", "250-02": "250-02_v1-03", ... }, "3793": { "250-31": "250-31_v1-04" } } }

This file serves as a sanity check, but does not provide any particular utility. If the file doesn't provide any value, I'm happy to convert the TiffItem class to an object.

Update the cogify create cog to support topo raster

30904ff

Wentao-Kuang added the container build pull request container for testing label Jan 8, 2025

Optional the tilematrix setting for topo-raster

da266fa

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 9, 2025

Wentao-Kuang added 2 commits January 10, 2025 11:27

Stop target srs if nonreprojection is set

ff0e4b4

Add imagery name from linz slug

5728e59

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 10, 2025

Overwrite epsg if noreprojection.

e7edc8e

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 10, 2025

Wentao-Kuang marked this pull request as ready for review January 12, 2025 21:01

Wentao-Kuang requested a review from a team as a code owner January 12, 2025 21:01

Wentao-Kuang requested review from blacha and tawera-manaena January 12, 2025 21:01

Wentao-Kuang changed the title ~~feat(cogify): Update the cogify create cog to support topo raster.~~ feat(cogify): Update the cogify create cog to support topo raster. BM-1116 Jan 12, 2025

blacha reviewed Jan 12, 2025

View reviewed changes

packages/cogify/src/cogify/cli/cli.cover.ts Outdated Show resolved Hide resolved

Wentao-Kuang and others added 2 commits January 13, 2025 10:49

Update packages/cogify/src/cogify/cli/cli.cover.ts

ea22f8c

Co-authored-by: Blayne Chard <[email protected]>

Fix linting

07e39b1

blacha reviewed Jan 12, 2025

View reviewed changes

packages/cogify/src/cogify/gdal.command.ts Outdated Show resolved Hide resolved

create topo raster cog

87fa033

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 13, 2025

Fix linting

73c2b71

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 13, 2025

Correct the order the input and output

e8a41a6

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 13, 2025

Fix the cmd.

f8c75ed

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 27, 2025

Fix the tmp url

268c442

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 27, 2025

blacha reviewed Jan 28, 2025

View reviewed changes

tawera-manaena and others added 3 commits January 29, 2025 12:04

fix(cogify): convert absolute StacItem link into a relative link (wip)

170ffda

Minor refinements.

9cb3c14

More refactoring to remove the types.ts

78ca1db

Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jan 29, 2025

refactor(cogify): simplify the StacItem creation function

24087c3

tawera-manaena mentioned this pull request Feb 11, 2025

[DO NOT MERGE] feat(cogify): migrate argo-task work to basemaps-cogify BM-1127 #3393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cogify): Update the cogify create cog to support topo raster. BM-1116 #3388

feat(cogify): Update the cogify create cog to support topo raster. BM-1116 #3388

Wentao-Kuang commented Jan 8, 2025 •

edited

Loading

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

Wentao-Kuang Jan 29, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

blacha Jan 28, 2025

tawera-manaena Jan 28, 2025

Argument	source
Type	string
Required	yes
Example	`s3://topographic-upload/TopoReleaseArchive/NZTopo50_GeoTif_Gridless/`


		const isPowerOfTwo = (x: number): boolean => (x & (x - 1)) === 0;
		const DEFAULT_TRIM_PIXEL_RIGHT = 1.7; // 1.7 pixels to trim from the right side of the topo raster imagery

	const slugs: { [key in EpsgCode]?: string } = {
	const slugs: Partial<Record<EpsgCode, string>> = {


		export const brokenTiffs = { noBounds: [] as string[], noEpsg: [] as string[], noSize: [] as string[] };

		export class TiffItem {

Argument	scale
Type	string
Required	yes
Options	`topo25`, `topo50`, or `topo250`

Argument	latestOnly
Type	boolean
Required	no
Options	`true` or `false`

feat(cogify): Update the cogify create cog to support topo raster. BM-1116 #3388

Are you sure you want to change the base?

feat(cogify): Update the cogify create cog to support topo raster. BM-1116 #3388

Conversation

Wentao-Kuang commented Jan 8, 2025 • edited Loading

Motivation

Modifications

CLI Command: cog Updates

New Arguments

CLI Command: topo

Arguments

Process

Outputs

Limitations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wentao-Kuang commented Jan 8, 2025 •

edited

Loading

CLI Command: `cog` Updates

CLI Command: `topo`