Skip to content

Commit

Permalink
feat(config): disable the empty tiff checker (#3023)
Browse files Browse the repository at this point in the history
#### Motivation

The empty tiff checker was designed to remove empty tiffs from the
configuration as we had 9,000 tiffs that were empty. It greatly slows
down the build pipeline.

We have since deleted the tiffs and have added logic to prevent them
being uploaded in future.


#### Modification

remove the `isEmpty` check when creating config from tiffs.


#### Checklist

_If not applicable, provide explanation of why._

- [ ] Tests updated
- [ ] Docs updated
- [ ] Issue linked in Title
  • Loading branch information
blacha authored Dec 6, 2023
1 parent 8bf9be9 commit 811bc79
Showing 1 changed file with 10 additions and 27 deletions.
37 changes: 10 additions & 27 deletions packages/config/src/json/json.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,6 @@ export function guessIdFromUri(uri: string): string | null {
}
}

/**
* If a tiff is smaller than this size, Validate that the tiff actually has meaningful amounts of data
*/
const SmallTiffSizeBytes = 256 * 1024;

/**
* Check to see if this tiff has any data
*
Expand Down Expand Up @@ -296,28 +291,16 @@ export class ConfigJson {
imageList.push({ ...imgBounds, name: tileName });
}
} else {
await Promise.all(
tiffTiles.map((t) => {
return Q(async () => {
if (t.tiff.size != null && t.tiff.size < SmallTiffSizeBytes) {
const isEmpty = await isEmptyTiff(t.tiff.path);
if (isEmpty) {
this.logger.warn({ uri: t.tiff.path, imageId: id, size: t.tiff.size }, 'Imagery:Empty');
return;
} else {
this.logger.trace({ uri: t.tiff.path, imageId: id, size: t.tiff.size }, 'Imagery:CheckSmall:NotEmpty');
}
}
const tileName = basename(t.tiff.path);

const tile = tileMatrix.tileToSourceBounds(t.tile);
// Expand the total bounds to cover this tile
if (bounds == null) bounds = Bounds.fromJson(tile);
else bounds = bounds.union(Bounds.fromJson(tile));
imageList.push({ ...tile, name: tileName });
});
}),
);
for (const t of tiffTiles) {
// TODO add the .tiff extension back in once the new basemaps-config workflow has been merged
const tileName = basename(t.tiff.path).replace('.tiff', '');

const tile = tileMatrix.tileToSourceBounds(t.tile);
// Expand the total bounds to cover this tile
if (bounds == null) bounds = Bounds.fromJson(tile);
else bounds = bounds.union(Bounds.fromJson(tile));
imageList.push({ ...tile, name: tileName });
}
}

// Sort the files by Z, X, Y
Expand Down

0 comments on commit 811bc79

Please sign in to comment.