+
+The `GeoArrowLoader` parses Apache Arrow columnar table format files, and looks for `GeoArrow` type extensions to parse geometries from the table.
+
+| Loader | Characteristic |
+| --------------------- | ------------------------------------------------------------------------- |
+| File Format | [IPC: Encapsulated Message Format](https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc) |
+| Data Format | [Geometry Table](/docs/specifications/category-table) |
+| File Extension | `.arrow` |
+| File Type | Binary |
+| Decoder Type | `load`, `parse`, `parseSync`, `parseInBatches` |
+| Worker Thread Support | Yes |
+| Streaming Support | Yes |
+
+## Usage
+
+```typescript
+import {GeoArrowLoader} from '@loaders.gl/arrow';
+import {load} from '@loaders.gl/core';
+
+const data = await load(url, GeoArrowLoader, options);
+```
+
+## Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
diff --git a/docs/modules/arrow/formats/geoarrow.md b/docs/modules/arrow/formats/geoarrow.md
index 38ed71f5a7..b79835d986 100644
--- a/docs/modules/arrow/formats/geoarrow.md
+++ b/docs/modules/arrow/formats/geoarrow.md
@@ -22,22 +22,22 @@ Geospatial tabular data where one or more columns contains feature geometries an
Note that GeoArrow is not a separate format from Apache Arrow rather, the GeoArrow specification simply describes additional conventions for metadata and layout of geospatial data. This means that a valid GeoArrow file is always a valid Arrow file. This is done through [Arrow extension type](https://arrow.apache.org/docs/format/Columnar.html#extension-types) definitions that ensure type-level metadata (e.g., CRS) is propagated when used in Arrow implementations.
+## Geometry Types
+
+| Geometry type | Read | Write | Description |
+| -------------------------- | ---- | ----- | -------------------- |
+| `geoarrow.point` | ✅ | ❌ | |
+| `geoarrow.multipoint` | ✅ | ❌ | |
+| `geoarrow.linestring` | ✅ | ❌ | |
+| `geoarrow.multilinestring` | ✅ | ❌ | |
+| `geoarrow.polygon` | ✅ | ❌ | |
+| `geoarrow.multipolygon` | ✅ | ❌ | |
+| `geoarrow.wkb` | ✅ | ❌ | `WKB` also supported |
+| `geoarrow.wkt` | ✅ | ❌ | `WKT` also supported |
+
## Relationship with GeoParquet
-The [GeoParquet specification](https://github.com/opengeospatial/geoparquet) is closely related to GeoArrow. Notable differences:
+The [GeoParquet](/docs/modules/parquet/formats/geoparquet) [specification](https://github.com/opengeospatial/geoparquet) is closely related to GeoArrow. Notable differences:
- GeoParquet is a file-level metadata specification
- GeoArrow is a field-level metadata and memory layout specification
-
-## Geometry Types
-
-| Geometry type | Read | Write | Description |
-| -------------------------- | ---- | ----- | ----------- |
-| `geoarrow.multipolygon` | ✅ | ❌ | |
-| `geoarrow.polygon` | ✅ | ❌ | |
-| `geoarrow.multipoint` | ✅ | ❌ | |
-| `geoarrow.point` | ✅ | ❌ | |
-| `geoarrow.multilinestring` | ✅ | ❌ | |
-| `geoarrow.linestring` | ✅ | ❌ | |
-| `geoarrow.wkb` | ❌ | ❌ | |
-| `geoarrow.wkt` | ❌ | ❌ | |
\ No newline at end of file
diff --git a/docs/modules/flatgeobuf/api-reference/flatgeobuf-loader.md b/docs/modules/flatgeobuf/api-reference/flatgeobuf-loader.md
index 571f9633fe..c67e79d2de 100644
--- a/docs/modules/flatgeobuf/api-reference/flatgeobuf-loader.md
+++ b/docs/modules/flatgeobuf/api-reference/flatgeobuf-loader.md
@@ -1,4 +1,4 @@
-# FlatGeobufLoader 🚧
+# FlatGeobufLoader
@@ -6,13 +6,13 @@
-Loader for the [FlatGeobuf](http://flatgeobuf.org/) format, a binary FlatBuffers-encoded format that defines geospatial geometries.
+Loader for the [FlatGeobuf](/docs/modules/flatgeobuf/formats/flatgeobuf) format, a binary FlatBuffers-encoded format that defines geospatial geometries.
| Loader | Characteristic |
| -------------- | --------------------------------------------------------------- |
| File Extension | `.fgb`, |
| File Type | Binary |
-| File Format | [FlatGeobuf](http://flatgeobuf.org/) |
+| File Format | [FlatGeobuf](/docs/modules/flatgeobuf/formats/flatgeobuf) |
| Data Format | [Geometry](/docs/specifications/category-gis) |
| Supported APIs | `load`, `loadInBatches`, `parse`, `parseSync`, `parseInBatches` |
@@ -39,4 +39,4 @@ The parser will return an array of [GeoJSON `features`](https://tools.ietf.org/h
## Attribution
-The `FlatGeobufLoader` wraps the [`flatgeobuf`](https://github.com/bjornharrtell/flatgeobuf) NPM module under the ISC license.
+The `FlatGeobufLoader` wraps the [`flatgeobuf`](https://github.com/bjornharrtell/flatgeobuf) NPM module which is published under the ISC license.
diff --git a/docs/modules/flatgeobuf/formats/flatgeobuf.md b/docs/modules/flatgeobuf/formats/flatgeobuf.md
new file mode 100644
index 0000000000..9c1b3ce43f
--- /dev/null
+++ b/docs/modules/flatgeobuf/formats/flatgeobuf.md
@@ -0,0 +1,17 @@
+# FlatGeobuf
+
+- *[`@loaders.gl/flatgeobuf`](/docs/moodules/flatgeobuf)*
+- *[FlatGeobuf](http://flatgeobuf.org/)*
+
+FlatGeobuf is a binary (FlatBuffers-encoded) format that defines geospatial geometries. It is row-oriented rather than columnar (like GeoParquet and GeoArrow) and offers a different set of trade-offs.
+
+FlatGeobuf was inspired by [geobuf](https://github.com/mapbox/geobuf) and [flatbush](https://github.com/mourner/flatbush).
+
+## Characteristics
+
+- binary
+- row oriented
+- supports appends, but no random writes
+
+Goals are to be suitable for large volumes of static data, significantly faster than legacy formats without size limitations for contents or metainformation and to be suitable for streaming/random access.
+
diff --git a/docs/modules/geopackage/README.md b/docs/modules/geopackage/README.md
index 18598c78d7..ddf31b3af7 100644
--- a/docs/modules/geopackage/README.md
+++ b/docs/modules/geopackage/README.md
@@ -1,4 +1,4 @@
-# Overview 🚧
+# Overview
diff --git a/docs/modules/geopackage/api-reference/geopackage-loader.md b/docs/modules/geopackage/api-reference/geopackage-loader.md
index dff6dae211..65bef30d5f 100644
--- a/docs/modules/geopackage/api-reference/geopackage-loader.md
+++ b/docs/modules/geopackage/api-reference/geopackage-loader.md
@@ -1,4 +1,4 @@
-# GeoPackageLoader 🚧
+# GeoPackageLoader
@@ -6,6 +6,10 @@
+:::caution
+The `GeoPackageLoader` depends on the [`sql.js`](https://github.com/sql-js/sql.js) npm module which has caused issues with certain JavaScript bundlers. It is recommended that you do your own tests before using the `GeoPackageLoader` in your project.
+:::
+
GeoPackage loader
| Loader | Characteristic |
diff --git a/docs/modules/las/README.md b/docs/modules/las/README.md
index e783d40f06..1b1f461c55 100644
--- a/docs/modules/las/README.md
+++ b/docs/modules/las/README.md
@@ -1,6 +1,11 @@
# Overview
-The `@loaders.gl/las` module handles the [LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities) (LAS) or its compressed version (LAZ), a public format for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+The `@loaders.gl/las` module supports the [LASER file format](/docs/modules/las/formats/las) (LAS) and its compressed version (LAZ).
+
+:::caution
+The `@loaders.gl/las` module only supports LAS/lAZ files up to LAS v1.3. It does not support LAS v1.4 files.
+For more detail, see the discussion in [Github Issues](https://github.com/visgl/loaders.gl/issues/591).
+:::
## Installation
diff --git a/docs/modules/las/api-reference/las-loader.md b/docs/modules/las/api-reference/las-loader.md
index 59ed17ee78..557b7f38f5 100644
--- a/docs/modules/las/api-reference/las-loader.md
+++ b/docs/modules/las/api-reference/las-loader.md
@@ -1,5 +1,10 @@
# LASLoader
+:::caution
+The `@loaders.gl/las` module only supports LAS/lAZ files up to LAS v1.3. It does not support LAS v1.4 files.
+For more detail, see the discussion in [Github Issues](https://github.com/visgl/loaders.gl/issues/591).
+:::
+
The `LASLoader` parses a point cloud in the LASER file format.
| Loader | Characteristic |
diff --git a/docs/modules/las/formats/las.md b/docs/modules/las/formats/las.md
index 1347004b3c..eb3376f018 100644
--- a/docs/modules/las/formats/las.md
+++ b/docs/modules/las/formats/las.md
@@ -1,3 +1,47 @@
# LAS / LAZ
-The `@loaders.gl/las` module handles the [LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities) (LAS) or its compressed version (LAZ), a public format for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+- *[`@loaders.gl/las`](/docs/modules/las)*
+- *[Wikipedia](https://en.wikipedia.org/wiki/LAS_file_format)* - *[LAS Spec](https://www.loc.gov/preservation/digital/formats/fdd/fdd000418.shtml)* - *[LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities)* - *[LASzip project](https://github.com/LASzip/LASzip)* - *[LAZ spec](https://www.cs.unc.edu/~isenburg/lastools/download/laszip.pdf)*
+
+The *LASER file format* (LAS) and its compressed version (LAZ) are public formats for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+
+## Variants
+
+LAS file format is not compressed. However there is an open source project (LASzip) which defined and implemented the open file format LAZ to losslessly compress LAS data.
+
+| Variant | Description |
+| ------- | --------------------- |
+| LAS | Uncompressed |
+| LAZ | Lossless compression. |
+
+## Version History
+
+| **Version** | **Date** | **loaders.gl Support** | **Description** |
+| ----------- | -------- | --------------------------- | --------------------------------------------------------------- |
+| 1.4 | | ❌ | 64 bit support |
+| 1.3 | | ✅ | Extended variable length records (EVLR) to hold longer metadata |
+| 1.2 | | ✅ | |
+| 1.1 | | ✅ | |
+| 1.0 | | ✅ | |
+
+Notes:
+- Work on LAS 2.0 was started but was suspended indefinitely.
+
+## File Structure
+
+A LAS file consists of sections:
+
+| Section | Description |
+| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Public header block | Describes format, number of points, extent of the point cloud and other generic data. |
+| Variable length records (VLR) | Any number of optional records (up to 64K bytes) to provide various data such as the spatial reference system used, metadata, waveform packet information and user application data. |
+| Point data records | Data for each of the individual points in the point cloud, including coordinates, classification (e.g. terrain or building), flight and scan data, etc. |
+| Extended variable length records (EVLR) | From v1.3. Similar to VLRs but located after the point data records and allow a much larger data payload per record due to the use of 8-byte size descriptors. |
+| Point data records | A LAS file contains point records in one of the point data record formats defined by the LAS specification |
+
+Notes:
+- As of LAS 1.4, there are 11 point data record formats (0 through 10) available. All point data records must be of the same format within the file. The various formats differ in the data fields available, such as GPS time, RGB and NIR color and wave packet information.
+- The 3D point coordinates are represented within the point data records by 32-bit integers, to which a scaling and offset defined in the public header must be applied in order to obtain the actual coordinates.
+- As the number of bytes used per point data record is explicitly given in the public header block, it is possible to add user-defined fields in "extra bytes" to the fields given by the specification-defined point data record formats. A standardized way of interpreting such extra bytes was introduced in the LAS 1.4 specification, in the form of a specific EVLR.
+
+
diff --git a/docs/modules/mvt/api-reference/tilejson-loader.md b/docs/modules/mvt/api-reference/tilejson-loader.md
index 928525e262..7c270088b9 100644
--- a/docs/modules/mvt/api-reference/tilejson-loader.md
+++ b/docs/modules/mvt/api-reference/tilejson-loader.md
@@ -4,7 +4,8 @@
-The `TileJSONLoader` parses header/metadata from a pmtiles archive
+The `TileJSONLoader` parses metadata from a TileJSON / tilestats file. It merges layer and field information from both tilestats and TileJSON and returns a strongly typed data structure.
+
| Loader | Characteristic |
| --------------------- | -------------------------------------------------- |
@@ -25,7 +26,12 @@ import {load} from '@loaders.gl/core';
const tileJSON = await load(url, TileJSONLoader, options);
```
+## Data Format
+
+See [TileJSON format](/docs/modules/mvt/formats/tilejson.md].
+
## Options
| Option | Type | Default | Description |
| ------ | ---- | ------- | ----------- |
+|
\ No newline at end of file
diff --git a/docs/modules/mvt/formats/mvt.md b/docs/modules/mvt/formats/mvt.md
index 665589668b..cf782b2312 100644
--- a/docs/modules/mvt/formats/mvt.md
+++ b/docs/modules/mvt/formats/mvt.md
@@ -1,12 +1,14 @@
# Mapbox Vector Tile
+- *[`@loaders.gl/mvt`](/docs/modules/mvt)*
- *[Mapbox Vector Tile Specification](https://github.com/mapbox/vector-tile-spec)*
-
A specification for encoding tiled vector data.
MVT is a protobuf-encoded format that defines geospatial geometries.
+tiles contain layers with features, the features can have geometries and properties.
+
## Metadata
It is often useful to have global metadata about a tileset. A common complementary format for encoding tileset metadata is [TileJSON](./tilejson).
diff --git a/docs/modules/mvt/formats/tilejson.md b/docs/modules/mvt/formats/tilejson.md
index 5c29814728..574fba8428 100644
--- a/docs/modules/mvt/formats/tilejson.md
+++ b/docs/modules/mvt/formats/tilejson.md
@@ -1,30 +1,87 @@
# TileJSON / Tilestats
-- *[TileJSON specification](https://github.com/mapbox/tilejson-spec/blob/master/3.0.0/README.md)*
-- *[Tilestats information](https://github.com/mapbox/mapbox-geostats)
-- *[Tilestats generation](https://github.com/mapbox/mapbox-geostats#output-the-stats)
+- *[`@loaders.gl/mvt`](/docs/modules/mvt)*
+- *[TileJSON specification](https://github.com/mapbox/tilejson-spec/blob/master/3.0.0/README.md)* - *[Tilestats information](https://github.com/mapbox/mapbox-geostats) - *[Tilestats generation](https://github.com/mapbox/mapbox-geostats#output-the-stats)
## TileJSON
-Metadata about a tileset.
+A TileJSON file contains JSON-encoded metadata about a vector tileset including which layers and fields (columns) can be found in the tiles.
-for representing metadata about multiple types of web-based map layers, to aid clients in configuration and browsing.
+The information in the TileJSON enables clients (such as web viewers) to understand the structure of data in the tileset up front, instead of piecing it together as tiles are loaded.
-As the name suggests, TileJSON is JSON encoded.
-## Tilestats
+While not all vector tilesets have a TileJSON file, when it is present there is normally a single TileJSON file per tileset, and this file is typically found at the root in TMS or XYZ based URL schemas, so that applications can make a speculative attempt to fetch it from a known place.
-Tilestats is a valuable inofficial "extension" to TileJSON. It provides column statistics, notably:
+## tilestats
+
+`tilestats`` is an inofficial "extension" to TileJSON. It provides column statistics, notably:
- the data type of each column
- min/max values for numeric columns (enabling e.g. global color scale calculations).
- a sample of values for each column
-Tilestats are not always available so applications must be prepared to work in their absence.
-However, tilestats is output by major tilers such as [tippecanoe](https://github.com/mapbox/mapbox-geostats#output-the-stats).
+tilestats provide "global" information about the data in the tileset that can allows for e.g.
+the creation of correct color maps that do not depend on the view (which requires knowing a priori the min and max values of each field / column).
+
+Note that tilestats are not always available for a given tileset, so applications must be prepared to work in their absence.
+However, tilestats are output by major tilers such as [tippecanoe](https://github.com/mapbox/mapbox-geostats#output-the-stats).
+
+## Format Description
+loaders.gl returns a typed `TileJSON` object, with merged tilestats information (if present).
+
+```typescript
+export type TileJSON
+```
## Fields
-| Data | TileJSON | tilestats | Description |
-| --- | --- | --- | --- |
-|
\ No newline at end of file
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ------------------ | -------- | --------- | --------------------------------------------------------------------- |
+| `name?` | `string` | | | Name of the tileset. |
+| `description?` | `string` | | | Short description of the tileset. |
+| `version?` | `string` | | | Version of the tileset. |
+| `tileFormat?` | `string` | | | Format of the tiles in the tileset.. |
+| `tilesetType?` | `string` | | | Type of tileset. |
+| `generator?` | `string` | | | Generating application. (e.g. tippecanoe adds this). |
+| `generatorOptions?` | `string` | | | Generating application options. (e.g. tippecanoe adds this). |
+| `scheme?` | `'xyz'` \| `'tms'` | | | Tile indexing scheme. Typically `tms`, i.e `z/x/y` coordinates. |
+| `tiles?` | `string[]` | | | Sharded URLs (can increased loading speed on HTTP 1 connections) |
+| `boundingBox?` | `[[w, s], [e, n]]` | | | limits of bounding box using axis units and order of specified CRS. |
+| `maxZoom?` | `number` | | | May be set to the maxZoom of the first layer |
+| `minZoom?` | `number` | | | May be set to the minZoom of the first layer |
+| `center?` | `number[]` | | | Center point of the data in the tileset |
+| `htmlAttribution?` | `string` | | | Attribution (can contain HTML syntax) |
+| `htmlLegend?` | `string` | | | Legend (can contain HTML syntax) |
+| `layers?` | `TileJSONLayer[]` | | | Layer information (combines tilestats (if present) and tilejson info) |
+| `metaJson?` | `any` | | | Any unparsed, nested JSON metadata |
+
+- `boundingBox` typing: `[min: [w: number, s: number], max: [e: number, n: number]]`
+
+```ts
+export type TileJSONLayer;
+```
+
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ----------------- | -------- | --------- | ------------------------------------------------------------------------------------ |
+| `name` | `string` | | | The name (id) of this layer (tilejson.vector_layers[].id / tilestats.layers[].layer) |
+| `description?` | `string` | | | The description of this layer (tilejson.layer.description) |
+| `featureCount?` | `number` | | | The number of features in this layer (tilestats.layer.count) |
+| `dominantGeometry?` | `string` | | | The dominant geometry type in this layer (tilestats.layer.geometry) |
+| `minZoom?` | `number` | | | An array of details about the first 100 attributes in this layer |
+| `maxZoom?` | `number` | | | |
+| `fields` | `TileJSONField[]` | | | |
+
+
+```ts
+export type TileJSONField;
+```
+
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ----------- | -------- | --------- | ---------------------------------------------------- |
+| `name` | `string` | | | The name of this attribute |
+| `description?` | `string` | | | |
+| `type` | `string` | | | |
+| `min?` | `number` | | | min value (if there are *any* numbers in the values) |
+| `max?` | `number` | | | max value (if there are *any* numbers in the values) |
+| `uniqueValueCount?` | `number` | | | Number of unique values across the tileset |
+| `values?` | `unknown[]` | | | An array of this attribute's first 100 unique values |
diff --git a/docs/modules/pmtiles/formats/pmtiles.md b/docs/modules/pmtiles/formats/pmtiles.md
index a71c97e939..f1d40c3c97 100644
--- a/docs/modules/pmtiles/formats/pmtiles.md
+++ b/docs/modules/pmtiles/formats/pmtiles.md
@@ -6,159 +6,38 @@ PMTiles is a single-file archive format for tiled data designed to enable indivi
## Overview
-TBA
+PMTiles is a general format for storing tiled data addressed by Z/X/Y coordinates in a single big archive file. This can be used to store cartographic basemap vector tiles, remote sensing observations, JPEG images, or more.
-## Versions
+PMTiles readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.
-## Version 3
+## Tile types
-- File Structure
-97% smaller overhead - Spec version 2 would always issue a 512 kilobyte initial request; version 3 reduces this to 16 kilobytes. What remains the same is that nearly any map tile can be retrieved in at most two additional requests.
-- Unlimited metadata - version 2 had a hard cap on the amount of JSON metadata of about 300 kilobytes; version 3 removes this limit. This is essential for tools like tippecanoe to store detailed column statistics. Essential archive information, such as tile type and compression methods, are stored in a binary header separate from application metadata.
-- Hilbert tile IDs - tiles internally are addressed by a single 64-bit Hilbert tile ID instead of Z/X/Y. See the blog post on Tile IDs for details.
-- Archive ordering - An optional clustered mode enforces that tile contents are laid out in Tile ID order.
-- Compressed directories and metadata - Directories used to fetch offsets of tile data consume about 10% the space of those in version 2. See the blog post on compressed directories for details.
-- JavaScript
-Compression - The TypeScript pmtiles library now includes a decompressor - fflate - to allow reading compressed vector tile archives directly in the browser. This reduces the size and latency of vector tiles by as much as 70%.
-- Tile Cancellation - All JavaScript plugins now support tile cancellation, meaning quick zooming across many levels will interrupt the loading of tiles that are never shown. This has a significant effect on the perceived user experience, as tiles at the end of a animation will appear earlier.
-- ETag support - clients can detect when files change on static storage by reading the ETag HTTP header. This means that PMTiles-based map applications can update datasets in place at low frequency without running into caching problems.
-
-## Version 3 Specification
-
-### File structure
-
-A PMTiles archive is a single-file archive of square tiles with five main sections:
-
-1. A fixed-size, 127-byte **Header** starting with `PMTiles` and then the spec version - currently `3` - that contains offsets to the next sections.
-2. A root **Directory**, described below. The Header and Root combined must be less than 16,384 bytes.
-3. JSON metadata.
-4. Optionally, a section of **Leaf Directories**, encoded the same way as the root.
-5. The tile data.
-
-### Entries
-
-A Directory is a list of `Entries`, in ascending order by `TileId`:
-
- Entry = (TileId uint64, Offset uint64, Length uint32, RunLength uint32)
-
-* `TileId` starts at 0 and corresponds to a cumulative position on the series of square Hilbert curves starting at z=0.
-* `Offset` is the position of the tile in the file relative to the start of the data section.
-* `Length` is the size of the tile in bytes.
-* `RunLength` is how many times this tile is repeated: the `TileId=5,RunLength=2` means that tile is present at IDs 5 and 6.
-* If `RunLength=0`, the offset/length points to a Leaf Directory where `TileId` is the first entry.
-
-### Directory Serialization
-
-Entries are stored in memory as integers, but serialized to disk using these compression steps:
-
-1. A little-endian varint indicating the # of entries
-2. Delta encoding of `TileId`
-3. Zeroing of `Offset`:
- * `0` if it is equal to the `Offset` + `Length` of the previous entry
- * `Offset+1` otherwise
-4. Varint encoding of all numbers
-5. Columnar ordering: all `TileId`s, all `RunLength`s, all `Length`s, then all `Offset`s
-6. Finally, general purpose compression as described by the `Header`'s `InternalCompression` field
-
-##3 Directory Hierarchy
-
-* The number of entries in the root directory and leaf directories is up to the implementation.
-* However, the compressed size of the header plus root directory is required in v3 to be under **16,384 bytes**. This is to allow latency-optimized clients to prefetch the root directory and guarantee it is complete. A sophisticated writer might need several attempts to optimize this.
-* Root size, leaf sizes and depth should be configurable by the user to optimize for different trade-offs: cost, bandwidth, latency.
+PMTiles is a container format and can in principle contain any type of quadtree tiles. A number of vector and image tile types are predefined.
-### Header Design
+| Type | MIME type | pmtiles | Description |
+| ---- | -------------------------------------- | ------- | --------------------------------------------------- |
+| MVT | `'application/vnd.mapbox-vector-tile'` | `1` | [Mapbox Vector Tile](/docs/modules/mvt/formats/mvt) |
+| PNG | `'image/png'` | `2` | |
+| JPEG | `'image/jpeg'` | `3` | |
+| WEBP | `'image/webp'` | `4` | |
+| AVIF | `'image/avif'` | `5` | |
+| - | `'application/octet-stream'` | - | Undefined / Custom types |
-*Certain fields belonging to metadata in v2 are promoted to fixed-size header fields. This allows a map container to be initialized to the desired extent or center without blocking on the JSON metadata, and allows proxies to return well-defined HTTP headers.*
-The `Header` is 127 bytes, with little-endian integer values:
+## Metadata
-| offset | description | width |
-| ------ | ----------------------------------------------------------------------------------------- | ----- |
-| 0 | magic number `PMTiles` | 7 |
-| 7 | spec version, currently `3` | 1 |
-| 8 | offset of root directory | 8 |
-| 16 | length of root directory | 8 |
-| 24 | offset of JSON metadata, possibly compressed by `InternalCompression` | 8 |
-| 32 | length of JSON metadata | 8 |
-| 40 | offset of leaf directories | 8 |
-| 48 | length of leaf directories | 8 |
-| 56 | offset of tile data | 8 |
-| 64 | length of tile data | 8 |
-| 72 | # of addressed tiles, 0 if unknown | 8 |
-| 80 | # of tile entries, 0 if unknown | 8 |
-| 88 | # of tile contents, 0 if unknown | 8 |
-| 96 | boolean clustered flag, `0x1` if true | 1 |
-| 97 | `InternalCompression` enum (0 = Unknown, 1 = None, 2 = Gzip, 3 = Brotli, 4 = Zstd) | 1 |
-| 98 | `TileCompression` enum | 1 |
-| 99 | tile type enum (0 = Unknown/Other, 1 = MVT (PBF Vector Tile), 2 = PNG, 3 = JPEG, 4 = WEBP | 1 |
-| 100 | min zoom | 1 |
-| 101 | max zoom | 1 |
-| 102 | min longitude (signed 32-bit integer: longitude * 10,000,000) | 4 |
-| 106 | min latitude | 4 |
-| 110 | max longitude | 4 |
-| 114 | max latitude | 4 |
-| 118 | center zoom | 1 |
-| 119 | center longitude | 4 |
-| 123 | center latitude | 4 |
+The pmtiles header has a field that can store JSON metadata. This means that for MVT pmtiles, [TileJSON](/docs/modules/mvts/formats/tilejson) is typically available, stored in the PMTiles header metadata field.
-### Notes
-* **# of addressed tiles**: the total number of tiles before run-length encoding, i.e. `Sum(RunLength)` over all entries.
-* **# of tile entries**: the total number of entries across all directories where `RunLength > 0`.
-* **# # of tile contents**: the number of referenced blobs in the tile section, or the unique # of offsets. If the archive is completely deduplicated, this is equal to the # of unique tile contents. If there is no deduplication, this is equal to the number of tile entries above.
-* **boolean clustered flag**: if true, blobs in the data section are ordered by Hilbert `TileId`. When writing with deduplication, this means that offsets are either contiguous with the previous offset+length, or refer to a lesser offset.
-* **compression enum**: Mandatory, tells the client how to decompress contents as well as provide correct `Content-Encoding` headers to browsers.
-* **tile type**: A hint as to the tile contents. Clients and proxies may use this to:
- * Automatically determine a visualization method
- * provide a conventional MIME type `Content-Type` HTTP header
- * Enforce a canonical extension e.g. `.mvt`, `png`, `jpeg`, `.webp` to prevent duplication in caches
+## Version History
-### Organization
+**Version 3**
-In most cases, the archive should be in the order `Header`, Root Directory, JSON Metadata, Leaf Directories, Tile Data. It is possible to relocate sections other than `Header` arbitrarily, but no current writers/readers take advantage of this. A future design may allow for reverse-ordered archives to enable single-pass writing.
-
-
-## Version 2
-
-*Note: this is deprecated in favor of spec version 3.*
-
-PMTiles is a binary serialization format designed for two main access patterns: over the network, via HTTP 1.1 Byte Serving (`Range:` requests), or via memory-mapped files on disk. **All integer values are little-endian.**
-
-A PMTiles archive is composed of:
-* a fixed-size 512,000 byte header section
-* Followed by any number of tiles in arbitrary format
-* Optionally followed by any number of *leaf directories*
-
-### Header
-* The header begins with a 2-byte magic number, "PM"
-* Followed by 2 bytes, the PMTiles specification version (currently 2).
-* Followed by 4 bytes, the length of metadata (M bytes)
-* Followed by 2 bytes, the number of entries in the *root directory* (N entries)
-* Followed by M bytes of metadata, which **must be a JSON string with bounds, minzoom and maxzoom properties (new in v2)**
-* Followed by N * 17 bytes, the root directory.
-
-### Directory structure
-
-A directory is a contiguous sequence of 17 byte entries. A directory can have at most 21,845 entries. **A directory must be sorted by Z, X and then Y order (new in v2).**
-
-An entry consists of:
-* 1 byte: the zoom level (Z) of the entry, with the top bit set to 1 instead of 0 to indicate the offset/length points to a leaf directory and not a tile.
-* 3 bytes: the X (column) of the entry.
-* 3 bytes: the Y (row) of the entry.
-* 6 bytes: the offset of where the tile begins in the archive.
-* 4 bytes: the length of the tile, in bytes.
-
-**All leaf directory entries follow non-leaf entries. All leaf directories in a single directory must have the same Z value. (new in v2).**
-
-### Notes
-* A full directory of 21,845 entries holds exactly a complete pyramid with 8 levels, or 1+4+16+64+256+1024+4096+16384.
-* A PMTiles archive with less than 21,845 tiles should have a root directory and no leaf directories.
-* Multiple tile entries can point to the same offset; this is useful for de-duplicating certain tiles, such as an empty "ocean" tile.
-* Analogously, multiple leaf directory entries can point to the same offset; this can avoid inefficiently-packed small leaf directories.
-* The tentative media type for PMTiles archives is `application/vnd.pmtiles`.
-
-### Implementation suggestions
-
-* PMTiles is designed to make implementing a writer simple. Reserve 512KB, then write all tiles, recording their entry information; then write all leaf directories; finally, rewind to 0 and write the header.
-* The order of tile data in the archive is unspecified; an optimized implementation should arrange tiles on a 2D space-filling curve.
-* PMTiles readers should cache directory entries by byte offset, not by Z/X/Y. This means that deduplicated leaf directories result in cache hits.
\ No newline at end of file
+- File Structure - smaller overhead
+- Unlimited metadata - version 2 had a hard cap on the amount of JSON metadata of about 300 kilobytes Allows tools like tippecanoe to store detailed column statistics. Essential archive information, such as tile type and compression methods, are stored in a binary header separate from application metadata.
+- Hilbert tile IDs - tiles internally are addressed by a single 64-bit Hilbert tile ID instead of Z/X/Y. See the blog post on Tile IDs for details.
+- Archive ordering - An optional clustered mode enforces that tile contents are laid out in Tile ID order.
+- Compressed directories and metadata - Directories used to fetch offsets of tile data consume about 10% the space of those in version 2.
+- JavaScript Compression - The TypeScript pmtiles library now includes a decompressor - fflate - to allow reading compressed vector tile archives directly in the browser. This reduces the size and latency of vector tiles by as much as 70%.
+- Tile Cancellation - All JavaScript plugins now support tile cancellation, meaning quick zooming across many levels will interrupt the loading of tiles that are never shown. This has a significant effect on the perceived user experience, as tiles at the end of a animation will appear earlier.
+- ETag support - clients can detect when files change on static storage by reading the ETag HTTP header. This means that PMTiles-based map applications can update datasets in place at low frequency without running into caching problems.
diff --git a/docs/upgrade-guide.md b/docs/upgrade-guide.md
index 3dcfa274c3..688ec0e999 100644
--- a/docs/upgrade-guide.md
+++ b/docs/upgrade-guide.md
@@ -1,6 +1,13 @@
# Upgrade Guide
-## Upgrading to loaders.gl v4.0
+## Upgrading to v4.1
+
+**@loaders.gl/wkt**
+
+- `WKBLoader`/`TWKBLoader`/`HexWKBLoader` - The default `shape` is now `geojson-geometry` rather than `binary-geometry`. If you were relying on `binary-geometry`, just add add a `shape: 'binary-geometry'` option, as in `load(..., WKBLoader, {wkb: {shape: 'binary-geometry}})`.
+- The `geometry` shape is deprecated, and now called `geojson-geometry`.
+
+## Upgrading to v4.0
**Node.js v18+**
diff --git a/docs/whats-new.mdx b/docs/whats-new.mdx
index 9fedf6a692..bef27771d4 100644
--- a/docs/whats-new.mdx
+++ b/docs/whats-new.mdx
@@ -1,5 +1,20 @@
# What's New
+## v4.1 (In development)
+
+Target Release Date: Early 2024
+
+- Tooling upgrades: Typescript 5.3, vite 5.0, docusauraus 3.0
+
+**@loaders.gl/arrow**
+
+- New [`GeoArrowLoader`](/docs/modules/arrow/api-reference/geoarrow-loader) supports loading [GeoArrow](/docs/modules/arrow/formats/geoarrow) files.
+- New documentation for [Arrow](/docs/modules/arrow/formats/arrow) and [GeoArrow](/docs/modules/arrow/formats/geoarrow) formats.
+
+**@loaders.gl/flatgeobuf**
+
+- Loading flatgeobuf into a `GeoJSONTable` now extracts a Schema object from the flatgeobuf header, and uses it to infer the types of the columns in the table.
+
## v4.0
Release Date: Oct 30, 2023
diff --git a/examples/get-started/bundle-with-nextjs/package.json b/examples/get-started/bundle-with-nextjs/package.json
index f38135f187..757b4a0960 100644
--- a/examples/get-started/bundle-with-nextjs/package.json
+++ b/examples/get-started/bundle-with-nextjs/package.json
@@ -23,6 +23,6 @@
"react": "18.2.0",
"react-dom": "18.2.0",
"sass": "^1.58.0",
- "typescript": "^5.0.4"
+ "typescript": "^5.3.0"
}
}
diff --git a/examples/get-started/bundle-with-rollup/package.json b/examples/get-started/bundle-with-rollup/package.json
index a2187c1721..bf5821b610 100644
--- a/examples/get-started/bundle-with-rollup/package.json
+++ b/examples/get-started/bundle-with-rollup/package.json
@@ -24,6 +24,6 @@
"@rollup/plugin-typescript": "^11.0.0",
"rollup": "^3.17.0",
"serve": "^14.2.0",
- "typescript": "^5.0.4"
+ "typescript": "^5.3.0"
}
}
diff --git a/examples/get-started/bundle-with-vite/package.json b/examples/get-started/bundle-with-vite/package.json
index 45c9af22c4..13d29a2226 100644
--- a/examples/get-started/bundle-with-vite/package.json
+++ b/examples/get-started/bundle-with-vite/package.json
@@ -13,7 +13,7 @@
"@loaders.gl/gltf": "^4.0.0"
},
"devDependencies": {
- "typescript": "^5.0.4",
- "vite": "^4.4.9"
+ "typescript": "^5.3.0",
+ "vite": "^5.0.0"
}
}
diff --git a/examples/get-started/bundle-with-webpack-5/package.json b/examples/get-started/bundle-with-webpack-5/package.json
index bbc9278720..b2cf00f737 100644
--- a/examples/get-started/bundle-with-webpack-5/package.json
+++ b/examples/get-started/bundle-with-webpack-5/package.json
@@ -15,7 +15,7 @@
"devDependencies": {
"html-webpack-plugin": "^5.5.0",
"ts-loader": "^9.4.2",
- "typescript": "^5.0.4",
+ "typescript": "^5.3.0",
"webpack": "^5.75.0",
"webpack-cli": "^5.0.1",
"webpack-dev-server": "^4.11.1"
diff --git a/examples/webpack.config.local.js b/examples/webpack.config.local.js
deleted file mode 100644
index 04b97ed7a3..0000000000
--- a/examples/webpack.config.local.js
+++ /dev/null
@@ -1,275 +0,0 @@
-// This file contains webpack configuration settings that allow
-// examples to be built against the source code in this repo instead
-// of building against their installed version.
-//
-// This enables using the examples to debug the main library source
-// without publishing or npm linking, with conveniences such hot reloading etc.
-const webpack = require('webpack');
-const resolve = require('path').resolve;
-const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
-const {getOcularConfig} = require('ocular-dev-tools');
-const ALIASES = getOcularConfig({root: resolve(__dirname, '..')}).aliases;
-const ROOT_DIR = resolve(__dirname, '..');
-const LERNA_INFO = require(resolve(ROOT_DIR, 'lerna.json'));
-
-const BABEL_CONFIG = {
- presets: ['@babel/env'],
- plugins: [
- [
- 'babel-plugin-inline-import',
- {
- extensions: ['.worker.js']
- }
- ],
- ['@babel/plugin-transform-runtime', {useESModules: true}],
- 'version-inline'
- ],
- ignore: ['**/*.worker.js']
-};
-
-const DECK_LINK_ALIASES = {
- // TODO - add all aliases
- '@deck.gl/core': resolve(ROOT_DIR, '../deck.gl/modules/core/src'),
- '@deck.gl/layers': resolve(ROOT_DIR, '../deck.gl/modules/layers/src'),
- '@deck.gl/mesh-layers': resolve(ROOT_DIR, '../deck.gl/modules/mesh-layers/src'),
- '@deck.gl/geo-layers': resolve(ROOT_DIR, '../deck.gl/modules/geo-layers/src'),
- '@deck.gl/react': resolve(ROOT_DIR, '../deck.gl/modules/react/src')
-};
-
-const MATH_LINK_ALIASES = {
- '@math.gl/core': resolve(ROOT_DIR, '../math.gl/modules/core/src'),
- '@math.gl/culling': resolve(ROOT_DIR, '../math.gl/modules/culling/src'),
- '@math.gl/geospatial': resolve(ROOT_DIR, '../math.gl/modules/geospatial/src')
-};
-
-const LUMA_LINK_ALIASES = {
- '@luma.gl/experimental': resolve(ROOT_DIR, '../luma.gl/modules/experimental/src'),
- '@luma.gl/constants': resolve(ROOT_DIR, '../luma.gl/modules/constants/src'),
- '@luma.gl/core': resolve(ROOT_DIR, '../luma.gl/modules/core/src'),
- '@luma.gl/debug': resolve(ROOT_DIR, '../luma.gl/modules/debug/src'),
- '@luma.gl/engine': resolve(ROOT_DIR, '../luma.gl/modules/engine/src'),
- '@luma.gl/gltools': resolve(ROOT_DIR, '../luma.gl/modules/gltools/src'),
- '@luma.gl/shadertools': resolve(ROOT_DIR, '../luma.gl/modules/shadertools/src'),
- '@luma.gl/webgl': resolve(ROOT_DIR, '../luma.gl/modules/webgl/src')
-};
-
-// Support for hot reloading changes to the library:
-const LOCAL_DEVELOPMENT_CONFIG = {
- mode: 'development',
-
- devtool: 'source-map',
-
- // suppress warnings about bundle size
- devServer: {
- stats: {
- warnings: false
- },
- contentBase: [resolve('.'), resolve('../../'), resolve('../../../')]
- },
-
- // this is required by draco
- node: {
- fs: 'empty'
- },
-
- resolve: {
- extensions: ['.ts', '.tsx', '.js', '.json'],
- // mainFields: ['esnext', 'browser', 'module', 'main'],
- // Imports the library from its src directory in this repo
- alias: Object.assign({}, ALIASES)
- },
-
- module: {
- rules: [
- {
- test: /\.ts$/,
- loader: 'babel-loader',
- exclude: [/node_modules/],
- options: {
- presets: ['@babel/preset-typescript', [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ], '@babel/preset-react']
- }
- },
- {
- test: /\.js$/,
- loader: 'babel-loader',
- exclude: [/node_modules/],
- options: {
- presets: [
- [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ], '@babel/preset-react']
- }
- },
- {
- // Unfortunately, webpack doesn't import library sourcemaps on its own...
- test: /\.js$/,
- use: ['source-map-loader'],
- enforce: 'pre'
- },
- {
- // This is required to handle inline worker!
- test: /worker.*\.js$/,
- exclude: /node_modules/,
- use: [
- {
- loader: 'babel-loader',
- options: BABEL_CONFIG
- }
- ]
- },
- // workers need to be completely transpiled
- {
- // Load worker tests
- test: /\.worker\.js$/,
- use: {
- loader: 'worker-loader'
- }
- }
- ]
- },
-
- plugins: [
- new webpack.DefinePlugin({
- __VERSION__: JSON.stringify(LERNA_INFO.version)
- })
- ]
-};
-
-function addLocalDependency(config, dependency) {
- config.resolve = config.resolve || {};
- config.resolve.alias = config.resolve.alias || {};
-
- switch (dependency) {
- case 'deck':
- Object.assign(config.resolve.alias, DECK_LINK_ALIASES);
- break;
- case 'luma':
- Object.assign(config.resolve.alias, LUMA_LINK_ALIASES);
- break;
- case 'math':
- Object.assign(config.resolve.alias, MATH_LINK_ALIASES);
- break;
- default:
- }
- // We need to have 1 `React` instance when running `yarn start-local-deck`
- Object.assign(config.resolve.alias, {
- react: resolve(ROOT_DIR, 'node_modules/react')
- });
-
- return config;
-}
-
-function addLocalDevSettings(config, opts) {
- // Merge local config with module config
- config = Object.assign({}, LOCAL_DEVELOPMENT_CONFIG, config);
- // Generate initial resolve object.
- config.resolve = config.resolve || {};
- // Generate initial alias object
- config.resolve.alias = config.resolve.alias || {};
- // Merge aliases from config with loacal config aliases
- Object.assign(config.resolve.alias, LOCAL_DEVELOPMENT_CONFIG.resolve.alias);
- // Use extensions from local config
- config.resolve.extensions = LOCAL_DEVELOPMENT_CONFIG.resolve.extensions;
- // Generate initial config mudule
- config.module = config.module || {};
- // Get module config rules
- const configRules = config.module.rules || [];
- // Merge local rules with module config rules
- config.module.rules = LOCAL_DEVELOPMENT_CONFIG.module.rules.concat(configRules);
- // Use initial config plugins
- config.plugins = config.plugins || [];
- // Do concatenation of local and module config plugins
- config.plugins = config.plugins.concat(LOCAL_DEVELOPMENT_CONFIG.plugins);
- // Uncomment to validate generated config
- // console.log(config);
-
- return config;
-}
-
-function addAnalyzerSettings(config) {
- config.mode = 'production';
-
- config.resolve = config.resolve || {};
- // 'esnext' picks up ES6 dist for smaller bundles
-
- config.plugins = config.plugins || [];
- config.plugins.push(new BundleAnalyzerPlugin());
- return config;
-}
-
-// Use non-transpiled vis.gl dependencies and disable regenerator transforms
-function addESNextSettings(config) {
- // Add 'esnext' to make sure vis.gl frameworks are imported with minimal transpilation
- config.resolve = config.resolve || {};
- config.resolve.mainFields = config.resolve.mainFields || ['browser', 'module', 'main'];
- config.resolve.mainFields.shift('esnext');
-
- // Look for babel plugin
- config.module = config.module || {};
- config.module.rules = config.module.rules || [];
- const babelRule = config.module.rules.find((rule) => rule.loader === 'babel-loader');
-
- // If found, inject excludes in @babel/present-env to prevent transpile
- if (babelRule && babelRule.options && babelRule.options.presets) {
- babelRule.options.presets = babelRule.options.presets.map((preset) => {
- if (preset === '@babel/preset-env') {
- return [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ];
- }
- return preset;
- });
- }
- return config;
-}
-
-module.exports =
- (baseConfig, opts = {}) =>
- (env) => {
- let config = baseConfig;
-
- /* eslint-disable no-console */
- if (env && env.help) {
- console.log(
- '--env.esnext: Use non-transpiled vis.gl dependencies and disable regenerator transforms'
- );
- console.log('--env.local: Build against local src for modules in this repo');
- console.log('--env.math,luma,deck: Build against local src for external repos');
- console.log('--env.analyze: Add bundle size analyzer plugin');
- }
-
- console.log('For documentation on build options, run: "yarn start --env.help"');
-
- /* eslint-enable no-console */
- if (env && env.esnext) {
- config = addESNextSettings(config);
- }
-
- if (env && env.local) {
- config = addLocalDevSettings(config, opts);
- }
-
- // Iterate over env keys and see if they match a local dependency
- for (const key in env || {}) {
- config = addLocalDependency(config, key);
- }
-
- if (env && env.analyze) {
- config = addAnalyzerSettings(config);
- }
-
- // uncomment to debug
- // console.warn(JSON.stringify(config, null, 2));
- return config;
- };
diff --git a/examples/website/3d-tiles/package.json b/examples/website/3d-tiles/package.json
index d4bb035d02..5695639c47 100644
--- a/examples/website/3d-tiles/package.json
+++ b/examples/website/3d-tiles/package.json
@@ -31,7 +31,7 @@
"styled-components": "^4.2.0"
},
"devDependencies": {
- "typescript": "^5.0.4",
- "vite": "^4.4.9"
+ "typescript": "^5.3.0",
+ "vite": "^5.0.0"
}
}
diff --git a/examples/website/geospatial/app.tsx b/examples/website/geospatial/app.tsx
index 3f3f3daa21..d390dd92de 100644
--- a/examples/website/geospatial/app.tsx
+++ b/examples/website/geospatial/app.tsx
@@ -5,32 +5,40 @@ import React, {useState, useEffect} from 'react';
import {createRoot} from 'react-dom/client';
import {Map} from 'react-map-gl';
-import maplibregl from 'maplibre-gl';
+import maplibregl, {Properties} from 'maplibre-gl';
import {DeckGL} from '@deck.gl/react/typed';
import {MapController} from '@deck.gl/core/typed';
import {GeoJsonLayer} from '@deck.gl/layers/typed';
import {ControlPanel} from './components/control-panel';
-import {FileUploader} from './components/file-uploader';
+// import {FileUploader} from './components/file-uploader';
import type {Example} from './examples';
import {INITIAL_LOADER_NAME, INITIAL_EXAMPLE_NAME, INITIAL_MAP_STYLE, EXAMPLES} from './examples';
import {Table, GeoJSON} from '@loaders.gl/schema';
import {Loader, load /* registerLoaders */} from '@loaders.gl/core';
-import {ParquetLoader, installBufferPolyfill} from '@loaders.gl/parquet';
+import {GeoArrowLoader} from '@loaders.gl/arrow';
+import {GeoParquetLoader, installBufferPolyfill} from '@loaders.gl/parquet';
import {FlatGeobufLoader} from '@loaders.gl/flatgeobuf';
+import {ShapefileLoader} from '@loaders.gl/shapefile';
+import {KMLLoader, GPXLoader, TCXLoader} from '@loaders.gl/kml';
+
+// GeoPackage depends on sql.js which has bundling issues in docusuarus.
// import {GeoPackageLoader} from '@loaders.gl/geopackage';
-import {ArrowLoader} from '@loaders.gl/arrow';
installBufferPolyfill();
const LOADERS: Loader[] = [
- ArrowLoader,
- ParquetLoader,
- FlatGeobufLoader
+ GeoArrowLoader,
+ GeoParquetLoader,
+ FlatGeobufLoader,
// GeoPackageLoader
+ ShapefileLoader,
+ KMLLoader,
+ GPXLoader,
+ TCXLoader
];
const LOADER_OPTIONS = {
worker: false,
@@ -48,6 +56,18 @@ const LOADER_OPTIONS = {
geopackage: {
shape: 'geojson-table'
// table: 'FEATURESriversds'
+ },
+ shapefile: {
+ shape: 'geojson-table'
+ },
+ kml: {
+ shape: 'geojson-table'
+ },
+ gpx: {
+ shape: 'geojson-table'
+ },
+ tcx: {
+ shape: 'geojson-table'
}
};
@@ -76,15 +96,14 @@ type AppState = {
};
/**
- *
+ * A Geospatial table map viewer
*/
export default function App(props: AppProps) {
-
const [state, setState] = useState({
// EXAMPLE STATE
examples: EXAMPLES,
- selectedExample: INITIAL_EXAMPLE_NAME,
- selectedLoader: INITIAL_LOADER_NAME,
+ selectedExample: null,
+ selectedLoader: null,
// CURRENT VIEW POINT / CAMERA POSITION
viewState: INITIAL_VIEW_STATE,
@@ -96,74 +115,82 @@ export default function App(props: AppProps) {
useEffect(() => {
let examples: Record> = {...EXAMPLES};
if (props.format) {
- // Move the preferred format examples to the "top"
- examples = {[props.format]: EXAMPLES[props.format], ...EXAMPLES};
- // Remove any keys
- for (const key of Object.keys(examples)) {
- if (key.endsWith('Test')) {
- delete examples[key];
- }
- }
+ // Keep only the preferred format examples
+ examples = {[props.format]: EXAMPLES[props.format]};
}
const selectedLoader = props.format || INITIAL_LOADER_NAME;
+ let selectedExample = props.format
+ ? Object.keys(examples[selectedLoader])[0]
+ : INITIAL_EXAMPLE_NAME;
- let selectedExample = INITIAL_EXAMPLE_NAME;
- if (props.format) {
- for (const exampleName of Object.keys(examples[selectedLoader])) {
- selectedExample = exampleName;
- break;
- }
- }
- setState({...state, examples, selectedExample, selectedLoader});
+ onExampleChange({
+ selectedLoader,
+ selectedExample,
+ example: examples[selectedLoader][selectedExample],
+ state,
+ setState
+ });
+ setState((state) => ({...state, examples, selectedExample, selectedLoader}));
}, [props.format]);
+ let schema = state.loadedTable?.schema
+ ? {metadata: state.loadedTable?.schema.metadata, ...state.loadedTable?.schema}
+ : null;
+
return (
+The GeoPackage module uses sql.js which has bundling issues with docusaurus,
+however the `GeoPackageLoader` has been used successfully with vite / esbuild.
+
+
+
\ No newline at end of file
diff --git a/website/src/examples/geospatial/gpx.mdx b/website/src/examples/geospatial/gpx.mdx
new file mode 100644
index 0000000000..ca9023c522
--- /dev/null
+++ b/website/src/examples/geospatial/gpx.mdx
@@ -0,0 +1,7 @@
+# GPX
+
+import Demo from 'examples/website/geospatial/app';
+
+
+
+
diff --git a/website/src/examples/geospatial/kml.mdx b/website/src/examples/geospatial/kml.mdx
new file mode 100644
index 0000000000..8de22f8115
--- /dev/null
+++ b/website/src/examples/geospatial/kml.mdx
@@ -0,0 +1,7 @@
+# KML
+
+import Demo from 'examples/website/geospatial/app';
+
+
+
+
diff --git a/website/src/examples/geospatial/shapefile.mdx b/website/src/examples/geospatial/shapefile.mdx
new file mode 100644
index 0000000000..679eaf10f7
--- /dev/null
+++ b/website/src/examples/geospatial/shapefile.mdx
@@ -0,0 +1,7 @@
+# Shapefile
+
+import Demo from 'examples/website/geospatial/app';
+
+
+
+
diff --git a/website/src/examples/geospatial/tcx.mdx b/website/src/examples/geospatial/tcx.mdx
new file mode 100644
index 0000000000..06b375513b
--- /dev/null
+++ b/website/src/examples/geospatial/tcx.mdx
@@ -0,0 +1,7 @@
+# TCX
+
+import Demo from 'examples/website/geospatial/app';
+
+
+
+
diff --git a/website/src/examples/index.mdx b/website/src/examples/index.mdx
index 53c03965cd..5446a32856 100644
--- a/website/src/examples/index.mdx
+++ b/website/src/examples/index.mdx
@@ -2,4 +2,7 @@
import {ExamplesIndex} from '../components';
+*Notes: These examples use [deck.gl](https://deck.gl) to visualize loaded data and only cover a subset of the available loaders.
+Check the [loader catalog](/docs/modules/arrow/api-reference/arrow-loader) to see if the format you need is available.*
+
`/images/examples/${item.docId || item.label.toLowerCase()}.jpg`} />
diff --git a/website/static/images/examples/flatgeobuf.jpg b/website/static/images/examples/flatgeobuf.jpg
index 7bc34d041e..e52fdd67bf 100644
Binary files a/website/static/images/examples/flatgeobuf.jpg and b/website/static/images/examples/flatgeobuf.jpg differ
diff --git a/website/static/images/examples/geoarrow.jpg b/website/static/images/examples/geoarrow.jpg
new file mode 100644
index 0000000000..205c9c2bc4
Binary files /dev/null and b/website/static/images/examples/geoarrow.jpg differ
diff --git a/website/static/images/examples/geojson.jpg b/website/static/images/examples/geojson.jpg
index ae79906471..bc628d74ab 100644
Binary files a/website/static/images/examples/geojson.jpg and b/website/static/images/examples/geojson.jpg differ
diff --git a/website/static/images/examples/geospatial/gpx.jpg b/website/static/images/examples/geospatial/gpx.jpg
new file mode 100644
index 0000000000..0c90803697
Binary files /dev/null and b/website/static/images/examples/geospatial/gpx.jpg differ
diff --git a/website/static/images/examples/geospatial/kml.jpg b/website/static/images/examples/geospatial/kml.jpg
new file mode 100644
index 0000000000..7f651cc0d8
Binary files /dev/null and b/website/static/images/examples/geospatial/kml.jpg differ
diff --git a/website/static/images/examples/geospatial/shapefile.jpg b/website/static/images/examples/geospatial/shapefile.jpg
new file mode 100644
index 0000000000..c540d21719
Binary files /dev/null and b/website/static/images/examples/geospatial/shapefile.jpg differ
diff --git a/website/static/images/examples/geospatial/tcx.jpg b/website/static/images/examples/geospatial/tcx.jpg
new file mode 100644
index 0000000000..3ed328c5a3
Binary files /dev/null and b/website/static/images/examples/geospatial/tcx.jpg differ
diff --git a/yarn.lock b/yarn.lock
index 095a409763..441147a1b7 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -11384,10 +11384,10 @@ typedarray@^0.0.6:
resolved "https://registry.yarnpkg.com/typedarray/-/typedarray-0.0.6.tgz#867ac74e3864187b1d3d47d996a78ec5c8830777"
integrity sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA==
-typescript@^5.2.2, typescript@~4.6.0:
- version "5.2.2"
- resolved "https://registry.yarnpkg.com/typescript/-/typescript-5.2.2.tgz#5ebb5e5a5b75f085f22bc3f8460fba308310fa78"
- integrity sha512-mI4WrpHsbCIcwT9cF4FZvr80QUeKvsUsUvKDoR+X/7XHQH98xYD8YHZg7ANtz2GtZt/CBq2QJ0thkGJMHfqc1w==
+typescript@^5.3.0, typescript@~4.6.0:
+ version "5.3.2"
+ resolved "https://registry.yarnpkg.com/typescript/-/typescript-5.3.2.tgz#00d1c7c1c46928c5845c1ee8d0cc2791031d4c43"
+ integrity sha512-6l+RyNy7oAHDfxC4FzSJcz9vnjTKxrLpDG5M2Vu4SHRVNg6xzqZp6LYSR9zjqQTu8DU/f5xwxUdADOkbrIX2gQ==
typical@^4.0.0:
version "4.0.0"
+
@@ -6,13 +6,13 @@
+:::caution
+The `GeoPackageLoader` depends on the [`sql.js`](https://github.com/sql-js/sql.js) npm module which has caused issues with certain JavaScript bundlers. It is recommended that you do your own tests before using the `GeoPackageLoader` in your project.
+:::
+
GeoPackage loader
| Loader | Characteristic |
diff --git a/docs/modules/las/README.md b/docs/modules/las/README.md
index e783d40f06..1b1f461c55 100644
--- a/docs/modules/las/README.md
+++ b/docs/modules/las/README.md
@@ -1,6 +1,11 @@
# Overview
-The `@loaders.gl/las` module handles the [LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities) (LAS) or its compressed version (LAZ), a public format for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+The `@loaders.gl/las` module supports the [LASER file format](/docs/modules/las/formats/las) (LAS) and its compressed version (LAZ).
+
+:::caution
+The `@loaders.gl/las` module only supports LAS/lAZ files up to LAS v1.3. It does not support LAS v1.4 files.
+For more detail, see the discussion in [Github Issues](https://github.com/visgl/loaders.gl/issues/591).
+:::
## Installation
diff --git a/docs/modules/las/api-reference/las-loader.md b/docs/modules/las/api-reference/las-loader.md
index 59ed17ee78..557b7f38f5 100644
--- a/docs/modules/las/api-reference/las-loader.md
+++ b/docs/modules/las/api-reference/las-loader.md
@@ -1,5 +1,10 @@
# LASLoader
+:::caution
+The `@loaders.gl/las` module only supports LAS/lAZ files up to LAS v1.3. It does not support LAS v1.4 files.
+For more detail, see the discussion in [Github Issues](https://github.com/visgl/loaders.gl/issues/591).
+:::
+
The `LASLoader` parses a point cloud in the LASER file format.
| Loader | Characteristic |
diff --git a/docs/modules/las/formats/las.md b/docs/modules/las/formats/las.md
index 1347004b3c..eb3376f018 100644
--- a/docs/modules/las/formats/las.md
+++ b/docs/modules/las/formats/las.md
@@ -1,3 +1,47 @@
# LAS / LAZ
-The `@loaders.gl/las` module handles the [LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities) (LAS) or its compressed version (LAZ), a public format for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+- *[`@loaders.gl/las`](/docs/modules/las)*
+- *[Wikipedia](https://en.wikipedia.org/wiki/LAS_file_format)* - *[LAS Spec](https://www.loc.gov/preservation/digital/formats/fdd/fdd000418.shtml)* - *[LASER file format](https://www.asprs.org/divisions-committees/lidar-division/laser-las-file-format-exchange-activities)* - *[LASzip project](https://github.com/LASzip/LASzip)* - *[LAZ spec](https://www.cs.unc.edu/~isenburg/lastools/download/laszip.pdf)*
+
+The *LASER file format* (LAS) and its compressed version (LAZ) are public formats for the interchange of 3-dimensional point cloud data data, developed for LIDAR mapping purposes.
+
+## Variants
+
+LAS file format is not compressed. However there is an open source project (LASzip) which defined and implemented the open file format LAZ to losslessly compress LAS data.
+
+| Variant | Description |
+| ------- | --------------------- |
+| LAS | Uncompressed |
+| LAZ | Lossless compression. |
+
+## Version History
+
+| **Version** | **Date** | **loaders.gl
-The `TileJSONLoader` parses header/metadata from a pmtiles archive
+The `TileJSONLoader` parses metadata from a TileJSON / tilestats file. It merges layer and field information from both tilestats and TileJSON and returns a strongly typed data structure.
+
| Loader | Characteristic |
| --------------------- | -------------------------------------------------- |
@@ -25,7 +26,12 @@ import {load} from '@loaders.gl/core';
const tileJSON = await load(url, TileJSONLoader, options);
```
+## Data Format
+
+See [TileJSON format](/docs/modules/mvt/formats/tilejson.md].
+
## Options
| Option | Type | Default | Description |
| ------ | ---- | ------- | ----------- |
+|
\ No newline at end of file
diff --git a/docs/modules/mvt/formats/mvt.md b/docs/modules/mvt/formats/mvt.md
index 665589668b..cf782b2312 100644
--- a/docs/modules/mvt/formats/mvt.md
+++ b/docs/modules/mvt/formats/mvt.md
@@ -1,12 +1,14 @@
# Mapbox Vector Tile
+- *[`@loaders.gl/mvt`](/docs/modules/mvt)*
- *[Mapbox Vector Tile Specification](https://github.com/mapbox/vector-tile-spec)*
-
A specification for encoding tiled vector data.
MVT is a protobuf-encoded format that defines geospatial geometries.
+tiles contain layers with features, the features can have geometries and properties.
+
## Metadata
It is often useful to have global metadata about a tileset. A common complementary format for encoding tileset metadata is [TileJSON](./tilejson).
diff --git a/docs/modules/mvt/formats/tilejson.md b/docs/modules/mvt/formats/tilejson.md
index 5c29814728..574fba8428 100644
--- a/docs/modules/mvt/formats/tilejson.md
+++ b/docs/modules/mvt/formats/tilejson.md
@@ -1,30 +1,87 @@
# TileJSON / Tilestats
-- *[TileJSON specification](https://github.com/mapbox/tilejson-spec/blob/master/3.0.0/README.md)*
-- *[Tilestats information](https://github.com/mapbox/mapbox-geostats)
-- *[Tilestats generation](https://github.com/mapbox/mapbox-geostats#output-the-stats)
+- *[`@loaders.gl/mvt`](/docs/modules/mvt)*
+- *[TileJSON specification](https://github.com/mapbox/tilejson-spec/blob/master/3.0.0/README.md)* - *[Tilestats information](https://github.com/mapbox/mapbox-geostats) - *[Tilestats generation](https://github.com/mapbox/mapbox-geostats#output-the-stats)
## TileJSON
-Metadata about a tileset.
+A TileJSON file contains JSON-encoded metadata about a vector tileset including which layers and fields (columns) can be found in the tiles.
-for representing metadata about multiple types of web-based map layers, to aid clients in configuration and browsing.
+The information in the TileJSON enables clients (such as web viewers) to understand the structure of data in the tileset up front, instead of piecing it together as tiles are loaded.
-As the name suggests, TileJSON is JSON encoded.
-## Tilestats
+While not all vector tilesets have a TileJSON file, when it is present there is normally a single TileJSON file per tileset, and this file is typically found at the root in TMS or XYZ based URL schemas, so that applications can make a speculative attempt to fetch it from a known place.
-Tilestats is a valuable inofficial "extension" to TileJSON. It provides column statistics, notably:
+## tilestats
+
+`tilestats`` is an inofficial "extension" to TileJSON. It provides column statistics, notably:
- the data type of each column
- min/max values for numeric columns (enabling e.g. global color scale calculations).
- a sample of values for each column
-Tilestats are not always available so applications must be prepared to work in their absence.
-However, tilestats is output by major tilers such as [tippecanoe](https://github.com/mapbox/mapbox-geostats#output-the-stats).
+tilestats provide "global" information about the data in the tileset that can allows for e.g.
+the creation of correct color maps that do not depend on the view (which requires knowing a priori the min and max values of each field / column).
+
+Note that tilestats are not always available for a given tileset, so applications must be prepared to work in their absence.
+However, tilestats are output by major tilers such as [tippecanoe](https://github.com/mapbox/mapbox-geostats#output-the-stats).
+
+## Format Description
+loaders.gl returns a typed `TileJSON` object, with merged tilestats information (if present).
+
+```typescript
+export type TileJSON
+```
## Fields
-| Data | TileJSON | tilestats | Description |
-| --- | --- | --- | --- |
-|
\ No newline at end of file
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ------------------ | -------- | --------- | --------------------------------------------------------------------- |
+| `name?` | `string` | | | Name of the tileset. |
+| `description?` | `string` | | | Short description of the tileset. |
+| `version?` | `string` | | | Version of the tileset. |
+| `tileFormat?` | `string` | | | Format of the tiles in the tileset.. |
+| `tilesetType?` | `string` | | | Type of tileset. |
+| `generator?` | `string` | | | Generating application. (e.g. tippecanoe adds this). |
+| `generatorOptions?` | `string` | | | Generating application options. (e.g. tippecanoe adds this). |
+| `scheme?` | `'xyz'` \| `'tms'` | | | Tile indexing scheme. Typically `tms`, i.e `z/x/y` coordinates. |
+| `tiles?` | `string[]` | | | Sharded URLs (can increased loading speed on HTTP 1 connections) |
+| `boundingBox?` | `[[w, s], [e, n]]` | | | limits of bounding box using axis units and order of specified CRS. |
+| `maxZoom?` | `number` | | | May be set to the maxZoom of the first layer |
+| `minZoom?` | `number` | | | May be set to the minZoom of the first layer |
+| `center?` | `number[]` | | | Center point of the data in the tileset |
+| `htmlAttribution?` | `string` | | | Attribution (can contain HTML syntax) |
+| `htmlLegend?` | `string` | | | Legend (can contain HTML syntax) |
+| `layers?` | `TileJSONLayer[]` | | | Layer information (combines tilestats (if present) and tilejson info) |
+| `metaJson?` | `any` | | | Any unparsed, nested JSON metadata |
+
+- `boundingBox` typing: `[min: [w: number, s: number], max: [e: number, n: number]]`
+
+```ts
+export type TileJSONLayer;
+```
+
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ----------------- | -------- | --------- | ------------------------------------------------------------------------------------ |
+| `name` | `string` | | | The name (id) of this layer (tilejson.vector_layers[].id / tilestats.layers[].layer) |
+| `description?` | `string` | | | The description of this layer (tilejson.layer.description) |
+| `featureCount?` | `number` | | | The number of features in this layer (tilestats.layer.count) |
+| `dominantGeometry?` | `string` | | | The dominant geometry type in this layer (tilestats.layer.geometry) |
+| `minZoom?` | `number` | | | An array of details about the first 100 attributes in this layer |
+| `maxZoom?` | `number` | | | |
+| `fields` | `TileJSONField[]` | | | |
+
+
+```ts
+export type TileJSONField;
+```
+
+| Data | Type | TileJSON | tilestats | Description |
+| ------------------- | ----------- | -------- | --------- | ---------------------------------------------------- |
+| `name` | `string` | | | The name of this attribute |
+| `description?` | `string` | | | |
+| `type` | `string` | | | |
+| `min?` | `number` | | | min value (if there are *any* numbers in the values) |
+| `max?` | `number` | | | max value (if there are *any* numbers in the values) |
+| `uniqueValueCount?` | `number` | | | Number of unique values across the tileset |
+| `values?` | `unknown[]` | | | An array of this attribute's first 100 unique values |
diff --git a/docs/modules/pmtiles/formats/pmtiles.md b/docs/modules/pmtiles/formats/pmtiles.md
index a71c97e939..f1d40c3c97 100644
--- a/docs/modules/pmtiles/formats/pmtiles.md
+++ b/docs/modules/pmtiles/formats/pmtiles.md
@@ -6,159 +6,38 @@ PMTiles is a single-file archive format for tiled data designed to enable indivi
## Overview
-TBA
+PMTiles is a general format for storing tiled data addressed by Z/X/Y coordinates in a single big archive file. This can be used to store cartographic basemap vector tiles, remote sensing observations, JPEG images, or more.
-## Versions
+PMTiles readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.
-## Version 3
+## Tile types
-- File Structure
-97% smaller overhead - Spec version 2 would always issue a 512 kilobyte initial request; version 3 reduces this to 16 kilobytes. What remains the same is that nearly any map tile can be retrieved in at most two additional requests.
-- Unlimited metadata - version 2 had a hard cap on the amount of JSON metadata of about 300 kilobytes; version 3 removes this limit. This is essential for tools like tippecanoe to store detailed column statistics. Essential archive information, such as tile type and compression methods, are stored in a binary header separate from application metadata.
-- Hilbert tile IDs - tiles internally are addressed by a single 64-bit Hilbert tile ID instead of Z/X/Y. See the blog post on Tile IDs for details.
-- Archive ordering - An optional clustered mode enforces that tile contents are laid out in Tile ID order.
-- Compressed directories and metadata - Directories used to fetch offsets of tile data consume about 10% the space of those in version 2. See the blog post on compressed directories for details.
-- JavaScript
-Compression - The TypeScript pmtiles library now includes a decompressor - fflate - to allow reading compressed vector tile archives directly in the browser. This reduces the size and latency of vector tiles by as much as 70%.
-- Tile Cancellation - All JavaScript plugins now support tile cancellation, meaning quick zooming across many levels will interrupt the loading of tiles that are never shown. This has a significant effect on the perceived user experience, as tiles at the end of a animation will appear earlier.
-- ETag support - clients can detect when files change on static storage by reading the ETag HTTP header. This means that PMTiles-based map applications can update datasets in place at low frequency without running into caching problems.
-
-## Version 3 Specification
-
-### File structure
-
-A PMTiles archive is a single-file archive of square tiles with five main sections:
-
-1. A fixed-size, 127-byte **Header** starting with `PMTiles` and then the spec version - currently `3` - that contains offsets to the next sections.
-2. A root **Directory**, described below. The Header and Root combined must be less than 16,384 bytes.
-3. JSON metadata.
-4. Optionally, a section of **Leaf Directories**, encoded the same way as the root.
-5. The tile data.
-
-### Entries
-
-A Directory is a list of `Entries`, in ascending order by `TileId`:
-
- Entry = (TileId uint64, Offset uint64, Length uint32, RunLength uint32)
-
-* `TileId` starts at 0 and corresponds to a cumulative position on the series of square Hilbert curves starting at z=0.
-* `Offset` is the position of the tile in the file relative to the start of the data section.
-* `Length` is the size of the tile in bytes.
-* `RunLength` is how many times this tile is repeated: the `TileId=5,RunLength=2` means that tile is present at IDs 5 and 6.
-* If `RunLength=0`, the offset/length points to a Leaf Directory where `TileId` is the first entry.
-
-### Directory Serialization
-
-Entries are stored in memory as integers, but serialized to disk using these compression steps:
-
-1. A little-endian varint indicating the # of entries
-2. Delta encoding of `TileId`
-3. Zeroing of `Offset`:
- * `0` if it is equal to the `Offset` + `Length` of the previous entry
- * `Offset+1` otherwise
-4. Varint encoding of all numbers
-5. Columnar ordering: all `TileId`s, all `RunLength`s, all `Length`s, then all `Offset`s
-6. Finally, general purpose compression as described by the `Header`'s `InternalCompression` field
-
-##3 Directory Hierarchy
-
-* The number of entries in the root directory and leaf directories is up to the implementation.
-* However, the compressed size of the header plus root directory is required in v3 to be under **16,384 bytes**. This is to allow latency-optimized clients to prefetch the root directory and guarantee it is complete. A sophisticated writer might need several attempts to optimize this.
-* Root size, leaf sizes and depth should be configurable by the user to optimize for different trade-offs: cost, bandwidth, latency.
+PMTiles is a container format and can in principle contain any type of quadtree tiles. A number of vector and image tile types are predefined.
-### Header Design
+| Type | MIME type | pmtiles | Description |
+| ---- | -------------------------------------- | ------- | --------------------------------------------------- |
+| MVT | `'application/vnd.mapbox-vector-tile'` | `1` | [Mapbox Vector Tile](/docs/modules/mvt/formats/mvt) |
+| PNG | `'image/png'` | `2` | |
+| JPEG | `'image/jpeg'` | `3` | |
+| WEBP | `'image/webp'` | `4` | |
+| AVIF | `'image/avif'` | `5` | |
+| - | `'application/octet-stream'` | - | Undefined / Custom types |
-*Certain fields belonging to metadata in v2 are promoted to fixed-size header fields. This allows a map container to be initialized to the desired extent or center without blocking on the JSON metadata, and allows proxies to return well-defined HTTP headers.*
-The `Header` is 127 bytes, with little-endian integer values:
+## Metadata
-| offset | description | width |
-| ------ | ----------------------------------------------------------------------------------------- | ----- |
-| 0 | magic number `PMTiles` | 7 |
-| 7 | spec version, currently `3` | 1 |
-| 8 | offset of root directory | 8 |
-| 16 | length of root directory | 8 |
-| 24 | offset of JSON metadata, possibly compressed by `InternalCompression` | 8 |
-| 32 | length of JSON metadata | 8 |
-| 40 | offset of leaf directories | 8 |
-| 48 | length of leaf directories | 8 |
-| 56 | offset of tile data | 8 |
-| 64 | length of tile data | 8 |
-| 72 | # of addressed tiles, 0 if unknown | 8 |
-| 80 | # of tile entries, 0 if unknown | 8 |
-| 88 | # of tile contents, 0 if unknown | 8 |
-| 96 | boolean clustered flag, `0x1` if true | 1 |
-| 97 | `InternalCompression` enum (0 = Unknown, 1 = None, 2 = Gzip, 3 = Brotli, 4 = Zstd) | 1 |
-| 98 | `TileCompression` enum | 1 |
-| 99 | tile type enum (0 = Unknown/Other, 1 = MVT (PBF Vector Tile), 2 = PNG, 3 = JPEG, 4 = WEBP | 1 |
-| 100 | min zoom | 1 |
-| 101 | max zoom | 1 |
-| 102 | min longitude (signed 32-bit integer: longitude * 10,000,000) | 4 |
-| 106 | min latitude | 4 |
-| 110 | max longitude | 4 |
-| 114 | max latitude | 4 |
-| 118 | center zoom | 1 |
-| 119 | center longitude | 4 |
-| 123 | center latitude | 4 |
+The pmtiles header has a field that can store JSON metadata. This means that for MVT pmtiles, [TileJSON](/docs/modules/mvts/formats/tilejson) is typically available, stored in the PMTiles header metadata field.
-### Notes
-* **# of addressed tiles**: the total number of tiles before run-length encoding, i.e. `Sum(RunLength)` over all entries.
-* **# of tile entries**: the total number of entries across all directories where `RunLength > 0`.
-* **# # of tile contents**: the number of referenced blobs in the tile section, or the unique # of offsets. If the archive is completely deduplicated, this is equal to the # of unique tile contents. If there is no deduplication, this is equal to the number of tile entries above.
-* **boolean clustered flag**: if true, blobs in the data section are ordered by Hilbert `TileId`. When writing with deduplication, this means that offsets are either contiguous with the previous offset+length, or refer to a lesser offset.
-* **compression enum**: Mandatory, tells the client how to decompress contents as well as provide correct `Content-Encoding` headers to browsers.
-* **tile type**: A hint as to the tile contents. Clients and proxies may use this to:
- * Automatically determine a visualization method
- * provide a conventional MIME type `Content-Type` HTTP header
- * Enforce a canonical extension e.g. `.mvt`, `png`, `jpeg`, `.webp` to prevent duplication in caches
+## Version History
-### Organization
+**Version 3**
-In most cases, the archive should be in the order `Header`, Root Directory, JSON Metadata, Leaf Directories, Tile Data. It is possible to relocate sections other than `Header` arbitrarily, but no current writers/readers take advantage of this. A future design may allow for reverse-ordered archives to enable single-pass writing.
-
-
-## Version 2
-
-*Note: this is deprecated in favor of spec version 3.*
-
-PMTiles is a binary serialization format designed for two main access patterns: over the network, via HTTP 1.1 Byte Serving (`Range:` requests), or via memory-mapped files on disk. **All integer values are little-endian.**
-
-A PMTiles archive is composed of:
-* a fixed-size 512,000 byte header section
-* Followed by any number of tiles in arbitrary format
-* Optionally followed by any number of *leaf directories*
-
-### Header
-* The header begins with a 2-byte magic number, "PM"
-* Followed by 2 bytes, the PMTiles specification version (currently 2).
-* Followed by 4 bytes, the length of metadata (M bytes)
-* Followed by 2 bytes, the number of entries in the *root directory* (N entries)
-* Followed by M bytes of metadata, which **must be a JSON string with bounds, minzoom and maxzoom properties (new in v2)**
-* Followed by N * 17 bytes, the root directory.
-
-### Directory structure
-
-A directory is a contiguous sequence of 17 byte entries. A directory can have at most 21,845 entries. **A directory must be sorted by Z, X and then Y order (new in v2).**
-
-An entry consists of:
-* 1 byte: the zoom level (Z) of the entry, with the top bit set to 1 instead of 0 to indicate the offset/length points to a leaf directory and not a tile.
-* 3 bytes: the X (column) of the entry.
-* 3 bytes: the Y (row) of the entry.
-* 6 bytes: the offset of where the tile begins in the archive.
-* 4 bytes: the length of the tile, in bytes.
-
-**All leaf directory entries follow non-leaf entries. All leaf directories in a single directory must have the same Z value. (new in v2).**
-
-### Notes
-* A full directory of 21,845 entries holds exactly a complete pyramid with 8 levels, or 1+4+16+64+256+1024+4096+16384.
-* A PMTiles archive with less than 21,845 tiles should have a root directory and no leaf directories.
-* Multiple tile entries can point to the same offset; this is useful for de-duplicating certain tiles, such as an empty "ocean" tile.
-* Analogously, multiple leaf directory entries can point to the same offset; this can avoid inefficiently-packed small leaf directories.
-* The tentative media type for PMTiles archives is `application/vnd.pmtiles`.
-
-### Implementation suggestions
-
-* PMTiles is designed to make implementing a writer simple. Reserve 512KB, then write all tiles, recording their entry information; then write all leaf directories; finally, rewind to 0 and write the header.
-* The order of tile data in the archive is unspecified; an optimized implementation should arrange tiles on a 2D space-filling curve.
-* PMTiles readers should cache directory entries by byte offset, not by Z/X/Y. This means that deduplicated leaf directories result in cache hits.
\ No newline at end of file
+- File Structure - smaller overhead
+- Unlimited metadata - version 2 had a hard cap on the amount of JSON metadata of about 300 kilobytes Allows tools like tippecanoe to store detailed column statistics. Essential archive information, such as tile type and compression methods, are stored in a binary header separate from application metadata.
+- Hilbert tile IDs - tiles internally are addressed by a single 64-bit Hilbert tile ID instead of Z/X/Y. See the blog post on Tile IDs for details.
+- Archive ordering - An optional clustered mode enforces that tile contents are laid out in Tile ID order.
+- Compressed directories and metadata - Directories used to fetch offsets of tile data consume about 10% the space of those in version 2.
+- JavaScript Compression - The TypeScript pmtiles library now includes a decompressor - fflate - to allow reading compressed vector tile archives directly in the browser. This reduces the size and latency of vector tiles by as much as 70%.
+- Tile Cancellation - All JavaScript plugins now support tile cancellation, meaning quick zooming across many levels will interrupt the loading of tiles that are never shown. This has a significant effect on the perceived user experience, as tiles at the end of a animation will appear earlier.
+- ETag support - clients can detect when files change on static storage by reading the ETag HTTP header. This means that PMTiles-based map applications can update datasets in place at low frequency without running into caching problems.
diff --git a/docs/upgrade-guide.md b/docs/upgrade-guide.md
index 3dcfa274c3..688ec0e999 100644
--- a/docs/upgrade-guide.md
+++ b/docs/upgrade-guide.md
@@ -1,6 +1,13 @@
# Upgrade Guide
-## Upgrading to loaders.gl v4.0
+## Upgrading to v4.1
+
+**@loaders.gl/wkt**
+
+- `WKBLoader`/`TWKBLoader`/`HexWKBLoader` - The default `shape` is now `geojson-geometry` rather than `binary-geometry`. If you were relying on `binary-geometry`, just add add a `shape: 'binary-geometry'` option, as in `load(..., WKBLoader, {wkb: {shape: 'binary-geometry}})`.
+- The `geometry` shape is deprecated, and now called `geojson-geometry`.
+
+## Upgrading to v4.0
**Node.js v18+**
diff --git a/docs/whats-new.mdx b/docs/whats-new.mdx
index 9fedf6a692..bef27771d4 100644
--- a/docs/whats-new.mdx
+++ b/docs/whats-new.mdx
@@ -1,5 +1,20 @@
# What's New
+## v4.1 (In development)
+
+Target Release Date: Early 2024
+
+- Tooling upgrades: Typescript 5.3, vite 5.0, docusauraus 3.0
+
+**@loaders.gl/arrow**
+
+- New [`GeoArrowLoader`](/docs/modules/arrow/api-reference/geoarrow-loader) supports loading [GeoArrow](/docs/modules/arrow/formats/geoarrow) files.
+- New documentation for [Arrow](/docs/modules/arrow/formats/arrow) and [GeoArrow](/docs/modules/arrow/formats/geoarrow) formats.
+
+**@loaders.gl/flatgeobuf**
+
+- Loading flatgeobuf into a `GeoJSONTable` now extracts a Schema object from the flatgeobuf header, and uses it to infer the types of the columns in the table.
+
## v4.0
Release Date: Oct 30, 2023
diff --git a/examples/get-started/bundle-with-nextjs/package.json b/examples/get-started/bundle-with-nextjs/package.json
index f38135f187..757b4a0960 100644
--- a/examples/get-started/bundle-with-nextjs/package.json
+++ b/examples/get-started/bundle-with-nextjs/package.json
@@ -23,6 +23,6 @@
"react": "18.2.0",
"react-dom": "18.2.0",
"sass": "^1.58.0",
- "typescript": "^5.0.4"
+ "typescript": "^5.3.0"
}
}
diff --git a/examples/get-started/bundle-with-rollup/package.json b/examples/get-started/bundle-with-rollup/package.json
index a2187c1721..bf5821b610 100644
--- a/examples/get-started/bundle-with-rollup/package.json
+++ b/examples/get-started/bundle-with-rollup/package.json
@@ -24,6 +24,6 @@
"@rollup/plugin-typescript": "^11.0.0",
"rollup": "^3.17.0",
"serve": "^14.2.0",
- "typescript": "^5.0.4"
+ "typescript": "^5.3.0"
}
}
diff --git a/examples/get-started/bundle-with-vite/package.json b/examples/get-started/bundle-with-vite/package.json
index 45c9af22c4..13d29a2226 100644
--- a/examples/get-started/bundle-with-vite/package.json
+++ b/examples/get-started/bundle-with-vite/package.json
@@ -13,7 +13,7 @@
"@loaders.gl/gltf": "^4.0.0"
},
"devDependencies": {
- "typescript": "^5.0.4",
- "vite": "^4.4.9"
+ "typescript": "^5.3.0",
+ "vite": "^5.0.0"
}
}
diff --git a/examples/get-started/bundle-with-webpack-5/package.json b/examples/get-started/bundle-with-webpack-5/package.json
index bbc9278720..b2cf00f737 100644
--- a/examples/get-started/bundle-with-webpack-5/package.json
+++ b/examples/get-started/bundle-with-webpack-5/package.json
@@ -15,7 +15,7 @@
"devDependencies": {
"html-webpack-plugin": "^5.5.0",
"ts-loader": "^9.4.2",
- "typescript": "^5.0.4",
+ "typescript": "^5.3.0",
"webpack": "^5.75.0",
"webpack-cli": "^5.0.1",
"webpack-dev-server": "^4.11.1"
diff --git a/examples/webpack.config.local.js b/examples/webpack.config.local.js
deleted file mode 100644
index 04b97ed7a3..0000000000
--- a/examples/webpack.config.local.js
+++ /dev/null
@@ -1,275 +0,0 @@
-// This file contains webpack configuration settings that allow
-// examples to be built against the source code in this repo instead
-// of building against their installed version.
-//
-// This enables using the examples to debug the main library source
-// without publishing or npm linking, with conveniences such hot reloading etc.
-const webpack = require('webpack');
-const resolve = require('path').resolve;
-const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
-const {getOcularConfig} = require('ocular-dev-tools');
-const ALIASES = getOcularConfig({root: resolve(__dirname, '..')}).aliases;
-const ROOT_DIR = resolve(__dirname, '..');
-const LERNA_INFO = require(resolve(ROOT_DIR, 'lerna.json'));
-
-const BABEL_CONFIG = {
- presets: ['@babel/env'],
- plugins: [
- [
- 'babel-plugin-inline-import',
- {
- extensions: ['.worker.js']
- }
- ],
- ['@babel/plugin-transform-runtime', {useESModules: true}],
- 'version-inline'
- ],
- ignore: ['**/*.worker.js']
-};
-
-const DECK_LINK_ALIASES = {
- // TODO - add all aliases
- '@deck.gl/core': resolve(ROOT_DIR, '../deck.gl/modules/core/src'),
- '@deck.gl/layers': resolve(ROOT_DIR, '../deck.gl/modules/layers/src'),
- '@deck.gl/mesh-layers': resolve(ROOT_DIR, '../deck.gl/modules/mesh-layers/src'),
- '@deck.gl/geo-layers': resolve(ROOT_DIR, '../deck.gl/modules/geo-layers/src'),
- '@deck.gl/react': resolve(ROOT_DIR, '../deck.gl/modules/react/src')
-};
-
-const MATH_LINK_ALIASES = {
- '@math.gl/core': resolve(ROOT_DIR, '../math.gl/modules/core/src'),
- '@math.gl/culling': resolve(ROOT_DIR, '../math.gl/modules/culling/src'),
- '@math.gl/geospatial': resolve(ROOT_DIR, '../math.gl/modules/geospatial/src')
-};
-
-const LUMA_LINK_ALIASES = {
- '@luma.gl/experimental': resolve(ROOT_DIR, '../luma.gl/modules/experimental/src'),
- '@luma.gl/constants': resolve(ROOT_DIR, '../luma.gl/modules/constants/src'),
- '@luma.gl/core': resolve(ROOT_DIR, '../luma.gl/modules/core/src'),
- '@luma.gl/debug': resolve(ROOT_DIR, '../luma.gl/modules/debug/src'),
- '@luma.gl/engine': resolve(ROOT_DIR, '../luma.gl/modules/engine/src'),
- '@luma.gl/gltools': resolve(ROOT_DIR, '../luma.gl/modules/gltools/src'),
- '@luma.gl/shadertools': resolve(ROOT_DIR, '../luma.gl/modules/shadertools/src'),
- '@luma.gl/webgl': resolve(ROOT_DIR, '../luma.gl/modules/webgl/src')
-};
-
-// Support for hot reloading changes to the library:
-const LOCAL_DEVELOPMENT_CONFIG = {
- mode: 'development',
-
- devtool: 'source-map',
-
- // suppress warnings about bundle size
- devServer: {
- stats: {
- warnings: false
- },
- contentBase: [resolve('.'), resolve('../../'), resolve('../../../')]
- },
-
- // this is required by draco
- node: {
- fs: 'empty'
- },
-
- resolve: {
- extensions: ['.ts', '.tsx', '.js', '.json'],
- // mainFields: ['esnext', 'browser', 'module', 'main'],
- // Imports the library from its src directory in this repo
- alias: Object.assign({}, ALIASES)
- },
-
- module: {
- rules: [
- {
- test: /\.ts$/,
- loader: 'babel-loader',
- exclude: [/node_modules/],
- options: {
- presets: ['@babel/preset-typescript', [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ], '@babel/preset-react']
- }
- },
- {
- test: /\.js$/,
- loader: 'babel-loader',
- exclude: [/node_modules/],
- options: {
- presets: [
- [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ], '@babel/preset-react']
- }
- },
- {
- // Unfortunately, webpack doesn't import library sourcemaps on its own...
- test: /\.js$/,
- use: ['source-map-loader'],
- enforce: 'pre'
- },
- {
- // This is required to handle inline worker!
- test: /worker.*\.js$/,
- exclude: /node_modules/,
- use: [
- {
- loader: 'babel-loader',
- options: BABEL_CONFIG
- }
- ]
- },
- // workers need to be completely transpiled
- {
- // Load worker tests
- test: /\.worker\.js$/,
- use: {
- loader: 'worker-loader'
- }
- }
- ]
- },
-
- plugins: [
- new webpack.DefinePlugin({
- __VERSION__: JSON.stringify(LERNA_INFO.version)
- })
- ]
-};
-
-function addLocalDependency(config, dependency) {
- config.resolve = config.resolve || {};
- config.resolve.alias = config.resolve.alias || {};
-
- switch (dependency) {
- case 'deck':
- Object.assign(config.resolve.alias, DECK_LINK_ALIASES);
- break;
- case 'luma':
- Object.assign(config.resolve.alias, LUMA_LINK_ALIASES);
- break;
- case 'math':
- Object.assign(config.resolve.alias, MATH_LINK_ALIASES);
- break;
- default:
- }
- // We need to have 1 `React` instance when running `yarn start-local-deck`
- Object.assign(config.resolve.alias, {
- react: resolve(ROOT_DIR, 'node_modules/react')
- });
-
- return config;
-}
-
-function addLocalDevSettings(config, opts) {
- // Merge local config with module config
- config = Object.assign({}, LOCAL_DEVELOPMENT_CONFIG, config);
- // Generate initial resolve object.
- config.resolve = config.resolve || {};
- // Generate initial alias object
- config.resolve.alias = config.resolve.alias || {};
- // Merge aliases from config with loacal config aliases
- Object.assign(config.resolve.alias, LOCAL_DEVELOPMENT_CONFIG.resolve.alias);
- // Use extensions from local config
- config.resolve.extensions = LOCAL_DEVELOPMENT_CONFIG.resolve.extensions;
- // Generate initial config mudule
- config.module = config.module || {};
- // Get module config rules
- const configRules = config.module.rules || [];
- // Merge local rules with module config rules
- config.module.rules = LOCAL_DEVELOPMENT_CONFIG.module.rules.concat(configRules);
- // Use initial config plugins
- config.plugins = config.plugins || [];
- // Do concatenation of local and module config plugins
- config.plugins = config.plugins.concat(LOCAL_DEVELOPMENT_CONFIG.plugins);
- // Uncomment to validate generated config
- // console.log(config);
-
- return config;
-}
-
-function addAnalyzerSettings(config) {
- config.mode = 'production';
-
- config.resolve = config.resolve || {};
- // 'esnext' picks up ES6 dist for smaller bundles
-
- config.plugins = config.plugins || [];
- config.plugins.push(new BundleAnalyzerPlugin());
- return config;
-}
-
-// Use non-transpiled vis.gl dependencies and disable regenerator transforms
-function addESNextSettings(config) {
- // Add 'esnext' to make sure vis.gl frameworks are imported with minimal transpilation
- config.resolve = config.resolve || {};
- config.resolve.mainFields = config.resolve.mainFields || ['browser', 'module', 'main'];
- config.resolve.mainFields.shift('esnext');
-
- // Look for babel plugin
- config.module = config.module || {};
- config.module.rules = config.module.rules || [];
- const babelRule = config.module.rules.find((rule) => rule.loader === 'babel-loader');
-
- // If found, inject excludes in @babel/present-env to prevent transpile
- if (babelRule && babelRule.options && babelRule.options.presets) {
- babelRule.options.presets = babelRule.options.presets.map((preset) => {
- if (preset === '@babel/preset-env') {
- return [
- '@babel/preset-env',
- {
- exclude: [/transform-async-to-generator/, /transform-regenerator/]
- }
- ];
- }
- return preset;
- });
- }
- return config;
-}
-
-module.exports =
- (baseConfig, opts = {}) =>
- (env) => {
- let config = baseConfig;
-
- /* eslint-disable no-console */
- if (env && env.help) {
- console.log(
- '--env.esnext: Use non-transpiled vis.gl dependencies and disable regenerator transforms'
- );
- console.log('--env.local: Build against local src for modules in this repo');
- console.log('--env.math,luma,deck: Build against local src for external repos');
- console.log('--env.analyze: Add bundle size analyzer plugin');
- }
-
- console.log('For documentation on build options, run: "yarn start --env.help"');
-
- /* eslint-enable no-console */
- if (env && env.esnext) {
- config = addESNextSettings(config);
- }
-
- if (env && env.local) {
- config = addLocalDevSettings(config, opts);
- }
-
- // Iterate over env keys and see if they match a local dependency
- for (const key in env || {}) {
- config = addLocalDependency(config, key);
- }
-
- if (env && env.analyze) {
- config = addAnalyzerSettings(config);
- }
-
- // uncomment to debug
- // console.warn(JSON.stringify(config, null, 2));
- return config;
- };
diff --git a/examples/website/3d-tiles/package.json b/examples/website/3d-tiles/package.json
index d4bb035d02..5695639c47 100644
--- a/examples/website/3d-tiles/package.json
+++ b/examples/website/3d-tiles/package.json
@@ -31,7 +31,7 @@
"styled-components": "^4.2.0"
},
"devDependencies": {
- "typescript": "^5.0.4",
- "vite": "^4.4.9"
+ "typescript": "^5.3.0",
+ "vite": "^5.0.0"
}
}
diff --git a/examples/website/geospatial/app.tsx b/examples/website/geospatial/app.tsx
index 3f3f3daa21..d390dd92de 100644
--- a/examples/website/geospatial/app.tsx
+++ b/examples/website/geospatial/app.tsx
@@ -5,32 +5,40 @@ import React, {useState, useEffect} from 'react';
import {createRoot} from 'react-dom/client';
import {Map} from 'react-map-gl';
-import maplibregl from 'maplibre-gl';
+import maplibregl, {Properties} from 'maplibre-gl';
import {DeckGL} from '@deck.gl/react/typed';
import {MapController} from '@deck.gl/core/typed';
import {GeoJsonLayer} from '@deck.gl/layers/typed';
import {ControlPanel} from './components/control-panel';
-import {FileUploader} from './components/file-uploader';
+// import {FileUploader} from './components/file-uploader';
import type {Example} from './examples';
import {INITIAL_LOADER_NAME, INITIAL_EXAMPLE_NAME, INITIAL_MAP_STYLE, EXAMPLES} from './examples';
import {Table, GeoJSON} from '@loaders.gl/schema';
import {Loader, load /* registerLoaders */} from '@loaders.gl/core';
-import {ParquetLoader, installBufferPolyfill} from '@loaders.gl/parquet';
+import {GeoArrowLoader} from '@loaders.gl/arrow';
+import {GeoParquetLoader, installBufferPolyfill} from '@loaders.gl/parquet';
import {FlatGeobufLoader} from '@loaders.gl/flatgeobuf';
+import {ShapefileLoader} from '@loaders.gl/shapefile';
+import {KMLLoader, GPXLoader, TCXLoader} from '@loaders.gl/kml';
+
+// GeoPackage depends on sql.js which has bundling issues in docusuarus.
// import {GeoPackageLoader} from '@loaders.gl/geopackage';
-import {ArrowLoader} from '@loaders.gl/arrow';
installBufferPolyfill();
const LOADERS: Loader[] = [
- ArrowLoader,
- ParquetLoader,
- FlatGeobufLoader
+ GeoArrowLoader,
+ GeoParquetLoader,
+ FlatGeobufLoader,
// GeoPackageLoader
+ ShapefileLoader,
+ KMLLoader,
+ GPXLoader,
+ TCXLoader
];
const LOADER_OPTIONS = {
worker: false,
@@ -48,6 +56,18 @@ const LOADER_OPTIONS = {
geopackage: {
shape: 'geojson-table'
// table: 'FEATURESriversds'
+ },
+ shapefile: {
+ shape: 'geojson-table'
+ },
+ kml: {
+ shape: 'geojson-table'
+ },
+ gpx: {
+ shape: 'geojson-table'
+ },
+ tcx: {
+ shape: 'geojson-table'
}
};
@@ -76,15 +96,14 @@ type AppState = {
};
/**
- *
+ * A Geospatial table map viewer
*/
export default function App(props: AppProps) {
-
const [state, setState] = useState