IPIP-378: Delegated Routing HTTP POST API

src/ipips/ipip-0378.md

@@ -0,0 +1,111 @@
+---
+title: "IPIP-0378: Delegated Routing HTTP POST API"
+date: 2024-02-05
+ipip: draft
+editors:
+  - name: Masih H. Derkani
+    github: masih
+  - name: Marcin Rataj
+    github: lidel
+    url: https://lidel.org/
+relatedIssues:
+  - https://github.com/ipfs/specs/pull/378
+order: 389
+tags: ['ipips']
+---
+
+## Summary
+
+This IPIP extends the [IPIP-337 HTTP Delegated Routing API](https://specs.ipfs.tech/ipips/ipip-0337/) to support announcement of content and peer provider records over `POST` requests.
+
+The work here was originally proposed as part of IPIP-337, and eventually was separated into its own IPIP in order to reduce the scope of original work, while enabling iterative release of the HTTP delegated routing APIs.
+
+## Motivation
+
+The IPFS interaction with DHT includes both read and write operations.
+A user can provide records, advertising the presence of content, as well as looking up providers for a given CID.
+
+The specification proposed by
+[IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337) offers an idiomatic
+first-class support for offloading the lookup portion of this interaction onto
+other processes and/or servers.
+
+Following the same motivations that inspired IPIP-337, this document expands the HTTP APIs to also
+offload the ability to provide records noto a third-party system.
+
+## Detailed design
+
+HTTP POST support is added
+to the [Delegated Content Routing HTTP API](https://specs.ipfs.tech/routing/http-routing-v1/)
+at `/routing/v1/providers` and `/routing/v1/peers`, along with complimentary
+sections that outline known formats followed by example payload.
+
+## Design rationale
+
+The rationale for the design of `POST` operations closely follows the reasoning
+listed in [IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337/).
+The design uses a human-readable request/response structure with extensibility in mind.
+The specification imposes no restrictions on the schema nor the protocol advertised in provider records.
+The hope is that such extensibility will encourage and inspire innovation for better transfer protocols.
+
+In order to reduce barrier for adoption, standard HTTP POST semantics are used,
+along with transport-agnostic signature of DAG-CBOR payload representation
+similar to the data field in IPNS. This ensures this IPIP does not introduce
+any new signature dependency because IPNS records are already supported on
+`/routing/v1` since [IPIP-379](https://specs.ipfs.tech/ipips/ipip-0379/).
+
+### User benefit
+
+Expanding the user benefits listed as part of
+[IPIP-337](https://specs.ipfs.tech/ipips/ipip-0337/#user-benefit), in the
+context of content routing write operations are typically more expensive than
+read operations. They involve book keeping such as TTL, gossip propagation,
+etc. Therefore, it is highly desirable to reduce the burden of advertising
+provider records onto the network by means of delegation through simple to use
+HTTP APIs.
+
+### Compatibility
+
+#### Backwards Compatibility
+
+##### DHT
+
+Records published through HTTP delegated routing will not benefit from implicit
+PeerID identity and libp2p handshake, and must be explicitly signed.
+
+At the time of writing of this IPIP the Amino DHT does not have a concept of
+explicit signature. That is why the `POST` APIs proposed here introduce a new
+DAG-CBOR data format for specifying provider records, and mechanism for signing
+them.
+
+##### Reframe
+
+The Reframe API was superseded by `/routing/v1`.
+See [IPIP-337/Backwards Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#backwards-compatibility).
+
+#### Forwards Compatibility
+
+[IPIP-337/Forwads Compatibility](https://specs.ipfs.tech/ipips/ipip-0337/#forwards-compatibility) still applies.
+
+The signature format introduced in this IPIP is not tied to HTTP semantics, and
+similar to IPNS records, could be propagated by various means, including adding
+support for delegated announcement to Amino DHT, where client sends
+announcement to HTTP API, and then API backend takes care of sending it
+to 20 peers.
+
+### Security
+
+This IPIP reuses semantics of IPNS Record signatures but with a different prefix, to avoid signature reuse attacks.
+
+ID, creation date and TTL facilitate signed mechanism for expiration.
+
+[IPIP-337/Security](https://specs.ipfs.tech/ipips/ipip-0337/#security) still applies.
+
+### Alternatives
+
+- Reframe (general-purpose RPC) was evaluated, see "Design rationale" section for rationale why it was not selected.
+- HTTP `PUT` was evaluated, see [rationale](https://github.com/ipfs/specs/pull/378#discussion_r1476257372) why `POST` was used instead.
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

src/routing/http-routing-v1.md

@@ -4,7 +4,7 @@ description: >
   Delegated routing is a mechanism for IPFS implementations to use for offloading
   content routing, peer routing and naming to another process/server. This specification describes
   an HTTP API for delegated routing of content, peers, and IPNS.
-date: 2023-08-31
+date: 2024-02-05
 maturity: reliable
 editors:
   - name: Gus Eggert
@@ -61,17 +61,17 @@ This API uses a standard version prefix in the path, such as `/v1/...`. If a bac
 
 ### `GET /routing/v1/providers/{cid}`
 
-#### Path Parameters
+#### `GET` Path Parameters
 
 - `cid` is the [CID](https://github.com/multiformats/cid) to fetch provider records for.
 
-#### Response Status Codes
+#### `GET` Response Status Codes
 
 - `200` (OK): the response body contains 0 or more records.
 - `404` (Not Found): must be returned if no matching records are found.
 - `422` (Unprocessable Entity): request does not conform to schema or semantic constraints.
 
-#### Response Body
+#### `GET` Response Body
 
 ```json
 {
@@ -93,6 +93,54 @@ The client SHOULD be able to make a request with `Accept: application/x-ndjson`
 
 Each object in the `Providers` list is a record conforming to a schema, usually the [Peer Schema](#peer-schema).
 
+### `POST /routing/v1/providers`
+
+#### `POST` Request Body
+
+```json
+{
+  "Providers": [
+    {
+      "Schema": "announcement",
+      ...
+    }
+  ]
+}
+```
+
+Each object in the `Providers` list is a *write provider record* entry.
+
+Server SHOULD accept  representing writes is [Announcement Schema](#announcement-schema).
+
+:::warn
+
+Since non-streaming results have to be buffered before sending,
+server SHOULD be no more than 100 `Providers` per `application/json` response.
+
+:::
+
+#### `POST` Response Status Codes
+
+- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records)
+- `400` (Bad Request): the server deems the request to be invalid and cannot process it
+- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints
+- `501` (Not Implemented): the server does not support providing records
+
+#### `POST` Response Body
+
+  ```json
+  {
+      "ProvideResults": [
+          { ... }
+      ]
+  }
+  ```
+
+- `ProvideResults` is a list of results in the same order as the `Providers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object
+  - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted.
+  - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed.
+- The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request
+
 ## Peer Routing API
 
 ### `GET /routing/v1/peers/{peer-id}`
@@ -114,10 +162,10 @@ represented as a CIDv1 encoded with `libp2p-key` codec.
 {
   "Peers": [
     {
-      "Schema": "<schema>",
-      "Protocols": ["<protocol-a>", "<protocol-b>", ...],
+      "Schema": "peer",
       "ID": "bafz...",
       "Addrs": ["/ip4/..."],
+      "Protocols": ["<protocol-a>", "<protocol-b>", ...],
       ...
     },
     ...
@@ -131,6 +179,52 @@ The client SHOULD be able to make a request with `Accept: application/x-ndjson`
 
 Each object in the `Peers` list is a record conforming to the [Peer Schema](#peer-schema).
 
+### `POST /routing/v1/peers`
+
+#### `POST` Request Body
+
+```json
+{
+  "Peers": [
+    {
+      "Schema": "announcement",
+      ...
+    }
+  ]
+}
+```
+
+Each object in the `Peers` list is a *write peer record* entry.
+
+Server SHOULD accept writes represented with [Announcement Schema](#announcement-schema).
+
+#### `POST` Response Status Codes
+
+- `200` (OK): the server processed the full list of provider records (possibly unsuccessfully, depending on the semantics of the particular records)
+- `400` (Bad Request): the server deems the request to be invalid and cannot process it
+- `422` (Unprocessable Entity): request does not conform to schema or semantic constraints
+- `501` (Not Implemented): the server does not support providing records
+
+#### `POST` Response Body
+
+  ```json
+  {
+      "PeersResults": [
+          { ... }
+      ]
+  }
+  ```
+
+- `PeersResults` is a list of results in the same order as the `Peers` in the request, and the schema of each object is determined by the `Schema` of the corresponding write object:
+  - Returned list MAY contain entry-specific information such as server-specific TTL, per-entry error message, etc. Fields which are not relevant, can be omitted.
+  - In error scenarios, a client can check for presence of non-empty `Error` field (top level, or per `ProvideResults` entry) to learn about the reason why POST failed.
+- The work for processing each provider record should be idempotent so that it can be retried without excessive cost in the case of full or partial failure of the request
+
+#### `POST` Response Status Codes
+
+- `200` (OK): processed - inspect response to see if there are any `Error` results.
+- `400` (Bad Request): unable to process POST request, make sure JSON schema and values are correct.
+
 ## IPNS API
 
 ### `GET /routing/v1/ipns/{name}`
@@ -225,7 +319,7 @@ limits, allowing every site to query the API for results:
 
 ```plaintext
 Access-Control-Allow-Origin: *
-Access-Control-Allow-Methods: GET, OPTIONS
+Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS
 ```
 
 ## Known Schemas
@@ -277,6 +371,98 @@ the case, the field MUST be ignored.
 
 :::
 
+### Announcement Schema
+
+The `announcement` schema can be used in `POST` operations to announce content providers or peer routing information.
+
+```json
+  {
+    "Schema": "announcement",
+    "Payload": {
+      "CID": "bafy..cid",
+      "Scope": "block",
+      "Timestamp": "YYYY-MM-DDT23:59:59Z",
+      "TTL": 0,
+      "ID": "12D3K...",
+      "Addrs": ["/ip4/...", ...],
+      "Protocols": ["foo", ...],
+      "Metadata": "mbase64-blob",
+    },
+    "Signature": "mbase64-signature"
+  }
+```
+
+#### Announcement Payload
+
+- `Payload`: is a map object with a subset of the below fields.
+  - `CID` is a string with multibase-encoded CID being provided (`/routing/v1/providers` only).
+    - This field is not present when used for `POST /routing/v1/peers`
+  - `Scope` (optional) is a string hint that provides semantic meaning about CID (`/routing/v1/providers` only):
+    - `block` announces only the individual block (this is the implicit default if `Scope` field is not present).
+    - `entity` announces CIDs required for enumerating entity behind the CID (e.g.: all blocks for UnixFS file or a minimum set of blocks to enumerate contents of HAMT-sharded UnixFS directory, only top level of directory tree, etc).
+    - `recursive` announces entire DAGs behind the CIDs (e.g.: entire DAG-CBOR DAG, or everything in UnixFS directory, including all files in all subdirectories).
+
+  - `Timestamp` is the current time, formatted as an ASCII string that follows notation from [rfc3339](https://specs.ipfs.tech/ipns/ipns-record/#ref-rfc3339).
+
+  - `TTL` is caching and expiration hint informing the server how long to keep the record available, specified as integer in milliseconds.
+    - If this value is unknown, the caller may skip this field or set it to 0. The server's default will be used.
+
+  - `ID` is a multibase-encoded Peer ID of the node that provides the content and also indicates the `libp2p-key` that SHOULD be used for verifying `Signature` field.
+    - ED25519 and other small public keys MUST be inlined inside of the `ID` field
+      with the identity multihash type.
+    - Key types that exceed 42 bytes (e.g. RSA) SHOULD NOT be inlined, the `ID`
+      field should only include the multihash of the key. The key itself SHOULD be
+      obtained out-of-band (e.g. by fetching the block via IPFS) and cached to
+      reduce the size of the signed `Payload`.
+
+      If support for big keys is needed in
+      the future, this spec can be updated to allow the client to provide the key
+      and key type out-of-band by adding optional `PublicKey` fields, and if the
+      Peer ID is a CID, then the server can verify the public key's authenticity
+      against the CID, and then proceed with the rest of the verification scheme.
+
+  - `Addrs` (optional) is an a list of string-encoded multiaddrs without `/p2p/peerID` suffix.
+
+  - `Protocols` (optional) is a list of strings with protocols supported by `ID` and/or `Addrs`, if known upfront.
+
+  - `Metadata` (optional) is a string with multibase-encoded binary metadata that should be passed as-is
+
+#### Announcement Signature
+
+- `Signature` is a string with multibase-encoded binary signature that provides integrity and authenticity of the `Payload` field.
+
+  - Signature is created by following below steps:
+    1. Convert `Payload` JSON to deterministic, ordered [DAG-CBOR](https://ipld.io/specs/codecs/dag-cbor/spec/) map notation
+       - Specification intention here is to use similar signature normalization as with DAG-CBOR `Data` field in IPNS Records, allowing for partial code and dependency reuse.
+    2. Prefix the DAG-CBOR bytes with ASCII string `routing-record:`
+    3. Sign the bytes with the private key of the Peer ID specified in the `Payload.ID`.
+       - Signing details for specific key types should follow [libp2p/peerid specs](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#key-types), unless stated otherwise.
+
+  - Client SHOULD sign every announcement.
+  - Servers SHOULD verify signature before accepting a record, unless running in a trusted environment.
+  - A [400 Bad Request](https://httpwg.org/specs/rfc9110.html#status.400)  response code SHOULD be returned if (in order):
+    - `Payload` serialized to DAG-CBOR is bigger than 2MiB
+    - `Signature` is not valid
+
+### Announcement Response Schema
+
+The `announcement-response` schema can be used as `POST` responses when announcing content providers or peer routing information. This schema allows the server to return additional TTL information if the TTL is not provided in the request, or if the server policy is to provide TTL different than the requested one.
+
+```json
+{
+  "Schema": "announcement-response",
+  "Error": "error in case there was error",
+  "TTL": 17280000
+}
+```
+
+- `Error` is a string representing the error that might have happened when announcing.
+
+- `TTL` in response is the time at which the server expects itself to drop the record
+  - If less than the `TTL` in the request, then the client SHOULD repeat announcement earlier, before the announcement TTL expires and is forgotten by the routing system
+  - If greater than the `TTL` in the request, then the server client SHOULD save resources and not repeat announcement until the announcement TTL expires and is forgotten by the routing system
+  - If `0`, the server makes no claims about the lifetime of the record
+
 ### Legacy Schemas
 
 Legacy schemas include `ID` and optional `Addrs` list just like