Merge pull request #127 from kinode-dao/develop

develop: 0.6

src/SUMMARY.md

@@ -14,11 +14,13 @@
   - [Networking Protocol](./networking_protocol.md)
   - [Public Key Infrastructure](./pki.md)
   - [HTTP Server & Client](./http_server_and_client.md)
+  - [Read+Write to Chain](./read_and_write_to_chain.md)
   - [Files](./files.md)
   - [Databases](./databases.md)
   - [Terminal](./terminal.md)
 - [Process Standard Library](./process_stdlib/overview.md)
 - [Kit: Development Tool**kit**](./kit-dev-toolkit.md)
+  - [Install](./kit/install.md)
   - [`boot-fake-node`](./kit/boot-fake-node.md)
   - [`new`](./kit/new.md)
   - [`build`](./kit/build.md)
@@ -34,6 +36,7 @@
   - [Frontend Time](./my_first_app/chapter_4.md)
   - [Sharing with the World](./my_first_app/chapter_5.md)
 - [In-Depth Guide: Chess App](./chess_app.md)
+  - [Environment Setup](./chess_app/setup.md)
   - [Chess Engine](./chess_app/chess_engine.md)
   - [Adding a Frontend](./chess_app/frontend.md)
   - [Putting Everything Together](./chess_app/putting_everything_together.md)
@@ -44,7 +47,10 @@
   - [Simple File Transfer Guide](./cookbook/file_transfer.md)
   - [Intro to Web UI with File Transfer](./cookbook/file_transfer_ui.md)
   - [Writing and Running Scripts](./cookbook/writing_scripts.md)
+  - [Reading Data from ETH](./cookbook/reading_data_from_eth.md)
+  - [Use ZK proofs with SP1](./cookbook/zk_with_sp1.md)
 - [API Reference](./api_reference.md)
+  - [ETH Provider API](./apis/eth_provider.md)
   - [HTTP API](./apis/http_authentication.md)
   - [HTTP Client API](./apis/http_client.md)
   - [HTTP Server API](./apis/http_server.md)

src/apis/eth_provider.md

@@ -0,0 +1,198 @@
+# ETH Provider API
+
+**Note: Most processes will not use this API directly. Instead, they will use the `eth` portion of the[`process_lib`](../process_stdlib/overview.md) library, which papers over this API and provides a set of types and functions which are much easier to natively use.
+This is mostly useful for re-implementing this module in a different client or performing niche actions unsupported by the library.**
+
+Processes can send two kinds of requests to `eth:distro:sys`: `EthAction` and `EthConfigAction`.
+The former only requires the capability to message the process, while the latter requires the root capability issued by `eth:distro:sys`.
+Most processes will only need to send `EthAction` requests.
+
+```rust
+/// The Action and Request type that can be made to eth:distro:sys. Any process with messaging
+/// capabilities can send this action to the eth provider.
+///
+/// Will be serialized and deserialized using `serde_json::to_vec` and `serde_json::from_slice`.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub enum EthAction {
+    /// Subscribe to logs with a custom filter. ID is to be used to unsubscribe.
+    /// Logs come in as alloy_rpc_types::pubsub::SubscriptionResults
+    SubscribeLogs {
+        sub_id: u64,
+        chain_id: u64,
+        kind: SubscriptionKind,
+        params: Params,
+    },
+    /// Kill a SubscribeLogs subscription of a given ID, to stop getting updates.
+    UnsubscribeLogs(u64),
+    /// Raw request. Used by kinode_process_lib.
+    Request {
+        chain_id: u64,
+        method: String,
+        params: serde_json::Value,
+    },
+}
+```
+
+The `Request` containing this action should always expect a response, since every action variant triggers one and relies on it to be useful.
+The ETH provider will respond with the following type:
+
+```rust
+/// The Response type which a process will get from requesting with an [`EthAction`] will be
+/// of this type, serialized and deserialized using `serde_json::to_vec`
+/// and `serde_json::from_slice`.
+///
+/// In the case of an [`EthAction::SubscribeLogs`] request, the response will indicate if
+/// the subscription was successfully created or not.
+#[derive(Debug, Serialize, Deserialize)]
+pub enum EthResponse {
+    Ok,
+    Response { value: serde_json::Value },
+    Err(EthError),
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub enum EthError {
+    /// provider module cannot parse message
+    MalformedRequest,
+    /// No RPC provider for the chain
+    NoRpcForChain,
+    /// Subscription closed
+    SubscriptionClosed(u64),
+    /// Invalid method
+    InvalidMethod(String),
+    /// Invalid parameters
+    InvalidParams,
+    /// Permission denied
+    PermissionDenied,
+    /// RPC timed out
+    RpcTimeout,
+    /// RPC gave garbage back
+    RpcMalformedResponse,
+}
+```
+
+The `EthAction::SubscribeLogs` request will receive a response of `EthResponse::Ok` if the subscription was successfully created, or `EthResponse::Err(EthError)` if it was not.
+Then, after the subscription is successfully created, the process will receive *Requests* from `eth:distro:sys` containing subscription updates.
+That request will look like this:
+
+```rust
+/// Incoming `Request` containing subscription updates or errors that processes will receive.
+/// Can deserialize all incoming requests from eth:distro:sys to this type.
+///
+/// Will be serialized and deserialized using `serde_json::to_vec` and `serde_json::from_slice`.
+pub type EthSubResult = Result<EthSub, EthSubError>;
+
+/// Incoming type for successful subscription updates.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct EthSub {
+    pub id: u64,
+    pub result: SubscriptionResult,
+}
+
+/// If your subscription is closed unexpectedly, you will receive this.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct EthSubError {
+    pub id: u64,
+    pub error: String,
+}
+```
+
+Again, for most processes, this is the entire API.
+The `eth` portion of the `process_lib` library will handle the serialization and deserialization of these types and provide a set of functions and types that are much easier to use.
+
+### Config API
+
+If a process has the `root` capability from `eth:distro:sys`, it can send `EthConfigAction` requests.
+These actions are used to adjust the underlying providers and relays used by the module, and its settings regarding acting as a relayer for other nodes (public/private/granular etc).
+
+```rust
+/// The action type used for configuring eth:distro:sys. Only processes which have the "root"
+/// capability from eth:distro:sys can successfully send this action.
+///
+/// NOTE: changes to config will not be persisted between boots, they must be saved in .env
+/// to be reflected between boots. TODO: can change this
+#[derive(Debug, Serialize, Deserialize)]
+pub enum EthConfigAction {
+    /// Add a new provider to the list of providers.
+    AddProvider(ProviderConfig),
+    /// Remove a provider from the list of providers.
+    /// The tuple is (chain_id, node_id/rpc_url).
+    RemoveProvider((u64, String)),
+    /// make our provider public
+    SetPublic,
+    /// make our provider not-public
+    SetPrivate,
+    /// add node to whitelist on a provider
+    AllowNode(String),
+    /// remove node from whitelist on a provider
+    UnallowNode(String),
+    /// add node to blacklist on a provider
+    DenyNode(String),
+    /// remove node from blacklist on a provider
+    UndenyNode(String),
+    /// Set the list of providers to a new list.
+    /// Replaces all existing saved provider configs.
+    SetProviders(SavedConfigs),
+    /// Get the list of current providers as a [`SavedConfigs`] object.
+    GetProviders,
+    /// Get the current access settings.
+    GetAccessSettings,
+    /// Get the state of calls and subscriptions. Used for debugging.
+    GetState,
+}
+
+pub type SavedConfigs = Vec<ProviderConfig>;
+
+/// Provider config. Can currently be a node or a ws provider instance.
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct ProviderConfig {
+    pub chain_id: u64,
+    pub trusted: bool,
+    pub provider: NodeOrRpcUrl,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub enum NodeOrRpcUrl {
+    Node {
+        kns_update: crate::core::KnsUpdate,
+        use_as_provider: bool, // for routers inside saved config
+    },
+    RpcUrl(String),
+}
+```
+
+`EthConfigAction` requests should always expect a response. The response body will look like this:
+```rust
+/// Response type from an [`EthConfigAction`] request.
+#[derive(Debug, Serialize, Deserialize)]
+pub enum EthConfigResponse {
+    Ok,
+    /// Response from a GetProviders request.
+    /// Note the [`crate::core::KnsUpdate`] will only have the correct `name` field.
+    /// The rest of the Update is not saved in this module.
+    Providers(SavedConfigs),
+    /// Response from a GetAccessSettings request.
+    AccessSettings(AccessSettings),
+    /// Permission denied due to missing capability
+    PermissionDenied,
+    /// Response from a GetState request
+    State {
+        active_subscriptions: HashMap<crate::core::Address, HashMap<u64, Option<String>>>, // None if local, Some(node_provider_name) if remote
+        outstanding_requests: HashSet<u64>,
+    },
+}
+
+/// Settings for our ETH provider
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct AccessSettings {
+    pub public: bool,           // whether or not other nodes can access through us
+    pub allow: HashSet<String>, // whitelist for access (only used if public == false)
+    pub deny: HashSet<String>,  // blacklist for access (always used)
+}
+```
+
+A successful `GetProviders` request will receive a response of `EthConfigResponse::Providers(SavedConfigs)`, and a successful `GetAccessSettings` request will receive a response of `EthConfigResponse::AccessSettings(AccessSettings)`.
+The other requests will receive a response of `EthConfigResponse::Ok` if they were successful, or `EthConfigResponse::PermissionDenied` if they were not.
+
+All of these types are serialized to a JSON string via `serde_json` and stored as bytes in the request/response body.
+[The source code for this API can be found in the `eth` section of the Kinode runtime library.](https://github.com/kinode-dao/kinode/blob/main/lib/src/eth.rs)

src/apis/http_authentication.md

@@ -25,15 +25,18 @@ The `lazy_load_blob` is the HTTP request body, and the `body` is an `IncomingHtt
 
 ```rs
 pub struct IncomingHttpRequest {
-    pub source_socket_addr: Option<String>, // will parse to SocketAddr
-    pub method: String,                     // will parse to http::Method
-    pub raw_path: String,
+    pub source_socket_addr: Option<String>,   // will parse to SocketAddr
+    pub method: String,                       // will parse to http::Method
+    pub url: String,                          // will parse to url::Url
+    pub bound_path: String,                   // the path that was originally bound
     pub headers: HashMap<String, String>,
+    pub url_params: HashMap<String, String>, // comes from route-recognizer
     pub query_params: HashMap<String, String>,
 }
 ```
 
-Note that `raw_path` is the host and full path of the original HTTP request that came in.
+Note that `url` is the host and full path of the original HTTP request that came in.
+`bound_path` is the matching path that was originally bound in `http_server`.
 
 ## Handling HTTP Requests
 
@@ -61,9 +64,9 @@ fn handle_http_server_request(
 
         // IMPORTANT BIT:
 
-        HttpServerRequest::Http(IncomingHttpRequest { method, raw_path, .. }) => {
+        HttpServerRequest::Http(IncomingHttpRequest { method, url, .. }) => {
             // Check the path
-            if raw_path.ends_with(&format!("{}{}", our.process.to_string(), "/messages")) {
+            if url.ends_with(&format!("{}{}", our.process.to_string(), "/messages")) {
                 // Match on the HTTP method
                 match method.as_str() {
                     // Get all messages

src/apis/http_server.md

@@ -44,6 +44,8 @@ pub enum HttpServerAction {
         /// lazy_load_blob bytes and serve them as the response to any request to this path.
         cache: bool,
     },
+    /// Unbind a previously-bound HTTP path
+    Unbind { path: String },
     /// Bind a path to receive incoming WebSocket connections.
     /// Doesn't need a cache since does not serve assets.
     WebSocketBind {
@@ -61,6 +63,8 @@ pub enum HttpServerAction {
         encrypted: bool,
         extension: bool,
     },
+    /// Unbind a previously-bound WebSocket path
+    WebSocketUnbind { path: String },
     /// Processes will RECEIVE this kind of request when a client connects to them.
     /// If a process does not want this websocket open, they should issue a *request*
     /// containing a [`type@HttpServerAction::WebSocketClose`] message and this channel ID.
@@ -140,6 +144,9 @@ If a process uses `Bind` or `SecureBind`, that process will need to field future
 
 If a process uses `WebSocketBind` or `WebSocketSecureBind`, future WebSocket connections to that path will be sent to the process, which is expected to issue a response that can then be sent to the client.
 
+Bindings can be removed using `Unbind` and `WebSocketUnbind` actions.
+Note that the HTTP server module will persist bindings until the node itself is restarted (and no later), so unbinding paths is usually not necessary unless cleaning up an old static resource.
+
 The incoming request, whether the binding is for HTTP or WebSocket, will look like this:
 ```rust
 /// HTTP Request type that can be shared over WASM boundary to apps.
@@ -168,10 +175,12 @@ pub enum HttpServerRequest {
 
 #[derive(Debug, Serialize, Deserialize)]
 pub struct IncomingHttpRequest {
-    pub source_socket_addr: Option<String>, // will parse to SocketAddr
-    pub method: String,                     // will parse to http::Method
-    pub raw_path: String,
+    pub source_socket_addr: Option<String>,   // will parse to SocketAddr
+    pub method: String,                       // will parse to http::Method
+    pub url: String,                          // will parse to url::Url
+    pub bound_path: String,                   // the path that was originally bound
     pub headers: HashMap<String, String>,
+    pub url_params: HashMap<String, String>, // comes from route-recognizer
     pub query_params: HashMap<String, String>,
     // BODY is stored in the lazy_load_blob, as bytes
 }

src/apis/websocket_authentication.md

@@ -97,7 +97,7 @@ fn handle_http_server_request(
         HttpServerRequest::WebSocketClose(_channel_id) => {
           channel_ids.remove(channel_id);
         }
-        HttpServerRequest::Http(IncomingHttpRequest { method, raw_path, .. }) => {
+        HttpServerRequest::Http(IncomingHttpRequest { method, url, bound_path, .. }) => {
             // Handle incoming HTTP requests here
         }
     };

src/chess_app.md

@@ -1,14 +1,4 @@
 # In-Depth Guide: Chess App
 
 This guide will walk you through building a very simple chess app on Kinode OS.
-The final result will look like [this](https://github.com/kinode-dao/kinode/tree/main/modules/chess): chess is in the basic runtime distribution so you can try it yourself.
-
-To prepare for this tutorial, follow the environment setup guide [here](../my_first_app/chapter_1.md), i.e. [start a fake node](../my_first_app/chapter_1.md#booting-a-fake-kinode-node) and then, in another terminal, run:
-```bash
-kit new my_chess
-cd my_chess
-kit build
-kit start-package -p 8080
-```
-
-Once you have the template app installed and can see it running on your testing node, continue...
+The final result will look like [this](https://github.com/kinode-dao/kinode/tree/main/kinode/packages/chess): chess is in the basic runtime distribution so you can try it yourself.

src/chess_app/chess_engine.md

@@ -25,7 +25,7 @@ wit_bindgen::generate!({
 call_init!(init);
 
 fn init(our: Address) {
-    println!("{our}: start");
+    println!("started");
 
     loop {
         let _ = await_message().map(|message| {
@@ -236,10 +236,7 @@ call_init!(init);
 
 fn init(our: Address) {
     // A little printout to show in terminal that the process has started.
-    println!(
-        "{} by {}: start",
-        our.process.process_name, our.process.publisher_node
-    );
+    println!("started");
 
     // Grab our state, then enter the main event loop.
     let mut state: ChessState = load_chess_state();
@@ -254,12 +251,12 @@ fn main_loop(our: &Address, state: &mut ChessState) {
         // this and surface it to the user.
         match await_message() {
             Err(send_error) => {
-                println!("{our}: got network error: {send_error:?}");
+                println!("got network error: {send_error:?}");
                 continue;
             }
             Ok(message) => match handle_request(&our, &message, state) {
                 Ok(()) => continue,
-                Err(e) => println!("{our}: error handling request: {:?}", e),
+                Err(e) => println!("error handling request: {:?}", e),
             },
         }
     }
@@ -312,7 +309,7 @@ fn handle_chess_request(
     state: &mut ChessState,
     action: &ChessRequest,
 ) -> anyhow::Result<()> {
-    println!("chess: handling action from {source_node}: {action:?}");
+    println!("handling action from {source_node}: {action:?}");
 
     // For simplicity's sake, we'll just use the node we're playing with as the game id.
     // This limits us to one active game per partner.
@@ -323,7 +320,7 @@ fn handle_chess_request(
             // Make a new game with source.node
             // This will replace any existing game with source.node!
             if state.games.contains_key(game_id) {
-                println!("chess: resetting game with {game_id} on their request!");
+                println!("resetting game with {game_id} on their request!");
             }
             let game = Game {
                 id: game_id.to_string(),

src/chess_app/frontend.md

@@ -65,7 +65,7 @@ In `my_chess/src/lib.rs`, inside `handle_request()`:
                 match handle_http_request(our, state, incoming) {
                     Ok(()) => Ok(()),
                     Err(e) => {
-                        println!("chess: error handling http request: {:?}", e);
+                        println!("error handling http request: {:?}", e);
                         http::send_response(
                             http::StatusCode::SERVICE_UNAVAILABLE,
                             None,