Merge pull request #69 from uqbar-dao/dr/net-api-and-wit-overview

add .wit explainer and net API

src/apis/kinode_wit.md

@@ -1 +1,48 @@
 # `kinode.wit`
+
+Throughout this book, readers will see references to [WIT](https://component-model.bytecodealliance.org/design/wit.html), the [WebAssembly Component Model](https://github.com/WebAssembly/component-model).
+WIT, or Wasm Interface Type, is a language for describing the types and functions that are available to a WebAssembly module.
+In conjunction with the Component Model itself, WIT allows for the creation of WebAssembly modules that can be used as components in a larger system.
+This standard has been under development for many years, and while still under construction, it's the perfect tool for building an operating-system-like environment for apps built in Wasm.
+
+Kinode OS uses WIT to present a standard interface for Kinode processes.
+This interface is a set of types and functions that are available to all processes.
+It also contains functions (well, just a single function: `init()`) that processes must implement in order to compile and run in Kinode OS.
+If one can generate WIT bindings in a language that compiles to Wasm, that language can be used to write Kinode processes.
+So far, we've written Kinode processes in Rust, Javascript, Go, and Python.
+
+To see exactly how to use WIT to write Kinode processes, see the [My First App](../my_first_app/chapter_1.md) chapter or the [Chess Tutorial](../chess_app/chess_engine.md).
+
+To see `kinode.wit` for itself, see the [file in the GitHub repo](https://github.com/uqbar-dao/kinode-wit/blob/master/kinode.wit).
+Since this interface applies to all processes, it's one of the places in the OS where breaking changes are most likely to make an impact.
+To that end, the version of the WIT file that a process uses must be compatible with the version of Kinode OS that it's running on.
+We intend to achieve perfect backwards compatibility upon first major release (1.0.0) of the OS and the WIT file.
+After that point, since processes signal the version of the WIT file they use, subsequent updates can be made without breaking existing processes needing to change the version they use.
+
+## Types
+
+[These 14 types](https://github.com/uqbar-dao/kinode-wit/blob/373542a9a94ae61a7d216159f9f7bdf9266cd935/kinode.wit#L8-L103) make up the entirety of the shared type system between processes and the kernel.
+Most types presented here are implemented in the [process standard library](../process_stdlib/overview.md) for ease of use.
+
+## Functions
+
+[These 15 functions](https://github.com/uqbar-dao/kinode-wit/blob/373542a9a94ae61a7d216159f9f7bdf9266cd935/kinode.wit#L105-L184) are available to processes.
+They are implemented in the kernel.
+Again, the process standard library makes it such that these functions often don't need to be directly called in processes, but they are always available.
+The functions are generally separated into 4 categories: system utilities, process management, capabilities management, and message I/O.
+Future versions of the WIT file will certainly add more functions, but the categories themselves are highly unlikely to change.
+
+System utilities are functions like `print_to_terminal`, whose role is to provide a way for processes to interact with the runtime in an idiosyncratic way.
+
+Process management functions are used to adjust a processes' state in the kernel.
+This includes its state-store and its on-exit behavior.
+This category is also responsible for functions that give processes the ability to spawn and manage child processes.
+
+Capabilities management functions relate to the capabilities-based security system imposed by the kernel on processes.
+Processes must acquire and manage capabilities in order to perform tasks external to themselves, such as messaging another process or writing to a file.
+See the [capabilities overview](../process-capabilities.md) for more details.
+
+Lastly, message I/O functions are used to send and receive messages between processes.
+Message-passing is the primary means by which processes communicate not only with themselves, but also with runtime modules which expose all kinds of I/O abilities.
+For example, handling an HTTP request involves sending and receiving messages to and from the `http_server:disto:sys` runtime module.
+Interacting with this module and others occurs through message I/O.

src/apis/net.md

@@ -1,3 +1,68 @@
 # Net API
 
-TODO Ben
+Most processes will not use this API directly.
+Instead, processes will make use of the networking protocol simply by sending messages to processes running on other nodes.
+This API is documented, rather, for those who wish to implement their own networking protocol.
+
+The networking API is implemented in the `net:distro:sys` process.
+
+For the specific networking protocol, see the [networking protocol](../networking_protocol.md) chapter.
+This chapter is rather to describe the message-based API that the `net:distro:sys` process exposes.
+
+`Net`, like all processes and runtime modules, is architected around a main message-receiving loop.
+The received `Request`s are handled in one of three ways:
+
+- If the `target.node` is "our domain", i.e. the domain name of the local node, and the `source.node` is also our domain, the message is parsed and treated as either a debugging command or one of the `NetActions` enum.
+
+- If the `target.node` is our domain, but the `source.node` is not, the message is either parsed as the `NetActions` enum, or if it fails to parse, is treated as a "hello" message and printed in the terminal, size permitting. This "hello" protocol simply attempts to display the `message.body` as a UTF-8 string and is mostly used for network debugging.
+
+- If the `source.node` is our domain, but the `target.node` is not, the message is sent to the target using the [networking protocol](../networking_protocol.md) implementation.
+
+Let's look at `NetActions`. Note that this message type can be received from remote or local processes.
+Different implementations of the networking protocol may reject actions depending on whether they were instigated locally or remotely, and also discriminate on what remote node sent the action.
+This is, for example, where a router would choose whether or not to perform routing for a specific node<>node connection.
+
+```rust
+enum NetActions {
+    /// Received from a router of ours when they have a new pending passthrough for us.
+    /// We should respond (if we desire) by using them to initialize a routed connection
+    /// with the NodeId in the string given.
+    ConnectionRequest(String),
+    /// can only receive from trusted source, for now just ourselves locally,
+    /// in the future could get from remote provider
+    KnsUpdate(KnsUpdate),
+    KnsBatchUpdate(Vec<KnsUpdate>),
+}
+
+struct KnsUpdate {
+    pub name: String, // actual username / domain name
+    pub owner: String,
+    pub node: String, // hex namehash of node
+    pub public_key: String,
+    pub ip: String,
+    pub port: u16,
+    pub routers: Vec<String>,
+}
+```
+
+This type must be parsed from a request body using MessagePack.
+`ConnectionRequest` is sent by remote nodes as part of the WebSockets networking protocol in order to ask a router to connect them to a node that they can't connect to directly.
+This is responded to with either an `Accepted` or `Rejected` variant of `NetResponses`.
+
+`KnsUpdate` and `KnsBatchUpdate` both are used as entry point by which the `net` module becomes aware of the Kinode PKI, or KNS.
+In the current distro these are only accepted from the local node, and specifically the `kns_indexer` distro package.
+
+
+Finally, let's look at the type parsed from a `Response`.
+
+```rust
+/// For now, only sent in response to a ConnectionRequest.
+enum NetResponses {
+    Accepted(NodeId),
+    Rejected(NodeId),
+}
+```
+
+This type must be also be parsed using MessagePack, this time from responses received by `net`.
+
+In the future, `NetActions` and `NetResponses` may both expand to cover message types required for implementing networking protocols other than the WebSockets one.

src/networking_protocol.md

@@ -118,6 +118,8 @@ The `source` is the initiator's node ID, as provided onchain.
 The `signature` must be created by the initiator's networking public key. The content is the routing target's node ID (i.e., the node which the initiator would like to establish an e2e encrypted connection with) concatenated with the router's node ID (i.e., the node which the initiator is sending the `RoutingRequest` to, which will serve as a router for the connection if it accepts).
 The `target` is the routing target's node ID that must be signed above.
 
+[TODO document the rejection/acceptance of RoutingRequests]
+
 Once a connection is established, the initiator sends an `e` message, containing an empty payload.
 
 The target responds with the `e, ee, s, es` pattern, including a `HandshakePayload` serialized with MessagePack.
@@ -169,6 +171,9 @@ These behaviors are necessary since they indicate that the networking informatio
 
 Connections may be closed due to inactivity or load-balancing. This behavior is implementation-specific.
 
+[TODO document the management of passthrough connections held open by routers]
+
+[TODO document the optionality of exposing IP vs. using a router, regardless of other node's status]
 
 ### 4. Connection Maintenance and Errors