p2p2

p2p Routing

• Each node has hard-coded list of at least one but most likely up to 5 bootstrap nodes. One bootstrap node must be available for new nodes to join the network.
• When a node starts up, it should use khd routing protocol to add itself to the bootstrap node.
• This will return a list of nodes that are 'close' (under xor arithmetic) to the node. These are effectively random nodes as node ids are random.
• A node should most likely only keep up to n=5 online neighbors from the close peers list at any time and have additional peers on stand-by in case it fails to connect to one of the neighbors.
• Similar ideas are found in libp2p and eth peer routing algorithms.
• As the bootstrap node will add the new node to its routing table, other nodes will try to connect to the new node as it will be close to them at random.
• So joining the network requires only 1 call searchPeer(self) and 1 online bootstrap node.

p2p Networking requirements

• Node Identity - Node id is its public key. There is really no need to derive a token/id from a public key in a way that requires another at least one network roundtrip to find the node's public key. The public key is already unique and public and so is perfectly suited to be used as the id.
• Encryption - end-to-end with forward-security (sym enc keys negotiated per session between nodes)
• Authentication - Receiver node should be able to auth sender node id
• Node Discovery - DHT: random discovery of neighbors for gossip protocol. Node should be able to dial any online node and connect to it.
• Bootstrapping - hard-coded nodes-set - start neighbors walk from this list.
• Base Protocol and multi-protocols - Support app-level protocols framed w base protocol.
• Protocols Multiplexing - Support multi-protocol messages over the same virtual connection.
• Protocols versioning - Support negotiation of protocol version between nodes.
• Gossip Protocol - Allow nodes to gossip over the base protocol

Wire protocol (serialization)

We are going with protobufs as wire format for all data instead of having to write our own wire format (such as eth hand-rolled RLP format). There's great Go support for this format.

p2p protocol support - design

• We have built our own p2p stack above golang tcp/ip stack.

P2P v2.0 design

Spec evolution based on r&d into libp2p, go net libs, and friends.

p2p network architecture

• Local Node
  • Implements p2p protocols: Can process responses to req by ided callbacks.
    • ping protocol
    • echo protocol
    • all app-level other protocols...
• DeMuxer - routes remote requests (and responses) back to protocols. Protocols register on demuxer.
• Swarm forward incoming protocol messages to muxer. Let protocol send message to a remote node. Connects to random nodes.
  • Manages all remote nodes, sessions and connections
  • Handles handshake / session management using the handshake protocol
  • Remote Node - type used internally by swarm to manage sessions and connections
  • RemoteNodeData - the type visible outside of swarm for identifying nodes
• NetworkSession
• Connection
  • msgio (prefix length encoding). shared buffers - used internally for all comm.
• Network
  • tcp for now
  • utp / upd soon

Node IDs and Keys

• Node (and account) ids should be their own public keys - greatly simplify design and tests - no need for key hash. A secp256k1 key is 32 bytes (like a 256bits hash anyhow.)
• Keys system is secp256k1 as it includes encrypt/decrypt
• We might switch to ed25519 based on crypto team review
• We use base58 encoding for string rep of keys (human readable)
• In proto messages keys/ids can be []byte - no need for string encoding /decoding as we are using a binary p2p format - greatly simplifies everything.

smp2p - The SpaceMesh p2p protocol

To libp2p or not to libp2p?

• go-libp2p is not in a good shape: can't use utp, node ids must be multi-hashes (not pub keys), there are serious open issues, kad/dht seems problematic as well.
• go-libp2p should be viewed as a collection of p2p go utils that we can pick and chose from as needed.
• Some of their design decisions are not great and are derived from the ipfs product requirements.

Transport

• tcp for now, udp or utp later

• All

Wire message format

We are using our own simple length-prefix binary format:

<32 bits big-endian data-length><message binary data (protobufs-bin-encoded)>

• We need to use length prefixed protobufs messages becuase protobufs data doesn't include length and we need this to allow multiple messages on the same tcp/ip connection . see p2p2.conn.go.

Network stack design

• 100% locks, mutexes and sync.* free - only using go channel concurrency patterns
• As little external deps as possible but don't rewrite enc code - bad idea.

USwarm - Core P2P Server

• Server should be modular to support different transport protocols (mainly tcp, udp)
• Responsible for establishing sessions with neighbors on startup (connecting to peers and getting neighbors)
• Gets wire-format messages from the transport and decodes them to higher-level protobuf messages and routes them to receivers
• Maintains connections with other peers (for reuse)
• Maintain session info for sessions with other peers. Session data includes an ephemeral session key used for encrypting / decrypting p2p messages.
• Handles sessions disconnections and can create a session with a peer
• Encapsulate session and implements the session protocol
• Supports gossip protocol - sends a message to all neighbors on behalf of clients
• Can reuse non-expiring session ids when connecting to a new node (after disconnection)

Unmarshaled top-level message format:

Session data:

data {
    sessionId: []byte      // if there's an established session between the peers, empty or missing otherwise
    payload: []byte        // encrypted bin data (binary protobuf) with sessoin key
    ... possible other fields ...
}

• Note that message only includes encrypted data and session id (random screen per session) and doesn't leak any other data

• payload is protocol-specific. e.g. session protocol or an app-level protocol.

• for app-level protocol it includes message-author data (might be diff than receiver)

• The p2p server is responsible to decrypt the payload using the local node private key or a session sym key for the session.

• The only connection with incoming data that doesn't have session id is a session protocol message

• All other messages will be rejected

Handshake protocol (session protocol)

• Format: Encrypted protobufs binary data with the destination public key (id)
• Used to establish ephemeral session key between peers
• Used by the core p2p server to establish sessions with remote nodes

Higher-level protocols

app-level protocol payload format:

payload {
    clientVersion  // sender client version
    sendTimestamp  // message sent time
    nonce          // used for mac
    mac            // auth that message data sent by claimed sender (sign of all params verifable via sender id/public key)
    gossip bool    // sender requests this message to be gossiped to neighbors
    data: []bytes  // message data []byte protobufs marshaled
}

data {
    clientVersion // author client version
    authorPubKey  // message writter pub key
    timestamp     // authored time
    nonce         // for mac
    mac           // all data bin sign by author
    protocol []string // protocol and message e.g.  ping/req/1.0
    reqId         // responses include req id for req callback on client
    /* ...protocol-specific fields go here... */
}

• Message is encrypted with session key
• Ony non-encrypted part as the session id (server knows which remote node is part of session)
• Message content is protobufs
• Message content include message author authenticated data (may be diff than sender)
• Protocol handlers register callbacks with the core server
• Server authenticate payload was sent by claimed sender
• Server authenticate data was created by claimed author
• Server calls the handler based on protocol / method message data for authenticated messages.
• Server may discard messages with a send timestamp that is too far aparat from local time.

Basic flows

• An app-level p2p protocol wants to send a message to a node and process response.

• Protocol sends message to the core server

• The core server sends the message if it has a session with the remote peer or tries to establish a new session with a peer.

• The response is called back on protocol impl callback.

• App-level protocol can query the server for a list of active sessions or neighbor peers

• Appl-level protocol may send a gossip message to all neighbors using the core server

Node discovery protocol (higher level kad/dht)

• Nodes discovery is just another app-level protocol - no need to special case it.
• We will implement go-kadh/dht for node discovery - critical piece of the p2p stack.

Protocols discovery protocol

• Allows a node to query all the remote node implement protocols and their versions and decide which protocols to use with the remote node.
• Versioning is on the req processing level to allow updating req/resp for specific methods - no need for a global version for protocol
• Returned data includes:
  • ping/req/v1
  • ping/req/v2
  • hello/req/v1
  • node-discovery/req/v1 Based on this the node knows which req the remote nodes can process and which protocols - in this example, ping and hello.

Session protocol

• Special protocol - used to establish an encrypted session between nodes.