1.3 Introduction CYFS for Developers: Core components and functional modules

[streetycat]

Core components and functional modules

I will introduce the distribution of project source code files, and then introduce the main core modules one by one.

Directory specification

The CYFS project directory is expanded as follows:

/--CYFS
    |--doc  // formal design documents
    |--scripts // some scripts for build/publish...
    |--src // the source code, We will specifically introduce the source code directory
    |--... // other files for solution: .gitignore, .eslintrc, package.json, .etc

All modules are divided into several categories and collected into corresponding directories in src:

1. 3rd
   The main code comes from an external project, but with a few custom modifications.

2. component
   Basic functional modules, most of which belong to this type, each implements its own functions, is referenced by other runnable instances, and has interdependence internally.

   If your module will be depended on by other modules, it should be collect in component. If part of your module will be depended on, you should split the part into a separate module.

   As examples:
   cyfs-base: It's the most basic module,Almost all modules depend on it.
   cyfs-core: It defines all normalized objects
   cyfs-util: Some auxiliary modules

   I will introduce them in other issues.

3. meta
   The modules for MetaChain, it's just a conceptual version?
   I haven't dabbled in this part yet, I will introduce it in future.

4. service
   Services in CYFS system, They run as independent processes on various CYFS devices.

   I will introduce them in other issues.

5. tests

   Some test cases and sample programs. We can help to supplement and improve, it's a good entrance to enter the CYFS world, and contribute to the cause of Web3.

6. tools

   Some tools to help users/developers to use the CYFS. We can design more useful tools.

7. misc

   Others.

Services

1. Devices

As introduced here, there are 2 types of devices in CYFS for users and developers. They are OOD(gateway) and runtime.The relationship is as follow:

Zone A			<-->	Zone Other
runtime A1	<-->	OOD(gateway) A		OOD(gateway) O	<-->	runtime O1
runtime A2	<-->				<-->	runtime O2
runtime ...	<-->				<-->	runtime ...
runtime An	<-->

2. Services

In a zone, several services is running on the OOD always. These services are running depend on the gateway, the gateway is also a service, It is equivalent to nginx for http, receive and forward the requests to the target services.

gateway	<-->	app-manager
	<-->	chunk-manager
	<-->	file-manager
	<-->	...

• app-manager: It manages the DecApps running on CYFS, install,uninstall,update,start,restart,stop,deamon.
• chunk-manager: The chunk-manager binary is the first experimental NDN implementation (completed in April 2020), which is quite different from the current NDN implementation. Currently this component is only used for Service and DecApp package download.
• file-manager: Similar to chunk-manager, it is the first experimental NON implementation (completed in March 2020), and is currently only used for Service and DecApp package downloads.
• ood-deamon: It will restart the gateway when the process is dead. It is an independent service that does not depend on gateway.
• ood-control: It is not a standalone binary service and is used by programs like ood-daemon or ood-installer. The purpose is to complete stack activation-related functions via the http interface.

3. DECApp

By the way, the server of a DECApp is also running as the services on OOD. They listen to requests from clients through the gateway and respond.

Thanks @weiqiushi for the correction

cyfs-base

This is the most basic component, and almost all other components depend on it.
It implement the follow functions:

1. The basic types or structures:

• AccessString: Structures for ACL.It refers to the design of file system permission in linux.It divides users into 6 groups:

  • CurrentDevice
  • CurrentZone
  • FriendZone
  • OthersZone
  • OwnerDec
  • OthersDec

  These groups can be combined arbitrarily to get the target group that the user wants to configure permissions.

  And we can configure 3 permissions:

  • Call
  • Write
  • Read

• Base36: Transform the BLOB with A human-readable string ignoring case. The string will be used case-insensitively, .eg:

  • Address Bar in browser
  • FileSystem of windows

• channel: This name is used in many occasions. Here it defines the operating environment of the CYFS system. Currently, the CYFS system has 3 operating environments:

  • nightly: The integrated environment for developers.
  • beta: The public test environment for users.
  • stable: The formal version release environment.
    Different environments are independent of each other, they are connected to different MetaChains and SNs, and cannot be interconnected.

• endpoint: Encapsulation of IpAddr, It's used to connect into the network.

• BuckyError: Define the Results for CYFS.

• Name: As the domain for http, we can register a readable nick name for any object.

• paths: The req-path(similar as the URI for http) or the path in RootState

• ports: The listening ports when the CYFS running.

• protocol_fields: The HTTP header fields, HTTP is used at RPC between the stack(runtime/gateway) and the other services or DECApp.

• time: Defines various standard timestamps, and the methods to convert between them

2. The framework for encode/decode:

• RawEncode/RawDecode: Encode with binary.
• Protobuf: Encode with protobuf
• JSON: Encode with JSON

3. Encapsulation of the crypto library:

• AES
• Hash: Only for SHA256?
• KeyPair(PrivateKey/PublicKey): RSA1024/2048/3072, and SECP256K1.
• Signature/Verify: Signature with the KeyPair

4. The 14 standard objects:

• Action: It seems a empty object, Maybe the design draft may not be finalized yet.
• AppGroup: It seems a empty object, Maybe the design draft may not be finalized yet.
• Chunk: The piece in any binary data. but there is only the Hash of the content is included in the Object system without the content itself.
• Device: The object describing a device,it includes all the parameters used to connect each other:
  _ Endpoint
  _ SN
  _ PN
  _ Protocol version * And we can give it a readable name.
  All the fields is mutable, so, they are saved in Body.The Device will be publish on the MetaChain, Therefore,we should query the latest version from MetaChain when we realized that it was no longer effective.
• Dir: The object describing a directory in filesystem.It includes some files in different sub-path, the file(even chunk) list will be included in the desc if the size is little enough, otherwise the content(file/chunk list) will be included in the body or in another file.If the content is not saved in desc, there should be a hash to verify the authenticity of the content.
• File: The object describing a file in filesystem.It includes 3 fields:
  • File length, immutable, saved in desc
  • File hash, immutable, saved in desc
  • The contained chunks, immutable, saved in body, So,we must verify the file hash instead of the chunkids.
• Diff: The delta between 2 file, it's usefull to describe different versions of the same file.
• Group: It's used to describe an organization,The main fields is the administrators and members, they are all mutable,we should update it through the MetaChain or other synchronization methods.There is a issue for it.
• People: Describing a natural person, There are only 3 fields currently:
  • name
  • icon
  • ood_list
    All the fields is mutable,and the new version will be pushed to the MetaChain.
• Relation: It seems a empty object, Maybe the design draft may not be finalized yet.
• Tx: It is the concept of tx in the blockchain, describing a transaction itself, the main field includes:
  • caller
  • message: transaction details
  • gas
• UnionAccount: Joint account of both parties(Provider and Consumer) to the transaction.They transfer the initial coin and set their respective shares,As the contract progresses, the share is revised.All the updates must be signed by both, and executed on the MetaChain.Both parties are free to withdraw their share of coins.The UionAccount object only describes the account itself, not the current state of the account. It has 3 fields only:
  • left: party to the transaction
  • right: another party to the transaction
  • service type: type of the transation
    How to update/query the state of the account?Maybe we can find it on the MetaChain
• Contract: Describe the commitment of both parties to the transaction (price, quality, etc.).
• ProofOfService: Describe the confirmation of the final transaction quantity and quality.Provider and consumer must sign on the contract and receipt.
  When there is a dispute in the transaction, one party presents the ProofOfService signed by the other party and submits it to MetaChain for arbitration, and the transaction can be completed according to the contract.
  The design of the specific service contract is complicated and different from each other, so I need to elaborate on it in another document.By the way,The contract seems a temporary design.
• ObjectMap: The container for objects, The ObjectId is calculate with all items, it will change when any item is updated, and we can regard the two versions before and after modification as two different objects.The ObjectMap can express three structures:
  • Map<Key, ObjectId>
  • Set
  • Diff: The delta between 2 ObjectMap, it can be stored in a file.

5. Other structures for object

• Area: Every object should belong to a different Area, and accept the legal supervision of the corresponding Area. The Area field is set when the object is generate, and is encoded in header 40 bits of the ObjectId.
• NamedObject: The basic trait for any Object.
• TypelessObject: It's a shell for any Object.It collects the common fields,and it includes the object content in the form of buffer.
• AnyNamedObject: It's a abstract shell for any Object. It can be generate from any object, and we can decode it to get the original object.It is often used with TypelessObject.
• NDNObject: It's a shell for all unstructured data(Dir,File,Diff)
• ObjectLink: **I don't understand its exact design intent.**Maybe it's a reference to another Object.
• ObjectSigns: It describes the signature information of an object, and the current signature independently signs Desc and Body respectively.I think there will be some risk,You are welcome to join the discussion.
• UniqueId: Binary with a length of 16 bytes, We can fill the ObjectDesc with the random UniqueId when we need to distinguish the object with same fields in ObjectDesc:
  • No fields in ObjectDesc
  • The fields in ObjectDesc is same, but the ObjectBody will be update in different time.They are 2 different objects.

6. cyfs-base-derive

Some usefull macro is implement in this crate:

• RawEncode
• RawDecode
• ProtobufEncode
• ProtobufDecode

7. cyfs-base-meta

The basic structures for MetaChain.

cyfs-core

I mentioned 4 object categories in previous part. The Core object is collected in the crate cyfs-core.Many object types have been collected, most of those support the functions of the current CYFS system.

1. common

• Text

  It's a common Object to describe a String.You can save a short(<=31Bytes) string in the ObjectId immediately building with the ObjectIdDataBuilder.Otherwise, you can use the Text object.

  It contains 3 fields:

  • id: immutable
  • header: immutable
  • value mutable
    I can't tell the design intent from the field name, maybe it need to rename or redesign?
    You can flexibly use these three fields according to your needs, for example:
  • Make a mutable Text,fill the id with key, and file the value with the content, header for other immutable annotation.
  • Make a immutable simple Text,fill the id with the content, header for immutable annotation, and value for mutable annotation.
  • Make a immutable large(>=64KBytes,Actually less for the header of Desc) Text, fill the value with the content, and fill the id with the hash of content to verify the value, header for immutable annotation.
  • Other usage

• Storage

  It's a common Object to describe a binary buffer(Vec),it can describe both the immutable and mutable buffer.
  It contains 3 fields:

  • id: immutable,set by the creator.
  • hash: immutable,the hash of the buffer,set None if the buffer is mutable to avoid the ObjectId is changed when the buffer is changed.
  • value: mutable, the content of the buffer.

2. app

   It supports the running of the system service AppManager,Various related entity objects and behavior objects are defined here.

• DecApp: Describe App Details, the main fields as follow:
  • id: immutable, UniqueId?
  • source: mutable, what?
  • icon: mutable
  • desc: mutable, the description for the App
  • source_desc: mutable, what?
  • tags: tags for the App
• AppCmd: Describe the control commands for APP, it has 2 fields:
  • app id
  • command code:
    • Add: add the app to the app list.
    • Remove: remove the app from the app list
    • Install: install the app in my zone
    • Uninstall: uninstall the app from my zone
    • Start: start the app
    • Stop
    • SetPermission: update the permission for the app
    • SetQuota: config the limit amout of resource for the app
    • SetAutoUpdate
• AppCmdList: Use this object when we need to execute multiple commands sequentially.
• AppList: APP list organized by category, the main fields as follow:
  • category: immutable, the name of the APP category
  • id: immutable, set by user to manage the APP category when the AppList is created, What's the different with category field? It's unique?
  • list of applications: mutable,The latest version is maintained by OOD in the zone.
• AppLocalList: What does the Local mean?
• AppLocalStatus: What does the Local mean?
• AppManagerAction: Is it related to the implementation logic of AppManager?
• AppSetting: Configuration for the app.
• AppStatus
  • id: immutable,DecAppId.
  • version: mutable
  • status: mutable, Why is it a boolean?
    The latest version is maintained by OOD in the zone.
• DefaultAppList
  APPs that are automatically installed after the CYFS system is installed.

3. group

There is a issue for it.The core objects for group is implemented in cyfs-core.

We use the blockchain structure to record the state evolution process of the Group.The consensus algorithm is Hotstuff.

• GroupProposal
  The object that prompts the infomation of group to update.It will answer the DECApp 3 questions:
  • Who want to update?
  • What does he want to update?
  • How to update?
    The fields is follow:
  • rpath: immutable, rpath the blockchain is running on.there is a blockchain is running for each rpath of the group.
  • method: immutable,the method provided by the DECApp,it will be called with the proposal.
  • params: immutable, the parameters for the method, it's defined and parsed by the DECApp.
  • meta_block_id: immutable,the highest block on MetaChain currently,Indicates that this proposal was made after the block on MetaChain.
  • effective_begining: immutable, when the proposal will effective.
  • effective_ending: immutable, when the proposal will invalid, The proposal should be discarded if it has not reached consensus after effective_ending.
  • payload: mutable, A buffer reserved for DECApp, the DECApp guarantees its correctness. It's usefull when the params is large.
• GroupQuorumCertificate
  The certificate to the blocks for Hotstuff.It has 2 types:
  • QC: the quorum certificate for the previous block.
    • block_id: immutable,current blockid.
    • prev_block_id: immutable,the previous block will be proven.
    • round: immutable,the round of the previous block.
    • votes: immutable,the set of signatures for the QC.
      • voter: the signer
      • signature: the signature
  • TC:the timeout certificate for last round.
    • round: immutable,the timeout round number
    • votes: immutable,the set of signatures for the round
      • voter: the signer
      • high_qc_round: the max round of the block which has collect the QC.
      • signature: the signature
• GroupConsensusBlock
  GroupConsensusBlock is used to describe the content of each block.
  The fields is follow:
  • rpath: immutable, rpath the blockchain is running on.there is a blockchain is running for each rpath of the group.
  • body_hash: immutable, the hash of GroupConsensusBlock, is used to verify the body.
  • result_state_id: immutable, the ObjectId of the result calculated by the DECApp,
  • height: immutable, the height of the block.
  • meta_block_id: immutable,the highest block on MetaChain currently,Indicates that this block was constructed after the block on MetaChain.
  • round: immutable,the round to vote for the block,used by Hotstuff.
  • group_chunk_id: immutable,the version of Group the block is depend on.
  • proposals: immutable,save in body, verify by body_hash,the set of the proposals(proposal/result/receipt/context) collected by the block.
  • qc: immutable,save in body, verify by body_hash,the quorum certificate for the previous block.
  • tc: immutable,save in body, verify by body_hash,the timeout certificate for last round.

4. nft

   NFT is a popular application scenario in recent years.CYFS collect and defined its own NFT objects.In theory,any named data is NFT in CYFS,but most NFTs are files in fact.So NFT is limited to file types.

   • NFTList
     Is a file list:
   • nft_list: immutable, file list.

5. trans

I don't know this part?

• TransContext

6. zone
   It's the manager of the devices belonging to the same owner.

• Zone
  The field is follow:
  • owner: immutable
  • ood_list: mutable
  • known_device_list: mutable
    How to maintain the latest version of the Body?

CyfsStack(cyfs-stack/cyfs-stack-loader/cyfs-lib)

CyfsStack integrates all functional modules provided by the CYFS system.Any service or DECApp will call the CyfsStack to use the CYFS.
There are 2 solutions:

• All services and DECApp are executed as plugins in the same process as CyfsStack.
  This has several disadvantages:
  • A dynamically loaded plug-in system, the developers have to spend more to learn it.
  • All services and DECApps will be affected by exceptions thrown by any DECApp, that is a disaster.
  • A DECApp can easily access the memory of another.
• Every service or DECApp is executed in a new process, they call the CyfsStack through the RPC.
  This solution is selected, and it's more friendly:
  • The developers can develop DECApp with their familiar technology.
  • DECApps can run independently in separate processes without interfering with each other.They can even run in separate dockers to keep resources isolated.
    It also has several disadvantages:
  • RPC in different programming languages is needed.but we can use it immediatly while anyone has complete it.
  • Performance will be lost in RPC.

RPC

1. Protocol for RPC

• The communication protocol of RPC adopts HTTP based on WebSocket or BDT.
• There is some common headers defined in cyfs-base/src/base/protocol_fields.rs.

2. Framework of RPC

There are client and service for RPC, the service is CyfsStack, and the client is SharedCyfsStack.

client	service
Implement in SharedCyfsStack	Implement in CyfsStack
Multiple instances	Only one instance for one device
Open in any DECApp process	Start with gateway(for OOD) or cyfs-runtime(for other devices)
Public for developers of DECApp	Invisible for developers of DECApp
Module named by cyfs-${function-name}-lib	Module named by cyfs-${function-name}

3. Data flow diagram

The data will flow from the DECApp to the processor in service.

• It will go through an indeterminate count of forwarding processes..
• The forwarding target may be other devices, or different modules on the current device.
• Each forwarding process has a input to receive the data, and a output to send the data except the initiator with only output and the ending processor with only input.
  • The inputs are usually named in the form of ${function-name}InputProcessor,and the parameter with data is named in ${function-name}InputRequest.
  • The outputs are usually named in the form of ${function-name}OutputProcessor,and the parameter with data is named in ${function-name}OutputRequest.
  • The forwarding process are usually named in the form of ${function-name}InputTransformer or ${function-name}OutputTransformer.

It's flow as follow:

graph TB
    subgraph DECApp
        User-->Interface[SharedCyfsStack: function with FunctionOutputRequest]
    end

    Interface-->FindTheTarget

    subgraph SharedCyfsStack
        FindTheTarget[find target device]-->RPC.send[RPC.send]
    end

    RPC.send.->RPC.recv
    RPC.send.->OtherDevice

    subgraph CyfsStack
        RPC.recv-->ObjectHttpListener
        ObjectHttpListener-->FunctionRequestHandlerEndpoint[FunctionRequestHandlerEndpoint.call]
        FunctionRequestHandlerEndpoint-->FunctionInputProcessor[FunctionInputProcessor.call_function]
        FunctionInputProcessor-->FindNextProcessor[output_processor = get_forward]
        FindNextProcessor-->InputOfOtherModule[input_processor]
        InputOfOtherModule-->NextN[next...]
        NextN-->Ending[ending_input_processor.call_function]
    end

    FindNextProcessor.->OtherDevice

    subgraph OtherDevice.CyfsStack
        OtherDevice[Other CyfsStack in different device]
    end

    OtherDevice.CyfsStack-.eq.-CyfsStack

Loading

We can integrate our own modules into the CYFS protocol stack according to the above framework.and you can also do it follow the issue step by step.

The crates

1. cyfs-stack

It implements the CyfsStack and all processors to route the data for each module.It's inaccessable immediatly for DECApp, so it will never be depended on by any DECApp.There are some directories named in ${function-name} or ${function-name}_api.They are implemented with the function module:

• ${function-name}: implement some basic structure, it's accessable for other modules.
• ${function-name}_api: it is the specific call routing logic of the function module.

2. cyfs-lib

It implements the SharedCyfsStack and some interface for the default function modules.It's the entry interface for DECApp to call the CyfsStack.So,most DECApp will depend on this crate.

If you are extending new functionality to CYFS,It's suggested to make a new crate named in cyfs-${function-name}-lib to instead of implement in cyfs-lib,you can export a interface to initialize your module with the SharedCyfsStack.

3. cyfs-stack-loader

There are so many parameters to initialize the CyfsStack, You are suggested to initialize the CyfsStack with cyfs-stack-loader if you want to initialize yourself, for example:

• Debuging your extended functions for CYFS.
• Make a tool like the zone-simulator.
  But you should only open the CyfsSharedStack if you are making a DECApp based on CYFS. it will connect to your CyfsStack initialized by cyfs-runtime or gateway.