diff --git a/doc/VRRP_HLD_Buzznik.md b/doc/VRRP_HLD_Buzznik.md new file mode 100755 index 0000000000..6981680e84 --- /dev/null +++ b/doc/VRRP_HLD_Buzznik.md @@ -0,0 +1,730 @@ +# Virtual Router Redundency Protocol (VRRP) HLD + +#### Rev 0.1 + + + +[TOC] + + + +# List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:--:|:--------:|:-----------------:|:------------------------------------------------------------:| +| 0.1 | 09/05/2019 | Dilip Kumar | Initial version | +| 0.2 | 09/11/2019 | Dilip Kumar | Addressed review comments from Ben and others. Minor edits to complete the unifished sections. | +| 0.3 | 10/10/2019 | Vijay Kumar | Added VRRP version 3 and VARP support | +| | | | | +| | | | | + +# Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +| -------- | ------------------------------------------- | +| ARP | Address Resolution Protocol | +| IP | Internet Protocol | +| LAG | Link Aggregation Group | +| LAN | Local Area Network | +| MAC | Media Access Control addresses | +| VIP | Virtual IP address | +| VLAN | Virtual Local Area Network | +| VMAC | Virtual MAC address | +| VRF | Virtual Routing and Forwarding | +| VRID | Virtual Router Identifier | +| VRRP | Virtual Router Redundency Protocol | +| VRRPv3 | Virtual Router Redundency Protocol verson 3 | +| IPv6 | Internet Protocol version 6 | +| VARP | Virtual Address Resolution Protocol | + +# About this Manual + +This document provides general overview of VRRP feature implementation based on RFC 5798 and VARP feature implementation in SONiC. + + + +# 1 Introduction and Scope + +Virtual Router Redundancy Protocol (VRRP) functionality is designed to eliminate the single point of +failure inherent in the static default routed environment. VRRP specifies an election protocol that +dynamically assigns responsibility of gateway router to one of the VRRP routers on a LAN. The VRRP +router controlling the IP(v6) address(es) associated with a virtual router is called the Master, and routes the +traffic. The election process provides dynamic fail-over in the forwarding responsibility should the Master +become unavailable. Any of the virtual router's IP(v6) addresses on a LAN can then be used as the default first +hop router by end-hosts. The advantage gained from using VRRP is a higher availability default path +without requiring configuration of dynamic routing or router discovery protocols on every end-host. + +Virtual-ARP (VARP) allows multiple switches to simultaneously route packets from a common IP address in an active-active router configuration. Each switch is configured with the same set of virtual IP addresses on corresponding VLAN interfaces and a common virtual MAC address. + + +# 2 Feature Requirements + +## 2.1 Functional Requirements + +Following requirements are addressed by the design presented in this document: + +1. Support VRRPv2 (IPv4), VRRPv3(IPv4, IPv6), VARP(IPv4, IPv6) + +2. Support multiple VRRP instances (groups) per interface + +3. Support VRRP on VLAN, PortChannel and Ethernet interfaces + +4. Support Uplink interface tracking + +5. Support preemption of a Master when a high priority VRRP node comes up + +6. Support configurable priority for VRRP instance + +7. Support configuration and management of various VRRP parameters + +8. VRRP support on non-default VRF + +9. Support REST access to VRRP objects + + + +Following requirements are beyond scope of this release. + +2. SNMP and gNMI access support to VRRP objects + + + +## 2.2 Configuration and Management Requirements + +This feature will support configuration and display CLIs to control and monitor VRRP parameters + +1. Support configuration of VRRP instances per interface + +2. Support configuration of VIP, priority, hello interval and preemption for VRRP instance + +3. Support configuration of uplink interface track + +4. Support configuration of VRRP protocol version + +5. Allow users to configure track interface weight to provide flexible policy of Master to Backup switchover + +6. Support display of various VRRP parameters as well as states using CLIs. + +7. Support configuration of VARP on interface + + + +## 2.3 Scalability Requirements + +1. Max number of VRRP instances: 128 +2. Max number of VRRP enabled interfaces: 128 +3. Max number of VRRP instances per interface: 16 +4. Max number of tracked interfaces per VRRP Instance: 8 +5. Max IP addresses per VRRP instance: 4 +6. Max number of VIP for VARP instance on an interface: 4 + + + +## 2.4 Warm Boot Requirements + +VRRP module is warm reboot compliant. That is, VRRP docker will be restarted as part of warm-reboot but will come up afresh and will build the state from scratch. The behavior of VRRP will be like Cold reboot. + +VARP module does not store any states. During warm-reboot router will be restarted. After restart VIP will be programmed in forwarding as per configuration. + + + +# 3 Feature Description + +## 3.1 Target Deployment use cases + +The following are some of the deployment use cases for VRRP and VARP + +- The Leaf nodes of the Clos network to provide first hop redundency to connected devices + + + +## 3.2 Functional Description of VRRP + +VRRP specifies an election protocol to provide the virtual router function described earlier. All protocol +messaging is performed using IP multicast datagrams, thus the protocol can operate over a variety of +multiaccess LAN technologies supporting IP multicast. Each VRRP virtual router has a single well-known +MAC address allocated to it. This document currently only details the mapping to networks using the IEEE +802 48-bit MAC address. The virtual router MAC address is used as the source in all periodic VRRP +messages sent by the Master router to enable bridge learning in an extended LAN. +A virtual router is defined by its virtual router identifier (VRID) and a set of IP addresses. A VRRP router +may associate a virtual router with its real addresses on an interface, and may also be configured with +additional virtual router mappings and priority for virtual routers it is willing to backup. The mapping +between VRID and addresses must be coordinated among all VRRP routers on a LAN. However, there is +no restriction against reusing a VRID with a different address mapping on different LANs. The scope of +each virtual router is restricted to a single LAN. +To minimize network traffic, only the Master for each virtual router sends periodic VRRP Advertisement +messages. A Backup router will not attempt to preempt the Master unless it has higher priority. This +eliminates service disruption unless a more preferred path becomes available. It's also possible to +administratively prohibit all preemption attempts. The only exception is that a VRRP router will always +become Master of any virtual router associated with addresses it owns. If the Master becomes unavailable +then the highest priority Backup will transition to Master after a short delay, providing a controlled +transition of the virtual router responsibility with minimal service interruption. +The VRRP protocol design provides rapid transition from Backup to Master to minimize service +interruption, and incorporates optimizations that reduce protocol complexity while guaranteeing +controlled Master transition for typical operational scenarios. The optimizations result in an election +protocol with minimal runtime state requirements, minimal active protocol states, and a single message +type and sender. The typical operational scenarios are defined to be two redundant routers and/or distinct +path preferences among each router. A side effect when these assumptions are violated (i.e., more than two redundant paths all with equal preference) is that duplicate packets may be forwarded for a brief period during Master election. However, the typical scenario assumptions are likely to cover the vast majority of deployments, loss of the Master router is infrequent, and the expected duration in Master election convergence is quite small ( << 1 second ). Thus the VRRP optimizations represent significant +simplifications in the protocol design while incurring an insignificant probability of brief network +degradation. +Though VRRP standard protocol present in RFC 3768 and RFC 5798 are complete in itself, there are few +limitations/drawbacks of the protocol: + +1. The point of failure which VRRP safeguards against is the interface on which VRRP instance is +present and the router as a whole. Even if all of the connectivity of master to the external network +fails, VRRP will still not trigger the failover to the backup gateway. Uplink interface tracking feature has been introduced to overcome this limitation +2. Only master router sends the advertisements, and backups just listen to them. Since backups do +not send any periodic message, it is difficult to ascertain that the backups are still active, and the +healthiness of the backup routers can not be known until the failover happens. +3. VRRP owner - if a VRRP instance's virtual IP address is same as the real interface IP address. VRRP owner has special privileges (priority is 255) and can preempt any other router acting as master. This usually causes unwanted temporary network disruption after the non-owner master had stabilized +after the failover from owner. +4. A non-owner master can not accept packets destined to the virtual IP address. Only the owner can +accept and respond to such packets. +5. Two virtual routers with same VRIDs but on different IP subnets can not co-exist on the same +VLAN because of the virtual MAC address clash. + +### 3.2.1 Virtual Router Owner + +VRRP instance whose virtual IP address (VIP) is same as real interface address is called the owner of virtual router and has the highest priority. VRRP owner is supported in SONIC's VRRP implementation. + +### 3.2.2 Virtual MAC Address + +Following virtual MAC addresses is used by the protocol (as per RFC 5798) + +IPv4 case: **00-00-5e-00-01-{vrid}** + +IPv6 case: **00-00-5e-00-02-{vrid}** + +where, vrid is user configured 1-byte virtual router identifier. VRID has interface scope; that is, VRID has to be unique among the VRRP instances on an interface. However, same VRID can be used for two or more virtual router instances across different interfaces. + +### 3.2.3 Preemption + +Preemption is turned on by default. Even if preemption is disabled, it does not affect the owner router since owner preempts the active master. Mastership switchover causes unnecessary temporary network disruption. + +### 3.2.4 VRRP Advertisement Frame + +VRRP control packets have IP protocol type as 112 (reserved for VRRP). IPv4 and IPv6 VRRP keepalives are sent to VRRP multicast address 224.0.0.18 and FF02::12 respectively. Source MAC in VRRP control packets is virtual MAC address + +### 3.2.5 ARP Request Handling + +Only master responds to the ARP requests for virtual IP address. In ARP replies sent by master, the source MAC in Ethernet header and ARP payload is virtual MAC address. + +### 3.2.6 Uplink Interface Tracking + +Interfaces other than the VRRP instance interface can be tracked for up/down events. When interface-tracking is enabled in the VRRP instance configuration, the tracked interface's operational status will be monitored. When a interface operational down event is detected on a tracked-interface, the track-priority/weight is subtracted from the current router’s priority value. Similarly, when interface operational up event is detected on the tracked-interface, the track-priority/weight is added to the router’s current priority value. + +The dynamic change of router priority can trigger mastership switchover if the preemption is enabled. **However, if the router is an owner, the mastership switchover will not happen**. + +Maximum number of interfaces that can be tracked for a virtual router instance is 8. + +## 3.3 Functional Description of VARP + +In most of Leaf-Spine deployments, redundancy in Spine layer is required to achieve high availability and to prevent network service disruption. Modern layer 2 networks adopted loop-free and balanced path networks using Multi Chassis Link Aggregation topologies with LACP port channels, leaving loop control methods (STP) as second protection layer. Spines also supports layer 3 networks, using ECMP in a scalable network topology. For unicast redundancy in layer 3, a common method is use Virtual Router Redundancy Protocol (VRRP) to provide a simple and unique gateway for Leaf level. Although VRRP provides redundancy, it is active-standby protocol and do not provide a balanced data traffic distribution over Multi Chassis Link Aggregated topologies. + +VARP provides better data traffic balancing and faster redundancy convergence, implementing active-active First Hop Router Redundancy to provide active/active unicast IP routing. + +The primary benefit of using VARP is that all configured routers are active and are able to perform routing. VARP also provides rapid failover in the event of a link or switch failure, while enabling the sharing of IP forwarding load between both switches. VARP requires configuring the same virtual-router IP address on the appropriate VLAN interfaces of both peers, as well as a global unique virtual-router MAC address. VARP functions by having both switches respond to ARP requests and GARP for a configured IP address with the “virtual-router” MAC address. This address is receive-only MAC address and no packet is ever sent with this address as its source. If IP routing is enabled, received packets will be routed as follows: when the DMAC of a packet destined to a remote network matches the configured “virtual-router” MAC address, each MLAG peer locally forwards the traffic to its next hop destination. + +### 3.3.1 Virtual MAC Address + +Administrator assigns virtual MAC address to the switch. The switch maps all virtual router IP addresses to this MAC address. The address is receive-only; the switch never sends packets with this address as the source. + +# 4 Feature Design + + + +## 4.1 Design Overview + +### 4.1.1 Basic Approach + +Keepalived (https://www.keepalived.org/) open source code is chosen for VRRP control plane code. + +![VRRP Keepalived Design](images/VRRP_Keepalived_design.PNG "Figure : Design") + + +### 4.1.2 Container + +A new container "vrrp" has been added to host VRRP protocol operation. VRRP source code is not maintained in SONIC repos. Instead, VRRP code is downloaded from keeplaived project repo at the compile time, patches are applied to it (for any fixes by us) and then compiled/linked with SONIC binary. + +New container "vrrp" handles both VRRP and VARP functionality. For VARP the configuration is programmed in kernel through 'vrrpmgrd' module and 'vrrpsyncd' programs the hardware entry in SAI. + +### 4.1.3 SAI Overview + +VRRP virtual router interface can be created by setting the attribute “SAI_ROUTER_INTERFACE_ATTR_IS_VIRTUAL” with **create_router_interface** API. + + * @brief RIF creation is a virtual RIF. + * + * Create a Virtual RIF object, which only programs the ingress router MAC. + * This simplifies the management of VRRP master router's configuration in + * SAI adapter, as defined by RFC 5798 (or similar proprietary protocols). + * Using a Virtual RIF allows SAI to optimize resources, so neighbor entries + * cannot be learned on a Virtual RIF. On a virtual RIF following attributes + * are invalid: ADMIN state, MTU size, packet action and multicast enable. + * Alternatively VRRP can also be configured using native RIF objects without + * using VIRTUAL attribute, with the expectation that SAI adapter will consume + * resources that will not be used. + * + * @type bool + * @flags CREATE_ONLY + * @default false + */ + + SAI_ROUTER_INTERFACE_ATTR_IS_VIRTUAL, +## 4.2 DB Changes + +At a high level below are some of the interactions between relevant components and the DB involved for VRRP support in SONiC architecture. + + + +![VRRP Arch](images/VRRP_architecture.PNG "Figure : Arch") +__Figure 1: VRRP Architecture__ + +### 4.2.1 CONFIG_DB changes + +**VRRP_TABLE** + +Producer: config manager + +Consumer: VrrpMgr + +Description: New table that stores VRRP configuration for per interface + VRID. + +Schema: + +``` +;New table +;holds the VRRP configuration per interface and VRID + +key = VRRP_TABLE:interface_name:vrid:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 + ; vrid is an integer +; field = value +vrid = 1*3DIGIT ; VRRP Instance Identifier +address_family = "IPv4"/"IPv6"; Address Family of VRRP instances +vip = ip_address ; Virtual IPv4/IPv6 address. This is a list of IPv4/IPv6 addresses +priority = vrrp_priority ; Priority of VRRP instance +adv_interval = 1*3DIGITS ; Advertisement interval for VRRP. Default = 1sec +state = vrrp_state ; String denoting the state of VRRP instance +version = vrrp_version ; VRRP version. VRRP for IPv6 will always be version 3 +pre_empt = "true"/"false" ; VRRP premption is enabled? Default is True +track_interface = track_interface ; List of interfaces tracked by a VRRP instance + |weight|; This is repeated for the configured tracking interfaces +``` + +Example:- + +**Key**: VRRP_TABLE:Vlan11:1 + +**Value**: + +​ 'vrid': '1', +​ 'vip': '4.1.1.100,', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '2', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet7|weight|20, + +​ PortChannel001|weight|40 + + + +Example:- Entery with multiple virtual IPs + +**Key**: VRRP_TABLE:Vlan11:1 + +**Value**: + +​ 'vrid': '1', +​ 'vip': '4.1.1.100,4.1.1.200,4.1.1.201,4.1.1.202,', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '2', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet7|weight|20, + +​ PortChannel001|weight|40 + + + +**Key**: VRRP_TABLE:Vlan12:2 + +**Value**: + +​ 'vrid': '2', +​ 'vip': 'fe80::100,4::100', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '3', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet8|weight|20, + +​ PortChannel002|weight|40 + + + +Example:- VRRP for IPv6 address + +**VARP_TABLE** + +Producer: config manager + +Consumer: VrrpMgr + +Description: New table that stores VARP configuration for per interface. + +Schema: + +``` +;New table +;holds the VARP configuration per interface + +key = VARP_TABLE:interface_name:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 +address_family = "IPv4"/"IPv6"; Address Family of VRRP instances +vip = ip_address ; Virtual IPv4/IPv6 address. This is a list of IPv4/IPv6 addresses +``` + + + +``` +;New table +;holds the VARP_GLOBAL configuration per router + +key = VARP_GLOBAL_TABLE:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 +vmac = mac ; Virtual mac address for all the VARP Virtual IP +``` + + + +Example:- + +**Key**: VRRP_TABLE:Vlan15 + +**Value**: +​ 'vip': '15.1.1.100,16.1.1.100', + +### 4.2.2 APP_DB Changes + +**VRRP_TABLE** + +Producer: VrrpMgr + +Consumer: VrrpOrch + +Description: This is a new table that contains VRRP and VARP state information + +Schema: + +``` +; New table +; holds the VRRP state and VMAC information + +key = VRRP_TABLE:interface_name:vip:type + interface_name ; interface name as a string. Vlan, Ethernet or PortChannel + vip ; virtual IP address in a.b.c.d/32 or a:b:c::d format + type ; IP(v6) address type string. + +; field = value +vmac = virtual_mac_address ; Virtual MAC address associated with VRRP instance +``` + +Example:- + +**Key**: VRRP_TABLE:Vlan1000:[40.10.8.101/32:ipv4] + +**Value**: "vmac":"00:00:5e:00:01:08" + +## 4.3 Modules Design and Flows + + + + + + +## 5 CLI + +SONIC Click based configuration and monitoring CLIs have been introduced in SONIC for VRRP + +### 5.1 Configuration Commands + +#### 5.1.1 Configuration Commands for IPv4 VRRP + +``` +configure interface vrrp add +This command adds/enables a IPv4 VRRP instance on an interface. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VRRP is to be enabled. +- vrid - VRRP instance identifier. + +configure interface vrrp remove +This command removes a IPv4 VRRP instance from an interface. + +configure interface vrrp vip add +This command adds a virtual IP address for a VRRP instance on an interface. User is allowed to create multiuple VIPs for a IPv4 VRRP instance. VIP must fall in the interface's subnet. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VIP is being added +- vrid - VRRP instance identifier. +- virtual_ip_address - VIP address in dotted decimal IPv4 address + +configure interface vrrp vip remove +This command deletes a already configured VIP from a IPv4 VRRP instance + +configure interface vrrp priority +This command configures priority for a VRRP instance +- priority - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +configure interface vrrp adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- interval - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +configure interface vrrp pre_empt enable +This command enables premeption of a Master when a higher priority VRRP router arrives +- enable - Enable premeption. Default is enabled. + +configure interface vrrp pre_empt disable +This command disables premeption of a Master when a higher priority VRRP router arrives +- disable - Disable premeption. Default is enabled. + +configure interface vrrp version +This command configures VRRP protocol version for IPv4 VRRP instances + +configure interface vrrp track_interface add +This command adds a track interface to a VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_interface - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +configure interface vrrp track_interface remove +This command removes an already configured track interface from a IPv4 VRRP Instance. +``` + +#### 5.1.1 Configuration Commands for IPv6 VRRP + +``` +configure interface vrrp6 add +This command adds/enables a IPv6 VRRP instance on an interface. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VRRP is to be enabled. +- vrid - VRRP instance identifier. + +configure interface vrrp6 remove +This command removes a IPv6 VRRP instance from an interface. + +configure interface vrrp6 vip add +This command adds a virtual IPv6 address for a IPv6 VRRP instance on an interface. User is allowed to create multiuple VIPs for a VRRP instance. VIP must fall in the interface's subnet. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VIP is being added +- vrid - VRRP instance identifier. +- virtual_ip_address - VIP address in dotted decimal IPv4 address + +configure interface vrrp6 vip remove +This command deletes a already configured VIP from a IPv6 VRRP instance + +configure interface vrrp6 priority +This command configures priority for a VRRP instance +- priority - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +configure interface vrrp6 adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- interval - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +configure interface vrrp6 pre_empt enable +This command enables premeption of a Master when a higher priority VRRP router arrives +- enable - Enable premeption. Default is enabled. + +configure interface vrrp6 pre_empt disable +This command disables premeption of a Master when a higher priority VRRP router arrives +- disable - Disable premeption. Default is enabled. + +configure interface vrrp6 version +This command configures VRRP protocol version for IPv4 VRRP instances + +configure interface vrrp6 track_interface add +This command adds a track interface to a IPv6 VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_interface - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +configure interface vrrp6 track_interface remove +This command removes an already configured track interface from a VRRP Instance. +``` + +SONIC kLISH based configuration and monitoring CLIs have been introduced in SONIC for VRRP + +``` +[no] configure vrrp address-family {ipv4 | ipv6} +This command configures/removes VRRP instance in IPv4 or IPv6 address-family. +- vrid - VRRP instance identifier. + +[no] vip +This command configures/removes a virtual IP address for a VRRP instance on an interface. User is allowed to create multiuple VIPs for a IPv4 VRRP instance. VIP must fall in the interface's subnet. For IPv6 VRRP instance VIP could be any link-local address. +- virtual_ip_address - VIP address + +priority +This command configures priority for a VRRP instance +- priority_value - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- int_value - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +[no] pre_empt +This command enables/disables premeption of a Master when a higher priority VRRP router arrives. + +[no] vrrpv3 +This command configures/removes VRRP version 3 for a IPv4 VRRP instance. IPv4 VRRP instances are version 2 by default. + +[no] track_interface + +This command adds/removes a track interface to a VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_ifname - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +``` + +SONIC kLISH based configuration and monitoring CLIs have been introduced in SONIC for VARP + +``` +[no] ip virtual-router address +This command configures/removes virtual IPv4 on an interface. +- vip - IPv4 virtual address. + +[no] ipv6 virtual-router address +This command configures/removes virtual IPv6 on an interface. +- vip - IPv6 virtual address. + +[no] ip virtual-router mac-address +This command configures/removes virtual mac address for all the IPv4/IPv6 virtual addresses. + + +``` + +#### + +### 5.2 Show Commands + +``` +show vrrp + - lists all the VRRP instances including their current state + Sample output:- +admin@sonic:~$ show vrrp +Interface_Name VRID State VIP Cfg_Prio Curr_Prio + Vlan1 1 Backup 4.1.1.100 100 120 + Vlan2 2 Master 4.1.2.100 100 100 + Vlan3 3 Backup 4.1.3.100 100 100 + Vlan4 4 Backup 4.1.4.100 100 100 + Vlan5 5 Master 4.1.5.100 100 100 + + +show vrrp + - This command displays data about a VRRP instance in detail + Sample output:- +admin@sonic:~$ show vrrp Vlan1 1 +Vlan1, VRID 1 +Version is 2 +State is Backup +Virtual IP address: + 4.1.1.100 +Virtual MAC address is 0000.5e00.0101 +Track interface: + Intfname State Priority + Ethernet7 Up 10 + PortChannel001 Up 10 +Configured Priority is 100, Current Priority is 120 +Advertisement interval is 1 sec +Preemption is enabled +``` + +## + +``` +show varp + - lists all the VARP instances including their current state + Sample output:- +IP virtual router is configured with MAC address: 24cd.5a29.cc31 +Interface IP Address Virtual IP Address Status Protocol +Vlan15 10.1.1.3/24 10.1.1.15 up up +Vlan15 10.1.1.3/24 10.1.1.16 up up +Vlan15 10.1.1.3/24 10.1.1.17 up up +Vlan20 10.12.1.6/24 10.12.1.51 up up +Vlan20 10.12.1.6/24 10.12.1.53 up up +Vlan20 10.12.1.6/24 10.12.1.55 up up + + +``` + + +## 6 Serviceability and Debug + +The existing logging mechanisms shall be used. Proposed debug framework shall be used for internal state dump. + +## 7 Warm Reboot Support + +Currently, warm-reboot is not supported for VRRP. That is, warm-reboot will simply restart the VRRP docker without VRRP storing any data for warm restart. + +VARP is stateless, during warm-reboot router will go down and restart with new configuration. + +## 8 Unit Test cases + +| Test Cases | UT Action and observation | Remarks | +| ---------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------- | +| Configuration | | | +| | Validate IPv4 VRRP config on Physical Interface | | +| | Validate IPv4 VRRP config on Ve Interface | | +| | Ensure that VMAC is generated based on VRID for IPv4 VRRP | | +| | Ensure VMAC is programmed in FDB and VIP in LPM | | +| | Validate IPv6 VRRP config on Physical Interface | | +| | Validate IPv6 VRRP config on Ve Interface | | +| | Ensure that VMAC is generated based on VRID for IPv6 VRRP | | +| | Ensure VMAC is programmed in FDB and VIP in LPM for IPv6 VRRP | | +| | Validate VRRP on more than one interface | | +| | Validate VRRP with more than one VIP per instance | | +| | Validate multiple VRRP instance per interface | | +| Failover | | | +| | Validate VRRP master failover with priority change | | +| | Validate VRRP master failover by disabling a vrrp group | Add 'disable' feature if needed and feasible | +| | Validate VRRP master failover by removing all virtual-ip | | +| | Validate VRRP master failover by removing interface IPv4/IPv6 address | | +| | Validate VRRP master failover by deleting master VRRP session | | +| | Validate VRRP master failover by bringing down interface | | +| | Validate VRRP master failover by deleting interface | | +| VRRP parameter changes | | | +| | Validate gratutious ARP | | +| | Validate adv interval | | +| | Validate version change for IPv4 VRRP instance | | +| ARP/ND resolve to VIP | | | +| | validate ARP resolution from host to VRRP session VIP | Hosts should get arp response with VMAC as source MAC | +| | Perform Master failover and check that ARP resolution to VIP will happen with VMAC | | +| Ping to VIP | | | +| | Validate ping to IVRRP from host, backup & non vrrp router works | | +| L3 Forwarding with VIP as GW | | | +| | Validate that IPv4/IPv6 traffic forwarding with VIP as gateway works from source to destination host | | +| | Perform VRRP master failover and check if traffic forwarding continues with new master | | +| VRRP Owner | | | +| | Configure VRRP for IPv4/IPv6 session with VIP same as interface IP and ensure this session becomes master and priority of session becomes max=255 | | +| | Ensure ping to VIP works in owner case | | +| | Ensure arp/nd resolution with VIP as gateway in owner case | ARP/ND should get resolved with VMAC and not interface MAC | +| | Ensure ping and traffic forwarding continues after owner failover and comes back | | +| | Traffic forwarding works with VIP as gateway | | +| Interface tracking | | | +| | Run below tests with Ping and traffic running | | +| | Enable interface tracking in a VRRP group with priority, bring down the port and ensure VRRP session priority comes down and session transition from Master to Backup State | | +| | Enable the tracked interface and check if session priority gets restored and session becomes master | | +| | Repeat this test on Port-channel interface | | +| | Repeat this test by deleting and re-adding port-channel interface | | \ No newline at end of file diff --git a/doc/aaa/Persistent HTTP Connections from CLI.md b/doc/aaa/Persistent HTTP Connections from CLI.md new file mode 100644 index 0000000000..4d455c44f7 --- /dev/null +++ b/doc/aaa/Persistent HTTP Connections from CLI.md @@ -0,0 +1,60 @@ +Persistent HTTP connections from CLI +==================================== + +The SONiC CLI provided by the management framework container runs an instance of +klish, which provides a fixed set of commands. The CLI is simply a front-end to +the REST server, and each command is mapped to a corresponding REST endpoint. + +# Command Flow + +When the user enters a specific command on the CLI, the corresponding `ACTION` +tag in the CLI XML specification shells out to a Python script with any +arguments and an optional template to format the returned values. This script +connects to the REST server on the local machine over HTTPS, retrieves and +formats the JSON response. The Python script then exits, terminating any HTTP +connection that had been set up. + +This is the current behavior, even without RBAC support, which means that every +command will need to set up a new HTTPS connection. However, when RBAC is +enabled, it is not likely to cause a noticeable performance impact, since the +system is already incurring the TLS overhead. + +As can be seen from the flow above, it is not possible to set up a persistent +HTTP connection, since every command spawns a new connection. + +# Alternative Designs + +This section describes some alternative designs that will enable the CLI to +create a persistent connection. + +## Proxy service + +As part of the management framework, we can add a "proxy" service that is +spawned with the CLI. This service will set up a secure HTTP connection for the +authenticated user, and create a local unix socket that is accessible only by +that user. The CLI XML will remain unchanged, but the Python ApiClient class +will be changed to connect to the local socket. + +This will still create independent HTTP connections, but they can be insecure +connections, while the proxy service will transfer the connections from the +insecure unix socket to the secure tunnel, which will reduce the TLS connection +time. + +**Note:** The security considerations have not been completely mapped out, and +this may open the system up to security holes. + +## Klish modification + +This approach considers modifying the Klish executable. When Klish is spawned, +it will set up the HTTPS connection and keep it alive as long as the CLI is +active. Each `ACTION` tag will call into klish functions that will connect to +the existing HTTPS connection. + +This approach is the most secure option, however, it needs heavy modification to +klish, and there are several unknowns at this time. + +## No modification - Buzznik + +This approach leaves the design as is for the Buzznik release. Every command +will continue to create a new HTTPS connection, as it does today, and will tear +down the connection on completion of the request. diff --git a/doc/aaa/SONiC RBAC HLD.md b/doc/aaa/SONiC RBAC HLD.md new file mode 100644 index 0000000000..2f6d4a3e37 --- /dev/null +++ b/doc/aaa/SONiC RBAC HLD.md @@ -0,0 +1,460 @@ + + +# Authentication and Role-Based Access Control + +# High Level Design Document +#### Rev 0.1 + +# Table of Contents +- [Revision History](#revision-history) +- [About this Manual](#about-this-manual) +- [Scope](#scope) +- [Definitions/Abbreviations](#definitionsabbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + - [1.1.1.1 NBI Authentication](#1111-nbi-authentication) + - [1.1.1.2 CLI Authentication to REST server](#1112-cli-authentication-to-rest-server) + - [1.1.1.3 Translib Enforcement of RBAC](#1113-translib-enforcement-of-rbac) + - [1.1.1.4 Linux Groups](#1114-linux-groups) + - [1.1.1.5 Certificate-based Authentication for gNMI and REST](#1115-certificate-based-authentication-for-gnmi-and-rest) + - [1.1.1.6 Local User Management and UserDB Sync](#1116-local-user-management-and-userdb-sync) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + - [1.1.3.1 REST Server](#1131-rest-server) + - [1.1.3.2 gNMI Server](#1132-gnmi-server) + - [1.1.3.3 Translib](#1133-translib) + + [1.1.4 Warm Boot Requirements](#114-warm-boot-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Switch State Service Design](#33-switch-state-service-design) + + [3.3.1 Orchestration Agent](#331-orchestration-agent) + + [3.3.2 Other Process](#332-other-process) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 User Interface](#36-user-interface) + + [3.6.1 Data Models](#361-data-models) + + [3.6.2 CLI](#362-cli) + - [3.6.2.1 Configuration Commands for User Management](#3621-configuration-commands-for-user-management) + - [3.6.2.2 Show Commands](#3622-show-commands) + - [3.6.2.3 Debug Commands](#3623-debug-commands) + - [3.6.2.4 IS-CLI Compliance](#3624-is-cli-compliance) + + [3.6.3 REST API Support](#363-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) +- [5 Error Handling](#5-error-handling) + * [5.1 REST Server](#51-rest-server) + * [5.2 gNMI server](#52-gnmi-server) + * [5.3 CLI](#53-cli) + * [5.4 Translib](#54-translib) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) +- [10 Internal Design Information](#10-internal-design-information) + +# Revision History +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/22/2019 | Jeff Yin | Initial version | +| 0.2 | 10/30/2019 | Jeff Yin | Revision after joint review with Broadcom/Dell | + +# About this Manual +This document provides a high-level design approach for authentication and RBAC in the SONiC Management Framework. + +For authentication, this document describes how the CLI and programmatic interfaces (REST, gNMI) -- collectively referred to in this document as the northbound interfaces (NBIs) -- will authenticate users and the supported credentials and methods. + +For authorization, this document describes a centralized authorization approach to be implemented in the Translib component of the SONiC Management Framework. + +# Scope +This document covers the interfaces and mechanisms by which NBIs will authenticate users who wish to access and configure the SONiC system via the Management Framework. It will also cover RBAC enforcement. + +This document will NOT extensively cover (or assumes the pre-existence of): +- Implementation of remote authentication and authorization (RADIUS, TACACS+, etc.) +- Public Key Infrastructure management: X.509v3 certificate installation, deletion, trust store management, etc. + + +# Definitions/Abbreviations + +| **Term** | **Meaning** | +|:--------------------------|:-------------------------------------| +| RBAC | Role-Based Access Control | +| AAA | Authentication, Authorization, Accounting | +| CLI | Command-Line Interface | +| REST | REpresentational State Transfer | +| gNMI | gRPC Network Management Interface | +| NBI | North-bound Interfaces (CLI, REST, gNMI) | + + + +# 1 Feature Overview + +## 1.1 Requirements + +- The SONiC Management Framework must support authenticated access for the various supported northbound interfaces (NBIs): CLI, REST, and gNMI. Since the CLI works with the REST server, it must also be authenticated with the REST server. +- The NBIs must pass along the username info to Translib, so that Translib can enforce role-based access (RBAC). +- For RBAC, Linux Groups will facilitate authorization. Initially, two roles will be supported: read/write and read-only. Additionally, remotely-authenticated users who map to a defined role will be authenticated as a static global user on the system. These limitations should be revisited at a later date. +- Local user management: CLIs and APIs for creating and managing local users on the system -- their usernames, passwords, and roles. + +### 1.1.1 Functional Requirements + +#### 1.1.1.1 NBI Authentication +A variety of authentication methods must be supported: +* **CLI** authentication is handled via the same mechanisms supported by SSH + - Password-based Authentication + - Public-key Authentication +* **REST** authentication + - Password-based Authentication with JWT token-based authentication + - Certificate-based Authentication with JWT token-based authentication + - The REST server must be enhanced to accept all types of authentication concurrently +* **gNMI** Authentication + - Password-based Authentication with Token-based authentication + - Certificate-based Authentication + +#### 1.1.1.2 CLI Authentication to REST server +Given that the CLI module works by issuing REST transactions to the Management Framework's REST server, the CLI module must also be able to authenticate with the REST server when REST authentication is enabled. This can be accomplished via the following steps: +1. When a user is created on the system, a self-signed certificate is generated with the username embedded into the Subject field. That self-signed certificate will be stored in the user's home directory, with access permissions limited to that user only. The self-signed certificate will be used for mutual authentication from the KLISH shell to the REST server. +**Design note:** Certificate-based authentication is desired over password-based authentication for the CLI-to-REST connection because it prevents the need to store and forward any plaintext passwords. Per-user certificates with strict file permissions are desired so as to ensure that users cannot drop into the Linux shell and perform operations as other than themselves (e.g., via `curl` with a shared certificate, or someone else's certificate). +2. When a user logs into the switch, the switch will authenticate the user by using the username and password credentials. +3. When the KLISH process is started, its UID and GID are set to those of the user that was authenticated by `sshd` or `login` (as through the console). +4. When the KLISH shell is spawned, it will create a persistent, local HTTPS connection over an internal socket, and send the REST request to authenticate this KLISH session as the logged-in user, which can be looked up through NSS by using `getpwuid()`. This authentication request will contain the user's client certificate. +5. The REST server will authenticate the user and return a token with the username encoded in it. The KLISH CLI must cache this token and use it for all future requests from this KLISH session. The REST server will also maintain this token to username mapping with it so as to identify all future requests as well. This will also allow the REST server in creating audit logs for all the requests sent from KLISH, as well as pass the username to Translib for RBAC enforcement. +6. KLISH CLI session will store the authentication token, and from then on, KLISH CLI will send REST requests using the persistent connection with the authentication token in the HTTP header to the REST server for all the CLI commands. +7. The KLISH session must be able to cleanly handle ctrl-c breaks while it is busy waiting on a response from the REST server. +8. When the user exits the KLISH CLI session, the HTTP persistent connection is closed. The REST server will clean up the corresponding authentication token for its corresponding KLISH CLI Client. + +#### 1.1.1.3 Translib Enforcement of RBAC +The Translib module in the [management framework](https://github.com/project-arlo/SONiC/blob/master/doc/mgmt/Management%20Framework.md) is a central point for all commands, regardless of the interface (CLI, REST, gNMI). Therefore, the RBAC enforcement will be done in Translib library. + +The CLI, REST, and gNMI will result in Translib calls with xpath and payload. Additionally, the REST and gNMI server modules must pass the username to the Translib. The RBAC checks will be enforced in the **Request Handler** of the Translib. The Request Handler processes the entire transaction atomically. Therefore, if even one operation in the transaction is not authorized, the entire transaction is rejected with the appropriate "Not authorized" error code. If the authorization succeeds, the transaction is passed to the Common Application module for further ‘transformation’ into the ABNF format. The rest of the flow of the transaction through the management framework is unmodified. + +At bootup time of the manageability framework (or gNMI container), it is recommended to cache the Privilege, Tenant and Resource Tables(RBAC tables) stored in the Config DB. This is because every command needs to access the information in the RBAC tables in order to authorize the access for the information in other tables. Alternately, instead of caching the entire RBAC tables, the information can be cached once the record is read from the DB. Additionally, the Translib must listen to change notifications on these RBAC Tables in order to keep its cache current. + +As described in section [1.1.1.4 Linux Groups](#1114-linux-groups), the enforcement of the users and roles in Linux will be done via the Linux groups. A user can use Linux commands on the Linux shell and create users and Linux groups (which represent the roles). This will mean that the information in the RBAC tables is no longer current. In order to keep the information in the RBAC tables and the Linux `/etc/passwd` file in sync, a service must run on the host (not the container) to keep the databases in sync. + +Since Translib is the main authority on authorized operations, this means that the NBIs cannot render to the user what they are and are not allowed to do. The CLI, therefore, renders the entire command tree to a user, even the commands they are not authorized to execute. + +#### 1.1.1.4 Linux Groups +Initially, only two roles will be supported: +- `admin` -- can perform read and write functions across all attributes (i.e., GET/PUT/PATCH/etc.) +- `operator` -- can only perform read functions on all attributes (i.e., GET only) +These privileges will be enforced by Translib. + +RBAC will be facilitated via Linux Groups: +- Local users + - Local user with `Operator` role is added into `operator` group + - Local user with `Admin` role is added into `admin` group and is a `sudoer`. +- Remote users + - Remote users with `Operator` role are mapped to the same global `remote-user` user who is part of `operator` group + - Remote users with `Admin` role are mapped to the same global `remote-user-su` user who is part of `admin` group and is a `sudoer` + - This means that all remote users will share the same accounts on the system, which also means they will share the same /home directory and certificate to log into the REST server. + + In the future, this "global user" approach will be revisited so that remote users are authenticated with their own username so that their activities may be properly audited. + +#### 1.1.1.5 Certificate-based Authentication for gNMI and REST +For the initial release, it will be assumed that certificates will be managed outside of the NBIs. That is, no CLIs or REST/gNMI interfaces will be implemented to support public key infrastructure management for certificates and certificate authorities. Certificates will be manually generated and copied into the system via Linux utilities. + +The exception to this is the self-signed certificates used for CLI authentication to the REST server. Those certificates will be auto-generated when a user is created on the system. These certificates should be copied to a user's home directory _as well as_ the trust store so that the REST server can use them for authenticating local CLI sessions. + +User certificates must be stored in a user's home directory. Their corresponding private keys must also be stored in the user's home directory, albeit with restricted permissions so that they are not readable by other users. + +The gNMI server will use a trust store for CA certificates from a location such as `/usr/local/share/ca-certificates`. The trust store itself must be managed by [existing Linux tools](https://manpages.debian.org/jessie/ca-certificates/update-ca-certificates.8.en.html). + +The gNMI server must implement a method by which a username can be determined from the presented client certificate, so that the username can thus be passed to Translib for RBAC enforcement. The username will be derived from the Subject field of the X.509v3 certificate. + +Users must be informed by way of documentation so that they know how to manage their certificate infrastructure in order to properly facilitate gNMI communication. + +The REST server must use the same certificate scheme as the gNMI server to validate client certificates. + +#### 1.1.1.6 Local User Management and UserDB Sync +An interface must be developed for local user management, so that administrators can add users and assign passwords and roles to them. Administrators with the appropriate role must be able to add/delete users, modify user passwords, and modify user roles. They must be able to do so through all of the NBIs, meaning that a YANG model and CLI tree must be developed. + +Users must be added to the Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). That is where a user is mapped to a Linux [User Identifier](https://en.wikipedia.org/wiki/User_identifier) (UID) and primary [Group Identifier](https://en.wikipedia.org/wiki/Group_identifier) (GID). When users are created they also need to be assigned roles. Roles are simply defined as Linux groups (`/etc/group`) and assigned to users as [Supplementary GIDs](https://en.wikipedia.org/wiki/Group_identifier#Supplementary_groups). + +When a user is created it also needs to be assigned certificates that will allow them to communicate with the REST server. Finally, all users need to be added to the REDIS database (see [section 3.2.6]()) where additional information about each user can be stored (e.g. *tenant*). + +Since these operations (i.e. creating Linux users, assigning certificates, etc.) are non-trivial, the process of creating users will be entrusted to the Host Account Management Daemon (**hamd**). + +##### 1.1.1.6.1 Host Account Management Daemon (hamd) + +The **hamd** process runs on the host. It is accessed via a DBus interface that provides the ability to access and/or modify the host's Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). Since DBus is a secured interface we can control which processes will be allowed to access **hamd**. + +The **hamd** process will provide the following DBus APIs to create/modify/delete user and group (role) accounts: + +- **useradd**: To create new users (similar to GNU [useradd](http://man7.org/linux/man-pages/man8/useradd.8.html)) +- **userdel**: To delete a user (similar to GNU [userdel](http://man7.org/linux/man-pages/man8/userdel.8.html)) +- **passwd**: To change a user password (similar to GNU [passwd](http://man7.org/linux/man-pages/man1/passwd.1.html)) +- **set_roles**: To set a user's roles (similar to GNU [usermod](http://man7.org/linux/man-pages/man8/usermod.8.html)) +- **groupadd**: To create new groups/roles (similar to GNU [groupadd](http://man7.org/linux/man-pages/man8/groupadd.8.html)) +- **groupdel**: To delete groups/roles (similar to GNU [groupdel](http://man7.org/linux/man-pages/man8/groupdel.8.html)) + +##### 1.1.1.6.2 User management with hamd + +Applications that need to manage users (for example **click** or **klish**) can do so by using **hamd**'s DBus **ham.accounts** interface. DBus services such as **hamd** publish their interfaces. This can be retrieved and analyzed at runtime in order to understand the used implementation. The resulting introspection data is in XML format. DBus debug tools such as **qdbus** can be used to retrieve this data. At the time this document was written, the DBus XML definition for the APIs defined in the previous section was: + +> ```xml +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> ``` + +##### 1.1.1.6.3 The hamctl shell program + +A utility program, **hamctl**, is provided to make it easier for operators to interact with **hamd** from a Linux shell (e.g. bash). This is primarily a debug tool and *should not be invoked from other programs*. Programs should use the DBus interface described above. + +Users logged in at a shell terminal can control **hamd** (e.g. ask it to create or delete a user) with **hamctl**. **hamctl** is sell-documented. Simply invoke "**hamctl --help**" to get the list of commands available. + +##### 1.1.1.6.4 Name Service Switch + +In addition to providing APIs to create/modify/delete user and group (role) accounts, **hamd** also provides APIs to simply read user and group (role) accounts. Here's the list: + +- **getpwnam**: To retrieve user credentials (similar to POSIX [getpwnam](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getpwuid**: To retrieve user credentials (similar to POSIX [getpwuid](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getgrnam**: To retrieve group/role credentials (similar to POSIX [getgrnam](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) +- **getgrgid**: To retrieve group/role credentials (similar to POSIX [getgrgid](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) + +These APIs, however, are meant to be invoked through [NSS](https://en.wikipedia.org/wiki/Name_Service_Switch) (name service switch). That is, applications running in containers can simply continue invoking the standard POSIX APIs (`getpwnam()`, `getgrnam()`, etc) and a Host Account Management NSS module will ensure that the credentials get retrieved from **hamd** running on the host. The HAM NSS module (`libnss_ham.so.2`) need be installed and configured (`/etc/nsswitch.conf`) in the containers that require access to the host's Linux database. + +### 1.1.2 Configuration and Management Requirements +An interface and accompanying CLI must be developed for local user management. Local users should be configurable like any other feature: via CLI, REST, and gNMI. Additionally, users may also be created and managed via Linux commands in the Bash shell. This will add additional complexity and require a service to sync between the Redis DB and the Linux user database. + + +### 1.1.3 Scalability Requirements +Adding authentication to NBIs will result in some performance overhead, especially when doing operations involving asymmetric cryptography. Care should be taken to leverage performance-enhancing features of a protocol wherever possible. + +#### 1.1.3.1 REST Server +- Persistent HTTP connections can be used to preserve TCP sessions, thereby avoiding handshake overhead. +- TLS session resumption can be used to preserve the TLS session layer, thereby avoiding TLS handshake overhead and repeated authentication operations (which can involve expensive asymmetric cryptographic operations) +- Token-based authentication via JSON Web Tokens (JWT) will be used to preserve sessions for users who have already authenticated with password-based authentication, so that they do not need to constantly re-use their passwords. + +#### 1.1.3.2 gNMI Server +- TLS session resumption can be used to preserve the TLS session layer, thereby avoiding TLS handshake overhead and repeated authentication operations (which can involve expensive asymmetric cryptographic operations) + +#### 1.1.3.3 Translib +- Translib will cache all the user information along with the privilege and resource information to avoid the overhead of querying them every time we receive a request. +- Will rely on notification to update any change in the user information, privilege or resource information + +### 1.1.4 Warm Boot Requirements +N/A + +## 1.2 Design Overview +### 1.2.1 Basic Approach +The code will extend the existing Klish (CLI) and REST Server modules in the sonic-mgmt-framework repository. Klish will be extended to enable authentication to the REST server (depending on the ultimately chosen approach), and the REST Server will need to be extended to map transactions to a user and pass that username data to the Translib. + +The gNMI server, which currently exists in the sonic-telemetry repository, needs to support passing the username down to Translib as well. + +The Translib code (also in sonic-mgmt-framework) will be extended to support RBAC via Linux Groups. It will receive username data from the REST/gNMI NBIs and perform the Group lookup for a given user. + +For user management, a service must run on the host to sync the Redis RBAC tables with the Linux user database and vice-versa. + +### 1.2.2 Container +SONiC Management Framework, gNMI Telemetry containers + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Enterprise networks that enforce authentication for their management interfaces. + +## 2.2 Functional Description +This feature enables authentication and Role-Based Access Control (RBAC) on the REST and gNMI programmatic interfaces that are provided by the SONiC Management Framework and Telemetry containers. With respect to authentication, these programmatic interfaces will support password-based authentication with tokens, and certificate-based authentication. + +Since the Klish CLI in the management framework communicates with the REST server in the back-end, the solution will also be extended to support REST authentication. + +RBAC will be enforced centrally in the management framework, so that users accessing the system through varying interfaces will be limited to the same, consistent set of operations and objects depending on their role. Users' roles will be mapped using Linux Groups. + +Users and their role (group) assignments may be managed via Linux tools or the NBIs. + +# 3 Design +## 3.1 Overview +(TODO/DELL: Draw a picture) + +## 3.2 DB Changes +### 3.2.1 CONFIG DB +To support this the following tables will be introduced in the CONFIG DB : +* **UserTable** + + This table contains the username to role mapping needed for enforcing the authorization checks. It has the following columns : + * *user* : This is the username being authorized. This is a string. + * *tenant* : This contains the tenant with which the user is associated. This is a string + * *role* : This specifies the role associated with the username in the tenant. This is a comma separated list of strings. + The UserTable is keyed on <***user, tenant***>. + + **Note**: The UserTable will _not_ store users' salted+hashed passwords due to security concerns surrounding access restrictions to the DB; instead, that information will be maintained in `/etc/shadow` as per Linux convention. + +* **PrivilegeTable** + + This table has provides the information about the type of operations that a particular role is authorized to perform. The authorization can be performed at the granularity of a feature, feature group, or the entire system. The table has the following columns : + * *role* : The role associated with the user that is being authorized. This is a string. + * *feature* : This is feature that the role is being authorized to access. The granularity of the feature can be : + * *feature* - A logical grouping of multiple commands. If the user is authorized to access a particular feature, the column contains the tag associated with that feature. (More on tagging later. This will be implemented in Phase 2 of RBAC.) + * *feature-group* - A logical grouping of multiple features. If the user is authorized to a feature-group, the column contains the name of the feature-group. (More on feature-group later. This will be implemented in Phase 2 of RBAC.) + * *entire-system* - If the user is being granted access to the entire system, the column contains *all* + * *permissions* : Defines the permissions associated with the role. This is a string. + * *none* - This is the default permissions that a role is created with. A role associated with *none* permission cannot access any resources on the system to read, or to modify them. + * *read-only* - The role only has read access to the resources associated with the *feature*. + * *read-write* - The role has permissions to read and write (create, modify and delete) the resources associated with the *feature*. + The PrivilegeTable is keyed on <***role, feature***> + +* **ResourceTable** + (To be implemented in Phase 2) + Though the resources are statically tagged with the features that they belong to, a ResourceTable is still needed so as to allow for future extensibility. It is possible that in the future, a customer wants a more granular control over the authorization and wants to either sub partition the features or override the default tagging associated with a feature. The ResourceTable will allow for this support in the future. In the Phase 2, this table will be create using the default tagging associated with the resources. + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *feature-name* : The tag of the feature this resource belongs to. + The ResourceTable is keyed on <***resource, feature***> + +* **TenantTable** + (To be implemented in Phase 3 or when Multi-tenancy is introduced) + In most systems today, a single SONiC system will serve multiple tenants. A tenant is a group of users who have a different privileges for resource instances. As SONiC becomes multitenant, RBAC needs to account for this when authorizing users. The TenantTable is needed to enable this and has the following columns : + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *tenant* : The tenant for which the resource partitioning is being done. This is a string. + * *instances* : The instances of the *resource* allocated to this *tenant*. This is a list of instances. + The TenantTable is keyed on <***resource, tenant***> + +### 3.2.2 APP DB +N/A + +### 3.2.3 STATE DB +N/A + +### 3.2.4 ASIC DB +N/A + +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent +N/A + +### 3.3.2 Other Process +N/A + +## 3.4 SyncD +To facilitate the sync between users in the UserDB and with users in Linux, a service must run on the host that listens for both changes to `/etc/passwd` and changes to UserDB, since users can be created via either interface (UserDB via NBIs / Linux commands). + +This service runs a process that uses the POSIX [inotify APIs](http://man7.org/linux/man-pages/man7/inotify.7.html) to register for file system events like changes to `/etc/passwd` and/or `/etc/shadow`. +Another approach would be to have a process started by `systemd` on changes to `/etc/passwd` or `/etc/group`, and that process would simply reconcile the Redis DB with what is found in those files. `systemd` allows starting processes based on file create/delete/modify. + +A user can be created either via CLI or REST. It is the responsibility of this service to ensure that when the user information is added to the User DB, the appropriate user and groups are also created in the `/etc/passwd` and `/etc/groups` files. +This way, the User DB information and the Linux groups information is always in sync. + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +TBD from developer +(TODO/DELL) + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands for User Management +Users may be managed via Linux tools like `useradd`, `usermod`, `passwd`, etc. They may also be managed via configuration. + +##### username +`username password role ` -- Configures a user on the system with a given name, password, and role. +* **name** is a text string of 1-32 alphanumeric characters +* **password-string** is a text string of 1-32 alphanumeric characters +* **role-string** is a text string consisting of a role name. In the initial release, the user is recommended to use "admin" and "operator" roles, as other roles will not be supported. A text string is desired instead of keywords so that in the future, more roles may be implemented and expanded. +* Configuring another a user with the same **name** should result in modification of the existing user. + +`no username ` -- Deletes a user from the system. +* **name** is a text string of 1-32 alphanumeric characters + +#### 3.6.2.2 Show Commands +N/A + +#### 3.6.2.3 Debug Commands +N/A + +#### 3.6.2.4 IS-CLI Compliance +N/A + +### 3.6.3 REST API Support +Ability to configure local users via REST API. + +# 4 Flow Diagrams +N/A + +# 5 Error Handling +## 5.1 REST Server +The REST server should return standard HTTP errors when authentication fails or if the user tries to access a forbidden resource or perform an unauthorized activity. + +## 5.2 gNMI server +The gNMI server should return standard gRPC errors when authentication fails. + +## 5.3 CLI +Authentication errors will be handled by SSH. However, the CLI must gracefully handle authorization failures from the REST server (the authorization failure would originate from Translib of course). While the CLI will render all of the available commands to a user, the user will actually only be able to execute a subset of them. This limitation is a result of the design decision to centralize RBAC in Translib. Nevertheless, the CLI must inform the user when they attempt to execute an unauthorized command. + +## 5.4 Translib +Translib will authorize the user and when the authorization fails will return appropriate error string to the REST/gNMI server. + +If a user authenticates but is not part of one of the pre-defined groups, they will not be allowed to do anything at all on the system. + +# 6 Serviceability and Debug +All operations performed by NBIs (CLI commands, REST/gNMI operations) should be logged/audited with usernames attached to the given operation(s) performed. + +Initially, users who are remotely authenticated will share a common role-specific username, so there will be a limitation here. + +# 7 Warm Boot Support +N/A + +# 8 Scalability +See previous section 1.1.3: Scalability Requirements + +# 9 Unit Test + +### Table 3: Test Cases +| **Test Case** | **Description** | +|:--------------------------|:-------------------------------------| +| REST with password | Authenticate to REST server with username/password and perform some operations | +| REST with token | Perform subsequent operations with token, ensure username/password are not re-prompted | +| REST authorized RBAC | Perform authorized operations as both `Admin` and `Operator` via REST | +| REST unauthorized RBAC | Attempt unauthorized operations as both `Admin` and `Operator` via REST | +| CLI with password | SSH to the system with username/password and execute some commands | +| CLI with RSA | SSH to the system with pubkey and execute some commands | +| CLI authorized RBAC | SSH to the system and perform authorized commands | +| CLI unauthorized RBAC | SSH to the system and perform unauthorized commands | +| RBAC no-group | Create a user and assign them to a non-predefined group; make sure they can't perform any operations | +| gNMI authentication | Test the same authentication methods as REST, but for gNMI instead | +| gNMI authorization | Test the same authorization as REST, but for gNMI instead | diff --git a/doc/copp/SONiC_CoPP.md b/doc/copp/SONiC_CoPP.md new file mode 100644 index 0000000000..4d89656dc9 --- /dev/null +++ b/doc/copp/SONiC_CoPP.md @@ -0,0 +1,982 @@ + +# SONiC Control Plane Policing +# High Level Design Document +#### Rev 0.1 + + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + * [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.1.1 Functional Requirements](#111-functional-requirements) + * [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + * [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.1.4 Warm Boot Requirements](#114-warm-boot-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) + * [1.2.3 SAI Overview](#123-sai-overview) + * [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) + * [2.3 Limitations](#23-limitations) + * [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + * [3.2.1 CONFIG_DB](#321-config_db) + * [3.2.2 APP_DB](#322-app_db) + * [3.2.3 STATE_DB](#323-state_db) + * [3.2.4 ASIC_DB](#324-asic_db) + * [3.2.5 COUNTER_DB](#325-counter_db) + * [3.3 Switch State Service Design](#33-switch-state-service-design) + * [3.4 SAI](#34-sai) + * [3.5 CLI](#35-cli) + * [3.5.1 Data Models](#351-data-models) + * [3.5.2 Configuration Commands](#352-configuration-commands) + * [3.5.3 Show Commands](#353-show-commands) + * [3.5.4 Clear Commands](#354-clear-commands) + * [3.5.5 Debug Commands](#355-debug-commands) + * [3.5.6 Rest API Support](#356-rest-api-support) + * [3.5.7 Example Configuration](#357-example-configuration) + * [4 Flow Diagrams](#4-flow-diagrams) + + * [5 Error Handling](#5-error-handling) + * [6 Serviceability and Debug](#6-serviceability-and-debug) + * [7 Warm Boot Support](#7-warm-boot-support) + * [8 Scalability](#8-scalability) + * [9 Unit Test](#9-unit-test) + * [9.1 Config Test Cases](#91-config-test-cases) + * [9.2 Functional Test Cases](#92-functional-test-cases) + * [9.3 Scaling Test Cases](#93-scaling-test-cases) + * [9.4 Warm Boot Test Cases](#94-warm-boot-test-cases) + * [9.5 Negative Test Cases](#95-negative-test-cases) + + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + + +# Revision +| Rev | Date | Authors | Change Description | +|:---:|:-----------:|:----------------------------:|-----------------------------------| +| 0.1 | 04/20/2020 | Michael Li, Aravindu Maneti | Initial version | + + +# About this Manual +This document provides general information about SONiC Control Plane Policing (CoPP) feature implementation in SONiC. +# Scope +This document describes the functionality and high level design of the Control Plane Policing (CoPP) feature in SONiC +with a focus on management and configuration CLIs. + + +# Definition/Abbreviation +### Table 1: Abbreviations +| Term | Meaning | +|--------|--------------------------------------------------------| +| QoS | Quality of Service | +| CoPP | Control Plane Policing | +| CIR | Committed Information Rate | +| CBS | Committed Burst Size | +| PIR | Peak Information Rate | +| PBS | Peak Burst Size | +| CVL | Configuration Validation Library | + + +# 1 Feature Overview +The Control Plane Policing (CoPP) feature allows management of the network traffic flow handled by the control plane CPU. CoPP is designed to protect the CPU from getting overwhelmed by a high rate of CPU bound traffic which can affect system performance, and also to define per-traffic type priorities and rate limits to ensure that the system can receive packets in an optimal manner. + +The switch creates a default "system CoPP policy" that is statically applied to the control plane to allow the configuration of trap IDs (traffic punted to CPU), CPU queue assignments, and policers to rate limit the traffic. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + 1. Support creation of multiple CoPP classifiers with protocol trap match types + 2. Support applying multiple CoPP classifiers to the system CoPP policy. + 3. Support configuration of CoPP policy actions such as assignment of the protocol trap to a specific CPU queue. + 4. Support addition and configuration of a policer to a CoPP policy flow to rate limit the bandwidth of each protocol trap matched by the classifier. + +### 1.1.2 Configuration and Management Requirements +1. Provide commands to support configuration CoPP classifiers and application to the system CoPP policy. + + +### 1.1.3 Scalability Requirements +1. Support a CoPP classifier for each protocol trap supported by SONiC. This allows configuration of a separate policer with a unique rate limit value for each protocol trap. (Max scaling is detailed in scaling section) +2. Multiple CoPP classifiers can be assigned to the same CPU queue. The max number of CPU queues will be silicon specific. + +### 1.1.4 Performance Requirements +1. The max aggregate packet rate should be tuned to protect the CPU from utilizing too much CPU bandwidth (**TODO: provide guidance on what is "too much CPU utilization"**) + +### 1.1.5 Warm Boot Requirements +CoPP functionalities are not expected to work during warm reboot but will be restored after warm reboot. Traffic to CPU will get dropped when CPU is reset until SAI drivers have restored the CPU pkt path. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +The SONiC management framework provide CLIs and interfaces for users to create CoPP classifiers, apply them to the system CoPP policy, and modify the associated trap actions and policers. A single system CoPP policy (which cannot be deleted) exists with a default configuration. The system CoPP policy is set in the SONiC CONFIG_DB COPP tables. + +The CoPP manager daemon handles the CoPP policy that the management framework has set in the SONiC CONFIG_DB COPP_TABLE. The CoPP mgmt daemon will apply each CONFIG_DB COPP_TABLE entry to the APP_DB COPP_TABLE. + +The CoPP Orchestration agent handles changes to the APP_DB COPP_TABLE (existing functionality). Each entry in the table provides configuration of several attributes such as: +- trap punt to the control plane CPU +- trap CPU queue assignment +- trap policer rate limiting + +Multiple CoPP classifiers can be created and applied to the system CoPP policy. Multiple protocol trap matches can be configured in a single CoPP classifier but the default CoPP configuration (see creating CoPP classifiers section) assigns each SONiC trap to a separate CoPP classifier with a unique CPU queue and policer. This provides several benefits: +- Ability to dynamically change policer values without affecting other traps +- Separate each CPU traffic type to its own CPU queue +- Provides HW level CPU queue stats for each CPU traffic type + +### 1.2.2 Container +CoPP manager and orchagent daemons reside in the SWSS container. + +### 1.2.3 SAI Overview +Existing host interface and policer SAI APIs are used to support the CoPP feature: + +https://github.com/opencomputeproject/SAI/blob/master/inc/saihostif.h
+https://github.com/opencomputeproject/SAI/blob/master/inc/saipolicer.h
+ + +# 2 Functionality + +## 2.1 Target Deployment Use Cases +See Feature Overview + +## 2.2 Functional Description + +See section 1.2.1 + +## 2.3 Limitations +1. Modifying CoPP classifier action attributes (ie queue, policer rates) causes service disruption to trap protocols assigned to the CoPP classifier (COPP_TABLE entry is deleted, then added) + +# 3 Design +## 3.1 Overview +A new CoPP manager is introduced to handle CONFIG_DB changes (ie from SONiC mgmt). When protocol classifiers are applied to the system CoPP policy, the CoPP manager will propagate the config to APP_DB for consumption by CoPP orchagent to program the traps and policers in hardware. + +## 3.2 DB Changes + +### 3.2.1 CONFIG_DB +Two COPP tables with fields similar to the APP_DB COPP_TABLE table is introduced in CONFIG_DB to configure CoPP parameters. In the rest of the document, the two tables will be referred to collectively as the CONFIG_DB COPP_TABLE. + +#### 3.2.1.1 COPP_TRAP table +The COPP_TRAP table allows configuration of a CoPP trap's packet match conditions such as specific trap IDs and feature specific trap attributes. A COPP_TRAP entry can reference a COPP_GROUP (trap_group field) that defines the actions taken when a packet hits the CoPP trap match condition. + +``` +key = "COPP_TRAP|name" +trap_ids = name ; list of trap ids (ie: bgp, lacp, arp, etc) +trap_group = ref_hash_key_reference ; reference to COPP_GROUP table entry + +;Feature specific attributes (sFlow) +genetlink_name = name ; "psample" for sFlow +genetlink_mcgrp_name = group_name; "packets" for sFlow +``` +#### 3.2.1.1.1 Feature specific COPP_TRAP attributes +Feature specific attributes (currently only genetlink parameters for sFlow) will be exposed in CONFIG_DB COPP_TRAP for schema backward compatibility but will not be configurable from mgmt and CLI. The CoPPMgr daemon will be responsible for adding these attributes to the COPP_TRAP table when receiving feature specific trap_ids such as "sflow". + +#### 3.2.1.2 COPP_GROUP +The COPP_GROUP table allows configuration of the CoPP trap's action attributes such as queue mappings, packet action, and policer attributes. +``` +key = "COPP_GROUP:name" +queue = number ; CPU queue +trap_action = packet_action; trap, drop, copy (see packet_action_map) +trap_priority = priority of trap when packet has multiple matches + +; settings for the embedded policer +meter_type = "packets" | "bytes" +mode = "sr_tcm" | "tr_tcm" | "storm" +color = "aware" | "blind" +green_action = packet_action +yellow_action = packet_action +red_action = packet_action +cir = number ; guaranteed rate in pps or bytes/sec +cbs = number ; guaranteed burst size in packets or bytes +pir = number ; max rate in pps or bytes/sec +pbs = number ; max burst size in packets or bytes +``` + +### 3.2.2 APP_DB + +#### 3.2.2.1 COPP_TABLE +No changes are introduced in "COPP_TABLE" in APP_DB (allows backward compatibility). + + key = "COPP_TABLE:name" + name_list = name | name,name_list + queue = number; strict queue priority. Higher number means higher priority. + trap_ids = name_list; Acceptable values: bgp, lacp, arp, lldp, snmp, ssh, ttl error, ip2me + trap_action = packet_action; trap action which will be applied to all trap_ids. + + ;Settings for embedded policer. NOTE - if no policer settings are specified, then no policer is created. + meter_type = "packets" | "bytes" + mode = "sr_tcm" | "tr_tcm" | "storm" + color = "aware" | "blind" + cbs = number ;packets or bytes depending on the meter_type value + cir = number ;packets or bytes depending on the meter_type value + pbs = number ;packets or bytes depending on the meter_type value + pir = number ;packets or bytes depending on the meter_type value + green_action = packet_action + yellow_action = packet_action + red_action = packet_action + + ;Feature specific attributes (sFlow) + genetlink_name = name ; "psample" for sFlow + genetlink_mcgrp_name = group_name; "packets" for sFlow + +### 3.2.3 STATE_DB +COPP_TRAP_TABLE and COPP_GROUP_TABLE are introduced in STATE_DB to set the entry state. Feature driven COPP config could delay COPP processing until the feature is enabled. This allows applications to check the state of COPP entry processing. + +``` +key = "COPP_GROUP_TABLE|name" +state = "ok" + +key = "COPP_TRAP_TABLE|name" +state = "ok" +``` + +### 3.2.4 ASIC_DB +No changes are introduced in ASIC_DB. +### 3.2.5 COUNTER_DB +No changes are introduced in COUNTER_DB. + +See the "show queue counter" in the CLI section on the usage monitor CPU queue counters. + +## 3.3 Switch State Service Design +### 3.3.1 CoPPMgr +Introduce a new CoPP manager, that subscribes for the CONFIG_DB CoPP Tables and Feature Tables. Based on the feature enablement, CoPPMgr handles the logic to resolve whether a CoPP table entry shall be written to APP_DB for orchagent consumption. Similar to existing swss managers, an entry with state "ok" shall be added to STATE_DB. + +Copp Manager daemon is responsible for the following activities: +- Subscribes to COPP_TABLE notifications and FEATURE table notifications in CONFIG_DB. +- Backend checks on syntax and field validations for COPP_TABLE in CONFIG_DB. +- Copy the COPP_TABLE entries from CONFIG_DB to APP_DB. After the entry is copied to APP_DB, set the STATE_DB COPP_TABLE state attribute to "ok". +- When an entry is removed from the CONFIG_DB COPP_TABLE, remove the associated CoPP entry in the APP_DB and STATE_DB. + + + +### 3.3.2 CoPPOrch +CoPPOrch shall only be a consumer of APP DB CoPP Table. It is not expected to handle feature logic and the current handling of features like sFlow, NAT shall be revisited and removed to be added as part of CoPPMgr + +CoPP Orchestration agent is responsible for the following activities: + - Subscribes to notifications on COPP_TABLE in APP_DB. + - Calls hostif SAI APIs to create host interface traps with parameters from COPP_TABLE entries + - Calls policer SAI APIs to create policers with parameters from COPP_TABLE entries + + +### 3.3.3 Default CoPP Tables +The default CoPP configuration is stored in JSON files that are loaded by the swssconfig utility when the swssconfig init bash script is started by the system supervisord daemon during bootup. + +The [swssconfig.sh script](https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-orchagent/swssconfig.sh) and [swssconfig utility](https://github.com/Azure/sonic-swss/blob/master/swssconfig/swssconfig.cpp) loads the following CoPP related JSON files from the default SWSS_CONFIG_DIR directory in the swss docker into the APP_DB (bypasses CONFIG_DB). + + /etc/swss/config.d/early-copp.config.json + /etc/swss/config.d/00-copp.config.json + +The 00-copp.config.json file (located at swssconfig/sample/00-copp.config.json) shall be modified to be compatible to CONFIG_DB schema (remove "OP" attributes used by `swssconfig` utility). Instead of using the `swssconfig` utility to load the CoPP file directly into APP_DB, the `swssconfig.sh` script will be modified to use `sonic-cfggen` script to load the CoPP JSON file to CONFIG_DB. + +``` +sonic-cfggen -j /etc/swss/config.d/00-copp.config.json --write-to-db +``` + +The swssconfig.sh script loads the default configuration to CONFIG_DB. Some time later, the function postStartAction() in the docker control script (docker_image_ctl.j2) loads the config_db.json. Any user defined CoPP configuration will overwrite the defaults loaded by the swssconfig.sh script earlier. + +``` + function postStartAction() + ... + if [ -r /etc/sonic/config_db.json ]; then + sonic-cfggen -j /etc/sonic/config_db.json --write-to-db + fi +``` +To handle potential merge conflicts when adding user defined CoPP configuration over the default CoPP configuration, the following limitations are imposed: +* The system CoPP classifiers cannot be deleted but the match conditions can be modified. Changes to the system CoPP classifiers is a modify operation on an existing COPP_TABLE entry. +* A protocol trap may only be added to a single CoPP classifier. This prevents a trap from being in both default and user defined CoPP configurations. + +### 3.3.3 Early CoPP Config JSON file +To address an issue with CoPP config JSON file processing delayed during fast_reboot processing of a scaled number of entries in FDB and ARP tables, an "early CoPP" JSON file is introduced to process traps before loading FDB and ARP tables. The early CoPP JSON file will be loaded directly to APP_DB. + +The same early CoPP entries will be loaded in CONFIG_DB later as well in postStartAction(). CoPPMgrd will prevent writing to APP_DB when there are no changes between CONFIG_DB and APP_DB entries. + +`/sonic-buildimage/dockers/docker-orchagent/swssconfig.sh` +``` +# Early COPP processing +SWSSCONFIG_EARLY_COPP="early-copp.config.json" +if [[ -f /etc/swss/config.d/$SWSSCONFIG_EARLY_COPP ]]; then + echo "Early COPP processing" + swssconfig /etc/swss/config.d/$SWSSCONFIG_EARLY_COPP + sleep 1 +fi + +# Restore FDB and ARP table ASAP +fast_reboot +``` + +## 3.4 SAI +No new SAI changes have been made to support this feature. The SAI attributes have been added here for reference. + +###### Table 2: Host Interface Group SAI attributes +| HostIF Group parameter | SAI attributes | +|--------------------------|--------------------------------------------------------| +| CPU Queue | SAI_HOSTIF_TRAP_GROUP_ATTR_QUEUE | +| Trap Action | SAI_HOSTIF_TRAP_GROUP_ATTR_PACKET_ACTION | +| Trap Priority | SAI_HOSTIF_TRAP_GROUP_ATTR_TRAP_PRIORITY | +| Policer ID | SAI_HOSTIF_TRAP_GROUP_ATTR_POLICER | + +###### Table 3: Host Interface trap SAI attributes +| Trap parameter | SAI attributes | +|--------------------------|--------------------------------------------------------| +| Type | SAI_HOSTIF_ATTR_TYPE | +| Name | SAI_HOSTIF_ATTR_NAME | +| Generic Netlink Name | SAI_HOSTIF_ATTR_GENETLINK_MCGRP_NAME | +| Name | SAI_HOSTIF_TRAP_NAME | + +###### Table 4: Policer SAI attributes +| Policer parameters | SAI attributes | +|----------------------------|------------------------------------------------------| +| Meter type | SAI_POLICER_ATTR_METER_TYPE | +| Mode | SAI_POLICER_ATTR_MODE | +| CIR | SAI_POLICER_ATTR_CIR | +| CBS | SAI_POLICER_ATTR_CBS | +| PIR | SAI_POLICER_ATTR_PIR | +| PBS | SAI_POLICER_ATTR_PBS | +| PBS | SAI_POLICER_ATTR_PBS | +| Color | SAI_POLICER_ATTR_COLOR_SOURCE | +| Green action | SAI_POLICER_ATTR_GREEN_PACKET_ACTION | +| Yellow action | SAI_POLICER_ATTR_YELLOW_PACKET_ACTION | +| Red action | SAI_POLICER_ATTR_RED_PACKET_ACTION | + + + +## 3.5 Manageability + +### 3.5.1 Data Models +The user facing COPP data model will build on top of the SONiC ACL flow based management models (based on OpenConfig ACL YANG model). + +The backend data model (SONiC YANG) will use the COPP_TABLE format in CONFIG_DB. See CONFIG_DB changes section. + +### 3.5.2 Configuration Commands + +The following KLISH CLI commands are used to configure CoPP policies. The CLI syntax follows the SONiC ACL flow based management (based on OpenConfig YANG model) and is also follows industry-standard CLIs
+ +#### 3.5.2.1 Creating CoPP Classifers + +| Mode | Config | +| ---- | --------------------------------------------------- | +| Syntax | sonic(config)# [**no**] **classifier** *NAME* **match-type** **copp** || +| Arguments | ***NAME***: String of 1-63 characters in length. Must begin with a alpha numeric character. Rest of the characters can be alpha numeric or hyphen (-) or underscore (\_). | + +The CoPP classifier name maps directly to a COPP_TRAP entry key in the CONFIG_DB. For example: +``` +sonic(config)# classifier copp-system-arp match-type copp + +CONFIG_DB: +{ + "COPP_TRAP:copp-system-arp": { + ... + }, +}, +``` +Table 1 shows a list of CoPP classifiers created by the system and the default CoPP policy configuration. + +The scaling notes provide hints on the derivation of default rate limits. Latency sensitive protocols are set to higher rates. Rates are subject to change based on performance and system testing. + +**NOTE**: The system CoPP classifiers cannot be deleted but the match conditions can be modified. User created CoPP classifiers can be deleted. + +**Table 1: Default system classifiers in CoPP policy** + +| Classifier | Queue | Rate (PPS) | Protocols | Scaling Notes +|----------------------|-------|--------------|-----------|------ +| copp-system-lacp | 25 | 1000 | lacp | 1 sec timer/port +| copp-system-udld | 24 | 1000 | udld | 1 sec timer/port +| copp-system-stp | 23 | 16000 | stp,pvrst | 16K port-vlan instances +| copp-system-bfd | 22 | 5000 | bfd,bfdv6 | 64x100ms sessions +| copp-system-ptp | 21 | 16000 | ptp | 16pps/port max +| copp-system-lldp | 20 | 1000 | lldp | 5 sec/port min +| copp-system-vrrp | 19 | 5000 | vrrp,vrrpv6 | +| copp-system-iccp | 18 | 5000 | iccp | +| copp-system-ospf | 17 | 10000 | ospf | +| copp-system-bgp | 16 | 10000 | bgp,bgpv6 | minimize convergence of high scale routes +| copp-system-pim | 15 | 10000 | pim | +| copp-system-igmp | 14 | 6000 | igmp_query| +| reserved | 13 | N/A | | +| reserved | 12 | N/A | | +| copp-system-suppress | 11 | 5000 | arp_suppress,nd_suppress | +| copp-system-arp | 10 | 6000 | arp_req,arp_resp,neigh_discovery | L3 perf tests +| copp-system-dhcp | 9 | 1000 | dhcp,dhcpv6 | +| copp-system-icmp | 8 | 1000 | icmp,icmpv6 | +| copp-system-ip2me | 7 | 6000 | ip2me | L3 perf, ZTP inband downloads +| copp-system-subnet | 6 | 6000 | subnet | L3 perf tests +| copp-system-nat | 5 | 600 | src_nat_miss,dest_nat_miss | limit exception case +| copp-system-mtu | 4 | 500 | l3_mtu_error | limit exception case +| copp-system-sflow | 3 | 16000 | sample_packet | detect large flows within 1 sec +| reserved | 2 | N/A | +| reserved | 1 | N/A | +| copp-system-default | 0 | 100 | any + + +#### 3.5.2.2 Adding match protocol traps to the CoPP Classifers +| Mode | Classifier| +| ------ | ----------------------------------------------------------- | +| Syntax | SONiC(config-classifier)# [**no**] **match protocol** *TRAP_ID* | +| Arguments | ***TRAP_ID***: See [trap_id_map](https://github.com/Azure/sonic-swss/blob/cdffff315f10f79d4f68668540cb80c32d84c9ea/orchagent/copporch.cpp#L39) and user_trap_id_map for valid values. Some examples include arp, bgp, lacp, lldp, ip2me. This is a mandatory field. + +**Note**: Modifying match conditions while the CoPP classifier is assigned to the system CoPP policy will cause service disruption to all traps assigned to the CoPP classifier (The COPP_TABLE entry will be removed, modified, and added back to APP_DB) + +**Note**: A protocol trap ID may only be added to a single CoPP classifier **(TODO: custom CVL logic to check this? or release note as invalid configuration).** + +Example config and corresponding entry written into the COPP_TRAP in CONFIG_DB +``` +sonic(config)# classifier copp-system-arp match-type copp +sonic(config-classifier)# match protocol arp_req +sonic(config-classifier)# match protocol arp_resp +sonic(config-classifier)# match protocol neigh_discovery + +CONFIG_DB: +{ + "COPP_TRAP:copp-system-arp": { + "trap_ids": "arp_req,arp_resp,neigh_discovery", + ... +}, +``` + +#### 3.5.2.3 Creating the CoPP Action Group + +| Mode | Config | +| ---- | --------------------------------------------------- | +| Syntax | sonic(config)# [**no**] **copp-action** *NAME* | +| Arguments | ***NAME***: String of 1-63 characters in length. Must begin with a alpha numeric character. Rest of the characters can be alpha numeric or hyphen (-) or underscore (\_). | + +The CoPP action group name maps directly to a COPP_GROUP entry key in the CONFIG_DB. For example: +``` +sonic(config)# copp-action copp-system-arp + +CONFIG_DB: +{ + "COPP_GROUP:copp-system-arp": { + ... + }, +}, +``` +**NOTE**: A default CoPP action group called "default" is created by the system and cannot be removed. This is used for the default CoPP action for unclassified traffic. The default CoPP action config is shown below. + +``` +"COPP_GROUP|default": { + "queue": "0", + "meter_type":"packets", + "mode":"sr_tcm", + "cir":"100", + "cbs":"100", + "red_action":"drop" +}, + +``` + +#### 3.5.2.3.1 Configuring the CoPP trap action +Optional configuration. Default trap action is "trap". + +| Mode | action +|-------------|------------------------------------------------------ +| Syntax | sonic(config-action)# [**no**] **set trap-action** *TRAP_ACTION* +| Arguments | ***TRAP_ACTION***: See [packet_action_map](https://github.com/Azure/sonic-swss/blob/cdffff315f10f79d4f68668540cb80c32d84c9ea/orchagent/copporch.cpp#L81): drop, forward, copy, copy_cancel, trap, log, deny, transit. + +The packet action maps directly to SAI Host interface packet action types. The [SAI switch packet action type](https://github.com/opencomputeproject/SAI/blob/befd00861459e343990a1e8b2acd8edc6be20b4f/inc/saiswitch.h#L68) describes the behavior of each action. + +#### 3.5.2.3.2 Configuring the CoPP trap action priority +Optional configuration. Default is same as CPU queue value. +| Mode | Policy +| ------ | ------ | +| Syntax | SONiC(config-policy)# [**no**] **set trap-priority** *PRIORITY* | +| Arguments | ***PRIORITY***: Priority number in range 0-1023. The trap_priority refers to the priority of a classifier rule when a packet has multiple matches. | + +#### 3.5.2.3.3 Configuring the CoPP action CPU queue +Optional configuration. Default CPU queue is 0. + +| Mode | action +|-------------|------------------------------------------------------ +| Syntax | sonic(config-action)# [**no**] **set trap-queue** *QUEUE_ID* +| Arguments | ***QUEUE_ID***: CPU queue value in range of 0-47. + +#### 3.5.2.3.4 Configuring the CoPP action policer +Optional configuration. No policer means no rate limit for this CoPP class flow. + +| Mode | action +|-------------|------------------------------------------------------ +| Syntax | sonic(config-action)# [**no**] **police** **cir** *CIR* [**cbs** *CBS*] [**pir** *PIR*] [**pbs** *PBS*] | +| Arguments | ***CIR***: Committed information rate in bps or pps (packets per second) depending on meter-type. CIR is mandatory.
***CBS***: Committed burst size in packets or bytes. The default value is 20% of the CIR. If configured by the user, it must be greater than or equal to CIR in packets or bytes.
***PIR***: Peak information rate in bps or pps. If configured by the user, it must be greater than CIR.
***PBS***: Peak burst size. The default value is 20% of the PIR value. If configured by the user, it must be greater than PIR and CBS value.| + +#### 3.5.2.3.5 Configuring the CoPP action policer meter type +Optional configuration with defaults specified in the table. + +| Mode | action +|-------------|------------------------------------------------------ +| Syntax | sonic(config-action)# [**no**] **police** [**meter-type** <**pps** \| **bps**>] +| Arguments | ***pps*** (default): packets per second
***bps***: bits per second + +#### 3.5.2.3.6 Configuring the CoPP action policer mode +Optional configuration with defaults specified in the table. + +| Mode | action +|-------------|------------------------------------------------------ +| Syntax | sonic(config-action)# [**no**] **police** [**mode** <**sr_tcm** \| **tr_tcm** \| **storm**> [**green** *TRAP_ACTION*] [**yellow** *TRAP_ACTION*] **red** *TRAP_ACTION*] +| Arguments | ***sr_tcm*** (default): Single Rate Three color marker, CIR, CBS and PBS, G, Y and R
***tr_tcm***: Two Rate Three color marker, CIR, CBS, PIR and PBS, G, Y and R
***storm***: Storm control mode. Single Rate Two color CIR, CBS, G and R
***TRAP_ACTION***: See [packet_action_map](http://10.59.132.240:8010/sonic_3.x/xref/sonic-swss/orchagent/copporch.cpp#92): drop, forward, copy, copy_cancel, trap, log, deny, transit.
Default green trap-action is "trap".
Default yellow trap-action is "trap".
Default red trap-action is "drop". + + +#### 3.5.2.4 Editing the CoPP Policy + +| Mode | Config +|-------------|------------------------------------------------------ +| Syntax | sonic(config)# **policy** **copp-system-policy** **type** **copp** +| Arguments | ***copp-system-policy***: Control Plane CoPP policy + +The only control plane policy is **copp-system-policy**, which cannot be deleted. The default **copp-system-policy** consists of the system classifers listed in table 1. + + +#### 3.5.2.5 Adding a CoPP classifier to the CoPP Policy +| Mode | Policy +| ------ | ------ | +| Syntax | SONiC(config-policy)# [**no**] **class** *NAME* | +| Arguments | ***NAME***: CoPP Classifier name. String of 1-63 characters in length. Must begin with a alpha numeric character. Rest of the characters can be alpha numeric or hyphen (-) or underscore (\_).| + +**NOTE**: The copp-system-default classifier (unclassified CPU traffic) always exists in the system CoPP policy and cannot be removed from the copp-system-policy. + +#### 3.5.2.6 Binding a CoPP action to the CoPP classifier +Binds CoPP action group to a classifier match group. CoPP action is mandatory when adding a CoPP classifier to the CoPP policy. +| Mode | Flow +| ------ | ------ | +| Syntax | SONiC(config-policy-flow)# **set copp-action** *NAME* | +| Arguments | ***NAME***: CoPP Action group name. String of 1-63 characters in length. Must begin with a alpha numeric character. Rest of the characters can be alpha numeric or hyphen (-) or underscore (\_).| + +This CLI triggers the CoPPMgrd to act on processing the COPP_TRAP and COPP_GROUP entries in CONFIG_DB and add/merge them to the corresponding COPP_TABLE in APP_DB. + +* Search for the COPP_GROUP name in the APP_DB COPP_TABLE and create the entry if it doesn't exist. Add the COPP_GROUP parameters to the COPP_TABLE. +* Add the traps from the COPP_TRAP table to the APP_DB COPP_TABLE trap_list if they don't already exist in the APP_DB COPP_TABLE trap_list. + +An example configuration is below: +``` +sonic(config)# policy copp-system-policy type copp +sonic(config-policy)# class copp-system-arp +sonic(config-policy-flow)# set copp-action copp-system-arp + +CONFIG_DB: +{ + ... + "COPP_TRAP:copp-system-arp": { + ... + "trap_group": "copp-system-arp" + }, + ... +}, +``` + +#### 3.5.2.7 CoPP Policy Binding +The **copp-system-policy** policy map is always applied to the CPU control plane. No commands are available to add or remove this assignment. + +#### 3.5.2.8 CoPP Configuration Examples + +##### 3.5.2.8.1 Configuring ARP related CoPP classifier + +``` +sonic(config)# classifier copp-system-arp match-type copp +sonic(config-classifier)# match protocol arp_req +sonic(config-classifier)# match protocol arp_resp +sonic(config-classifier)# match protocol neigh_discovery +sonic(config-classifier)# exit + +CONFIG_DB: +{ + ... + "COPP_TRAP:copp-system-arp": { + "trap_ids": "arp_req,arp_resp,neigh_discovery" + }, + ... +}, +``` +##### 3.5.2.8.2 Configuring ARP related CoPP actions + +``` +sonic(config)# copp-action copp-system-arp +sonic(config-action)# set trap-action copy +sonic(config-action)# set trap-priority 3 +sonic(config-action)# set trap-queue 3 +sonic(config-action)# police meter-type pps +sonic(config-action)# police mode sr_tcm red drop +sonic(config-action)# police cir 6000 cbs 6000 +sonic(config-action)# exit + +CONFIG_DB: +{ + ... + "COPP_GROUP:copp-system-arp": { + "trap_action":"copy", + "trap_priority":"3", + "queue": "3", + "meter_type":"packets", + "mode":"sr_tcm", + "cir":"6000", + "cbs":"6000", + "red_action":"drop" + }, + ... +}, +``` + +##### 3.5.2.8.3 Binding ARP classifier and action to CoPP system policy +``` +sonic(config)# policy copp-system-policy type copp +sonic(config-policy)# class copp-system-arp +sonic(config-policy-flow)# set copp-action copp-system-arp +sonic(config-policy-flow)# exit + +CONFIG_DB: +{ + ... + "COPP_TRAP:copp-system-arp": { + "trap_ids": "arp_req,arp_resp,neigh_discovery", + "trap_group": "copp-system-arp" + }, + ... +}, +``` + +### 3.5.3 Show Commands + +#### 3.5.3.1 show CoPP classifiers +Dumps the configured CoPP classifiers +- Example: +``` +sonic# show classifier match-type copp +Classifier copp-system-bgp match-type copp + protocol bgp + protocol bgpv6 +Classifier copp-system-arp match-type copp + protocol arp_req + protocol arp_resp + protocol neigh_discovery +... +``` +An alternate non-ISCLI version of the command keeps all COPP related show commands under "show copp ..." +``` +sonic# show copp classifiers +``` + +#### 3.5.3.2 show CoPP protocols +Dumps a list of supported CoPP classifier protocol trap IDs +- Example: +``` +sonic# show copp protocols +Classifier match-type copp protocols + protocol bgp + protocol bgpv6 + protocol arp_req + protocol arp_resp + protocol neigh_discovery +... +``` +#### 3.5.3.1 show CoPP action groups +Dumps the configured CoPP action groups +- Example: +``` +sonic# show copp actions +CoPP action group copp-system-bgp + trap-action trap + trap-priority 5 + trap-queue 5 + police cir 10000 cbs 10000 + meter-type: packets + mode: sr_tcm + red-action: drop +CoPP action group copp-system-arp + trap-action: trap + trap-priority: 4 + trap-queue: 4 + police cir 6000 cbs 6000 + meter-type: packets + mode: sr_tcm + red-action: drop +... +``` +#### 3.5.3.3 show CoPP policy +Dumps the CoPP policy **copp-system-policy**. +``` +sonic# show policy type copp +Policy copp-system-policy Type copp + Flow copp-system-bgp + Action copp-system-bgp + trap-action trap + trap-priority 5 + trap-queue 5 + police cir 10000 cbs 10000 + meter-type: packets + mode: sr_tcm + red-action: drop + Flow copp-system-arp + Action copp-system-arp + trap-action: trap + trap-priority: 4 + trap-queue: 4 + police cir 6000 cbs 6000 + meter-type: packets + mode: sr_tcm + red-action: drop + ... +``` +An alternate non-ISCLI version of the command keeps all COPP related show commands under "show copp ..." +``` +sonic# show copp policy +``` +##### 3.5.3.4.1 Trap statistics enhancement +In a future enhancement, add policer statistics to each classifier flow output. + + +#### 3.5.3.5 show queue counters CPU +Show queue counters CLI can be used to monitor CPU queuing and trap policing functions. +- Usage: +``` + show queue counters CPU [queue ] +``` +- Example: +``` +sonic# show queue counters CPU + Port TxQ Counter/pkts Counter/bytes Drop/pkts Drop/bytes +------ ----- -------------- --------------- ----------- ------------ + CPU MC0 0 0 0 0 + CPU MC1 0 0 0 0 + CPU MC2 0 0 0 0 + CPU MC3 0 0 0 0 + CPU MC4 0 0 0 0 + CPU MC5 0 0 0 0 + CPU MC6 0 0 0 0 + CPU MC7 0 0 0 0 + CPU MC8 0 0 0 0 + CPU MC9 0 0 0 0 + CPU MC10 0 0 0 0 + CPU MC11 0 0 0 0 + CPU MC12 0 0 0 0 + CPU MC13 0 0 0 0 + CPU MC14 0 0 0 0 + CPU MC15 0 0 0 0 + CPU MC16 0 0 0 0 + CPU MC17 0 0 0 0 + CPU MC18 0 0 0 0 + CPU MC19 0 0 0 0 + CPU MC20 0 0 0 0 + CPU MC21 0 0 0 0 + CPU MC22 0 0 0 0 + CPU MC23 0 0 0 0 + CPU MC24 0 0 0 0 + CPU MC25 0 0 0 0 + CPU MC26 0 0 0 0 + CPU MC27 0 0 0 0 + CPU MC28 0 0 0 0 + CPU MC29 0 0 0 0 + CPU MC30 0 0 0 0 + CPU MC31 0 0 0 0 + CPU MC32 0 0 0 0 + CPU MC33 0 0 0 0 + CPU MC34 0 0 0 0 + CPU MC35 0 0 0 0 + CPU MC36 0 0 0 0 + CPU MC37 0 0 0 0 + CPU MC38 0 0 0 0 + CPU MC39 0 0 0 0 + CPU MC40 0 0 0 0 + CPU MC41 0 0 0 0 + CPU MC42 0 0 0 0 + CPU MC43 0 0 0 0 + CPU MC44 0 0 0 0 + CPU MC45 0 0 0 0 + CPU MC46 0 0 0 0 + CPU MC47 0 0 0 0 +``` +#### 3.5.3.6 show running-config copp +Dumps the CoPP running configuration +``` +sonic# show running-config copp +! +copp-action copp-system-bgp + set trap-action trap + set trap-queue 5 + police meter-type pps + police mode sr_tcm red drop + police cir 10000 cbs 10000 +! +copp-action copp-system-arp + set trap-action copy + set trap-queue 3 + police meter-type pps + police mode sr_tcm red drop + police cir 6000 cbs 6000 +! +policy copp-system-policy type copp + class copp-system-bgp + copp-action copp-system-bgp + class copp-system-arp + copp-action copp-system-arp + ... +``` + +### 3.5.4 Clear Commands +- Usage: +``` + clear queue counters CPU [queue ] +``` + +#### 3.5.5 IS-CLI Compliance +The following table maps SONIC CLI commands to corresponding IS-CLI commands. The compliance column identifies how the command comply to the IS-CLI syntax: +- **IS-CLI drop-in replace** ? meaning that it follows exactly the format of a pre-existing IS-CLI command. +- **IS-CLI-like** ? meaning that the exact format of the IS-CLI command could not be followed, but the command is similar to other commands for IS-CLI (e.g. IS-CLI may not offer the exact option, but the command can be positioned is a similar manner as others for the related feature). +- **SONIC** - meaning that no IS-CLI-like command could be found, so the command is derived specifically for SONIC. + + +| CLI Command | Compliance | IS-CLI Command
(if applicable) | Reference to IS-CLI cmd
(if applicable) | +| -------------- | --------- | ----------------------------------------------- | ------------------------------------------------------------ | +| **classifier** *NAME* **match-type** **copp** | IS-CLI-like | **class-map type copp match-any** *NAME* | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands#ww1154073) | +| **match protocol** *TRAP_ID* | IS-CLI-like | **match protocol** *TRAP_ID* | [Control Plane Config](https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/15-mt/qos-plcshp-15-mt-book/qos-plcshp-cpp.html) | +| **policy** **copp-system-policy** **type** **copp** | IS-CLI-like | **policy-map type copp** **copp-system-policy** | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands) | +| **class** *NAME* | IS-CLI-like | **class** *NAME* | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands) | +| **copp-action** *NAME* | SONiC | N/A | N/A | +| **set trap-action** *TRAP_ACTION* | SONIC | N/A | N/A | +| **set trap-queue** *QUEUE_ID* | IS-CLI-like | **set traffic class** *QUEUE_ID* | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands) | +| **police** **cir** *CIR* [**cbs** *CBS*] [**pir** *PIR*] [**pbs** *PBS*] | IS-CLI-like | **police** **cir** *CIR* [**cb** *CBS*] | [Control Plane Config](https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_010001.html#task_1082584) | +| **police** [**meter-type** <**pps** \| **bps**>] | IS-CLI-like | **police** **cir** [**pps\|kbps**] *CIR* [**cb** *CBS*] | [Control Plane Config](https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_010001.html#task_1082584) | +| **police** [**mode** <**sr_tcm** \| **tr_tcm** \| **storm**> [**green** *TRAP_ACTION*] [**yellow** *TRAP_ACTION*] **red** *TRAP_ACTION*] | IS-CLI-like | N/A | N/A +| show classifier match-type copp | IS-CLI-like | show class-map type control-plane *NAME* | [Control Plane Config](https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_010001.html#task_1082584) | +| show copp protocols | SONIC | N/A | N/A | +| show copp actions | SONIC | N/A | N/A | +| show policy type copp | IS-CLI-like | show policy-map copp copp-system-policy | [Control Plane Config](https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/security/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Security_Configuration_Guide_chapter_010001.html#task_1082584) | +| show queue counters CPU [queue ] | IS-CLI-like | show policy-map copp copp-system-policy | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands) | +| clear queue counters CPU [queue ] | IS-CLI-like | clear policy-map interface control-plane counters copp-system-policy | [Control Plane Config](https://www.arista.com/en/um-eos/eos-section-30-7-traffic-management-configuration-commands) + +### 3.5.5 Restrictions +* A protocol trap may only be added to a single CoPP classifier. Adding traps to multiple CoPP classifiers that are added to the CoPP policy is user error. + +### 3.5.5 Debug Commands +Not applicable +### 3.5.6 REST API Support +The COPP data model will build on top of the SONiC ACL flow based management models (based on OpenConfig ACL YANG model). + +### 3.5.7 Example Configuration + +# 4 Flow Diagrams + +## 4.1 Initial Config +The following flow captures scenarios for ```boot-up``` sequence and ```config reload```. Default CoPP Tables shall be present in ```00-copp.config.json``` and if the same entry is present in ```config_db.json```, it is expected to be overwritten by ```config_db``` entry. This model ensures user-configuration gets priority over default configuration. + +![Init CoPP Config](../../images/copp/copp_flow_init.png "Figure 1: Init CoPP Config") + +## 4.2 CoPP Manager flow +The following flow captures CoPP management functionality. + +![User Driven CoPP Config](../../images/copp/copp_flow_user.png "Figure 2: User Driven CoPP Config") + + +Corresponding CLI: +``` +sonic(config)# classifier class-user match-type copp +sonic(config-classifier)# match protocol user_trap +sonic(config-classifier)# exit + +sonic(config)# copp-action action-user +sonic(config-action)# set trap-queue 1 +sonic(config-action)# police cir 1000 +sonic(config-action)# exit + +sonic(config)# policy copp-system-policy type copp +sonic(config-policy)# class class-user +sonic(config-policy-flow)# set copp-action action-user +sonic(config-policy-flow)# exit +``` + +## 4.3 Feature Driven CoPP Config flow +The following flow captures CoPP manager functionality. +![Feature Driven CoPP Config](../../images/copp/copp_flow_feature.png "Figure 3: Feature Driven CoPP Config") + +# 5 Error Handling +- Invalid config errors will be displayed via console and configuration will be rejected +- Internal processing errors within SwSS will be logged in syslog with ERROR level +- SAI interaction errors will be logged in syslog + +# 6 Serviceability and Debug +Debug output will be captured as part of tech support. +- Internal processing errors within SwSS will be logged in syslog with ERROR level +- SAI interaction errors will be logged in syslog + +CoPPMgr daemon restartability will be added in SWSS docker. + +# 7 Warm Boot Support +CoPP functionalities are not expected to work during warm reboot but will be restored after warm reboot. Traffic to CPU will get dropped when CPU is reset until SAI drivers have restored the CPU pkt path. + + +# 8 Scalability +- Limit CoPP classifiers to 64 +- Minimum number of CoPP policers supported must be equal to max number of CoPP protocol traps supported in SONiC +- Max number of CPU queues is ASIC specific + + + +# 9 Unit Test + +## 9.1 Config Test Cases +1. Verify creating CoPP classifier +2. Verify deleting a CoPP classifier +3. Verify adding one (or) more than one match protocol_traps to a CoPP classifier +4. Verify removing match protocol_traps from a CoPP classifier +5. Verify creating CoPP action group +6. Verify deleting CoPP action group +7. Verify adding CoPP action attributes(trap_action, queue, policer) +8. Verify modifying CoPP action policer attributes +9. Verify adding classifier and binding CoPP action to CoPP policy +10. Verify deleting classifier from CoPP policy +11. Verify modifying CoPP action policer attributes when bound to classifier + +## 9.2 Functional Test Cases + +1. Verify that protocol packets are rate-limited as per default configured policies
+2. Verify that protocol packets go to the right CPU queue as per default configured policies
+3. Verify that unmatched traffic to CPU is rate-limited after deleting IP2ME classifier +4. Verify that protocol packets are rate-limited when an existing classifier action is modified + +## 9.3 Scaling Test Cases +1. Verify creating of maximum number of classifiers + +## 9.4 Warm Boot Test Cases +1. Verify that CoPP config is restored after warmboot. +2. Verify that traffic to CPU continues to get rate-limited as per configured CoPP policy after warmboot +2. Verify that CoPP policy is active after warmboot on the protocols that had classifers before warmboot + +## 9.5 Negative Test Cases +1. Verify that you cannot delete default CoPP policy "copp-system-policy" +2. Verify that you cannot create new policies of type 'copp' +3. Kill CoppMgrd and observe process restart. +4. Verify that a protocol can be added to a single classifier. Protocol that is added to a classifier is not allowed to be added to another classifier +5. Verify that 'default' classifier is not removable from CoPP policy + +# 10 BRCM Internal Design Information +Internal BRCM information to be removed before sharing with the community + +## 10.1 BCM Shell Debug Commands + +**show counters** + +The BCM show counters CLI provides rate output (pkts/sec) per interface queue that can be used to automated testing validation of policer rate limit function. +``` +admin@sonic:~$ bcmcmd "show c cpu" +MC_PERQ_PKT(3).cpu0 : 3 +1 +MC_PERQ_PKT(10).cpu0 : 23,312 +23,312 6,000/s +MC_PERQ_BYTE(3).cpu0 : 204 +68 +MC_PERQ_BYTE(10).cpu0 : 3,077,184 +3,077,184 791,964/s +``` + +## 10.2 Chip support +Supported on TD2, TD3, TH1, TH2, TH3. + +### 10.2.1 Scalability +* Max number of CPU queues is ASIC specific (48 CPU queues available in most BRCM ASICs) + +## 10.3 CLI syntax references +* https://www.arista.com/en/um-eos/eos-section-30-5-traffic-management-configuration--trident-platform-switches +* https://tools.cisco.com/security/center/resources/copp_best_practices + + +## 11 Open Issues +- Need to converge with MSFT PR on COPP Management (https://github.com/Azure/SONiC/pull/606). Especially related to default COPP configuration (includes init_cfg.json infra that has not been pulled into BRCM SONiC yet) +- How to handle mini COPP during fast-reboot if using MSFT way of creating default COPP cfg using jinja + diff --git a/doc/ifa-hld.md b/doc/ifa-hld.md new file mode 100644 index 0000000000..b4e97db320 --- /dev/null +++ b/doc/ifa-hld.md @@ -0,0 +1,284 @@ +# Feature Name +Inband Flow Analyzer. +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/30/2019 | Srinadh Penugonda| Initial version | + +# About this Manual +This document provides general information about the Inband Flow Analyzer feature implementation in SONiC. +# Scope +This document describes the north bound interface and unit tests for Inband Flow Analyzer feature. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| IFA | Inband Flow Analyzer | +| TAM | Telemetry and Monitoring | +# 1 Feature Overview + +Inband Flow Analyzer is a flexible packet and flow monitoring inband telemtry solution. The feature allows configuration of IFA sessions that provide Inband telemetry over sampled traffic to collectors. + +It provides mechanism to monitor and analyze when packets enter/exit the network, the path packets and flows take through the network, the rate at which packets arrive at each hop and how log packets spend at each hop etc., Out of band management technhiques can not measure such details. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to IFA. + +1.1 IFA feature is accomplished by configuring IFA session on various nodes that act as ingress, +intermediate and egress devices. Device role is per flow in a node and a single node can act as ingress +device for one flow and intermediate device for another flow. + +2.0.0.1 TAM device identifier to uniquely identify a device in network and insert the same in INT header. +2.0.0.2 ACL configuration to identify a flow and sample packets from that flow to insert IFA headers. +2.0.0.3 TAM collector configuration that can be attached to IFA flow on egress device to forward telemetry +data. +3.0 UI commands available to configure TAM device identifier, TAM collector and IFA configuration. +3.1 UI commands available to show TAM device identifier, TAM collector, IFA configuration +3.2 UI commands available to clear IFA configuration +4.0 The maximum number of IFA 􀃖ows are platform dependent. +4.1 Only one collector can be con􀃕gured in a device. +4.5 Some platforms may require provisioning to enable IFA. 'ifa -config -enable' command to be issued to +provision such platforms for IFA functionality. 'ifa -config -disable' command can be issued to disable +provisioning of IFA on such platforms. + +### 1.1.2 Configuration and Management Requirements +1. CLI configuration/show support +2. REST API support +3. gNMI support + +## 1.2 Design Overview +### 1.2.1 Basic Approach + +As there is no opeconfig/ietf yang file exists for this feature, it is decided to go with sonic yang. + +### 1.2.2 Container +The changes are in sonic-management-framework container. + +There will be additional files added. +1. XML file for the CLI +2. Python script to handle CLI request (actioner) +3. Jinja template to render CLI output (renderer) +4. YANG models + sonic-ifa.yang + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Whenever operator want to track packet latency/congestion metrics. +## 2.2 Functional Description +The management UI provides an user friendly interface to configure Inband Flow Analyzer. +# 3 Design +## 3.1 Overview +The packet action field in acl rule indicates the type of IFA device: ingress and egress. 'int_insert' makes the device as ingress IAF device and 'int_delete' as egress IFA device. Ingress IAF device makes a sample of a flow and tags them for analysis and data collection. Egress device is responsible for terminating IFA flow by summarizing the telemtry data of the entire path and sending it to the collector. + +The device identifer uniquely identifies the device in the network and inserts the ID in the IFA header. + +Collectors receive the telemetry data from egress devices. + +Flow configuration will contain sampling rate at which rate traffic will be sampled. + +IFA feature can be enabled or disabled. +## 3.6 User Interface +### 3.6.1 Data Models + +https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/sonic/sonic-ifa.yang + +module: sonic-ifa + +--rw sonic-ifa + +--rw TAM_INT_IFA_FEATURE_TABLE + | +--rw TAM_INT_IFA_FEATURE_TABLE_LIST* [name] + | +--rw name enumeration + | +--rw enable? boolean + +--rw TAM_INT_IFA_FLOW_TABLE + +--rw TAM_INT_IFA_FLOW_TABLE_LIST* [name] + +--rw name string + +--rw acl-table-name string + +--rw acl-rule-name string + +--rw sampling-rate? uint16 + +--rw collector-name? string + +https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/sonic/sonic-tam.yang + +module: sonic-tam + +--rw sonic-tam + +--rw TAM_DEVICE_TABLE + | +--rw TAM_DEVICE_TABLE_LIST* [name] + | +--rw name enumeration + | +--rw deviceid? uint16 + +--rw TAM_COLLECTOR_TABLE + +--rw TAM_COLLECTOR_TABLE_LIST* [name] + +--rw name string + +--rw ipaddress-type? enumeration + +--rw ipaddress? inet:ip-address + +--rw port? inet:port-number + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +1. Command : confg tam device-id +Attribute : +The command is used to configure TAM device identifier. + +2. Command : config tam collector +Attribute(s) : {collector-name} ip-type ip-addr
port +The command is used to configure TAM collector and IFA report will be forwarded to the collector. + +3. Command : config tam int-ifa feature +Attribute : +The command is used to enable or disable the IFA feature. + +4. Command : config tam int-ifa flow +Attribute(s) : acl-rule acl-table { sampling-rate collector } +The command is used to specify flow criteria to match against incoming flow and tag with IFA data. When sampling rate is specified, one packet will be sampled out of its value. When collector is specified, IFA report will be forwarded to it. + +5. Command : config-tam no device-id +The command is used to clear user configured device identifier. Default device identifier is used. + +6. Command : config-tam no collector +Attribute : +The command is used to delete previously configured collector information. + +7. Command : config-tam-int-ifa no flow +Attribute : +The command is used to delete previously configure flow information. + +#### 3.6.2.2 Show Commands +1. Command : show tam device +The command is used to show TAM device identifier. + +2. Command : show tam collector +Attribute : { | all } +The command is used to show TAM collector information. + +3. Command : show tam int-ifa flow +Attribute(s) : { | all } +The command is used to display configured IFA flow information. + +4. Command : show tam int-ifa statistics +Attribute(s) : { | all } +The command is used to display statisitcs of IFA flow. + +5. Command : show tam int-ifa status +The command is used to display status of TAM. + +#### 3.6.2.3 Debug Commands +N/A + +### 3.6.3 REST API Support + +1. Get Device Information +sonic-tam:sonic-tam/TAM_DEVICE_TABLE + +2. Get Collector information +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE + +3. Get particular collector information +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST={name} + +4. Get IFA feature information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FEATURE_TABLE + +5. Get IFA flow information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE + +6. Get particular IFA flow information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} + +7. Set device identifier +sonic-tam:sonic-tam/TAM_DEVICE_TABLE/TAM_DEVICE_TABLE_LIST={device}/deviceid +{ + "sonic-tam:deviceid": 0 +} + +8. Set TAM collector +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST +{ + "sonic-tam:TAM_COLLECTOR_TABLE_LIST": [ + { + "name": "string", + "ipaddress-type": "ipv4", + "v4addr": "string", + "v6addr": "string", + "port": 0 + } + ] +} + +9. Set TAM INT IFA feature +sonic-ifa:sonic-ifa/TAM_INT_IFA_FEATURE_TABLE/TAM_INT_IFA_FEATURE_TABLE_LIST={feature}/enable +{ + "sonic-ifa:enable": true +} + +10. Set TAM INT IFA flow +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} +{ + "sonic-ifa:TAM_INT_IFA_FLOW_TABLE_LIST": [ + { + "name": "string", + "acl-table-name": "string", + "acl-rule-name": "string", + "sampling-rate": 0, + "collector": "string" + } + ] +} + +11. Delete TAM device identifier +sonic-tam:sonic-tam/TAM_DEVICE_TABLE/TAM_DEVICE_TABLE_LIST={device}/deviceid + +12. Delete TAM collector +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST={name} + +13. Delete IFA flow +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} + + +# 4 Flow Diagrams +N/A +# 5 Error Handling +N/A +# 6 Serviceability and Debug +TBD + +# 7 Warm Boot Support +N/A +# 8 Scalability +N/A +# 9 Unit Test +| Test Name | Test Description | +| :------ | :----- | +| Create TAM DEvice Identifier | Verify device-id is configured. Verify a value with more than five digits will be rejected | +| Delete TAM device Identifier | Verify when device-id is deleted, it defaults to default value of 0 | +| Create TAM collector | Verify TAM collector is configured ( ip address validation is does by application ) | +| Delete TAM collector | Verify TAM collector can be deleted | +| Enable TAM INT IFA Feature | Verify user can enable/disable IFA feature | +| Create IFA Flow | Verify IFA flow is configured. Verify configuration fails when user uses invalid acl table/rule. Verify configuration fails when user uses invalid collector name | +| Delete IFA flow | Verify IFA flow is deleted | +| Show TAM Device | Verify configured device identifier is displayed with the show command | +| Show TAM collector | Verify all collectors are displayed with 'all' keyword. Verify specified collector is displayed with the name. Verify command fails when an invalid flow name is used to display | +| Show TAM INT IFA flow | Verify all flow information is displayed when used with 'all'. Verify particular flow is displayed when name is supplied. Verify command fails when an non-existent flow name is given | +| show tam int ifa supported | Verify feature status is correctly displayed | + + +# 10 Internal Design Information +N/A diff --git a/doc/mgmt/Developer Guide.md b/doc/mgmt/Developer Guide.md new file mode 100644 index 0000000000..ffa8d57be4 --- /dev/null +++ b/doc/mgmt/Developer Guide.md @@ -0,0 +1,802 @@ +# SONiC Management Framework Developer Guide + +## Rev 0.6 + +## Table of Contents + +* [List of Tables](#list-of-tables) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definition/Abbreviation](#definitionabbreviation) +* [Table 1: Abbreviations](#table-1-abbreviations) +* [1 Architecture](#1-Architecture) + * [1.1 Requirements](#11-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) +* [2 Developer Workflow](#2-developer-workflow) + * [2.1 ABNF Schema](#21-abnf-schema) + * [2.2 YANG Identification](#22-yang-identification) + * [2.2.1 Standard YANG](#221-standard-yang) + * [2.2.2 SONiC YANG](#222-sonic-yang) + * [2.3 Code Generation](#23-code-generation) + * [2.4 Config Translation App](#24-config-translation-app) + * [2.4.1 Transformer](#241-transformer) + * [2.4.1.1 Annotation File](#2411-annotation-file) + * [2.4.1.2 Annotate YANG extensions](#2412-annotate-YANG-extensions) + * [2.4.1.3 Manifest file](#2413-manifest-file) + * [2.4.1.4 Special handling](#2414-special-handling) + * [2.4.2 App Module](#242-app-module) + * [2.5 Config Validation Library](#25-config-validation-library) + * [2.6 Industry Standard CLI](#26-Industry-Standard-cli) + * [2.6.1 CLI components](#261-cli-components) + * [2.6.2 CLI development steps](#262-cli-development-steps) + * [2.6.3 Enhancements to Klish](#263-enhancements-to-klish) + * [2.6.4 Preprocess XML files](#264-preprocess-xml-files) + * [2.6.5 CLI directory structure](#265-cli-directory-structure) + * [2.7 gNMI](#27-gnmi) + * [2.8 Unit Testing](#28-unit-testing) + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:-----------------------:|----------------------------------------------------------------------| +| 0.1 | 09/12/2019 | Anand Kumar Subramanian | Initial version | +| 0.2 | 09/16/2019 | Prabhu Sreenivasan | Added references, prerequisites and updated section 2.1.2 SONiC YANG | +| 0.3 | 09/16/2019 | Partha Dutta | Updated SONiC YANG Guideline link | +| 0.4 | 09/18/2019 | Anand Kumar Subramanian | Updated transformer section | +| 0.5 | 09/20/2019 | Mohammed Faraaz C | Updated REST Unit testing | +| 0.6 | 09/20/2019 | Prabhu Sreenivasan | Updated reference links and yang path | +| 0.7 | 09/23/2019 | Partha Dutta | Updated SONiC YANG, CVL, gNMI section | + +## About this Manual + +This document provides developer guidelines for Management framework feature development in SONiC. + +## Scope + +This document describes the steps the feature developers need to follow to develop a CLI, REST and gNMI for a given feature using the Management framework. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| CVL | Config Validation Library | +| NBI | North Bound Interface | +| ABNF | Augmented Backus-Naur Form | +| YANG | Yet Another Next Generation | +| JSON | Java Script Object Notation | +| XML | eXtensible Markup Language | +| gNMI | gRPC Network Management Interface | +| YGOT | YANG Go Tools | +| SONiC YANG | YANG file which describes Redis DB schema for a given feature | +| RPC | Remote Procedure Call | + +## References + +| Document | Link | +|:--------:|:-------| +| SONiC YANG model guideline | https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md | +| SONiC Management Framework HLD | https://github.com/Azure/SONiC/pull/436 | +| RFC 7950 | August 2016 | https://tools.ietf.org/html/rfc7950 | + +## Prerequisite + +1) Knowledge of YANG syntax + +2) GO Programming language + +3) SONiC Management Framework Architecture + +## 1 Architecture + +![Management Framework Architecture diagram](https://github.com/project-arlo/SONiC/blob/master/doc/mgmt/images/Mgmt_Frmk_Arch.jpg) + +In the above architecture diagram developer needs to write all the elements in green. Following are blocks that the developer has to write in order to get CLI, REST and gNMI support for the new feature being developed. + +1) YANG Models + +2) ABNF schema and SONiC YANG + +3) Transformer - includes YANG annotations and custom translation + +4) KLISH XML including Actioner and Renderer scripts + +## 2 Developer Workflow + +Following are the steps to be followed by the application developer. + +### 2.1 ABNF Schema + +For the existing feature ABNF schema must be used as a reference to develop the SONiC YANG. + +Define the ABNF schema for the new feature. It is suggested to keep it in line with YANG model selected below. It is also suggested to keep the field names in the ABNF schema same as that of the leafs and leaf lists in the YANG model to help in easy translation. Ideally defining ABNF schema and configuration objects in SONiC YANG can go hand in hand to keep one definition in line with other. + +### 2.2 YANG Identification + +Identify the standard/SONiC YANG model to be used for northbound APIs. + +#### 2.2.1 Standard YANG + +OpenConfig model is preferred. IETF YANG model can be used if there is no OpenConfig model for the feature. + +Feature implementation may have to support additional configuration/state knobs than the standard YANGs. In such cases, the developer will have to write a custom extension YANG. Extension YANGs will add custom configuration/state knobs on top of standard YANG tree using YANG deviation. + +Below is a hypothetical example of extending openconfig ACL configuration by disabling description, limiting number of characters of ACL name and adding a new alt-name config. + + module openconfig-acl-deviations { + namespace "http://github.com/Azure/sonic-oc-acl-deviations"; + prefix oc-acl-deviations; + import openconfig-acl { prefix oc-acl; } + + /* Mark ACL description not supported */ + deviation /oc-acl:acl/acl-sets/acl-set/config/description { + deviate not-supported; + } + + /* Restrict ACL name to max 127 characters */ + deviation /oc-acl:acl/acl-sets/acl-set/config/name { + deviate replace { + type string { + length "1..127"; + } + } + } + + /* Add an alt-name config to ACL */ + augment /oc-acl:acl/acl-sets/acl-set/config { + leaf alt-name { + type string; + } + } + } + +Note: You should avoid deviations/extensions to standard YANG models as much as possible. Add them only if necessary for your feature. NSM apps that depend of standard modules may not be able to discover the SONiC switches if standard YANG models are modified too much. + +#### 2.2.2 SONiC YANG + +SONiC YANGs are custom YANGs developed for data (i.e. ABNF JSON) modeling in SONiC. SONiC YANGs captures the Redis DB schema associated with a given feature. This YANG helps SONiC validate syntax and semantics of a feature configuration. SONiC YANGs are mandatory for Management Framework to work and it needs to capture all supported configuration pertaining to a feature. SONiC YANG shall capture operational state data syntax, RPC, notification when used as Northbound Management Interface YANG. + +Please refer to [SONiC YANG Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) for detailed guidelines on writing SONiC YANG. Please refer to existing YANG models (in "models/yang/sonic/" folder in 'sonic-mgmt-framework' repository) already written for features like ACL etc. You can also refer to few test YANGs available in "src/cvl/testdata/schema" folder. Once you have written a SONiC YANG following the guideline, place it in "models/yang/sonic/" folder. + +### 2.3 Code Generation + +Generate the REST server stubs and REST Client SDKs for YANG based APIs by placing the main YANG modules under sonic-mgmt-framework/models/yang directory and compiling. By placing YANG module in this directory and compiling, YAML (swagger spec) is generated for the YANG. Also the YGOT GO structures for the YANG are also automatically generated during the build time. If there is YANG which is augmenting or adding deviations for the main YANG module, this augmenting/deviation YANG should also be placed in sonic-mgmt-framework/models/yang directory itself. + +Place all dependent YANG modules such as submodules or YANGs which will define typedefs etc under sonic-mgmt-framework/models/yang/common directory. By placing YANG module in this directory, YAML (swagger spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-framework/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory. +Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-framework/models/yang/common. + +Generation of Rest-server stubs and client SDKs will automatically happen when make command is executed as part of the build. + +### 2.4 Config Translation App + +Developer writes the Config translation App. Config translation App translates the data in [Northbound API schema](#21-YANG-identification) to the native [ABNF schema](#22-abnf-schema) and vice versa. All Northbound API services like REST, GNMI, NETCONF will invoke this app to read and write data. + +Config Translation can be done using + +1. Transformer (Preferred) or +2. App module (Not preferred) + +#### 2.4.1 Transformer + +Transformer provides a generic infrastructure for Translib to programmatically translate YANG to ABNF/Redis schema and vice versa, using YANG extensions to define translation hints along the YANG paths. At run time, the translation hints are mapped to an in-memory Transformer Spec that provides two-way mapping between YANG and ABNF/Redis schema for Transformer to perform data translation. + +In case that SONiC YANG modules are used by NBI applications, the Transformer performs 1:1 mapping between a YANG object and a SONiC DB object without a need to write special translation codes or any translation hints. + +If you use the openconfig YANGs for NBI applications, you need special handling to translate the data between YANG and ABNF schema. In such case, you have to annotate YANG extensions and write callbacks to perform translations where required. + +In either case, the default application [common-app.go](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/translib/common_app.go) generically handles both set and get requests with the returned data from Transformer. + +##### 2.4.1.1 Annotation File + +The goYANG package is extended to generate the template annotation file for any input YANG file. A new output format type "annotate" can be used to generate the template annotation file. + +The goYANG usage is as below: + +Usage: goYANG [-?] [--format FORMAT] [--ignore-circdep] [--path DIR[,DIR...]] [--trace TRACEFILE] [FORMAT OPTIONS] [SOURCE] [...] + -?, --help display help + --format=FORMAT + format to display: annotate, tree, types + --ignore-circdep + ignore circular dependencies between submodules + --path=DIR[,DIR...] + comma separated list of directories to add to search path + --trace=TRACEFILE + write trace into to TRACEFILE + +Formats: + annotate - generate template file for YANG annotations + + tree - display in a tree format + + types - display found types + --types_debug display debug information + --types_verbose + include base information + +Add $(SONIC_MGMT_FRAMEWORK)/gopkgs/bin to the PATH to run the goYANG binary. + +e.g. + +goYANG --format=annotate --path=/path/to/yang/models openconfig-acl.yang > openconfig-acl-annot.yang + +[Annotation file example](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/models/yang/annotations/openconfig-acl-annot.yang) + +Sample output: + + module openconfig-acl-annot { + + YANG-version "1" + + namespace "http://openconfig.net/yang/annotation"; + prefix "oc-acl-annot" + + import openconfig-packet-match { prefix oc-pkt-match } + import openconfig-interfaces { prefix oc-if } + import openconfig-yang-types { prefix oc-yang } + import openconfig-extensions { prefix oc-ext } + + deviation oc-acl:openconfig-acl { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state/oc-acl:counter-capability { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set { + deviate add { + } + } + + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set/oc-acl:type { + deviate add { + } + } + ... + ... + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:config { + deviate add { + } + } + } + +##### 2.4.1.2 Annotate YANG extensions + +The translation hints can be defined as YANG extensions to support simple table/field name mapping or more complex data translation by overloading the default methods. + +[Extensions](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/models/yang/common/sonic-extensions.yang) + +| Extensions | Usage | Note +|-------------------------------------------------------|-------------------------------------------|---------------------------------------------------------------------------------------| +| sonic-ext:table-name [string] | Map a YANG list to TABLE name | Processed by the default transformer method. | +| | | Argument could be one or more table names mapped to the given YANG list node | +| | | The table-name is inherited to all descendant nodes unless another one is defined. | +| sonic-ext:field-name [string] | Map a YANG leafy to FIELD name | Processed by the default transformer method | +| sonic-ext:key-delimiter [string] | Override the default delimiter, “|” | Processed by the default transformer method | +| | | Used to concatenate multiple YANG keys into a single DB key | +| sonic-ext:key-transformer [function] | Overloading the default method to generate| Used when the key values in a YANG list are different from ones in DB TABLE | +| | DB key(s) | A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, | +| | | Db2Yangxxx | +| sonic-ext:field-transformer [function] | Overloading default method for field | Used when the leaf/leaf-list values defined in a YANG list are different from the | +| | generation | field values in DB | +| | | A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, | +| | | Db2Yangxxx | +| sonic-ext:subtree-transformer [function] | Overloading default method for the current| Allows the sub-tree transformer to take full control of translation. Note that, if any| +| | subtree | other extensions are annotated to the nodes on the subtree, they are not effective. | +| | | The subtree-transformer is inherited to all descendant nodes unless another one is | +| | | defined, i.e. the scope of subtree-transformer callback is limited to the current and | +| | | descendant nodes along the YANG path until a new subtree transformer is annotated | +| | | A pair of callbacks should be implemented to support 2 way translation - YangToDBxxx, | +| | | DbToYangxxx | +|(Coming soon)sonic-ext:db-access-transformer [function]| Overloading default (get) method to read | Allows the readonly transformer to take full control of DB access and translation | +| | db and populate to ygot structure | from DB to Yang | +| | | Used for “GET” operation, applicable to both rw/ro YANG container/lists | +| | | Note that, unless annotated, the default transformer will read DB and populate to the | +| | | YGOT optionally with the help of DbToYang subtree transformer. | +| | | The readonly-transformer is inherited to all descendant nodes unless another one. | +| sonic-ext:db-name [string] | DB name to access data – “APPL_DB”, | Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG | +| | “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, | Processed by Transformer core to traverse database | +| | “FLEX_COUNTER_DB”, “STATE_DB”. The | The db-name is inherited to all descendant nodes unless another one. | +| | default db-name is CONFIG_DB | Must be defined with the table-name | +|(Coming soon) sonic-ext:post-transformer [function] | A special hook to update the DB requests | Analogous to the postponed YangToDB subtree callback that is invoked at the very end | +| | right before passing to common-app | by the Transformer. | +| | | Used to add/update additional data to the maps returned from Transformer before | +| | | passing to common-app, e.g. add a default acl rule | +| | | Note that the post-transformer can be annotated only to the top-level container(s) | +| | | within each module, and called once for the given node during translation | +| sonic-ext:get-validate [function] | A special hook to validate YANG nodes, | Allows developers to instruct Transformer to choose a YANG node among multiple nodes, | +| | to populate data read from database | while constructing the response payload | | +| | | Typically used to check the “when” condition to validate YANG node among multiple | +| | | nodes to choose only valid nodes from sibling nodes. | + +The template annotation file generated from section [2.4.1.1](#2411-annotation-file) can be used by the app developer to add extensions to the YANG paths as needed to transform data between YANG and ABNF format. + +Following is the guide to find which extensions can be annotated in implementing the model. + + 1. If the translation is simple mapping between YANG list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter, key-transformer, field-transformer + + 2. If the translation requires a complex translation with codes, use the extension, subtree-transformer, to take control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope + + 3. If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-name extension with a set of table names, and annotate the subtree-transformer to dynamically choose tables based on URI/payload + + 4. In GET operation, Transformer can read data by TABLE and optionally KEYs learned from URI. But, in case that you need to access database with your own method, annotate the db-access-transformer to take a control during data access, e.g. when you read interface counters, you need to get OID for the interface name, then subsequently read the counters by the OID. + +##### 2.4.1.3 Manifest file + +Add the list of YANG modules and annotation files to the transformer manifest file located at sonic-mgmt-framework/config/transformer/models_list + + e.g. for openconfig ACL + openconfig-acl-annot.yang + openconfig-acl.yang + +##### 2.4.1.4 Special handling + +Implement the overload methods if you need translate data with special handling. There are three types of transformer special handling functions: key-transformer, field-transformer and subtree-transformer. Argument of each transformer is used by the transformer core to dynamically lookup and invoke the function during data translation. A function name is formed by an argument string prefixed by a reserved literal – “YANGToDb_” or “DbToYANG_”, which distinguishes the direction in which the convertion happens. + +Here is a data structures passed from Transformer to overloaded methods. + + type XfmrParams struct { + d *db.DB + dbs [db.MaxDB]*db.DB + curDb db.DBNum + ygRoot *ygot.GoStruct + uri string + oper int + key string + dbDataMap *map[db.DBNum]map[string]map[string]db.Value + param interface{} + } + +e.g. for YANG extension defined in the annotation file +sonic-ext:subtree-transformer "acl_port_bindings_xfmr"; +A pair of functions shall be implemented for two way translation: YANGToDb_acl_port_bindings_xfmr() and DbToYANG_acl_port_bindings_xfmr(). + + e.g. xfmr_acl.go + + var YangToDb_acl_port_bindings_xfmr SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value, error) { + var err error + res_map := make(map[string]map[string]db.Value) + aclTableMap := make(map[string]db.Value) + log.Info("YangToDb_acl_port_bindings_xfmr: ", inParams.ygRoot, inParams.uri) + + aclObj := getAclRoot(inParams.ygRoot) + if aclObj.Interfaces == nil { + return res_map, err + } + . . . + } + var DbToYang_acl_port_bindings_xfmr SubTreeXfmrDbToYang = func(inParams XfmrParams) error { + var err error + data := (*inParams.dbDataMap)[inParams.curDb] + log.Info("DbToYang_acl_port_bindings_xfmr: ", data, inParams.ygRoot) + . . . + } + +Mapping openconfig ENUM to Redis ABNF data is typically handled by field transformer. Below example shows how. + e.g. E_OpenconfigAcl_FORWARDING_ACTION + + var ACL_FORWARDING_ACTION_MAP = map[string]string{ + strconv.FormatInt(int64(ocbinds.OpenconfigAcl_FORWARDING_ACTION_ACCEPT), 10): "FORWARD", + strconv.FormatInt(int64(ocbinds.OpenconfigAcl_FORWARDING_ACTION_DROP), 10): "DROP", + strconv.FormatInt(int64(ocbinds.OpenconfigAcl_FORWARDING_ACTION_REJECT), 10): "REDIRECT", + } + + . . . + + var YangToDb_acl_forwarding_action_xfmr FieldXfmrYangToDb = func(inParams XfmrParams) (map[string]string, error) { + res_map := make(map[string]string) + var err error + if inParams.param == nil { + res_map["PACKET_ACTION"] = "" + return res_map, err + } + action, _ := inParams.param.(ocbinds.E_OpenconfigAcl_FORWARDING_ACTION) + log.Info("YangToDb_acl_forwarding_action_xfmr: ", inParams.ygRoot, " Xpath: ", inParams.uri, " forwarding_action: ", action) + res_map["PACKET_ACTION"] = findInMap(ACL_FORWARDING_ACTION_MAP, strconv.FormatInt(int64(action), 10)) + return res_map, err + } + var DbToYang_acl_forwarding_action_xfmr FieldXfmrDbtoYang = func(inParams XfmrParams) (map[string]interface{}, error) { + var err error + result := make(map[string]interface{}) + data := (*inParams.dbDataMap)[inParams.curDb] + log.Info("DbToYang_acl_forwarding_action_xfmr", data, inParams.ygRoot) + oc_action := findInMap(ACL_FORWARDING_ACTION_MAP, data[RULE_TABLE][inParams.key].Field["PACKET_ACTION"]) + n, err := strconv.ParseInt(oc_action, 10, 64) + result["forwarding-action"] = ocbinds.E_OpenconfigAcl_FORWARDING_ACTION(n).ΛMap()["E_OpenconfigAcl_FORWARDING_ACTION"][n].Name + return result, err + } + +Overloaded transformer functions are grouped by YANG module, to have a separate GO file. At init(), the functions need to be bind for dynamic invocation. + e.g. xfmr_acl.go + + func init () { + XlateFuncBind("YANGToDb_acl_entry_key_xfmr", YANGToDb_acl_entry_key_xfmr) + XlateFuncBind("DbToYANG_acl_entry_key_xfmr", DbToYANG_acl_entry_key_xfmr) + XlateFuncBind("YANGToDb_acl_l2_ethertype_xfmr", YANGToDb_acl_l2_ethertype_xfmr) + XlateFuncBind("DbToYANG_acl_l2_ethertype_xfmr", DbToYANG_acl_l2_ethertype_xfmr) + XlateFuncBind("YANGToDb_acl_ip_protocol_xfmr", + . . . + } + +#### 2.4.2 App Module + +Instead of using the transformer, developers can write the complete App module that aids in the conversion of the incoming request to the SONiC ABNF format and vice versa. Following are the steps to be performed. + +1. Define a structure to hold all the incoming information as well as the translation information +Example: + + type AclApp struct { + pathInfo *PathInfo + ygotRoot *ygot.GoStruct + ygotTarget *interface{} + + aclTs *db.TableSpec + ruleTs *db.TableSpec + + aclTableMap map[string]db.Value + ruleTableMap map[string]map[string]db.Value + } + +2. App modules will implement an init function which registers itself with the translib. In addition it should also add the YANG models it supports here to serve the capabilities API of gNMI. + Example: + func init () { + log.Info("Init called for ACL module") + err := register("/openconfig-acl:acl", + &appInfo{appType: reflect.TypeOf(AclApp{}), + ygotRootType: reflect.TypeOf(ocbinds.OpenconfigAcl_Acl{}), + isNative: false, + tablesToWatch: []*db.TableSpec{&db.TableSpec{Name: ACL_TABLE}, &db.TableSpec{Name: RULE_TABLE}}}) + + if err != nil { + log.Fatal("Register ACL App module with App Interface failed with error=", err) + } + + err = appinterface.AddModel(&gnmi.ModelData{Name:"openconfig-acl", + Organization:"OpenConfig working group", + Version:"1.0.2"}) + if err != nil { + log.Fatal("Adding model data to appinterface failed with error=", err) + } + } + +App Modules will implement the following interface functions that will enable conversion from YGOT struct to ABNF format. + + initialize(data appData) + translateCreate(d *db.DB) ([]db.WatchKeys, error) + translateUpdate(d *db.DB) ([]db.WatchKeys, error) + translateReplace(d *db.DB) ([]db.WatchKeys, error) + translateDelete(d *db.DB) ([]db.WatchKeys, error) + translateGet(dbs [db.MaxDB]*db.DB) error + translateSubscribe(dbs [db.MaxDB]*db.DB, path string) (*notificationOpts, *notificationInfo, error) + processCreate(d *db.DB) (SetResponse, error) + processUpdate(d *db.DB) (SetResponse, error) + processReplace(d *db.DB) (SetResponse, error) + processDelete(d *db.DB) (SetResponse, error) + processGet(dbs [db.MaxDB]*db.DB) (GetResponse, error) + +initialize – Populate the app structure that we created in step 1 with the incoming appData which contains path, payload, YGOT root structure, YGOT target structure. + +translate(CRUD) – Convert the information in the YGOT root and target structure to the corresponding ABNF key value pair and store it in the structure created in step 1 + +translateGet – Convert the incoming path to corresponding ABNF keys to be got from the redis DB using the DB access layer APIs similar to the python SWSSSDK. + +translateSubscribe – Convert the incoming path in the argument to corresponding ABNF keys as part of the notificationInfo structure. In addition to this subscribe parameters like type of subscription supported, SAMPLE interval can also be specified as part of the notificationOpts structure. + +process(CRUD) – Write the converted data from translate(CRUD) function into the Redis DB using the DB access layer APIs + +### 2.5 Config Validation Library + +Config Validation Library (CVL) performs the automatic syntactic and semantic validation based on the constraints defined in SONiC YANG. Note that CVL uses CVL YANG derived from SONiC YANG by stripping off read-only state objects, RPC and Notification objects. This happens during build time automatically with the help of pyang plugin and therefore no manual intervention is needed. + +If any specific validation can't be achieved through constraints, CVL provides options to develop custom validation code (this is WIP in 'buzznik' and more details would be updated soon) which is loaded as a plug-in in CVL. CVL generates stub function based on YANG extension and you need to implement functional validation code inside this stub function and place the file inside 'src/cvl/custom_validation' folder. + +CVL allows platform specific validation in following ways: + +* Add YANG 'deviation' files (e.g. sonic-acl-deviation.yang for ACL YANG) per platform and place it in 'models/yang/sonic/deviation' folder for static platform validation. These files are automatically picked up during build time and processed. +* Implement platform specific custom validation through CVL plug-in for dynamic platform validation. + +### 2.6 Industry Standard CLI + +Open source Klish is integrated to sonic-mgmt-framework to provide the command line interface tool to perform network operations more efficiently in SONiC. Klish will provide the core functionality of command parsing, syntax validation, command help and command auto-completion. + +Open Source [klish](http://libcode.org/projects/klish/.) is used here. + +#### 2.6.1 CLI components + +1. CLI Parser engine: Open source Klish + +2. XML files: +XML files, defined by developer, that describe the CLI command structure. +Klish uses XML based command tree inputs to build the parser command tree. +All CLI commands to be supported are specified in XML format in module specific XML file. +XML files can be defined with macros and entity references, preprocessed by scripts to generate the expanded XML files. + +3. Actioner scripts: Script that will form the request body and invoke the swagger client API. + +4. Renderer: Script that will receive the JSON response from Swagger CLI API and use the jinja2 template file to render(display) the CLI output in the desired format. + +#### 2.6.2 CLI development steps + +Following are the steps to add a new CLI command + +1. Create a CLI XML .xml file that defines the CLI command structure. This file defines the following + * CLI command format + * Parameters that the command requires + * Help string to be displayed for the command and parameters + * Datatype of the parameters. + * View name for which the command needs to be available. Eg: configure-view(config mode) or enable-view(exec mode) + +Example: + + + + + + + + if test "${access-list-name}" = ""; then + python $SONIC_CLI_ROOT/sonic-cli.py get_acl_acl_sets show_access_list.j2 + else + python $SONIC_CLI_ROOT/sonic-cli.py get_acl_set_acl_entries ${access-list-name} ACL_IPV4 show_access_list.j2 + fi + + + +2. Write/Update an actioner script: The actioner script prepares the message body having the JSON request and invokes the swagger client API. The actioner script is invoked by the klish and the input parameters are passed to it from the XML file. + + **Example**: + Refer the tag in the above example command. + The actioner scripts are placed in the following location: + sonic-mgmt-framework/src/CLI/actioner/ + One actioner script will be written per module. + Eg: sonic_cli_if.py can be used to handle the interface cases. + +3. Write/Update Renderer scripts and templates. The JSON response from the swagger client API is received by the actioner and passes the response to the renderer script. The renderer script will invoke the jinja2 template with the JSON response. The template will parse the JSON response and generate the CLI output. Refer files in the below path for an example of usage + + **Renderer path**: + sonic-mgmt-framework/src/CLI/renderer + **Renderer script**: + scripts/render_cli.py + **Renderer template**: + templates/show_access_list.j2 + + Every show command response can have a separate template file. + +#### 2.6.3 Enhancements to Klish + +Additional enhancements can be done to open source klish as below. Enhancements may include defining a new data types for CLI command parameters, define new macros that can be referenced for CLI commands structure that have repetitive definitions and defining platform specific entity values. + +1. PTYPES + New parameter types (PTYPES) can be defined and used in the CLI XML files. + * PTYPE represents a parameter type, a syntactical template which parameters reference. + * sonic-clish.xsd defines the tag and the attributes associated with the PTYPE tag. + * Whenever a new attribute or tag introduced the xsd rules should also be updated. + * sonic-clish.xsd is avilable at sonic-mgmt-framework/src/CLI/clitree/scripts/ + * The klish source supports certain primitive PTYPEs. + * New user defined PTYPEs can be added to the SONIC project and can be defined in sonic_types.xml file. + * sonic_types.xml is available in sonic-mgmt-framework/src/CLI/clitree/cli-xml/sonic_types.xml + + **Example**: STRING_32 + + + +2. MACROS + * Macros can be introduced by defining them in _macro.xml + * Macros are used where repetitive command snippets (or command parameter snippets) with minor variations are needed in XML files. + * It is possible to create a macro definition in a _macro.xml file and use the macro in the .xml file. + * For cases where variations in the values of these macro options are needed, the XML attributes can be passed as arguments to macro and substituted as values inside the macro definition. + * As part of parser tree preprocessing during Make, the referenced macro is expanded with the macro definitions and then the parser file is exported to the target XML files in CLI/target/command-tree to be consumed by Klish parser. + * One macro file can be written per module. + * The macro files are placed at sonic-mgmt-framework/src/CLI/clitree/macro/ + * Macros can also be nested with reference to another macro definition. + + **Example**: + + + + + + + + + * The previous macro “IPV4-SRC-OPTIONS“ is used by the macro below "IPV4-ACL". + + + + + + + + + + + + + + + + + + +3. Entities + + * Entities can be defined and referenced in the XML files. + * One common use case for entities is to specify platform specific parameters and parameter value limits in XML files. + * To define an entity based parameter value range limit, three tasks must be done: + + 1. Define the feature value as entity in mgmt_clish_feature_master.xsd + **Example**: + + + + This contains the allowed feature names for any platform specific customizable feature names. + If you are adding a new platform customizable feature, add a record in the enumerated list here. + + + + + + + + + + This contains the allowed feature-value names for any platform specific customizable feature-value + strings. If you are adding a new platform customizable entity, add a record in the enumerated list here. + + + + + + + + 2. Set the attribute 'value' to the right limit on respective platform configuration file that can be defined statically (used at build time). + Refer to a dummy platform (platform_dummy.xml) configuration file. + The platform_dummy.xml file can be located at sonic-mgmt-framework/src/CLI/clitree/scripts/. + + **Example**: + + MAX_MTU + + 3. Use the ENTITY just like any other XML ENTITY in the CLI parser XML file. + +#### 2.6.4 Preprocess XML files + +* The preprocessing scripts are invoked at compile time and no action is required to add a new CLI command. +* This section gives information about what is done during preprocessing stage. + + * The preprocessing scripts preprocess the raw CLI XML files and generate a target XML file that can be consumed by the klish open source parser. + * The inputs to the preprocessing scripts are the raw CLI XML files, macro files and other utility files like platform specifics. + * The cli-xml files are validated as part of compilation. + * The 'xmllint' binary is used to validate all the processed XML files (i.e. after macro substitution and pipe processing) against the detailed schema kept at sonic-clish.xsd + * The following preprocessing scripts are introduced: + + * **klish_ins_def_cmd.py**: This script is used to append the "exit" and "end" commands to the views of the Klish XML files + * **klish_insert_pipe.py**: This script extends every show and get COMMAND with pipe option + * **klish_platform_features_process.sh**: Validate all platform XML files. Generate the entity.xml files. + * **klish_replace_macro.py**: This script does macro replacement on the XML files which are used by klish to define CLI structure. + +#### 2.6.5 CLI directory structure + +sonic-mgmt-framework/src + +- actioner +- clicfg +- clitree + - cli-xml + - macro + - scripts +- klish + - patches + - klish-2.1.4 + - scripts +- renderer + - scripts + - templates + +Below directories are used to collect XML files and utility scripts to generate the target XML files for CLI command tree build-out. + +**clitree/cli-xml** - contains unprocessed/raw CLI XML files defined by developer + +**clitree/macro** - contains macro definitions used/referenced in CLI XML files defined by developer + +**clitree/scripts** - contains utility scripts to process raw CLI XML files defined by developer into processed CLI XMLs to be consumed by Klish Parser. + +**clitree/Makefile** - rules to preprocess the raw CLI XMLs and validate the processed output against the DTD in sonic-clish.xsd + +**clicfg** - files for platform specific entity substitution. + +**renderer** - templates and scripts to be used in rendering the show commands output. + +After compilation the processed CLI XMLs can be found in sonic-mgmt-framework/build/cli/command-tree + +### 2.7 gNMI + +There is no specific steps required for gNMI. + +### 2.8 Unit Testing +#### 2.8.1 REST Server + +REST server provides a test UI (Swagger UI) will display the operations defined in OpenAPI spec along with options to test it inline. To launch the Swagger UI, open “https://IP:80/ui” from the browser. Browser may complain that the certificate is not trusted. This is expected since the REST Server uses a temporary self signed certificate by default. Ignore the warning and proceed, this loads a home page as shown below, which contains links for OpenAPIs both derived from yang models and also manually written OpenAPI specs. + +![Swagger UI Home Page](images/homepage.jpeg) +Upon clicking of any one of the link, The Swagger UI correspoding to that openAPI spec will be opened as shown below. + +![OpenAPI Page](images/openapis.jpeg) + +Expand any section to view description, parameters and schema/sample data.. + +Use “Try it out” button test the API from the Swagger UI itself. It opens a form to enter URI parameters and body data. Enter values and hit “Execute” button. REST API will be invoked and results are shown. + +### 2.8.2 gNMI Unit Testing +gNMI unit testing can be performed using open-source gNMI tools. 'telemetry' docker already contains such tools e.g. gnmi_get, gnmi_set, gnmi_cli etc. You can directly invoke these tools from linux shell of a switch as below : + +``` +docker exec -it telemetry gnmi_get -xpath /openconfig-interfaces:interfaces/interface[name=Ethernet0] -target_addr 127.0.0.1:8080 -insecure + +``` + +* Use 'gnmi_get' tool to perform a get request against the gNMI target in switch. For sample get request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_get.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_get.result respectively. +* Use 'gnmi_set' tool to perform a set request against the gNMI target in switch. For sample set request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_set.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_set.result respectively. +* Use 'gnmi_cli' to get the capabilities of gNMI target. For example : + +``` +docker exec -it telemetry gnmi_cli -capabilities -insecure -logtostderr -address 127.0.0.1:8080 +``` +* Use 'gnmi_cli' to perform subscription request. For sample subscribe request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/ethernet_oper_status_subscribe.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/ethernet_oper_status_subscribe.result respectively. + diff --git a/doc/mgmt/Docker to Host communication.md b/doc/mgmt/Docker to Host communication.md new file mode 100644 index 0000000000..85b19d0231 --- /dev/null +++ b/doc/mgmt/Docker to Host communication.md @@ -0,0 +1,204 @@ +# Feature Name +Docker to Host communication + +# High Level Design Document +#### Rev 0.2 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/28/2019 | Nirenjan Krishnan | Initial version | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.2 | 12/08/2019 | Mike Lazar | Add details about architecture | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.3 | 12/16/2019 | Mike Lazar | Add security and logging info | + + +# About this Manual +This document provides general information about the Docker to Host +communication feature in SONiC. + +# Scope +This document describes the high level design of Docker to Host communication. +This describes the infrastructure provided by the feature, and example usage, +however, it does not describe the individual host-specific features. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|---------------------------------------------------| +| D-Bus | Desktop Bus: https://en.wikipedia.org/wiki/D-Bus | + +# 1 Feature Overview + +The management framework runs within a Docker container, and performs actions +translating the user CLIs or REST requests to actions. Most of these actions +perform some operation on the Redis database, but some of them require +operations to be done on the host, i.e., outside the container. This document +describes the host server, and translib API that are used to communicate between +the Docker container and the host. + + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +* The SONiC Management Framework and Telemetry containers must be able to issue + requests to the host, and return the responses from the host. +* The individual applications that need access to the host must be able to + create a host module and easily issue requests and get responses back from the + host. +* The host communication API shall be available in Translib, and shall provide + both synchronous and asynchronous communication methods. +* It shall be possible to configure the identity of the Linux user accounts who have access to a D-Bus socket. +* It shall be possible to configure containers in such a way that only certain containers (e.g. SONiC Mgmt.) + have access to the D-Bus socket. + +### 1.1.2 Configuration and Management Requirements + +N/A + +### 1.1.3 Scalability Requirements + +N/A + +### 1.1.4 Warm Boot Requirements + +N/A + +## 1.2 Design Overview +### 1.2.1 Basic Approach + +The code will extend the existing Translib modules to provide a D-Bus based +query API to issue requests to the host. The host service will be a Python based +application which listens on known D-Bus endpoints.https://en.wikipedia.org/wiki/D-Bus + +The individual app modules can extend the host service by providing a small +Python snippet that will register against their application endpoint. + +### 1.2.2 Container + +SONiC Management Framework, gNMI Telemetry containers + +### 1.2.3 SAI Overview + +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +All deployments + +## 2.2 Functional Description + +This feature enables management applications to issue +requests to the host to perform actions such as: +* image install / upgrade +* ZTP enable/disable +* initiate reboot and warm reboot using existing scripts +* create show-tech tar file using existing show-tech script +* config save/reload using existing scripts + +# 3 Design +## 3.1 Overview + +The feature extends the SONiC management framework to add a D-Bus service on the +host. This service will register against a known endpoint, and will service +requests to the endpoint. Application modules will add snippets to the host +service, which will automatically register their endpoints, and the app module +in the container can use the APIs provided in Translib to send the request to +the host, and either wait for the response (if the request was synchronous), or +receive a channel and wait for the request to return the response on the +channel (asynchronous request). + +The architecture of a D-Bus host service in a SONiC environment is illustrated in the diagram below: +![](images/docker-to-host-services-architecture.jpg) + +Note. The Linux D-Bus implementation uses Unix domain sockets for client to D-Bus service communications. +All containers that use D-Bus services will bind mount +(-v /var/run/dbus:/var/run/dbus:rw) the host directory where D-Bus service sockets are created. +This ensures that only the desired containers access the D-Bus host services. + +D-Bus provides a reliable communication channel between client (SONiC management container) and service (native host OS) – all actions are acknowledged and can provide return values. It should be noted that acknowledgements are important for operations such as “image upgrade” or “config-save”. In addition, D-Bus methods can return values of many types – not just ACKs. For instance, they can return strings, useful to return the output of a command. + +### 3.1.1 Security of D-Bus Communications +In addition to standard Linux security mechanisms for file/Unix socket access rights (read/write), D-Bus provides a separate security layer, using the D-Bus service configuration files. +This allows finer grain access control to D-Bus objects and methods - D-Bus can restrict access only to certain Linux users. + +### 3.1.2 Command Logging + +It is possible to track and log the user name and the command that the user has requested. +The log record is created in the system log. + +## 3.2 DB Changes +### 3.2.1 CONFIG DB +N/A +### 3.2.2 APP DB +N/A +### 3.2.3 STATE DB +N/A +### 3.2.4 ASIC DB +N/A +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent +N/A +### 3.3.2 Other Process +N/A +## 3.4 SyncD +N/A +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +N/A +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +N/A +#### 3.6.2.2 Show Commands +N/A +#### 3.6.2.3 Debug Commands +N/A +#### 3.6.2.4 IS-CLI Compliance +N/A +### 3.6.3 REST API Support +N/A + +# 4 Flow Diagrams + +![](images/docker-to-host-service.svg) + +# 5 Error Handling + +The `hostQuery` and `hostQueryAsync` APIs return a standard Go `error` object, +which can be used to handle any errors that are returned by the D-Bus +infrastructure. + +# 6 Serviceability and Debug +N/A + +# 7 Warm Boot Support +N/A + +# 8 Scalability +N/A + +# 9 Unit Test +List unit test cases added for this feature including warm boot. + +# 10 Internal Design Information +N/A diff --git a/doc/mgmt/Management Framework.md b/doc/mgmt/Management Framework.md new file mode 100644 index 0000000000..c61ba7d589 --- /dev/null +++ b/doc/mgmt/Management Framework.md @@ -0,0 +1,2568 @@ +# SONiC Management Framework + +## High level design document + +### Rev 0.17 + +## Table of Contents + +* [List of Tables](#list-of-tables) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definition/Abbreviation](#definitionabbreviation) +* [Table 1: Abbreviations](#table-1-abbreviations) +* [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) +* [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) +* [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.1.1 Build time flow](#311-build-time-flow) + * [3.1.2 Run time flow](#312-run-time-flow) + * [3.1.2.1 CLI](#3121-cli) + * [3.1.2.2 REST](#3122-REST) + * [3.1.2.3 gNMI](#3123-gnmi) + * [3.2 SONiC Management Framework Components](#32-sonic-management-framework-components) + * [3.2.1 Build time components](#321-build-time-components) + * [3.2.1.1 Yang to OpenAPI converter](#3211-yang-to-openapi-converter) + * [3.2.1.1.1 Overview](#32111-overview) + * [3.2.1.1.2 Supported HTTP verbs](#32112-supported-http-verbs) + * [3.2.1.1.3 Supported Data Nodes](#32113-supported-data-nodes) + * [3.2.1.1.4 Data Type Details](#32114-data-type-details) + * [3.2.1.1.5 Special Handling](#32115-special-handling) + * [3.2.1.1.6 Future enhancements](#32115-future-enhancements) + * [3.2.1.2 OpenAPI generator](#3212-OpenAPI-generator) + * [3.2.1.3 YGOT generator](#3213-YGOT-generator) + * [3.2.1.4 pyang compiler](#3214-pyang-compiler) + * [3.2.2 Run time components](#322-run-time-components) + * [3.2.2.1 CLI](#3221-cli) + * [3.2.2.2 REST Client SDK](#3222-REST-client-sdk) + * [3.2.2.3 gNMI Client](#3223-gnmi-client) + * [3.2.2.4 REST server](#3224-REST-server) + * [3.2.2.4.1 Transport options](#32241-transport-options) + * [3.2.2.4.2 Translib linking](#32242-Translib-linking) + * [3.2.2.4.3 Media Types](#32243-media-types) + * [3.2.2.4.4 Payload Validations](#32244-payload-validations) + * [3.2.2.4.5 Concurrency](#32245-concurrency) + * [3.2.2.4.6 API Versioning](#32246-api-versioning) + * [3.2.2.4.7 RESTCONF Entity-tag](#32247-restconf-entity-tag) + * [3.2.2.4.8 RESTCONF Discovery](#32248-restconf-discovery) + * [3.2.2.4.9 RESTCONF Query Parameters](#32249-restconf-query-parameters) + * [3.2.2.4.10 RESTCONF Operations](#322410-restconf-operations) + * [3.2.2.4.11 RESTCONF Notifications](#322411-restconf-notifications) + * [3.2.2.4.12 Authentication](#322412-authentication) + * [3.2.2.4.13 Error Response](#322413-error-response) + * [3.2.2.4.14 DB Schema](#322414-db-schema) + * [3.2.2.4.15 API Documentation](#322415-api-documentation) + * [3.2.2.5 gNMI server](#3225-gnmi-server) + * [3.2.2.5.1 Files changed/added](#32251-files-changed/added) + * [3.2.2.5.2 Sample Requests](#32253-sample-requests) + * [3.2.2.6 Translib](#3226-Translib) + * [3.2.2.6.1 App Interface](#32261-app-interface) + * [3.2.2.6.2 Translib Request Handler](#32262-Translib-request-handler) + * [3.2.2.6.3 YGOT request binder](#32263-YGOT-request-binder) + * [3.2.2.6.4 YANG Model Versioning](#32264-yang-model-versioning) + * [3.2.2.6.5 DB access layer](#32265-db-access-layer) + * [3.2.2.7 Transformer](#3227-transformer) + * [3.2.2.7.1 Components](#32271-components) + * [3.2.2.7.2 Design](#32272-design) + * [3.2.2.7.3 Process](#32273-process) + * [3.2.2.7.4 Common App](#32274-common-app) + * [3.2.2.7.5 YANG Extensions](#32275-yang-extensions) + * [3.2.2.7.6 Public Functions](#32276-public-functions) + * [3.2.2.7.7 Overloaded Modules](#32277-overloaded-modules) + * [3.2.2.7.8 Utilities](#32278-utilities) + * [3.2.2.8 Config Validation Library (CVL)](#3228-config-validation-library-cvl) + * [3.2.2.8.1 Architecture](#32281-architecture) + * [3.2.2.8.2 Validation types](#32282-validation-types) + * [3.2.2.8.3 CVL APIs](#32283-cvl-apis) + * [3.2.2.9 Redis DB](#3229-redis-db) + * [3.2.2.10 Non DB data provider](#32210-non-db-data-provider) +* [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 REST SET flow](#41-REST-set-flow) + * [4.2 REST GET flow](#42-REST-get-flow) + * [4.3 Translib Initialization flow](#43-Translib-initialization-flow) + * [4.4 gNMI flow](#44-gNMI-flow) + * [4.5 CVL flow](#45-CVL-flow) +* [5 Developer Work flow](#5-Developer-Work-flow) + * [5.1 Developer work flow for custom (SONiC/CVL) YANG](#51-Developer-work-flow-for-custom-SONiCCVL-YANG) + * [5.1.1 Define Config Validation YANG schema](#511-Define-Config-Validation-YANG-schema) + * [5.1.2 Generation of REST server stubs and Client SDKs for YANG based APIs](#512-Generation-of-REST-server-stubs-and-Client-SDKs-for-YANG-based-APIs) + * [5.1.3 Config Translation App (Go language)](#513-Config-Translation-App-Go-language) + * [5.1.4 IS CLI](#514-IS-CLI) + * [5.1.5 gNMI](#515-gNMI) + * [5.2 Developer work flow for standard (OpenConfig/IETF) YANG](#52-Developer-work-flow-for-standard-OpenConfigIETF-YANG) + * [5.2.1 Identify the standard YANG module for the feature for northbound APIs](#521-Identify-the-standard-YANG-module-for-the-feature-for-northbound-APIs) + * [5.2.2 Define the Redis schema for the new feature. (not applicable for legacy/existing feature)](#522-Define-the-Redis-schema-for-the-new-feature-not-applicable-for-legacyexisting-feature) + * [5.2.3 Define Config Validation YANG schema](#523-Define-Config-Validation-YANG-schema) + * [5.2.4 Generation of REST server stubs and Client SDKs for YANG based APIs](#524-Generation-of-REST-server-stubs-and-Client-SDKs-for-YANG-based-APIs) + * [5.2.5 Config Translation App (Go language)](#525-Config-Translation-App-Go-language) + * [5.2.6 IS CLI](#526-IS-CLI) + * [5.2.7 gNMI](#527-gNMI) +* [6 Error Handling](#6-error-handling) +* [7 Serviceability and Debug](#7-serviceability-and-debug) +* [8 Warm Boot Support](#8-warm-boot-support) +* [9 Scalability](#9-scalability) +* [10 Unit Test](#10-unit-test) +* [11 Appendix A](#11-appendix-a) +* [12 Appendix B](#11-appendix-b) + + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:-----------------------:|-----------------------------------| +| 0.1 | 06/13/2019 | Anand Kumar Subramanian | Initial version | +| 0.2 | 07/05/2019 | Prabhu Sreenivasan | Added gNMI, CLI content from DELL | +| 0.3 | 08/05/2019 | Senthil Kumar Ganesan | Updated gNMI content | +| 0.4 | 08/07/2019 | Arun Barboza | Clarifications on Table CAS | +| 0.5 | 08/07/2019 | Anand Kumar Subramanian | Translib Subscribe support | +| 0.6 | 08/08/2019 | Kwangsuk Kim | Updated Developer Workflow and CLI sections | +| 0.7 | 08/09/2019 | Partha Dutta | Updated Basic Approach under Design Overview | +| 0.8 | 08/15/2019 | Anand Kumar Subramanian | Addressed review comments | +| 0.9 | 08/19/2019 | Partha Dutta | Addressed review comments related to CVL | +| 0.10 | 09/25/2019 | Kwangsuk Kim | Updated Transformer section | +| 0.11 | 09/30/2019 | Partha Dutta | Updated as per SONiC YANG guideline | +| 0.12 | 10/19/2019 | Senthil Kumar Ganesan | Added Appendix B | +| 0.13 | 11/27/2019 | Anand Kumar Subramanian | Added new APIs in translib | +| 0.14 | 12/03/2019 | Sachin Holla | RESTCONF yang library and other enhancements | +| 0.15 | 12/19/2019 | Partha Dutta | Added new CVL API, platform and custom validation details | +| 0.16 | 04/08/2020 | Sachin Holla | API versioning enhancement | +| 0.17 | 04/08/2020 | Mohammed Faraaz | OpenAPI 3.0 enhancements | +| 0.18 | 04/09/2020 | Kwangsuk Kim | Updated CLI and Transformer enhancement | + +## About this Manual + +This document provides general information about the Management framework feature implementation in SONiC. + +## Scope + +This document describes the high level design of Management framework feature. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| CVL | Config Validation Library | +| NBI | North Bound Interface | +| ABNF | Augmented Backus-Naur Form | +| YANG | Yet Another Next Generation | +| JSON | Java Script Object Notation | +| XML | eXtensible Markup Language | +| gNMI | gRPC Network Management Interface | +| YGOT | YANG Go Tools | + +## 1 Feature Overview + +Management framework is a SONiC application which is responsible for providing various common North Bound Interfaces (NBIs) for the purposes of managing configuration and status on SONiC switches. The application manages coordination of NBI’s to provide a coherent way to validate, apply and show configuration. + +### 1.1 Requirements + +* Must provide support for: + + 1. Standard [YANG](https://tools.ietf.org/html/rfc7950) models (e.g. OpenConfig, IETF, IEEE) + 2. Custom YANG models ([SONiC YANG](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md)) + 3. Industry-standard CLI / Cisco like CLI + +* Must provide support for [OpenAPI spec](https://OpenAPI.io/specification/) to generate REST server side code +* Must provide support for NBIs such as: + + 1. CLI + 2. gNMI + 3. REST/RESTCONF + +* Must support the following security features: + + 1. Certificate-based authentication + 2. User/password based authentication + 3. Role based authorization + +* Ease of use for developer workflow + + 1. Specify data model and auto-generate as much as possible from there + +* Must support Validation and Error Handling - data model, platform capability/scale, dynamic resources +* SNMP integration in SONiC is left for future study + +### 1.2 Design Overview + +Management framework makes use of the translation library (Translib) written in golang to convert the data models exposed to the management clients into the Redis ABNF schema format. Supported management servers can make use of the Translib to convert the incoming payload to SONiC ABNF schema and vice versa depending on the incoming request. Translib will cater to the needs of REST and gNMI servers. Later the Translib can be enhanced to support other management servers if needed. This framework will support both standard and custom YANG models for communication with the corresponding management servers. Management framework will also take care of maintaining data consistency, when writes are performed from two different management servers at the same time. Management framework will provide a mechanism to authenticate and authorize any incoming requests. Management framework will also take care of validating the requests before writing them into the Redis DB. Config Validation Library is used for syntactic and semantic validation of ABNF JSON based on YANG derived from Redis ABNF schema. + +#### 1.2.1 Basic Approach + +* Management framework takes comprehensive approach catering: + * Standard based YANG Models and custom YANG + * Open API spec + * Industry standard CLI + * Config Validation +* REST server, gNMI server, App module and Translib - all in Go +* Translation by using the Translib Library and application specific modules +* Marshalling and unmarshalling using YGOT +* Redis updated using CAS(Check-and-Set) trans. (No locking, No rollback) +* Config Validation by using YANG model from ABNF schema +* CLI with Klish framework + +#### 1.2.2 Container + +The management framework is designed to run in a single container named “sonic-mgmt-framework”. The container includes the REST server linked with Translib, and CLI process. +The gNMI support requires the gNMI server which is provided as a part of sonic-telemetry container. We would like to rename this container as the sonic-gnmi container as now it can perform configurations as well through the gNMI server. +Will introduce a new container sonic-mgmt-common to host the common code that is used both in the mgmt-framework and sonic-telemetry container. This new repo will compile into static libraries that will be used in the other two repos. This way sonic-telemetry repo can be compiled without the mgmt-framework being present in the code base. + +## 2 Functionality + +### 2.1 Target Deployment Use Cases + +1. Industry Standard CLI which will use REST client to talk to the corresponding servers to send and receive data. +2. REST client through which the user can perform POST, PUT, PATCH, DELETE, GET operations on the supported YANG paths. +3. gNMI client with support for capabilities, get, set, and subscribe based on the supported YANG models. + +## 3 Design + +### 3.1 Overview + +The SONiC management framework comprises two workflows: + +1. Build time flow +2. Run time flow + +as show in the architecture diagram below. + +![Management Framework Architecture diagram](images/Mgmt_Frmk_Arch.jpg) + +#### 3.1.1 Build time flow + +The Developer starts by defining the desired management objects and the access APIs to provide for the target application. This can be done in one of the two ways: - +1) A YANG data model +2) An OpenAPI spec + +This can be an independent choice on an application by application basis. However note that using YANG allows for richer data modelling, and therefore superior data validation. + +1. In case of YANG, if the developer chooses standard YANG model (Openconfig, IETF etc.), a separate SONiC YANG model has to be written based on Redis ABNF schema for validating Redis configuration and transformer hints should be written in a deviation file for standard YANG model to Redis DB coversion and vice versa (refer to [3.2.2.7 Transformer](#3227-transformer) for details). However, if custom SONiC YANG model is written based on guidelines, CVL YANG is automatically derived from it and the same is used for validation purpose and there is no need of writing any deviation file for transformer hints. Based on the given YANG model as input, the pyang compiler generates the corresponding OpenAPI spec which is in turn given to the OpenAPI generator to generate the REST client SDK and REST server stubs in golang. The YANG data model is also provided to the [YGOT](https://github.com/openconfig/YGOT) generator to create the YGOT bindings. These are used on the interface between Translib and the selected App module. Specifically, Translib populates the binding structures based upon the incoming server payload, and the App module processes the structure accordingly. Additionally, a YANG annotation file must also be provided, for data models that do not map directly to the SONiC YANG structure. The requests in this case will be populated into the YGOT structures and passed to App module for conversion. The App module uses the YANG annotations to help convert and map YANG objects to DB objects and vice-versa. + +2. In case of OpenAPI spec, it is directly given to the [OpenAPI](https://OpenAPI.io) generator to generate the REST client SDK and REST server stubs in golang. In this case the REST server takes care of validating the incoming request to be OpenAPI compliant before giving the same to Translib. There is no YANG, and therefore no YGOT bindings are generated or processed, and so the Translib infra will invoke the App module functions with the path and the raw JSON for App modules to convert. For configuration validation purpose, SONiC YANG model has to be written based on Redis ABNF schema. + +#### 3.1.2 Run time flow + +##### 3.1.2.1 CLI + +1. CLI uses the KLISH framework to provide a CLI shell. The CLI request is converted to a corresponding REST client request using c, and is sent to the REST server. +2. The OpenAPI generated REST server handles all the REST requests from CLI and invokes a common handler for all the create, update, replace, delete and get operations along with path and payload. This common handler converts all the requests into Translib arguments and invokes the corresponding Translib provided APIs. +3. Translib API then calls the corresponding handler defined in Common App module, subsequently invokes Transformer functions with YGOT structured request to perform model to model transformation, i.e. Openconfig Model to SONiC model, to generate the request DB map. +4. Common App handlers pass the request DB map to DB Access module to access REDIS Database, process the request DB map. The response to GET operation is passed to Transformer to perform reverse transformation to Openconfig Model, and wired out to CLI as JSON response. +5. Further processing of CLI commands will be handled by Translib componenets that will be discussed in more detail in the later sections. + +##### 3.1.2.2 REST + +1. REST client will use the OpenAPI generated client SDK to send the request to the REST server. +2. From then on the flow is similar to the one seen in the CLI. + +##### 3.1.2.3 gNMI + +GNMI service defines a gRPC-based protocol for the modification and retrieval of configuration from a target device, as well as the control and generation of telemetry streams from a target device to a data collection system. Refer [GNMI spec](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md) + +![GNMI Service High level diagram (Proposed)](images/GNMI_Server.png) + +1. Existing SONiC telemetry framework has been extended to support the new GNMI services. +2. All 4 GNMI services are supported: Get, Set, Capabilities and Subscribe. +3. A new transl data client is added to process the incoming YANG-based request (either standard or proprietary) +4. The new transl data client relies on Translib infra provided to translate, get, set of the YANG objects. + +More details on the GNMI server, Client and workflow provided later in the document. + +### 3.2 SONiC Management Framework Components + +Management framework components can be classified into + +1. Build time components +2. Run time components + +#### 3.2.1 Build time components + +Following are the build time components of the management framework + +1. Pyang compiler (for YANG to OpenAPI conversion) +2. OpenAPI generator +3. YGOT generator +4. pyang compiler (for YANG to YIN conversion) + +##### 3.2.1.1 YANG to OpenAPI converter + +##### 3.2.1.1.1 Overview + +Open source Python-based YANG parser called pyang is used for YANG parsing and building a Python object dictionary. A custom plugin is developed to translate this Python object dictionary into an OpenAPI spec. The OpenAPI specs are [3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.3.md#schemaObject) complaint. + +URI format and payload is RESTCONF complaint and is based on the [RFC8040](https://tools.ietf.org/html/rfc8040). The Request and Response body is in JSON format in this release. + +##### 3.2.1.1.2 Supported HTTP verbs + +Following table lists the HTTP methods generated for different types of YANG nodes. + + YANG node type | HTTP methods +------------------------|----------------- +Configuration data | POST, PUT, PATCH, DELETE, GET, HEAD +Non configuration data | GET, HEAD +YANG RPC | POST + +##### 3.2.1.1.3 Supported Data Nodes + +For each of the below-listed Data keywords nodes in the YANG model, the OpenAPI (path) will be generated + +* Container +* List +* Leaf +* Leaf-list +* Rpc + +##### 3.2.1.1.4 Data Type Details + + YANG Type | OpenAPI Type +--------------|------------------ +int8 | Integer +int16 | Integer +int32 | Integer +int64 | Integer +uint8 | Integer +uint16 | Integer +uint32 | Integer +uint64 | Integer +decimal64 | Number +String | string +Enum | Enum +Identityref | String (Future can be Enum) +long | Integer +Boolean | Boolean +Binary | String with Format as Binary +bits | integer + +* All list keys will be made mandatory in the payload and URI +* YANG mandatory statements will be mapped to the required statement in OpenAPI +* Default values, Enums are mapped to Default and Enums statements of OpenAPI +* Since OpenAPI spec does not provide a way to define an unsigned types, a custom vendor extension x-yang-type is introduced which will contain the actual yang data type of a leaf/leaf-list. This vendor extension will be part of type object. + +Example below + +``` + /restconf/data/ietf-snmp:snmp/tlstm/cert-to-name={id}: + put: + tags: + - ietf-snmp + parameters: + - name: id + in: path + required: true + schema: + format: int32 + maximum: 4294967295 + minimum: 0 + type: integer + **x-yang-type: uint32** +``` +* As seen in above example all Integer types will have maximum and minimum fields. If the range statement is defined on the YANG node, then the value for maximum and minimum is derived from the range statement otherwise default range specified by YANG will be considered. And also a custom vendor extension called x-range will include a copy of range statement's value. If a type has one more range statements (may be indirectly using typedefs) all the range statements will be included as part of x-range extension separated by '|' character. + +Exampe below +``` + - format: int32 + **maximum: 2147483647** + **minimum: 1** + type: integer + **x-range: 1..2147483647 | 4..10** + x-yang-type: int32 +``` +* A string data types will have a vendor extension x-pattern whose value will be a regex. This field will only be present when the YANG node has it. Although OpenAPI 3.0 has a pattern field under schema object, but it is not used because the regex required for OpenAPI 3.0 is a W3C complaint regex but Openconfig models follow POSIX style regex. If a type has one more regex statements (may be indirectly using typedefs) all the regex statements will be included as part of x-pattern extension, they will be ANDed together. + +Example below +``` + - maxLength: 18446744073709551615 + minLength: 0 + type: string + **x-pattern: (([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?** + x-yang-type: string +``` + +* As seen in above example all String types will have maxLength and minLength fields. If the length statement is defined on the YANG node, then the value for maxLength and minLength is derived from the length statement otherwise default length specified by YANG will be considered. And also a custom vendor extension called x-length will include a copy of length statement's value. If a type has one more length statements (may be indirectly using typedefs) all the length statements will be included as part of x-length extension separated by '|' character. + +* An Union data type is supported through oneOf directive. oneOf is a JSON schema directive supported by openAPI 3.0. It takes an array of objects and at any given time only one of item can be used. + +Example below +``` + - name: security-model + in: path + required: true + schema: + oneOf: + - enum: + - v1 + - v2c + - usm + - tsm + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-yang-type: string + - format: int32 + maximum: 2147483647 + minimum: 1 + type: integer + x-range: 1..2147483647 + x-yang-type: int32 +``` + +##### 3.2.1.1.5 Special handling + +* A list and leaf-list will have maxItems and minItems properties set if a YANG node defines max-elements and min-elements statements. + +Example below +``` + security-model: + type: array + items: + oneOf: + - enum: + - v1 + - v2c + - usm + - tsm + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-yang-type: string + - format: int32 + maximum: 2147483647 + minimum: -2147483648 + type: integer + x-range: 1..2147483647 + x-yang-type: int32 + minItems: 1 +``` +* A Choice-case statement is supported using oneOf directive. + +Example below +``` + ietf-snmp:auth: + type: object + properties: {} + **oneOf:** + - type: object + properties: + md5: + type: object + properties: + key: + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-pattern: ([0-9a-fA-F]{2}(:[0-9a-fA-F]{2})*)? + x-yang-type: string + required: + - key + - type: object + properties: + sha: + type: object + properties: + key: + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-pattern: ([0-9a-fA-F]{2}(:[0-9a-fA-F]{2})*)? + x-yang-type: string + required: + - key +``` + +##### 3.2.1.1.6 Future enhancements + +* Support for additional YANG actions and notifications(if required). +* Support for RESTCONF query parameters such as depth, filter, etc +* Convert Yang identities in to enums + +##### 3.2.1.2 OpenAPI generator + +OpenAPI-codegen tool (github.com/OpenAPITools/openapi-generator) is used to generate REST server and client code from the OpenAPI definitions. It consumes the OpenAPI definitions generated from YANG files and any other manually written OpenAPI definition files. + +REST server is generated in Go language. Customized OpenAPI-codegen templates are used to make each server stub invoke a common request handler function. The common request handler will invoke Translib APIs to service the request. + +REST client is generated in Python language. Client applications can generate the REST client in any language using standard OpenAPI-codegen tool. + +##### 3.2.1.3 YGOT generator + +YGOT generator generates Go binding structures for the management YANG. The generated Go binding structures are consumed by the Translib to validate the incoming payload and help in conversion of Redis data to management YANG specific JSON output payload. + +##### 3.2.1.4 pyang compiler + +Open source pyang tool is used to compile YANG models and generate output file in required format. For example SONiC YANG model is compiled and YIN schema file is generated for validation purpose. Similarly, after compiling YANG model, OpenAPI spec file is generated for generating REST client SDK and server code using OpenAPI-codegen. + +#### 3.2.2 Run time components + +Following are the run time components in the management framework + +1. CLI +2. REST Client SDK +3. REST server +4. gNMI Client +5. gNMI server +6. Translib +7. Config Validation Library (CVL) +8. Redis DB +9. Non DB data provider + +##### 3.2.2.1 CLI + +Open source Klish is integrated into sonic-mgmt-framework container to provide the command line interface tool to perform more efficient network operations more efficiently in SONiC. Klish will provide the core functionality of command parsing, syntax validation, command help and command auto-completion. The following diagram shows how the CLI commands are built, processed, and executed. + +![CLI components Interaction Diagram](images/cli_interactions.jpg) + +1. CLI command input from user +2. Klish invokes the actioner defined in command tag +3. Actioner invokes either sub shell or the built-in methods to make a REST client API call using C based libcurl or Python Requests package to the REST server. +4. Receive response from OpenAPI clientAPI and pass it to renderer scripts. +5. Renderer scripts processes the JSON response from REST Client and optionally formats the output using a Jinja template +6. CLI output is rendered to the terminal. + +###### 3.2.2.1.1 CLI components + +CLI consists of the following components. + +* **Open source Klish** - CLI parser framework to support Command Line Interface Shell +* **XML files** to define CLI command line options and actions + * Klish uses XML to define CLI commands to build the command tree. Klish provides modular CLI tree configurations to build command trees from multiple XML files. XML elements can be defined with macros and entity references, which are then preprocessed by utility scripts to generate the expanded XML files that are ultimately used by Klish. +* **Actioner** - Python scripts or built-in methods defined as a command `action`, to form the request body and invoke the OpenAPI client API +* **Renderer** - Python scripts defined with Jinja templates. Receives the JSON response from Swagger API and use the jinja2 template file to render and format CLI output. +* **Preprocess scripts** - Validates XML files and applies some processing from a developer-friendly form into a "raw" form that is compatible with Klish. + +###### 3.2.2.1.2 Preprocessing XML files + +Multiple scripts are executed at build time to preprocess special XML tags/attributes - MACRO substitution, adding pipe processing, etc. - and generate target XML files in a form that is consumable by Klish. + +The XML files are also validated as part of compilation. `xmllint` is used to validate all the processed XML files after macro substitution and pipe processing against the detailed schema defined in `sonic-clish.xsd`. Once the XML files are fully validated and preprocessed, the target XML files are generated in the folder `${workspace}/sonic-mgmt-framework/build/cli/target/command-tree`. + +The following preprocessing scripts are introduced: +* `klish_ins_def_cmd.py` - append the "exit" and "end" commands to the views of the Klish XML files +* `klish_insert_pipe.py` - extend every show and get COMMAND with pipe option +* `klish_platform_features_process.sh` - validate all platform XML files. Generate the entity.XML files. +* `klish_replace_macro.py` – perform macro substitution on the Klish XML files + +###### 3.2.2.1.3 MACROs + +There are some CLI commands that can have the same set of options, where the set of XML tags would need to be repeated in all those CLI commands. Macros can be used to avoid such repetitions and to keep the options in one place, so that it is possible to make a reference to a macro in multiple command definitions. There are cases where we may need to use variations in the values of these macro options. In such cases, it is also possible to pass those XML attributes as an argument to macros and substitute those values inside the macro definition. The macro definition is referred as the ```` and ```` tags. `klish_replace_macro.py` is used to process macros at the compile time to expand the references to the target XML files. +The macro definition files are located in the folder `${workspace}/sonic-mgmt-framework/src/CLI/clitree/macro`. + +Example: + +Before macro substitution: + +```XML + + + + + ... + +``` + +After macro substitution: + +```XML + + + + + + + + + ... + +``` + +###### 3.2.2.1.4 ENTITY +XML files can include an ENTITY that refers to a predefined value. Entity is typically used to define platform specific values and processed by `klish_platform_features_process.sh` to prepend the ENTITY values to the target XML files. By default, there is a default file called `platform_dummy.XML` that defines a platform default ENTITY list. Note that the platform specific is not supported yet. + +Example: `platform_dummy.XML` + +```XML + + START_PORT_ID + MAX_PORT_IDma + START_SUB_PORT_ID + MAX_SUB_PORT_ID + MAX_MTU + +``` + +The ENTITY name can be referenced in command definitions. For example, PTYPE for RANGE_MTU, used for interface commands: + +```XML + +``` + +###### 3.2.2.1.5 Actioner + +The Actioner is used to convert CLI commands to the corresponding OpenAPI client requests on the target resources. It is defined with the `` tag in the XML file. There are three different methods available in the Actioner: sub-shell, builtin clish_restcl and clish_pyobj. CLI builtin functions "clish_pyobj" and "clish_restcl" can be applied to reduce time take to execute a command by eliminating the sub-shell interpreter overhead. + +1) Spawn a sub-shell to interpret Python scripts +Klish spawns a sub-shell to interpret the Python scripts defined in a command's `` tag. +The sub-shell runs the wrapper script `sonic_cli_.py` +``` + sonic_cli_.py [parameters . . .] +``` +The `sonic_cli_.py` has a dispatch function to call a OpenAPI client method with parameters passed from user input. + +Example: +```XML + + + + + + if test "${direction-switch}" = "in"; then + Python $SONIC_CLI_ROOT/target/sonic-cli.py post_list_base_interfaces_interface ${access-list-name} ACL_IPV4 ${iface} ingress + else + Python $SONIC_CLI_ROOT/target/sonic-cli.py post_list_base_interfaces_interface ${access-list-name} ACL_IPV4 ${iface} egress + fi + + + ... +``` + +2) Invoke the built-in function, clish_restcl, to use libcurl to make REST client call +This builtin uses libcurl to connect to the REST server +Format of ACTION tag argument: +oper= url= body={..} +oper can be PUT/POST/PATCH/DELETE +body is optional. +oper GET is not supported as we currently don't handle rendering using jinja templates. +Example: +``` +oper=PATCH url=/restconf/data/openconfig-interfaces:interfaces/interface=Vlan${vlan-id}/config body={"openconfig-interfaces:config": {"name": "Vlan${vlan-id}"}} +``` + +3) Invoke the built-in function, clish_pyobj, to use embedding Python to make REST client call +This builtin uses embedded python approach. + +Format of ACTION tag argument: + +( is the actioner script without the extension. The name cannot include "-") + +Example: +``` +sonic_cli_if patch_openconfig_interfaces_interfaces_interface_config Vlan${vlan-id} +``` + +###### 3.2.2.1.6 Renderer scripts + +The actioner script receives the JSON output from the OpenAPI client API and invokes the renderer script. The renderer script will send the JSON response to the jinja2 template file to parse the response and generate the CLI output. + +Example: "show acl" + +``` +{% set acl_sets = acl_out['openconfig_aclacl']['acl_sets']['acl_set'] %} + {% for acl_set in acl_sets %} + Name: {{ acl_set['state']['description'] }} + {% endfor %} +``` + +###### 3.2.2.1.7 Workflow (to add a new CLI) + +The following steps are to be followed when a new CLI is to be added. +1. Create an XML file that defines CLI command and parameters that the command requires. +2. Define the CLI help string to be displayed and datatype for the parameters. New parameter types (PTYPES), macros, and entities can be defined and used in the XML files. Valid XML tags are defined in the `sonic-clish.xsd` file. +3. Add the actioner to `` tag to run the wrapper script with the OpenAPI client method name and parameters +4. Add the code to the wrapper script to construct the payload in `generate_body()` and handle the response +5. For ‘show’ commands, create a Jinja template to format the output + + +##### 3.2.2.2 REST Client SDK + +Framework provides OpenAPI-codegen generated Python client SDK. Developers can generate client SDK code in other programming languages from the OpenAPI definitions as required. + +Client applications can use OpenAPI generated client SDK or any other REST client tool to communicate with the REST server. + +##### 3.2.2.3 gNMI Client + +SONiC Telemetry service provides the gNMI server, while the client must be provided by the user. gNMI is typically used as a programmatic interface and therefore it is typically called directly from programming language environments using their respective gRPC libraries (https://github.com/grpc/grpc). For testing and scripting purposes, several CLI programs are provided as well. + +GNMI clients developed by google and modified by JipanYANG.(github.com/jipanYANG/gnxi/gnmi_get, github.com/jipanYANG/gnxi/gnmi_set) +was taken and further modified to support new features. gnmi_cli from openconfig (https://github.com/openconfig/gnmi/tree/master/cmd/gnmi_cli) is also taken for testing subscribe and capabilities operations. Finally, a new client was developed gnoi_client, for test gNOI RPC operations. + +Note: Although the gRPC protocol allows for many encodings to be used, our usage is restricted to JSON_IETF encoding. + +Supported RPC Operations: +------------------------- +- Get: Get one or more paths and have value(s) returned in a GetResponse. +- Set: Update, replace or delete objects + + Update: List of one or more objects to update + + Replace: List of one or objects to replace existing objects, any unspecified fields wil be defaulted. + + Delete: List of one or more object paths to delete +- Capabilities: Return gNMI version and list of supported models and model versions. + + A gNMI Extension field has been added as well to return the SONiC model "bundle version". +- Subscribe: + + Subscribe to paths using either streaming or poll, or once based subscription, with either full current state or updated values only. + * Once: Get single subscription message. + * Poll: Get one subscription message for each poll request from the client. + * Stream: Get one subscription message for each object update (called ON_CHANGE mode), or at each sample interval if using sample mode. target_defined uses the values pre-configured for that particular object. Not all paths support ON_CHANGE mode due to performance considerations, while all paths will support sample mode and therefore target defined as well since it will default to sample if ON_CHANGE is not supported. + + +Example Client Operations: +-------------------------- +Using opensource clients, these are example client operations. The .json test payload files are available here: https://github.com/project-arlo/sonic-mgmt-common/tree/master/src/Translib/test + +Get: +---- +`./gnmi_get -xpath /openconfig-acl:acl/interfaces -target_addr 127.0.0.1:8080 -alsologtostderr -insecure true -pretty` + +Set: +---- +Replace: +-------- + `./gnmi_set -replace /openconfig-acl:acl/:@./test/01_create_MyACL1_MyACL2.json -target_addr 127.0.0.1:8080 -alsologtostderr -insecure true -pretty` +Delete: +------- + `./gnmi_set -delete /openconfig-acl:acl/ -target_addr 127.0.0.1:8080 -insecure` + +Subscribe: +---------- +Streaming sample based: +----------------------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type s -streaming_sample_interval 3 -streaming_type SAMPLE -q /openconfig-acl:acl/ -v 0 -target YANG` + +Poll based: +----------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type p -polling_interval 1s -count 5 -q /openconfig-acl:acl/ -v 0 -target YANG` + +Once based: +----------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type o -q /openconfig-acl:acl/ -v 0 -target YANG` + + +gNOI: +-------------- +gNOI (gRPC Network Operations Interface) extends the gNMI server, adding new custom RPC's to execute management functions on the switch. + +gNOI Clear Neighbor: +------------------- +``` +gnoi_client -module Sonic -rpc clearNeighbors -jsonin '{"sonic-neighbor:input": {"force": true, "ip": "4.4.4.1"}}' -insecure +``` + +##### 3.2.2.4 REST Server + +The management REST server is a HTTP server implemented in Go language. +It supports following operations: + +* RESTCONF APIs for YANG data +* REST APIs for manual OpenAPI definitions + +###### 3.2.2.4.1 Transport options + +REST server supports only HTTPS transport and listens on default port 443. +Server port can be changed through an entry in ConfigDB REST_SERVER table. + +By default a temporary self signed certificate is used as TLS server certificate. +It can be overridden by specifiying a valid TLS private key and certificate file +paths through the REST_SERVER table. + +REST_SERVER table schema is described in [DB Schema](#322414-db-schema) section. + +###### 3.2.2.4.2 Translib linking + +REST server will statically link with Translib. For each REST request, the server +invokes Translib API which then invokes appropriate App module to process the request. +Below is the mapping of HTTP operations to Translib APIs: + + HTTP Method | Translib API | Request data | Response data +-------------|------------------|---------------|--------------- + GET | translib.Get | path | status, payload + POST | translib.Create | path, payload | status + POST (for YANG RPC) | translib.Action | path, payload | status, payload + PATCH | translib.Update | path, payload | status + PUT | translib.Replace | path, payload | status + DELETE | translib.Delete | path | status + HEAD | translib.Get | path | status, (payload ignored) + OPTIONS | - | - | - + +More details about Translib APIs are in section [3.2.2.6](#3_2_2_6-Translib). + +REST OPTIONS requests do not invoke any Translib API. They return list of supported HTTP methods +for requested path in the "Allow" response header. If PATCH method was supported, the response +will also include an "Accept-Patch" header with the value "application/yang-data+json". + +###### 3.2.2.4.3 Media Types + +YANG defined RESTCONF APIs support **application/yang-data+json** media type. +Request and response payloads follow [RFC7951](https://tools.ietf.org/html/rfc7951) +defined encoding rules. Media type **application/yang-data+xml** is not supported +in first release. + +OpenAPI defined REST APIs can use any media type depending on App module +implementation. However content type negotiation is not supported by REST server. +A REST API should not be designed to consume or produce multiuple content types. +OpenAPI definition for each REST API should have maximun one cone media type in +its "consumes" and "produces" statements. + +###### 3.2.2.4.4 Payload Validations + +REST server does not validate request payload for YANG defined RESTCONF APIs. +Payload will be validated automatically in lower layers when it gets loaded +into YGOT bindings. + +For OpenAPI defined REST APIs the REST server will provide limited payload +validation. JSON request payloads (content type **application/json**) will be +validated against the schema defined in OpenAPI. Response data and non-JSON +request data are not validated by the REST server - this is left to the App module. + +###### 3.2.2.4.5 Concurrency + +REST server will accept concurrent requests. Translib provides appropriate locking mechanism - parallel reads and sequential writes. + +###### 3.2.2.4.6 API Versioning + +REST server will allow clients to specify API version through a custom HTTP header **Accept-Version**. + + Accept-Version: 1.0.3 + +Version text should be in **MAJOR.MINOR.PATCH** format. REST server will extract version text from +the request header and pass it to the Translib API as **ClientVersion** argument. +Section [3.2.2.6.4.2](#322642-version-checks) explains Translib version checking logic. + +Version checks are bypassed if Accept-Version header is not present in the request. + +For YANG defined RESTCONF APIs, the server's version can be discovered through the standard YANG +module library API "GET /restconf/data/ietf-yang-library:modules-state/module-set-id". +Client should not pass Accept-Version header for this API! This is safe since module-set-id +is a standard API defined by [RFC7895](https://tools.ietf.org/html/rfc7895) for this very purpose. + +API Versioning is not supported for manual OpenAPI definitions in this release. +Accept-Version header value will be ignored even if specified. + +###### 3.2.2.4.7 RESTCONF Entity-tag + +REST server does not maintain entity-tag and last-modified timestamp for configuration data nodes. +GET and HEAD responses does not include "ETag" and "Last-Modified" headers. + +[RFC7232](https://tools.ietf.org/html/rfc7232) style HTTP conditional requests is also not supported. +REST server ignores "If-Match", "If-Modified-Since" like conditional request headers. + +###### 3.2.2.4.8 RESTCONF Discovery + +REST server supports following APIs for clients to discover various RESTCONF protocol features +as described in [RFC8040](https://tools.ietf.org/html/rfc8040). + +Method | Path | Purpose +-------|------------------------------------------------|--------------------------- +GET | /.well-known/host-meta | RESTCONF root path +GET | /restconf/yang-library-version | YANG library version +GET | /restconf/data/ietf-yang-library:modules-state | RFC7895 YANG module library +GET | /restconf/data/ietf-restconf-monitoring:restconf-state/capabilities | RESTCONF Capabilities +GET | /models/yang/{filename} | YANG download + +###### 3.2.2.4.8.1 RESTCONF root + +Server supports RESTCONF root path discovery through "GET /.well-known/host-meta" API +as described in [RFC8040, section 3.1](https://tools.ietf.org/html/rfc8040#page-18). +RESTCONF root path is "/restconf". + +###### 3.2.2.4.8.2 Yang library version + +REST server supports "GET /restconf/yang-library-version" API to advertise YANG library version. +Response will indicate version "2016-06-21" as described in [RFC8040, section 3.3.3](https://tools.ietf.org/html/rfc8040#section-3.3.3). +This advertises that the server supports [RFC7895](https://tools.ietf.org/html/rfc7895) compliant +YANG library operations. + +###### 3.2.2.4.8.3 YANG module library + +REST server allows clients to discover and download all YANG modules supported by the server +via "GET /restconf/data/ietf-yang-library:modules-state" API. Response data includes YANG module +information as per [RFC7895](https://tools.ietf.org/html/rfc7895) requirements. + +REST server allows clients to download the YANG files via "GET /models/yang/{filename}" API. +YANG module library response includes full download URL for every YANG module entry. + +Note - RFC7895 has been recently obsoleted by [RFC8525](https://tools.ietf.org/html/rfc8525). +It supports YANG library with multi data stores and data store schemas. However these features +are not available in SONiC. Hence REST server does not implement RFC8525 YANG library APIs (for now). + +###### 3.2.2.4.8.4 RESTCONF capabilities + +REST server supports "GET /restconf/data/ietf-restconf-monitoring:restconf-state/capabilities" API to +advertise its capabilities as described in [RFC8040, section 9.1](https://tools.ietf.org/html/rfc8040#section-9.1). +REsponse includes below mentioned capability information. + + urn:ietf:params:restconf:capability:defaults:1.0?basic-mode=report-all + +###### 3.2.2.4.9 RESTCONF Query Parameters + +RESTCONF Query Parameters will be supported in future release. All query parameters will be ignored by REST server in this release. + +###### 3.2.2.4.10 RESTCONF Operations + +REST server supports invoking YANG RPCs through "POST /restconf/operations/{rpc_name}" API +as described in [RFC8040, section 4.4.2](https://tools.ietf.org/html/rfc8040#section-4.4.2). +Operations modeled through YANG v1.1 action statement are not supported in this release. + +YGOT binding objects are not available for YANG RPC input and output data model. +Hence payload validation is not performed by REST server or Translib for these APIs. +App modules will receive raw JSON data. + +###### 3.2.2.4.11 RESTCONF Notifications + +RESTCONF Notification are not supported by framework. Clients can use gNMI for monitoring and notifications. + +###### 3.2.2.4.12 Authentication + +REST server supporsts following authentication modes. + +* HTTP Basic authentication (username/password authentication) +* HTTP Bearer token authentication with JSON Web Token (JWT) format. +* TLS Certificate authentication +* Any combination of above 3 modes +* No authentication + +Details are in [SONiC RBAC HLD](https://github.com/project-arlo/SONiC/blob/master/doc/aaa/SONiC%20RBAC%20HLD.md). + +By default HTTP Baisc and bearer token authentication modes are enabled. +It can be overridden through ConfigDB [REST_SERVER table](#322414-db-schema) entry. + +###### 3.2.2.4.13 Error Response + +REST server sends back HTTP client error (4xx) or server error (5xx) status when request processing +fails. Response status and payload will be as per RESTCONF specifications - [RCF8040, section7](https://tools.ietf.org/html/rfc8040#page-73). +Error response data will be a JSON with below structure. Response Content-Type will be +"application/yang-data+json". + + +---- errors + +---- error* + +---- error-type "protocol" or "application" + +---- error-tag string + +---- error-app-tag? string + +---- error-path? xpath + +---- error-message? string + +Note: REST server will not populate error-app-tag and error-path fields in this release. It can be +enhanced in a future release. A sample error response: + + { + "ietf-restconf:errors" : { + "error" : [ + { + "error-type" : "application", + "error-tag" : "invalid-value", + "error-message" : "VLAN 100 not found" + } + ] + } + } + +**error-type** can be either "protocol" or "application", indicating the origin of the error. +RESTCONF defines two more error-type enums "transport" and "rpc"; they are not used by REST server. + +**error-tag** indicates nature of error as described in [RFC8040, section 7](https://tools.ietf.org/html/rfc8040#page-74). + +**error-message** field carries a human friendly error message that can be displayed to the end +user. This is an optional field; system error do not include error-message, or have generic +messages like "Internal error". App module developer should use human friendly messages while +returning application errors. In case of CVL constraint violation the REST server will pick +the error message from the YANG "error-message" statement of CVL schema YANG. + +Table below lists possible error conditions with response status and data returned by REST server. + +Method | Error condition | Status | error-type | error-tag | error-message +--------|--------------------------|--------|-------------|------------------|---------------------- +*any* | Incorrect request data | 400 | protocol | invalid-value | +*write* | Bad content-type | 415 | protocol | invalid-value | Unsupported content-type +*write* | OpenAPI schema validation fails | 400 | protocol| invalid-value | Content not as per schema +*write* | YGOT schema validation fails | 400 | protocol| invalid-value | *YGOT returned message* +*any* | Invalid user credentials | 401 | protocol | access-denied | Authentication failed +*write* | User is not an admin | 403 | protocol | access-denied | Authorization failed +*write* | Translib commit failure | 409 | protocol | in-use | +*any* | Bad Accept-Version value | 400 | protocol | invalid-value | Invalid Accept-Version +*any* | Version ckeck fails | 400 | protocol | operation-not-supported | Unsupported client version *X.Y.Z* +*any* | Unknown HTTP server failure | 500 | protocol | operation-failed | Internal error +*any* | Not supported by App module | 405 | application | operation-not-supported | *App module returned message* +*any* | Incorrect payload | 400 | application | invalid-value | *App module returned message* +*any* | Resource not found | 404 | application | invalid-value | *App module returned message* +POST | Resource exists | 409 | application | resource-denied | *App module returned message* +*any* | Unknown error in Translib | 500 | application | operation-failed | Internal error +*any* | Unknown App module failure | 500 | application | operation-failed | *App module returned message* +*any* | CVL constraint failure | 500 | application | invalid-value | *error-message defined in SONiC YANG* + + +###### 3.2.2.4.14 DB Schema + +A new table "REST_SERVER" is introduced in ConfigDB for maintaining REST server configurations. + + key = REST_SERVER:default ; REST server configurations. + ;field = value + port = 1*5DIGIT ; server port - defaults to 443 + client_auth = "none"/"password"/"jwt"/"cert" + ; Client authentication mode. + ; none: No authentication, all clients + ; are allowed. Should be used only + ; for debugging. + ; password: HTTP Basic authentication. + ; jwt : HTTP Bearer Token authentication with + ; JSON Web Token format. + ; cert: Certificate based authentication. + ; Requires REST_SERVER['certs']['ca_crt'] configuration. + ; Any combination of "password", "jwt" and "cert" modes can be + ; enabled by specifying a comma separated values. + ; Eg: "password,jwt" enables both password and jwt modes. + log_level = DIGIT ; Verbosity for glog.V logs + + + key = REST_SERVER:certs ; Server certificate configurations + ;field = value + server_crt = STRING ; Path to TLS certificate file + server_key = STRING ; Path to TLS private key file + ca_crt = STRING ; Path to the CA certificate to be used for + ; client certificate validation. + +###### 3.2.2.4.15 API Documentation + +REST server will provide [OpenAPI UI](https://github.com/OpenAPI-api/OpenAPI-ui) based online +documentation and test UI for all REST APIs it supports. Documentation can be accessed by launching +URL **https://REST_SERVER_IP/ui** in a browser. This page will list all supported OpenAPI +definition files (both YANG generated and manual) along with link to open OpenAPI UI for them. + + +##### 3.2.2.5 gNMI server + +1. gNMI server is part of the telemetry process that supports telemtry as well as gNMI. +2. The gRPC server opens a TCP port and allows only valid mutually authenticated TLS connections, which requires valid Client, server and CA Certificates be installed as well a properly configured DNS. Multiple simultaneous connections are allowed to gNMI server. +3. The gNMI Agent uses the db client, as well as the non-db client to access and modify data directly in the Redis DB. +4. The Translib client is used to provide alternative models of access such as Openconfig models as opposed to the native Redis schema, as long as the Translib supports these models. Translib offers bidirectional translation between the native Redis model and the desired north bound model, as well as notifications/updates on these model objects to support telemetry and asynchronous updates, alarms and events. Translib should also provide information about what models it supports so that information can be returned in gNMI Capabilities response. +5. The gNMI server defines the four RPC functions as required by the gNMI Specification: Get, Set, Capabilities and Subscribe. +6. Since the db, non-db and Translib clients offer the functionality to support these functions, gNMI only has to translate the paths and object payloads into the correct parameters for the client calls and package the results back into the response gNMI objects to return to the gNMI Client, which is a straightforward operation, since no additional processing of the data is expected to be done in the gNMI server itself. When new models are added to Translib, no additional work should be required to support them in gNMI server. +7. All operations in a Set request are processed in a single transaction that will either succeed or fail as one operation. The db, non-db and Translib clients must support a Bulk operation in order to achieve the transactional behavior. gNMI server then must use this Bulk operation for Set requests. +8. Subscribe operations: Once, Poll and Stream require that the gRPC connection remain open until the subscription is completed. This means many connections must be supported. Subscribe offers several options, such as only sending object updates (not the whole object) which requires support form the db clients. Subscribe also allows for periodic sampling defined by the client. This must be handled in the gNMI agent itself. This requires a timer for each subscribe connection of this type in order to periodically poll the db client and return the result in a Subscribe Response. These timers should be destroyed when the subscription gRPC connection is closed. + +###### 3.2.2.5.1 Files changed/added: + + |-- gnmi_server + | |-- client_subscribe.go + | |-- server.go ------------------- MODIFIED (Handles creation of transl_data_client for GET/SET/CAPABILITY) + | |-- server_test.go + |-- sonic_data_client + | |-- db_client.go ---------------- MODIFIED (Common interface Stub code for new functions as all data clients implement common interface functions) + | |-- non_db_client.go ------------ MODIFIED (Common interface Stub code for new functions as all data clients implement common interface functions) + | |-- transl_data_client.go ------- ADDED (Specific processing for GET/SET/CAPABILITY for transl data clients) + | |-- trie.go + | |-- virtual_db.go + | + |-- transl_utils -------------------- ADDED + |-- transl_utils.go ------------- ADDED (Layer for invoking Translib APIs) + +###### 3.2.2.5.2 Sample Requests + +go run gnmi_get.go -xpath /openconfig-acl:acl/acl-sets/acl-set[name=MyACL4][type=ACL_IPV4]/acl-entries/acl-entry[sequence-id=1] -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +go run gnmi_set.go -replace /openconfig-acl:acl/acl-sets/acl-set[name=MyACL4][type=ACL_IPV4]/acl-entries/acl-entry=2/actions/config:@openconfig.JSON -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +go run gnmi_capabilities.go -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +##### 3.2.2.6 Translib + +Translib is a library that adapts management server requests to SONiC data providers and vice versa. Translib exposes the following APIs for the management servers to consume. + + func Create(req SetRequest) (SetResponse, error) + func Update(req SetRequest) (SetResponse, error) + func Replace(req SetRequest) (SetResponse, error) + func Delete(req SetRequest) (SetResponse, error) + func Get(req GetRequest) (GetResponse, error) + func Action(req ActionRequest) (ActionResponse, error) + func Bulk(req BulkRequest) (BulkResponse, error) + func Subscribe(req SubscribeRequest) ([]*IsSubscribeResponse, error) + func IsSubscribeSupported(req IsSubscribeRequest) ([]*IsSubscribeResponse, error) + func GetModels() ([]ModelData, error) + + Translib Structures: + type ErrSource int + + const ( + ProtoErr ErrSource = iota + AppErr + ) + + type SetRequest struct { + Path string + Payload []byte + User string + ClientVersion Version + } + + type SetResponse struct { + ErrSrc ErrSource + Err error + } + + type GetRequest struct { + Path string + User string + ClientVersion Version + } + + type GetResponse struct { + Payload []byte + ErrSrc ErrSource + } + + type ActionRequest struct { + Path string + Payload []byte + User string + ClientVersion Version + } + + type ActionResponse struct { + Payload []byte + ErrSrc ErrSource + } + + type BulkRequest struct { + DeleteRequest []SetRequest + ReplaceRequest []SetRequest + UpdateRequest []SetRequest + CreateRequest []SetRequest + User string + ClientVersion Version + } + + type BulkResponse struct { + DeleteResponse []SetResponse + ReplaceResponse []SetResponse + UpdateResponse []SetResponse + CreateResponse []SetResponse + } + + type SubscribeRequest struct { + Paths []string + Q *queue.PriorityQueue + Stop chan struct{} + User string + ClientVersion Version + } + + type SubscribeResponse struct { + Path string + Payload []byte + Timestamp int64 + SyncComplete bool + IsTerminated bool + } + + type NotificationType int + + const ( + Sample NotificationType = iota + OnChange + ) + + type IsSubscribeRequest struct { + Paths []string + User string + } + + type IsSubscribeResponse struct { + Path string + IsOnChangeSupported bool + MinInterval int + Err error + PreferredType NotificationType + } + + type ModelData struct { + Name string + Org string + Ver string + } + + type Version struct { + Major uint32 + Minor uint32 + Patch uint32 + } + +Translib has the following sub modules to help in the translation of data + +1. App Interface +2. Translib Request Handlers +3. YGOT request binder +4. DB access layer +5. App Modules +6. Transformer + +###### 3.2.2.6.1 App Interface + +App Interface helps in identifying the App module responsible for servicing the incoming request. It provides the following APIs for the App modules to register themselves with the App interface during the initialization of the app modules. + + func Register(path string, appInfo *AppInfo) error + This method can be used by any App module to register itself with the Translib infra. + Input Parameters: + path - base path of the model that this App module services + appInfo - This contains the reflect types of the App module structure that needs to be instantiated for each request, corresponding YGOT structure reflect type to instantiate the corresponding YGOT structure and boolean indicating if this is native App module to differentiate between OpenAPI spec servicing App module and the YANG serving App module. + Returns: + error - error string + func AddModel(model *gnmi.ModelData) error + This method can be used to register the models that the App module supports with the Translib infra. + Input Parameters: + model - Filled ModelData structure containing the Name, Organisation and version of the model that is being supported. + Returns: + error - error string + + App Interface Structures: + //Structure containing App module information + type AppInfo struct { + AppType reflect.Type + YGOTRootType reflect.Type + IsNative bool + tablesToWatch []*db.TableSpec + } + + Example Usages: + func init () { + log.Info("Init called for ACL module") + err := register("/openconfig-acl:acl", + &appInfo{appType: reflect.TypeOf(AclApp{}), + ygotRootType: reflect.TypeOf(ocbinds.OpenconfigAcl_Acl{}), + isNative: false, + tablesToWatch: []*db.TableSpec{&db.TableSpec{Name: ACL_TABLE}, &db.TableSpec{Name: RULE_TABLE}}}) + + if err != nil { + log.Fatal("Register ACL App module with App Interface failed with error=", err) + } + + err = appinterface.AddModel(&gnmi.ModelData{Name:"openconfig-acl", + Organization:"OpenConfig working group", + Version:"1.0.2"}) + if err != nil { + log.Fatal("Adding model data to appinterface failed with error=", err) + } + } + + type AclApp struct { + path string + YGOTRoot *YGOT.GoStruct + YGOTTarget *interface{} + } + +Translib request handlers use the App interface to get all the App module information depending on the incoming path as part of the requests. + +###### 3.2.2.6.2 Translib Request Handler + +These are the handlers for the APIs exposed by the Translib. Whenever a request lands in the request handler, the handler uses the App interface to get the App module that can process the request based on the incoming path. It then uses the YGOT binder module, if needed, to convert the incoming path and payload from the request into YGOT structures. The filled YGOT structures are given to the App Modules for processing. The Translib also interacts with the DB access layer to start, commit and abort a transaction. + +###### 3.2.2.6.3 YGOT request binder + +The YGOT request binder module uses the YGOT tools to perform the unmarshalling and validation. YGOT (YANG Go Tools) is an open source tool and it has collection of Go utilities which are used to + + 1. Generate a set of Go structures for bindings for the given YANG modules at build time + 2. Unmarshall the given request into the Go structure objects. These objects follows the same hierarchical structure defined in the YANG model, and it's simply a data instance tree of the given request, but represented using the generated Go structures + 3. Validate the contents of the Go structures against the YANG model (e.g., validating range and regular expression constraints). + 4. Render the Go structure objects to an output format - such as JSON. + +This RequestBinder module exposes the below mentioned APIs which will be used to unmarshall the request into Go structure objects, and validate the request + + func getRequestBinder(uri *string, payload *[]byte, opcode int, appRootNodeType *reflect.Type) *requestBinder + This method is used to create the requestBinder object which keeps the given request information such as uri, payload, App module root type, and unmarshall the same into object bindings + Input parameters: + uri - path of the target object in the request. + payload - payload content of given the request and the type is byte array + opcode - type of the operation (CREATE, DELETE, UPDATE, REPLACE) of the given request, and the type is enum + appRootNodeType - pointer to the reflect.Type object of the App module root node's YGOT structure object + Returns: + requestBinder - pointer to the requestBinder object instance + + func (binder *requestBinder) unMarshall() (*YGOT.GoStruct, *interface{}, error) + This method is be used to unmarshall the request into Go structure objects, and validates the request against YANG model schema + Returns: + YGOT.GoStruct - root Go structure object of type Device. + interface{} - pointer to the interface type of the Go structure object instance of the given target path + error - error object to describe the error if the unmarshalling fails, otherwise nil + +Utilities methods: +These utilities methods provides below mentioned common operations on the YGOT structure which are needed by the App module + + func getParentNode(targetUri *string, deviceObj *ocbinds.Device) (*interface{}, *YANG.Entry, error) + This method is used to get parent object of the given target object's uri path + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + Returns + interface{} - pointer to the parent object of the given target object's URI path + YANG.Entry - pointer to the YANG schema of the parent object + error - error object to describe the error if this methods fails to return the parent object, otherwise nil + + func getNodeName(targetUri *string, deviceObj *ocbinds.Device) (string, error) + This method is used to get the YANG node name of the given target object's uri path. + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + Returns: + string - YANG node name of the given target object + error - error object to describe the error if this methods fails to return the parent object, otherwise nil + + func getObjectFieldName(targetUri *string, deviceObj *ocbinds.Device, YGOTTarget *interface{}) (string, error) + This method is used to get the go structure object field name of the given target object. + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + YGOTTarget - pointer to the interface type of the target object. + Returns: + string - object field name of the given target object + error - error object to describe the error if this methods fails to perform the desired operation, otherwise nil + +###### 3.2.2.6.4 YANG Model Versioning + +###### 3.2.2.6.4.1 Version Management + +Translib maintains a "YANG bundle version" which is the collective version number for all the YANG +modules deployed in the Management Framework Service. This version is maintained in a configuration +file. It uses **MAJOR.MINOR.PATCH** syntax as per [Sematic Versioning](https://semver.org) specification. +Developer will have to update the bundle version number when he makes any YANG change. + +**Major version** is fixed to 1 in this release. +It will be incremented in future releases when any YANG model is changed in a non backward +compatible manner. Following are the candidates: + +* Delete, rename or relocate data node +* Change list key attributes +* Change data type of a node to an incompatible type +* Change leafref target + +**Minor version** is incremented if the YANG change modifies the API in a backward +compatible way. Patch version is reset to 0. +Candidate YANG changes for this category are: + +* Add new YANG module +* Add new YANG data nodes +* Mark a YANG data node as deprecated +* Change data type of a node to a compatible type +* Add new enum or identity + +**Patch version** is incremented for cosmetic fixes that do not change YANG API. +Candidate YANG changes for this category are: + +* Change description, beautification. +* Expand pattern or range of a node to wider set. +* Change must expression to accept more cases. +* Error message or error tag changes. + +###### 3.2.2.6.4.2 Version Checks + +Version check ensures that translib processes the requests only from compatible clients. +All translib APIs accept an optional **ClientVersion** argument. +Clients can pass the YANG bundle version that they are capable of handling through this argument. +If specified, request will be processed only if below criteria is satisfied. + + Base_Version <= ClientVersion <= YANG_Bundle_Version + +Base Version is **1.0.0** for current release. +It will change in a future release when major version is incremented. + +Version check is bypassed if client does not send its version number in the request. +However some requests may succeed and some may fail due to input or schema mismatch. + +###### 3.2.2.6.5 DB access layer + +The DB access layer implements a wrapper over the [go-redis](https://github.com/go-redis/redis) package +enhancing the functionality in the following ways: + + * Provide a sonic-py-swsssdk like API in Go + * Enable support for concurrent access via Redis CAS (Check-And-Set) + transactions. + * Invoke the CVL for validation before write operations to the Redis DB + +The APIs are broadly classified into the following areas: + + * Initialization/Close: NewDB(), DeleteDB() + * Read : GetEntry(), GetKeys(), GetKeysPattern(), GetTable() + * Write : SetEntry(), CreateEntry(), ModEntry(), DeleteEntry() + DeleteEntryField() + * Transactions : StartTx(), CommitTx(), AbortTx(), AppendWatchTx() + * Map : GetMap(), GetMapAll() + * Subscriptions : SubscribeDB(), UnsubscribeDB() + * Publish : Publish() + +Detail Method Signature: + Please refer to the code for the detailed method signatures. + +Concurrent Access via Redis CAS transactions: + + Upto 4 levels of concurrent write access support. + + 1. Table based watch keys (Recommended): + At App module registration, the set of Tables that are to be managed by + the module are provided. External (i.e. non-Management-Framework) + applications may choose to watch and set these same table keys to detect + and prevent concurrent write access. The format of the table key is + "CONFIG_DB_UPDATED_". (Eg: CONFIG_DB_UPDATED_ACL_TABLE) + + 2. Row based watch keys: + For every transaction, the App module provides a list of keys that it + would need exclusive access to for the transaction to succeed. Hence, + this is more complex for the app modules to implement. The external + applications need not be aware of table based keys. However, the + concurrent modification of yet to be created keys (i.e. keys which are + not in the DB, but might be created by a concurrent application) may not + be detected. + + 3. A combination of 1. and 2.: + More complex, but easier concurrent write access detection. + + 4. None: + For applications not needing concurrent write access protections. + + +DB access layer, Redis, CVL Interaction: + + DB access | PySWSSSDK API | RedisDB Call at | CVL Call at + | | at CommitTx | invocation + ----------------|------------------|-------------------|-------------------- + SetEntry(k,v) | set_entry(k,v) | HMSET(fields in v)|If HGETALL=no entry + | | HDEL(fields !in v | ValidateEditConfig + | | but in | (OP_CREATE) + | | previous HGETALL)| + | | |Else + | | | ValidateEditConfig( + | | | {OP_UPDATE, + | | | DEL_FIELDS}) + ----------------|------------------|-------------------|-------------------- + CreateEntry(k,v)| none | HMSET(fields in v)| ValidateEditConfig( + | | | OP_CREATE) + ----------------|------------------|-------------------|-------------------- + ModEntry(k,v) | mod_entry(k,v) | HMSET(fields in v)| ValidateEditConfig( + | | | OP_UPDATE) + ----------------|------------------|-------------------|-------------------- + DeleteEntry(k,v)|set,mod_entry(k,0)| DEL | ValidateEditConfig( + | | | OP_DELETE) + ----------------|------------------|-------------------|-------------------- + DeleteEntryField| none | HDEL(fields) | ValidateEditConfig( + (k,v) | | | DEL_FIELDS) + +Notes: + 1. SetEntry(), CreateEntry(), ModEntry() with empty fields in v, will map to + DeleteEntry(). [ There may be an option to disable this in the future. ] + + +##### 3.2.2.7 Transformer + +Transformer provides the underlying infrastructure for developers to translate data from YANG to ABNF/Redis schema and vice versa, using YANG extensions annotated to YANG paths to provide translation methods. At run time, the YANG extensions are mapped to an in-memory Transformer Spec that provides two-way mapping between YANG and ABNF/Redis schema for Transformer to perform data translation while processing SET/GET operations. + +In case that SONiC YANG modules are used by NBI applications, the Transformer performs 1:1 mapping between a YANG object and a SONiC DB object without a need to write special translation codes. If the openconfig YANGs are used by NBI applications, you may need special handling to translate data between YANG and ABNF schema. In such case, you can annotate YANG extensions and write callbacks to perform translations where required. + +For special handling, a developer needs to provide: +1. An annotation file to define YANG extensions on YANG paths where translation required +e.g. In openconfig-acl.yang, ACL FORWARDING_ACTION "ACCEPT" mapped to "FORWARD", ACL sequence id ‘1’ mapped to ‘RULE_1’ in Redis DB etc. +2. Transformer callbacks to perform translation + +###### 3.2.2.7.1 Components + +Transformer consists of the following components and data: +* **Transformer Spec:** a collection of translation hints +* **Spec Builder:** loads YANG and annotation files to dynamically build YANG schema tree and Transformer Spec. +* **Transformer Core:** perform main transformer tasks, i.e. encode/decode YGOT, traverse the payload, lookup Transformer spec, call Transformer methods, construct the results, error reporting etc. +* **Built-in Default Transformer method:** perform static translation +* **Overloaded Transformer methods:** callback functions invoked by Transformer core to perform complex translation with developer supplied translation logic +* **Ouput framer:** aggregate the translated pieces returned from default and overloaded methods to construct the output payload +* **Method overloader:** dynamically lookup and invoke the overloaded transformer methods during data translation +* **YANG schema tree:** provides the Transformer with the schema information that can be accessed by Transformer to get node information, like default values, parent/descendant nodes, etc. + +![Transformer Components](images/transformer_components_v1.png) + +###### 3.2.2.7.2 Design + +Requests from Northbound Interfaces (NBI) are processed by Translib public APIs - Create, Replace, Update, Delete, (CRUD) and Get - that call a specific method on the common app module. The common app calls Transformer APIs to translate the request, then use the translated data to proceed to DB/CVL layer to set or get data in Redis DB. The **common app** as a default application module generically handles both SET (CRUD) and GET, and Subscribe requests with Transformer. Note that a specific app module can be registered to the Translib to handle the requests if needed. + +![Transformer Design](images/transformer_design.PNG) + +At Transformer init, it loads YANG modules pertaining to the applications. Transformer parses YANG modules with the extensions to dynamically build an in-memory schema tree and transformer spec. + +Below structure is defined for the transformer spec: + +```YANG +type yangXpathInfo struct { + yangDataType string + tableName *string + xfmrTbl *string + childTable []string + dbEntry *yang.Entry + yangEntry *yang.Entry + keyXpath map[int]*[]string + delim string + fieldName string + xfmrFunc string + xfmrField string + xfmrPost string + validateFunc string + rpcFunc string + xfmrKey string + keyName *string + dbIndex db.DBNum + keyLevel int + isKey bool + defVal string + hasChildSubTree bool +} +``` + +When a request lands at the common app in the form of a YGOT structure from the Translib request handler, the request is passed to Transformer that decodes the YGOT structure to read the request payload and look up the spec to get translation hints. Note that the Transformer cannot be used for OpenAPI spec. The Transformer Spec is structured with a two-way mapping to allow Transformer to map YANG-based data to ABNF data and vice-versa via reverse lookup. The reverse mapping is used to populate the data read from Redis DB to YANG structure for the response to get operation. + +Transformer has a built-in default transformer method to perform static, simple translation from YANG to ABNF or vice versa. It performs simple mapping - e.g. a direct name/value mapping, table/key/field name - which can be customized by a YANG extension. + +Additionally, for more complex translations of non-ABNF YANG models, i.e. OpenConfig models, Transformer also allows developers to overload the default method by specifying a callback fucntion in YANG extensions, to perform translations with developer-supplied translation codes as callback functions. Transformer dynamically invokes those functions instead of using default method. Each transformer callback must be defined to support two-way translation, i.e, YangToDb_ and DbToYang_, which are invoked by Transformer core. + +###### 3.2.2.7.3 Process + +CRUD requests (configuration) are processed via the following steps: + +1. App module calls transformer, passing it a YGOT populated Go structure to translate YANG to ABNF +2. App module calls CVL API to get all depenent table list to watch tables, and get the ordered table list +3. Transformer allocates buffer with 3-dimensional map: `[table-name][key-values][attributes]` +4. Transformer decodes YGOT structure and traverses the incoming request to get the YANG node name +5. Transformer looks up the Transformer Spec to check if a translation hint exists for the given path +6. If no spec or hint is found, the name and value are copied as-is +7. If a hint is found, check the hint to perform the action, either simple data translation or invoke external callbacks +8. Repeat steps 4 through 7 until traversal is completed +9. Invoke any annotated post-Transformer functions +10. Transformer aggregates the results to returns to App module +11. App module proceeds to update DB to ensure DB update in the order learnt from step 2 + +GET requests are processed via the following steps: +1. App module asks the transformer to translate the URL to the keyspec to the query target + ```YANG + type KeySpec struct { + DbNum db.DBNum + Ts db.TableSpec + Key db.Key + Child []KeySpec + IgnoreParentKey bool + } + ``` +2. Transformer proceeds to traverse the DB with the keyspec to get the results +3. Transformer translate the results from ABNF to YANG, with default transformer method or callbacks +4. Transformer aggregate the translated results, return to the App module to unmarshall the JSON payload + +###### 3.2.2.7.4 Common App + +The Common App is a default app that handles the GET/SET/Subscribe requests for SONiC or OpenConfig YANG modules unless an app module is registered to Translib. + +Here is a diagram to show how the common app supports SET(CRUD)/GET requests. + +![sequence diagram_for_set](images/crud_v1.png) + +![sequence_diagram_for_get](images/get_v1.png) + +If a request is associated with multiple tables, the common app module processes the DB updates in the table order learned from CVL layer. +e.g. in case that the sonic-acl.yang used by NBI and the payload with CREATE operation has a data including both ACL_TABLE and ACL_RULE, the common app updates the CONFIG-DB to create an ACL TABLE instance, followed by ACL_RULE entry creation in this order. DELETE operates in the reverse order, i.e. ACL_RULE followed by ACL_TABLE. + +###### 3.2.2.7.5 YANG Extensions + +The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by external callbacks. + +---------- + +1. `sonic-ext:table-name [string]`: +Map a YANG container/list to TABLE name, processed by the default transformer method. Argument is a table name statically mapped to the given YANG container or list node. +The table-name is inherited to all descendant nodes unless another one is defined. + +2. `sonic-ext:field-name [string]`: +Map a YANG leafy - leaf or leaf-list - node to FIELD name, processed by the default transformer method + +3. `sonic-ext:key-delimiter [string]`: +Override the default key delimiters used in Redis DB, processed by the default transformer method. +Default delimiters are used by Transformer unless the extension is defined - CONFIG_DB: "|", APPL_DB: ":", ASIC_DB: "|", COUNTERS_DB: ":", FLEX_COUNTER_DB: "|", STATE_DB: "|" + +4. `sonic-ext:key-name [string]`: +Fixed key name, used for YANG container mapped to TABLE with a fixed key, processed by the default transformer method. Used to define a fixed key, mainly for container mapped to TABLE key +e.g. Redis can have a hash “STP|GLOBAL” +```YANG +container global + sonic-ext:table-name “STP” + sonic-ext:key-name “GLOBAL” +``` +5. `sonic-ext:key-transformer [function]`: +Overloading default method with a callback to generate DB keys(s), used when the key values in a YANG list are different from ones in DB TABLE. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +6. `sonic-ext:field-transformer [function]`: +Overloading default method with a callback to generate FIELD value, used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +7. `sonic-ext:subtree-transformer [function]`: +Overloading default method with a callback for the current subtree, allows the sub-tree transformer to take full control of translation. Note that, if any other extensions, e.g. table-name etc., are annotated to the nodes on the subtree, they are not effective. +The subtree-transformer is inherited to all descendant nodes unless another one is defined, i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes along the YANG path until a new subtree transformer is annotated. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +8. `sonic-ext:db-name [string]`: +DB name to access data – “APPL_DB”, “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, “FLEX_COUNTER_DB”, “STATE_DB”. The default db-name is CONFIG_DB, Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG. Processed by Transformer core to traverse database. +The db-name is inherited to all descendant nodes unless another one. Must be defined with the table-name + +9. `sonic-ext:post-transformer [function]`: +A special hook to update the DB requests right before passing to common-app, analogous to the postponed YangToDB subtree callback that is invoked at the very end by the Transformer. +Used to add/update additional data to the maps returned from Transformer before passing to common-app, e.g. add a default acl rule +Note that the post-transformer can be annotated only to the top-level container(s) within each module, and called once for the given node during translation + +10. `sonic-ext:table-transformer [function]`: +Dynamically map a YANG container/list to TABLE name(s), allows the table-transformer to map a YANG list/container to table names. +Used to dynamically map a YANG list/container to table names based on URI and payload. +The table-transformer is inherited to all descendant nodes unless another one is defined + +11. `sonic-ext:get-validate [function]`: +A special hook to validate YANG nodes, to populate data read from database, allows developers to instruct Transformer to choose a YANG node among multiple nodes, while constructing the response payload. +Typically used to check the “when” condition to validate YANG node among multiple nodes to choose only valid nodes from sibling nodes. + +---------- + + +Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - **YangToDB***function*, **DbToYang***function*. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only **DbToYang***function*. + +The template annotation file can be generated and used by the developers to define extensions to the yang paths as needed to translate data between YANG and ABNF format. Refer to the 3.2.2.7.8 Utilities. + +Here is the general guide you can check to find which extensions can be annotated in implementing your model. +```YANG +1) If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter +2) If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope +3) If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload +4) In Get operation access to non CONFIG_DB, you can use the db-name extension +5) In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function +6) In case of mapping a container to TABLE/KET, you can use the key-name along with the table-name extension +``` + +###### 3.2.2.7.6 Public Functions + +`XlateToDb()` and `GetAndXlateFromDb` are used by the common app to request translations. + +```go +func XlateToDb(path string, opcode int, d *db.DB, yg *ygot.GoStruct, yt *interface{}) (map[string]map[string]db.Value, error) {} + +func GetAndXlateFromDB(xpath string, uri *ygot.GoStruct, dbs [db.MaxDB]*db.DB) ([]byte, error) {} +``` + +###### 3.2.2.7.7 Overloaded Methods + +The function prototypes for external transformer callbacks are defined in the following- + +```go +type XfmrParams struct { + d *db.DB + dbs [db.MaxDB]*db.DB + curDb db.DBNum + ygRoot *ygot.GoStruct + uri string + requestUri string //original uri using which a curl/NBI request is made + oper int + key string + dbDataMap *map[db.DBNum]map[string]map[string]db.Value + subOpDataMap map[int]*RedisDbMap // used to add an in-flight data with a sub-op + param interface{} + txCache *sync.Map + skipOrdTblChk *bool +} + +/** + * KeyXfmrYangToDb type is defined to use for conversion of Yang key to DB Key + * Transformer function definition. + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: Database keys to access db entry, error + **/ +type KeyXfmrYangToDb func (inParams XfmrParams) (string, error) +/** + * KeyXfmrDbToYang type is defined to use for conversion of DB key to Yang key + * Transformer function definition. + * Param: XfmrParams structure having Database info, operation, Database keys to access db entry + * Return: multi dimensional map to hold the yang key attributes of complete xpath, error + **/ +type KeyXfmrDbToYang func (inParams XfmrParams) (map[string]interface{}, error) + +/** + * FieldXfmrYangToDb type is defined to use for conversion of yang Field to DB field + * Transformer function definition. + * Param: Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ +type FieldXfmrYangToDb func (inParams XfmrParams) (map[string]string, error) +/** + * FieldXfmrDbtoYang type is defined to use for conversion of DB field to Yang field + * Transformer function definition. + * Param: XfmrParams structure having Database info, operation, DB data in multidimensional map, output param YgotRoot + * Return: error + **/ +type FieldXfmrDbtoYang func (inParams XfmrParams) (map[string]interface{}, error) + +/** + * SubTreeXfmrYangToDb type is defined to use for handling the yang subtree to DB + * Transformer function definition. + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ +type SubTreeXfmrYangToDb func (inParams XfmrParams) (map[string]map[string]db.Value, error) +/** + * SubTreeXfmrDbToYang type is defined to use for handling the DB to Yang subtree + * Transformer function definition. + * Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri + * Return : error + **/ +type SubTreeXfmrDbToYang func (inParams XfmrParams) (error) +/** + * ValidateCallpoint is used to validate a YANG node during data translation back to YANG as a response to GET + * Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri + * Return : bool + **/ +type ValidateCallpoint func (inParams XfmrParams) (bool) +/** + * RpcCallpoint is used to invoke a callback for action + * Param : []byte input payload, dbi indices + * Return : []byte output payload, error + **/ +type RpcCallpoint func (body []byte, dbs [db.MaxDB]*db.DB) ([]byte, error) +/** + * PostXfmrFunc type is defined to use for handling any default handling operations required as part of the CREATE + * Transformer function definition. + * Param: XfmrParams structure having database pointers, current db, operation, DB data in multidimensional map, YgotRoot, uri + * Return: multi dimensional map to hold the DB data, error + **/ +type PostXfmrFunc func (inParams XfmrParams) (map[string]map[string]db.Value, error) +/** + * TableXfmrFunc type is defined to use for table transformer function for dynamic derviation of redis table. + * Param: XfmrParams structure having database pointers, current db, operation, DB data in multidimensional map, YgotRoot, uri + * Return: List of table names, error + **/ +type TableXfmrFunc func (inParams XfmrParams) ([]string, error) + +``` + +###### 3.2.2.7.8 Utilities + +The goyang package is extended to generate the template annotation file for any input yang files. A new output format type "annotate" can be used to generate the template annotation file.The goyang usage is as below: + +``` +Usage: goyang [-?] [--format FORMAT] [--ignore-circdep] [--path DIR[,DIR...]] [--trace TRACEFILE] [FORMAT OPTIONS] [SOURCE] [...] + -?, --help display help + --format=FORMAT + format to display: annotate, tree, types + --ignore-circdep + ignore circular dependencies between submodules + --path=DIR[,DIR...] + comma separated list of directories to add to search path + --trace=TRACEFILE + write trace into to TRACEFILE + +Formats: + annotate - generate template file for yang annotations + + tree - display in a tree format + + types - display found types + --types_debug display debug information + --types_verbose + include base information +``` +The $(SONIC_MGMT_FRAMEWORK)/gopkgs/bin is added to the PATH to run the goyang binary. + +For example: + +``` +goyang --format=annotate --path=/path/to/yang/models openconfig-acl.yang > openconfig-acl-annot.yang + +Sample output: +module openconfig-acl-annot { + + yang-version "1" + + namespace "http://openconfig.net/yang/annotation"; + prefix "oc-acl-annot" + + import openconfig-packet-match { prefix oc-pkt-match } + import openconfig-interfaces { prefix oc-if } + import openconfig-yang-types { prefix oc-yang } + import openconfig-extensions { prefix oc-ext } + + deviation oc-acl:openconfig-acl { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state/oc-acl:counter-capability { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set/oc-acl:type { + deviate add { + } + } +... +... + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:config { + deviate add { + } + } +} +``` + + +##### 3.2.2.8 Config Validation Library (CVL) + +Config Validation Library (CVL) is an independent library to validate ABNF schema based SONiC (Redis) configuration. This library can be used by component like [Cfg-gen](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-config-engine/sonic-cfggen), Translib, [ZTP](https://github.com/Azure/SONiC/blob/master/doc/ztp/ztp.md) etc. to validate SONiC configuration data before it is written to Redis DB. Ideally, any component importing config_db.json file into Redis DB can invoke CVL API to validate the configuration. + +CVL uses SONiC YANG models written based on ABNF schema along with various constraints. These native YANG models are simple and have a very close mapping to the associated ABNF schema. Custom YANG extensions (annotations) are used for custom validation purpose. Specific YANG extensions (rather metadata) are used to translate ABNF data to YANG data. In the context of CVL these YANG models are called CVL YANG models and generated from SONiC YANG during build time. Opensource [libyang](https://github.com/CESNET/libyang) library is used to perform YANG data validation. + +SONiC YANG can be used as Northbound YANG for management interface by adding other data definitions such as state data (read only data), RPC or Notification as needed.Such YANG models are called SONiC NBI YANG models. Since CVL validates configuration data only, these data definition statements are ignored by CVL. During build time CVL YANG is actually generated from SONiC NBI YANG models with the help of CVL specific pyang plugin. + +CVL supports multiple user-defined Redis database instances based on the [Multi DB instance HLD](https://github.com/Azure/SONiC/blob/master/doc/database/multi_database_instances.md) specification. During CVL initialization time, the DB configuration file is read from the predefined location and details are stored in internal data structure. CVL implements internal API for connecting to a Redis DB instance using the details stored in its internal data structure. + +###### 3.2.2.8.1 Architecture + +![CVL architecture](images/CVL_Arch.jpg) + +1. During build time, developer writes SONiC YANG schema based on ABNF schema following [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) and adds metadata and constraints as needed. Custom YANG extensions are defined for this purpose. +2. The YANG models are compiled using Pyang compiler. CVL specific YIN schema files are derived from SONiC YANG with the help of CVL specific pyang plugin. Finally, generated YIN files are packaged in the build. +3. During boot up/initialization sequence, YIN schemas generated from SONiC YANG models are parsed and schema tree is build using libyang API. +4. Application calls CVL APIs to validate the configuration. In case of Translib library DB Access layer calls appropriate CVL APIs to validate the configuration. +5. ABNF JSON goes through a translator and YANG data is generated. Metadata embedded in the YANG schema are used to help this translation process. +6. Then YANG data is fed to libyang for performing syntax validation first. If error occurs, CVL returns appropriate error code and details to application without proceeding further. +7. If syntax validation is successful, CVL uses dependent data from translated YANG data or if needed, fetches the dependent data from Redis DB. Dependent data refers to the data needed to validate constraint expressed in YANG syntax such as 'leafref', 'must', 'when' etc. +8. Finally translated YANG data and dependent data are merged and fed to libyang for performing semantics validation. If error occurs, CVL returns appropriate error code and details to application, else success is returned. +9. Platform validation is specific syntax and semantics validation only performed with the help of dynamic platform data as input. + +###### 3.2.2.8.2 Validation types + +Config Validator does Syntactic, Semantic validation and Platform Validation as per YIN schema with metadata. + +###### 3.2.2.8.2.1 Syntactic Validation + +Following are some of the syntactic validations supported by the config validation library + +* Basic data type +* Enum +* Ranges +* Pattern matching +* Check for mandatory field +* Check for default field +* Check for number of keys are their types +* Check for table size etc. + +###### 3.2.2.8.2.2 Semantic Validation + +* Check for key reference existence in other table +* Check any conditions between fields within same table +* Check any conditions between fields across different table + +###### 3.2.2.8.2.3 Platform specific validation + +There can be two types of platform constraint validation + +###### 3.2.2.8.2.3.1 Static Platform Constraint Validation + +* Platform constraints (range, enum, ‘must’/’when’ expression etc.) are expressed in SONiC YANG deviation model for each feature. + +Example of 'deviation' : + +``` +module sonic-acl-deviation { + ...... + ...... + deviation /sacl:sonic-acl/sacl:ACL_TABLE/sacl:ACL_TABLE_LIST { + deviate add { + max-elements 3; + } + } + + deviation /sacl:sonic-acl/sacl:ACL_RULE/sacl:ACL_RULE_LIST { + deviate add { + max-elements 768; + } + } +} +``` + +* All deviation models are compiled along with corresponding CVL YANG model and are placed in platform specific schema folder. At runtime based on detected platform from provisioned “DEVICE_METADATA:platform” field, deviation files are applied. + +* Here is the sample folder structure for platform specific deviation files. +``` + models/yang/sonic/platform/ + |-- accton_as5712 + | |--- sonic-acl-deviation.yang + |-- quanta_ix8 + |--- sonic-acl-deviation.yang +``` + +###### 3.2.2.8.2.3.2 Dynamic Platform Constraint Validation + +###### 3.2.2.8.2.3.2.1 Platform data is available in Redis DB table. + +* Based on Redis DB, platform specific YANG data defintion can be added in SONiC YANG models. Constraints like ‘must’ or ‘when’ are used in feature YANG by cross-referencing platform YANG models. + +###### 3.2.2.8.2.3.2.2 Platform data is available through APIs + +* If constraints cannot be expressed using YANG syntax or platform data is available through feature/component API (APIs exposed by a feature to query platform specific constants, resource limitation etc.), custom validation needs to be hooked up in SONiC YANG model through custom YANG extension. +* Feature developer implements the custom validation functions with functional validation code. The validation function may call feature/component API and fetch required parameter for checking constraints. +* Based on YANG extension syntax, CVL will call the appropriate custom validation function along with YANG instance data to be validated. Below is the custom validation context structure definition. + +``` + //Custom validation context passed to custom validation function + type CustValidationCtxt struct { + ReqData []CVLEditConfigData //All request data + CurCfg *CVLEditConfigData //Current request data for which validation should be done + YNodeName string //YANG node name + YNodeVal string //YANG node value, leaf-list will have "," separated value + YCur *xmlquery.Node //YANG data tree + RClient *redis.Client //Redis client + } +``` + +###### 3.2.2.8.3 CVL APIs + +``` + //Structure for key and data in API + type CVLEditConfigData struct { + VType CVLValidateType //Validation type + VOp CVLOperation //Operation type + Key string //Key format : "PORT|Ethernet4" + Data map[string]string //Value : {"alias": "40GE0/28", "mtu" : 9100, "admin_status": down} + } + + /* CVL Error Structure. */ + type CVLErrorInfo struct { + TableName string /* Table having error */ + ErrCode CVLRetCode /* Error Code describing type of error. */ + Keys []string /* Keys of the Table having error. */ + Value string /* Field Value throwing error */ + Field string /* Field Name throwing error . */ + Msg string /* Detailed error message. */ + ConstraintErrMsg string /* Constraint error message. */ + } + + /* Structure for dependent entry to be deleted */ + type CVLDepDataForDelete struct { + RefKey string /* Ref Key which is getting deleted */ + Entry map[string]map[string]string /* Entry or field which should be deleted as a result */ + } + + /* Maintain time stats for call to ValidateEditConfig(). */ + type ValidationTimeStats struct { + Hits uint /* Total number of times ValidateEditConfig() called */ + Time time.Duration /* Total time spent in ValidateEditConfig() */ + Peak time.Duration /* Highest time spent in ValidateEditConfig() */ + } + + /* Validation type */ + type CVLValidateType uint + const ( + VALIDATE_NONE CVLValidateType = iota //Data is used as dependent data + VALIDATE_SYNTAX //Syntax is checked and data is used as dependent data + VALIDATE_SEMANTICS //Semantics is checked + VALIDATE_ALL //Syntax and Semantics are checked + ) + + /* Validation operation */ + type CVLOperation uint + const ( + OP_NONE CVLOperation = 0 //Used to just validate the config without any operation + OP_CREATE = 1 << 0//For Create operation + OP_UPDATE = 1 << 1//For Update operation + OP_DELETE = 1 << 2//For Delete operation + ) + + /* Error code */ + type CVLRetCode int + const ( + CVL_SUCCESS CVLRetCode = iota + CVL_SYNTAX_ERROR /* Generic syntax error */ + CVL_SEMANTIC_ERROR /* Generic semantic error */ + CVL_ERROR /* Generic error */ + CVL_SYNTAX_MISSING_FIELD /* Missing field */ + CVL_SYNTAX_INVALID_FIELD /* Invalid Field */ + CVL_SYNTAX_INVALID_INPUT_DATA /*Invalid Input Data */ + CVL_SYNTAX_MULTIPLE_INSTANCE /* Multiple Field Instances */ + CVL_SYNTAX_DUPLICATE /* Duplicate Fields */ + CVL_SYNTAX_ENUM_INVALID /* Invalid enum value */ + CVL_SYNTAX_ENUM_INVALID_NAME /* Invalid enum name */ + CVL_SYNTAX_ENUM_WHITESPACE /* Enum name with leading/trailing whitespaces */ + CVL_SYNTAX_OUT_OF_RANGE /* Value out of range/length/pattern (data) */ + CVL_SYNTAX_MINIMUM_INVALID /* min-elements constraint not honored */ + CVL_SYNTAX_MAXIMUM_INVALID /* max-elements constraint not honored */ + CVL_SEMANTIC_DEPENDENT_DATA_MISSING /* Dependent Data is missing */ + CVL_SEMANTIC_MANDATORY_DATA_MISSING /* Mandatory Data is missing */ + CVL_SEMANTIC_KEY_ALREADY_EXIST /* Key already existing. */ + CVL_SEMANTIC_KEY_NOT_EXIST /* Key is missing. */ + CVL_SEMANTIC_KEY_DUPLICATE /* Duplicate key. */ + CVL_SEMANTIC_KEY_INVALID /* Invaid key */ + CVL_NOT_IMPLEMENTED /* Not implemented */ + CVL_INTERNAL_UNKNOWN /*Internal unknown error */ + CVL_FAILURE /* Generic failure */ + ) +``` + +1. func Initialize() CVLRetCode + - Initialize the library only once, subsequent calls does not affect once library is already initialized . This automatically called when if ‘cvl’ package is imported. + +2. func Finish() + - Clean up the library resources. This should ideally be called when no more validation is needed or process is about to exit. + +3. func (c *CVL) ValidateConfig(jsonData string) CVLRetCode + - Just validates json buffer containing multiple row instances of the same table, data instance from different tables. All dependency are provided in the payload. This is useful for bulk data validation. + +4. func (c *CVL) ValidateEditConfig(cfgData []CVLEditConfigData) (cvlErr CVLErrorInfo, ret CVLRetCode) + - Validates the JSON data for create/update/delete operation. Syntax or Semantics Validation can be done separately or together. Related data should be given as dependent data for validation to be successful. + +5. func (c *CVL) ValidateKeys(key []string) CVLRetCode + - Just validates the key and checks if it exists in the DB. It checks whether the key value is following schema format. Key should have table name as prefix. + +6. func (c *CVL) ValidateFields(key string, field string, value string) CVLRetCode + - Just validates the field:value pair in table. Key should have table name as prefix. + +7. func (c *CVL) SortDepTables(inTableList []string) ([]string, CVLRetCode) + - Sort the list of given tables as per their dependency imposed by leafref. + +8. func (c *CVL) GetOrderedTables(yangModule string) ([]string, CVLRetCode) + - Get the sorted list of tables in a given YANG module based on leafref relation. + +9. func (c *CVL) GetDepTables(yangModule string, tableName string) ([]string, CVLRetCode) + - Get the list of dependent tables for a given table in a YANG module. + +10. func (c *CVL) GetDepDataForDelete(redisKey string) ([]CVLDepDataForDelete) + - Get the dependent data (Redis keys) to be deleted or modified for a given entry getting deleted. + +11. func GetValidationTimeStats() (ValidationTimeStats) + - Provides statistics details of time spent in ValidateEditConfig(). + +12. func ClearValidationTimeStats() + - Clears statistics details of time spent in ValidateEditConfig(). + +##### 3.2.2.9 Redis DB + +Please see [3.2.2.6.5 DB access layer](#32265-db-access-layer) + +##### 3.2.2.10 Non DB data provider + +Currently, it is up to each App module to perform the proprietary access +mechanism for the app specific configuration. + +## 4 Flow Diagrams + +### 4.1 REST SET flow + +![REST SET flow](images/write.jpg) + +1. REST client can send any of the write commands such as POST, PUT, PATCH or DELETE and it will be handled by the REST Gateway. +2. All handlers in the REST gateway will invoke a command request handler. +3. Authentication and authorization of the commands are done here. +4. Request handler invokes one of the write APIs exposed by the Translib. +5. Translib infra populates the YGOT structure with the payload of the request and performs a syntactic validation +6. Translib acquires the write lock (mutex lock) to avoid another write happening from the same process at the same time. +7. Translib infra gets the App module corresponding to the incoming URI. +8. Translib infra calls the initialize function of the App module with the YGOT structures, path and payload. +9. App module caches the incoming data into the app structure. +10. App module calls Transformer function to translate the request from cached YGOT structures into Redis ABNF format. It also gets all the keys that will be affected as part of this request. +11. App module returns the list of keys that it wants to keep a watch on along with the status. +12. Translib infra invokes the start transaction request exposed by the DB access layer. +13. DB access layer performs a WATCH of all the keys in the Redis DB. If any of these keys are modified externally then the EXEC call in step 26 will fail. +14. Status being returned from Redis. +15. Status being returned from DB access layer. +16. Translib then invokes the processWrite API on the App module. +17. App modules perform writes of the translated data to the DB access layer. +18. DB access layer validates the writes using CVL and then caches them. +19. Status being returned from DB access layer. +20. Status being returned from App module. +21. Translib infra invokes the commit transaction on the DB access layer. +22. DB access layer first invokes MULTI request on the Redis DB indicating there are multiple writes coming in, so commit everything together. All writes succeed or nothing succeeds. +23. Status returned from Redis. +24. Pipeline of all the cached writes are executed from the DB access layer. +25. Status retuned from Redis. +26. EXEC call is made to the Redis DB. Here if the call fails, it indicates that one of the keys that we watched has changed and none of the writes will go into the Redis DB. +27. Status returned from Redis DB. +28. Status returned from DB access layer. +29. Write lock acquired in Step 6 is released. +30. Status returned from the Translib infra. +31. REST Status returned from the Request handler. +32. REST response is sent by the REST gateway to the REST client. + +### 4.2 REST GET flow + +![REST GET flow](images/read.jpg) + +1. REST GET request from the REST client is sent to the REST Gateway. +2. REST Gateway invokes a common request handler. +3. Authentication of the incoming request is performed. +4. Request handler calls the Translib exposed GET API with the uri of the request. +5. Translib infra gets the App module corresponding to the incoming uri. +6. Translib infra calls the initialize function of the App module with the YGOT structures and path. App module caches them. +7. Status retuned from App module. +8. App module queries Transformer to translate the path to the Redis keys that need to be queried. +9. Status returned from App module. +10. Translib infra calls the processGet function on the App module +11. App modules calls read APIs exposed by the DB access layer to read data from the Redis DB. +12. Data is read from the Redis DB is returned to the App module +13. App module fills the YGOT structure with the data from the Redis DB and validated the filled YGOT structure for the syntax. +14. App module converts the YGOT structures to JSON format. +15. IETF JSON payload is returned to the Translib infra. +16. IETF JSON payload is returned to the request handler. +17. Response is returned to REST gateway. +18. REST response is returned to the REST client from the REST gateway. + +### 4.3 Translib Initialization flow + +![Translib Initialization flow](images/Init.jpg) + +1. App module 1 `init` is invoked +2. App module 1 calls `Register` function exposed by Translib infra to register itself with the Translib. +3. App module 2 `init` is invoked +4. App module 2 calls `Register` function exposed by Translib infra to register itself with the Translib. +5. App module N `init` is invoked +6. App module N calls `Register` function exposed by Translib infra to register itself with the Translib. + +This way multiple app modules initialize with the Translib infra during boot up. + +### 4.4 gNMI flow + +![GNMI flow](images/GNMI_flow.jpg) + +1. GNMI requests land in their respective GET/SET handlers which then redirect the requests to corresponding data clients. +2. If user does not provide target field then by default the request lands to the transl_data_client. +3. Next, the transl_data_client provides higher level abstraction along with collating the responses for multiple paths. +4. Transl Utils layer invokes Translib API's which in turn invoke App-Module API's and data is retrieved and modified in/from Redis Db/non-DB as required. + +### 4.5 CVL flow + +![CVL flow](images/CVL_flow.jpg) + +Above is the sequence diagram explaining the CVL steps. Note that interaction between DB Access layer and Redis including transactions is not shown here for brevity. + +1. REST/GNMI invokes one of the write APIs exposed by the Translib. +2. Translib infra populates the YGOT structure with the payload of the request and performs a syntactic validation. +3. Translib acquires the write lock (mutex lock) to avoid another write happening from the same process at the same time. +4. Translib infra gets the App module corresponding to the incoming uri. +5. Translib infra calls the initialize function of the App module with the YGOT structures, path and payload. +6. App module calls Transformer to translate the request from cached YGOT structure into Redis ABNF format. It also gets all the keys that will be affected as part of this request. +7. App modules returns the list of keys that it wants to keep a watch on along with the status. +8. Translib infra invokes the start transaction request exposed by the DB access layer. +9. Status being returned from DB access layer. +10. Translib then invokes the processWrite API on the App module. +11. App modules perform writes of the translated data to the DB access layer. +12. DB access layer calls validateWrite for CREATE/UPDATE/DELETE operation. It is called with keys and Redis/ABNF payload. +13. validateSyntax() feeds Redis data to translator internally which produces YANG XML. This is fed to libyang for validating the syntax. +14. If it is successful, control goes to next step, else error is returned to DB access layer. The next step is to ensure that keys are present in Redis DB for Update/Delete operation. But keys should not be present for Create operation. +15. Status is returned after checking keys. +16. CVL gets dependent data from incoming Redis payload. For example if ACL_TABLE and ACL_RULE is getting created in a single request. +17. Otherwise dependent should be present in Redis DB, query is sent to Redis to fetch it. +18. Redis returns response to the query. +19. Finally request data and dependent is merged and validateSemantics() is called. +20. If above step is successful, success is returned or else failure is returned with error details. +21. DB Access layer forwards the status response to App mpdule. +22. App module forwards the status response to Translib infra. +23. Translib infra invokes the commit transaction on the DB access layer. +24. Status is returned from DB access layer after performing commit operation. +25. Write lock acquired in Step 3 is released. +26. Final response is returned from the Translib infra to REST/GNMI. + +## 5 Developer Work flow +Developer work flow differs for standard YANG (IETF/OpenConfig) vs proprietary YANG used for a feature. When a standards-based YANG model is chosen for a new feature, the associated Redis DB design should take the design of this model into account - the closer the mapping between these, then the less translation logic is required in the Management path. This simplifies the work flow as translation intelligence can be avoided as both Redis schema and NB YANG schema are aligned. + +Where the YANG does not map directly to the Redis DB, the management framework provides mechanisms to represent complex translations via developer written custom functions. + +SONiC YANG should always be developed for a new feature, and should be purely based on the schema of config objects in DB. + +For the case where developer prefers to write a non-standard YANG model for a new or existing SONIC feature, the YANG should be written aligned to Redis schema such that the same YANG can be used for both northbound and CVL. This simplifies the developer work flow (explained in section 5.1) + + +### 5.1 Developer work flow for non Standards-based SONiC YANG + + Typical steps for writing a non-standard YANG for management framework are given below. +- Write the SONiC YANG based upon the Redis DB design. +- Add 'must','when' expressions to capture the dependency between config objects. +- Add required non-config, notification and RPC objects to the YANG. +- Add meta data for transformer. + +#### 5.1.1 Define SONiC YANG schema +Redis schema needs to be expressed in SONiC proprietary YANG model with all data types and constraints. Appropriate custom YANG extensions need to be used for expressing metadata. The YANG model is used by Config Validation Library(CVL) to provide automatic syntactic and semantic validation and Translib for Northbound management interface. + +Custom validation code needs to be written if some of the constraints cannot be expressed in YANG syntax. + +Please refer to [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) for detiailed guidelines on writing SONiC YANG. + +#### 5.1.2 Generation of REST server stubs and Client SDKs for YANG based APIs + +* Place the main YANG modules under sonic-mgmt-common/models/yang directory. + * By placing YANG module in this directory, on the next build, OpenAPI YAML (OpenAPI spec) is generated for the YANG. + * If there is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-common/models/yang directory itself. +* Place all dependent YANG modules which are imported into the main YANG module such as submodules or YANGs which define typedefs, etc under sonic-mgmt-common/models/yang/common directory. + * By placing YANG module in this directory, OpenAPI YAML (OpenAPI spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-common/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory. + * Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-common/models/yang/common. +* Generation of REST server stubs and client SDKs will automatically happen when make command is executed as part of the build. + +#### 5.1.3 Config Translation App (Go language) +Config Translation App consists of two parts - Transformer and App module. They translate the data in Northbound API schema to the native Redis schema (refer to section 5.1.1) and vice versa. All Northbound API services like REST, GNMI, NETCONF invoke this App to read and write data through Translib API calls. + +Key features: + +* Go language. +* YANG to Redis and vice-versa data translation is handled by Transformer. The data translation happens based on the YANG model developed as per section 5.1.1. +* The processing of data is taken care by App module + * App consumes/produces YANG data through [YGOT](https://github.com/openconfig/YGOT) structures. + * Framework provides Go language APIs for Redis DB access. APIs are similar to existing Python APIs defined in sonic-py-swsssdk repo. + * The Transformer converts between these formats. + + * For read operation + * App module receives the YANG path to read in a Go struct + * App module uses the Transformer to get a DB entry(s) reference from this path + * App module reads the Redis entry(s) + * App module uses the Transformer to convert the returned DB object(s) to a Go struct + * App module returns this to TransLib + * For write operations + * App module receives the target YANG path and data in a Go struct + * App module uses the Transformer to translates the Go struct into appropriate Redis calls + * DB Access layer takes care of DB transactions - write everything or none + +* REST server provides a test UI for quick UT of App modules. This UI lists all REST APIs for a YANG and provides option to try them out. +* Pytest automation integration can make use of direct REST calls or CLI (which also makes use of REST APIs internally). Framework generates REST client SDK to facilitate direct REST API calls. + +#### 5.1.4 IS CLI +IS CLI is achieved using KLISH framework. + +* CLI tree is expressed in the XML file with node data types and hierarchy along with different modes. +* Actioner handler needs to be hooked up in XML for corresponding CLI syntax. Actioner handler should be developed to call client SDK APIs (i.e one action handler might need to call multiple client SDK APIs.) +* Show command output formatting is achieved using [Jinja](http://jinja.pocoo.org/) templates. So, the developer needs to check if an existing template can be used or new template needs to be written. + +#### 5.1.5 gNMI +There is no specific steps required for gNMI. + + +### 5.2 Developer work flow for standard-based (e.g. OpenConfig/IETF) YANG + +#### 5.2.1 Identify the standard YANG module for the feature for northbound APIs. +The Developer starts by selecting the standard YANG to be the basis for the Management data model. For SONiC, OpenConfig is the preferred source. However, in the absence of a suitable model there, other standards can be considered (e.g. IETF, IEEE, OEM de-facto standards). + +The SONiC feature implementation may not exactly match the chosen model - there may be parts of the YANG that are not implemented by (or have limitations in) the feature, or the Developer may wish to expose feature objects that are not covered by the YANG. In such cases, the Developer should extend the model accordingly. The preferred method is to write a deviation file and then use modification statements there (e.g. deviate, augment) accordingly. + +#### 5.2.2 Define the Redis schema for the new feature. (not applicable for legacy/existing feature) +The Redis DB design should take the YANG design into account, and try to stay as close to it as possible. This simplifies the translation processes within the Management implementation. Where this is not possible or appropriate, custom translation code must be provided. + + +#### 5.2.3 Define SONiC YANG schema +Redis schema needs to be expressed in SONiC YANG model with all data types and constraints. Appropriate custom YANG extensions need to be used for expressing this metadata. The YANG model is used by Config Validation Library(CVL)to provide automatic syntactic and semantic validation. + +Custom validation code needs to be written if some of the constraints cannot be expressed in YANG syntax. + +Please refer to [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) for detiailed guidelines on writing SONiC YANG. + +#### 5.2.4 Generation of REST server stubs and Client SDKs for YANG based APIs + +* Place the main YANG modules under sonic-mgmt-common/models/yang directory. + * By placing YANG module in this directory, YAML (OpenAPI spec) is generated for the YANG. + * If there is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-common/models/yang directory itself. +* Place all dependent YANG modules such as submodules or YANGs which define typedefs, etc under sonic-mgmt-common/models/yang/common directory. + * By placing YANG module in this directory, YAML (OpenAPI spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-common/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory. + * Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-common/models/yang/common. +* Generation of REST-server stubs and client SDKs will automatically happen when make command is executed as part of the build. + + +#### 5.2.5 Config Translation App (Go language) +Config Translation App consists of two parts - Transformer and App module. They translate the data in Northbound API schema (defined in step#1) to the native Redis schema (defined in step#2) and vice versa. All Northbound API services like REST, GNMI, NETCONF invoke this App to read and write data. + +Key features: + +* Go language. +* YANG to Redis and vice-versa data translation is handled by Transformer. In order to facilitate data translation, the developer needs to provide just the YANG file for the data model + * YANG file for the data model + * Optionally, a YANG annotation file (refer to [3.2.2.7.8 Utilities](#32278-utilities)) to define translation hints to map YANG objects to DB objects. These translation hints are external callbacks for performing complex translation whereas simple translations are handled by Transformer's built-in methods. The annotation file is also placed in `sonic-mgmt-common/models/yang` + * Code to define the translation callbacks, in `sonic-mgmt-common/src/translib/transformer` +* The processing of data is taken care by App module + * App consumes/produces YANG data through [YGOT](https://github.com/openconfig/YGOT) structures. + * Framework provides Go language APIs for Redis DB access. APIs are similar to existing Python APIs defined in sonic-py-swsssdk repo. + * For read operation + * App module receives the YANG path to read in a Go struct + * App module uses the Transformer to get a DB entry(s) reference from this path + * App module reads the Redis entry(s) + * App module uses the Transformer to convert the returned DB object(s) to a Go struct + * App module returns this to TransLib + * For write operations + * App module receives the target YANG path and data as YGOT tree + * App module translates the YGOT tree data into appropriate Redis calls using reference from Transformer + * App module also handles additional complex logic like transaction ordering or checking for dependencies + * Translation Framework takes care of DB transactions - write everything or none +* REST server provides a test UI for quick UT of translation app. This UI lists all REST APIs for a YANG and provide option to try them out. REST server invokes Translation Apps. +* Spytest automation integration can make use of direct REST calls or CLI (which also makes use of REST internally - step#5). Framework generates REST client SDK to facilitate direct REST calls. + + +#### 5.2.6 IS CLI +IS CLI is achieved using KLISH framework and steps are same as in SONiC YANG model. Please refer to section [5.1.4 IS CLI](#514-IS-CLI). + +#### 5.2.7 gNMI +There is no specific steps required for gNMI. + + +## 6 Error Handling + +Validation is done at both north bound interface and against database schema. Appropriate error code is returned for invalid configuration. +All application errors are logged into syslog. + +## 7 Serviceability and Debug + +1. Detailed syslog messages to help trace a failure. +2. Debug commands will be added when debug framework becomes available. +3. CPU profiling enable/disable with SIGUR1 signal. + +## 8 Warm Boot Support + +Management Framework does not disrupt data plane traffic during warmboot. No special handling required for warmboot. + +## 9 Scalability + +Manageability framework will be scalable to handle huge payloads conforming to the standard/SONiC yang models. + +## 10 Unit Test + +#### GNMI +1. Verify that gnmi_get is working at Toplevel module +2. Verify thet gnmi_get is working for each ACL Table +3. Verify gnmi_get working for each ACL Rule: +4. Verify that gnmi_get is working for all ACL interfaces +5. Verify that gnmi_get is working for each ACL interface name +6. Verify that gnmi_get fails for non-existent ACL name and type +7. Verify that TopLevel node can be deleted +8. Verify that a particular ACL Table can be deleted +9. Verify that ACL rule can be deleted +10. Verify that ACL table can be created +11. Verify that ACL rule can be created +12. Verify that ACL binding can be created +13. Verify that creating rule on non existent ACL gives error +14. Verify that giving invalid interface number is payload gives error. +15. Verify that GNMI capabalities is returning correctly. + +#### Request Binder (YGOT) +1. create a YGOT object binding for the uri ends with container +2. create a YGOT object binding for the uri ends with leaf +3. create a YGOT object binding for the uri ends with list +4. create a YGOT object binding for the uri ends with leaf-list +5. create a YGOT object binding for the uri which has keys +6. create a YGOT object binding for the uri which has keys and ends with list with keys +7. validate the uri which has the correct number of keys +8. validate the uri which has the invalid node name +9. validate the uri which has the invalid key value +10. validate the uri which has the incorrect number of keys +11. validate the uri which has the invalid leaf value +12. validate the payload which has the incorrect number of keys +13. validate the payload which has the invalid node name +14. validate the payload which has the invalid leaf value +15. validate the uri and the payload with the "CREATE" operation +16. validate the uri and the payload with the "UPDATE" operation +17. validate the uri and the payload with the "DELETE" operation +18. validate the uri and the payload with the "REPLACE" operation +19. validate the getNodeName method for LIST node +20. validate the getNodeName method for leaf node +21. validate the getNodeName method for leaf-list node +22. validate the getParentNode method for LIST node +23. validate the getParentNode method for leaf node +24. validate the getParentNode method for leaf-list node +25. validate the getObjectFieldName method for LIST node +26. validate the getObjectFieldName method for leaf node +27. validate the getObjectFieldName method for leaf-list node + +#### DB access layer +1. Create, and close a DB connection. (NewDB(), DeleteDB()) +2. Get an entry (GetEntry()) +3. Set an entry without Transaction (SetEntry()) +4. Delete an entry without Transaction (DeleteEntry()) +5. Get a Table (GetTable()) +6. Set an entry with Transaction (StartTx(), SetEntry(), CommitTx()) +7. Delete an entry with Transaction (StartTx(), DeleteEntry(), CommitTx()) +8. Abort Transaction. (StartTx(), DeleteEntry(), AbortTx()) +9. Get multiple keys (GetKeys()) +10. Delete multiple keys (DeleteKeys()) +11. Delete Table (DeleteTable()) +12. Set an entry with Transaction using WatchKeys Check-And-Set(CAS) +13. Set an entry with Transaction using Table CAS +14. Set an entry with Transaction using WatchKeys, and Table CAS +15. Set an entry with Transaction with empty WatchKeys, and Table CAS +16. Negative Test(NT): Fail a Transaction using WatchKeys CAS +17. NT: Fail a Transaction using Table CAS +18. NT: Abort an Transaction with empty WatchKeys/Table CAS +19. NT: Check V logs, Error logs +20. NT: GetEntry() EntryNotExist. + +#### ACL app (via REST) +1. Verify that if no ACL and Rules configured, top level GET request should return empty response +2. Verify that bulk request for ACLs, multiple Rules within each ACLs and interface bindings are getting created with POST request at top level +3. Verify that all ACLs and Rules and interface bindings are shown with top level GET request +5. Verify that GET returns all Rules for single ACL +6. Verify that GET returns Rules details for single Rule +7. Verify that GET returns all interfaces at top level ACL-interfaces +8. Verify that GET returns one interface binding +9. Verify that single or multiple new Rule(s) can be added to existing ACL using POST/PATCH request +10. Verify that single or mutiple new ACLs can be added using POST/PATCH request +11. Verify that single or multiple new interface bindings can be added to existing ACL using POST/PATCH request +12. Verify that single Rule is deleted from an ACL with DELETE request +13. Verify that single ACL along with all its Rules and bindings are deleted with DELETE request +14. Verify that single interface binding is deleted with DELETE request +15. Verify that all ACLs and Rules and interface bindings are deleted with top level DELETE request +16. Verify that CVL throws error is ACL is created with name and type same as existing ACL with POST request +17. Verify that CVL throws error is RULE is created with SeqId, ACL name and type same as existing Rule with POST request +18. Verify that GET returns error for non exising ACL or Rule +19. Verify that CVL returns errors on creating rule under non existent ACL using POST request +20. Verify that CVL returns error on giving invalid interface number in payload during binding creation + +#### CVL +1. Check if CVL validation passes when data is given as JSON file +2. Check if CVL Validation passes for Tables with repeated keys like QUEUE,WRED_PROFILE and SCHEDULER +3. Check if CVL throws error when bad schema is passed +4. Check if debug trace level is changed as per updated conf file on receiving SIGUSR2 +5. Check must constraint for DELETE throws failure if condition fails, (if acl is a bound to port, deleting the acl rule throws error due to must constraint) +6. Check if CVL Validation passes when data has cascaded leafref dependency (Vlan Member->Vlan->Port) +7. Check if Proper Error Tag is returned when must condition is not satisfied +8. Check if CVL Validation passes if Redis is loaded with dependent data for UPDATE operation. +9. Check is CVL Error is returned when any mandatory node is not provided. +10. Check if CVL validation passes when global cache is updated for PORT Table for "must" expressions. +11. Check if CVL is able to validate JSON data given in JSON file for VLAN , ACL models +12. Check if CVL initialization is successful +13. Check if CVL is able to validate JSON data given in string format for CABLE LENGTH +14. Check if CVL failure is returned if input JSON data has incorrect key +15. Check if CVL is returning CVL_SUCCESS for Create operation if Dependent Data is present in Redis +16. Check if CVL is returning CVL_FAILURE for Create operation with invalid field for CABLE_LENGTH . +17. Check is CVL Error is returned for any invalid field in leaf +18. Check is Valid CVL_SUCCESS is returned for Valid field for ACL_TABLE when data is given in Test Structure +19. Check is Valid CVL_SUCCESS is returned for Valid field for ACL_RULE where Dependent data is provided in same session +20. Check if CVL is returning CVL_FAILURE for Create operation with invalid Enum vaue +21. Check if CVL validation fails when incorrect IP address prefix is provided. +22. Check is CVL validation fails when incorrect IP address is provided. +23. Check is CVL validation fails when out of bound are provided. +24. Check is CVL validation fails when invalid IP protocol +25. Check is CVL validation fails when out of range values are provided. +26. Check if CVL validation fails when incorrect key name is provided . +27. Check if CVL validation passes is any allowed special character is list name. +28. Check if CVL validation fails when key names contains junk characters. +29. Check if CVL validation fails when additional extra node is provided +30. Check is CVL validation passes when JSON data is given as buffer for DEVICE METEADATA +31. Check if CVL validation fails when key name does not contain separators. +32. Check if CVL validation fails when one of the keys is missing for Create operation +33. Check if CVL validation fails when there are no keys between separators for Create operation +34. Check if CVL validation fails when missing dependent data is provided for Update operation in same transaction +35. Check if CVL validation fails when missing dependent data is provided for Create operation in same transaction. +36. Check if CVL validation fails when there are no keys between separators for DELETE operation +37. Check if CVL validation fails when there are no keys between separators for UPDATE operation +38. Check if CVL validation fails when invalid key separators are provided for Delete operation +39. Check if CVL validation fails if UPDATE operation is given with invalid enum value +40. Check if CVL validation fails if UPDATE operation is given with invalid key containing missing keys +41. Check if CVL validation passes with dependent data present in same transaction for DELETE operation. +42. Check if CVL validation fails if DELETE operation is given with missing key for DELETE operation. +43. Check if CVL validation fails if UPDATE operation is given with missing key +44. Check if CVL validation fails when an existing key is provided in CREATE operation +45. Check if CVL validation passes for INTERFACE table +46. Check if CVL validation fails when configuration not satisfying must constraint is provided +47. Check if CVL validation passes when Redis has valid dependent data for UPDATE operation +48. Check if CVL validation fails when two different sequences are passed(Create and Update is same transaction) +49. Check if CVL validation fails for UPDATE operation when Redis does not have dependent data. +50. Check if CVL validation passes with valid dependent data given for CREATE operation. +51. Check if CVL validation fails when user tries to delete non existent key +52. Check if CVL Validation passes if Cache contains dependent data populated in same sessions but separate transaction. +53. Check if CVL Validation passes if Cache data dependent data that is populated across sessions +54. Check if CVL Validation fails when incorrect dependent Data is provided for CREATE operation +55. Check if CVL validation passes when valid dependent data is provided for CREATE operation +56. Check if Proper Error Tag is returned when must condition is not satisfied in "range" +57. Check if Proper Error Tag is returned when must condition is not satisfied in "length" +58. Check if Proper Error Tag is returned when must condition is not satisfied in "pattern" +59. Check if DELETE fails when ACL Table is tried to Rule or when DELETE tries to delete TABLE with non-empty leafref +60. Check if validation fails when non-existent dependent data is provided. +61. Check if CVL validation fails when DELETE tries to delete leafref of another table(delete ACL table referenced by ACL rule) +62. Check if CVL Validation fails when unrelated chained dependent data is given. +63. Check if CVL Validation fails when VLAN range is out of bound and proper error message is returned +64. Check if Logs are printed as per configuration in log configuration file. +65. Check if DELETE operation is performed on single field +66. Check if CVL validation passes when valid dependent data is provided using a JSON file. +67. Check if CVL validation is passed when when delete is performed on Table and then connected leafref +68. Check if CVL validation is passes when JSON data can be given in file format +69. Check if CVL Finish operation is successful +70. Check if CVL validation passes when Entry can be deleted and created in same transaction +71. Check if CVL validation passes when two UPDATE operation are given +72. Check if CVL validation passes when 'leafref' points to a key in another table having multiple keys. +73. Check if CVL validation passes when 'leafref' points to non-key in another table. +74. Check if CVL validation passes when 'leafref' points to a key which is drived in predicate from another table in cascaded fashion. +75. Check if CVL validation passes when 'must' condition involves checking with fields having default value which are not provided in request a data. +76. Check if CVL validation passes when 'must' condition has predicate field/key value derived from another table using another predicate in cascaded fashion. +77. Check if CVL validation passes for 'when' condition present within a leaf/leaf-list. +78. Check if CVL validation passes for 'when' condition present in choice/case node. +79. Check if CVL validation passes if 'max-elements' is present in a YANG list. +80. Check if CVL can sort the list of given tables as per their dependency imposed by leafref. +81. Check if CVL can return sorted list of tables in a given YANG module based on leafref relation. +82. Check if CVL can return the list of dependent tables for a given table in a YANG module. +83. Check if CVL can return the dependent data(Redis keys) to be deleted or modified for a given entry getting deleted. + +## 11 Appendix A + +Following are the list of Open source tools used in Management framework + +1. [Gorilla/mux](https://github.com/gorilla/mux) +2. [Go datastructures](https://github.com/Workiva/go-datastructures/tree/master/queue) +3. [OpenAPI](https://OpenAPI.io) +4. [gNMI client](https://github.com/jipanyang/gnxi) +5. [goyang](https://github.com/openconfig/goyang) +6. [YGOT](https://github.com/openconfig/ygot/ygot) +7. [GO Redis](https://github.com/go-redis/redis) +8. [Logging](https://github.com/google/glog) +9. [Profiling](https://github.com/pkg/profile) +10. [Validation](https://gopkg.in/go-playground/validator.v9) +11. [JSON query](https://github.com/antchfx/jsonquery) +12. [XML query](https://github.com/antchfx/xmlquery) +13. [Sorting](https://github.com/facette/natsort) +14. [pyangbind](https://github.com/robshakir/pyangbind) +15. [libYang](https://github.com/CESNET/libyang) + + +## 12 Appendix B + +Following are the list of Open source libraries used in telemetry container. +Always refer to the [Makefile](https://github.com/Azure/sonic-telemetry/blob/master/Makefile) for the sonic-telemetry container for current package list. + + +1. [GRPC](https://google.golang.org/grpc) +2. [GNMI](https://github.com/openconfig/gnmi/proto/gnmi) +3. [Protobuf](https://github.com/golang/protobuf/proto) +4. [goyang](https://github.com/openconfig/goyang) +5. [GO Redis](https://github.com/go-redis/redis) +6. [YGOT](https://github.com/openconfig/ygot/ygot) +7. [Logging](https://github.com/google/glog) +8. [GO Context](https://golang.org/x/net/context) +9. [Credentials](https://google.golang.org/grpc/credentials) +10. [Validation](https://gopkg.in/go-playground/validator.v9) +11. [GNXI utils](https://github.com/google/gnxi/utils) +12. [Gorilla/mux](https://github.com/gorilla/mux) +13. [jipanyang/xpath](https://github.com/jipanyang/gnxi/utils/xpath) +14. [c9s/procinfo](https://github.com/c9s/goprocinfo/linux) +15. [Workkiva/Queue](https://github.com/Workiva/go-datastructures/queue) +16. [jipanyang/gnmi client](https://github.com/jipanyang/gnmi/client/gnmi) +17. [xeipuuv/gojsonschema](https://github.com/xeipuuv/gojsonschema) diff --git a/doc/mgmt/REST_DOC_GEN_HLD.md b/doc/mgmt/REST_DOC_GEN_HLD.md new file mode 100644 index 0000000000..dd43fbc2a6 --- /dev/null +++ b/doc/mgmt/REST_DOC_GEN_HLD.md @@ -0,0 +1,98 @@ +# The RESTCONF API document generator design +# High Level Design Document +### Rev 0.1 + +# Table of Contents + * [Revision](#revision) + * [About this Manual](#about-this-manual) + * [Scope](#scope) + * [Definitions/Abbreviation](#definitionsabbreviation) + * [Introduction](#introduction) + + * [1 Overview](#1-overview) + * [1.1 Design Diagram](#11-system-chart) + * [1.2 Modules description](#12-modules-description) + * [1.2.1 Pyang](#121-pyang) + * [1.2.2 OpenAPI plugin](#122-openapi-plugin) + * [1.2.3 Markdown formatter](#123-markdown-formatter) + * [2 Requirements](#2-requirements) + * [3 Document Generation Approach](#3-document-generation-approach) + * [3.1 Auto Generated document](#31-auto-generated-document) + * [3.2 Manually written template document](#32-manually-written-template-document) + * [3.3 Final Document](#33-final-document) + +###### Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 01/08/2020 | Mohammed Faraaz | Initial version | + +# About this Manual +This document provides general information about the design of the [RESTCONF 8040](https://tools.ietf.org/html/rfc8040) API document generator and the REST document syntax. + +# Scope +This document describes only the high level design of the [RESTCONF 8040](https://tools.ietf.org/html/rfc8040) API document related components. It is intending to provide high level details about Pyang and its associated OpenAPI plugin. + +# Introduction +The Existing OpenAPI plugin for Pyang is enhanced to generate a document for RESTCONF API, it uses the same Utils and tools used by OpenAPI Plugin. The generated document will be in markdown(.md) format. + +# Definitions/Abbreviation + +| Definitions/Abbreviation | Description | +|--------------------------|--------------------------------------------| +| REST | Representational state transfer | +| API | Application Programmable Interface | + +# 1 Overview + +## 1.1 Design + +Following diagram describes a RESTCONF API document generator design + +![Design](https://docs.google.com/drawings/d/1mIejWXE6kqnxcN933_VMDKgtIeyobT0sw-w0DErkRHQ/edit?folder=0AM8ib1ydVn2cUk9PVA) + +Note: +Temporary using G-link above will change it actual file name after review. + +## 1.2 Modules description + +### 1.2.1 Pyang + +Pyang is an open source Yang Parser written in Python, It parses the yang modules and provides yang data in a programmable object (python objects) format, this data is passed on to the OpenAPI plugin for OpenAPI spec and RESTCONF document generation. + +### 1.2.2 OpenAPI Plugin + +This is an extension plugin written on top of Pyang parser, it walks over the generated Yang data (present in a form of Python object) and generates a yang tree along with required metadata for the generation of OpenAPI spec (swagger spec 2.0) and a RESTCONF document for each yang module in a markdown (.md) format. + +### 1.2.3 Markdown Formatter + +It is an extension/module inside OpenAPI plugin, it walks over the yang tree produced and uses the same Utils and metadata generated by OpenAPI generator and generates a REST document in a markdown(.md) Format. + +# 2 Requirements + +* Tool shall generate RESTCONF compatible URIs and payload. +* Tool shall output the Request and Response payload for the URIs in a JSON format. +* Tool shall fetch the Description from the yang and describe the URIs in a generated document +* Tool shall list and describe the Parameters which will be part of URIs +* Tool shall generate a RESTCONF API document in a markdown(.md) format. +* Tool shall as part of build and can be turned off using an option in Makefile. +* Tool shall generate documents for both Standard and Sonic yang models. + +# 3 Document Generation Approach + +## 3.1 Auto Generated document + +The RESTCONF document generation tool will generate one markdown (.md) file per Yang module. +These auto-generated documents will be placed under sonic-mgmt-framework/build/restconf_md directory. +The auto-generated documents will have following sections +* Configuration APIs - Describes Configuration URIs i.e. config:true Yang statement. +* Operational-state APIs - Describes Operational-state URIs i.e. config:false Yang statement. +* Operations APIs - Describes RPCs defined in yang model. + +## 3.2 Manually written template document + +The base document known as RESTCONF_documentation_index.template is a manually written jinja based markdown document which contains a static sections describing RESTCONF protocol itself, authentication, supported media types, RESTCONF capabilities, Response messages, protocol and transport requirements. This is a checked-in file. + +## 3.3 Final Document + +The Final RESTCONF document called RESTCONF_documentation_index.md is auto-generated during build time which contains contents of static file described in above [3.2](#32-manually-written-template-document) and the hyperlinks to the RESTCONF document for individual Yang models generated in section [3.1](#31-auto-Generated-document). + diff --git a/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md b/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md new file mode 100644 index 0000000000..83d2aa0323 --- /dev/null +++ b/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md @@ -0,0 +1,298 @@ +# Show techsupport +Diagnostic information aggregated presentation +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/06/2019 | Kerry Meyer | Initial version | + +# About this Manual +This document provides general information about presentation of aggregated diagnostic information for the SONiC subsystem via the Management Framework infrastructure. +# Scope +This document describes the high level design for the "show techsupport" command implementation under the control of the Management Framework infrastructure. It is intended to cover the general approach and method for providing a flexible collection of diagnostic information items and a mechanism for adding new items to the list of information to be provided. It also considers the basic mechanisms to be used for the various types of information to be aggregated. It does not address specific details for collection of all supported classes of information. + +# Definition/Abbreviation + +# 1 Feature Overview + +Provide Management Framework functionality to process the "show techsupport" command: + - Create an aggregated file containing the information items needed for analysis and diagnosis of problems occurring during switch operations. + - Support reduction of aggregated log file information via an optional "--since" parameter specifying the desired logging start time. + +NOTE: The underlying feature for which this Management Framework feature provides "front end" client interfaces is unchanged by the addition of these interfaces. (The "since " option available through these interfaces, however, is restricted to the IETF/YANG date/time format.) Please refer to the following document for a description of the "show techsupport" base feature: + +https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md#troubleshooting-commands + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide a Management Framework based interface for the "show tech-support" command. + +### 1.1.2 Configuration and Management Requirements +Provide the ability to invoke the command via the following client interfaces: + + - Management Framework CLI (same syntax as the existing Click-based + API except for tighter restriction of the "DateTime" format to + conform with the Yang/IETF DateTime standard) + - REST API + - gNOI + +(See Section 3 for additional details.) + +### 1.1.3 Scalability Requirements + +Time and storage space constraints: The large number of information items collected and the potentially large size of some of the items (e.g. interface information display in a large system) present an exposure to the risk of long processing times and significant demands on disk storage space. The Management Framework interface invokes the same command used for the Click-based interface. It adds no significant additional overhead or processing time. The storage space requirements are unchanged. +### 1.1.4 Warm Boot Requirements +N/A +## 1.2 Design Overview +### 1.2.1 Basic Approach +This feature will be implemented using the Management Framework infrastructure supplemented with customized access mechanisms for handling "non-DB" data items. + +### 1.2.2 Container +The user interface (front end) portion of this feature is implemented within the Management Framework container. + +### 1.2.3 SAI Overview +N/A (non-hardware feature) + +# 2 Functionality +## 2.1 Target Deployment Use Cases +This feature provides a quick and simple mechanism for network administrators or other personnel with no detailed knowledge of switch internal details to gather an extensive set of information items by using a single command. These items provide critical information to help development and sustaining engineering teams in the analysis and debugging of problems encountered in deployment and test environments. +## 2.2 Functional Description +The set of items to be gathered for a given software release is defined by the development team. It is specified in a way that enables run-time access to the desired set of information items to be collected. The definition of the set of information items to be collected includes specification of the access function to be used for each item in the list to gather the information, format it as needed, and pack it into the output file. The location of the resulting output file is provided to the requesting client at the completion of command execution. + +The output file name has the following form: + +` +/var/dump/sonic_dump_sonic_YYYYMMDD_HHMMSS.tar.gz +` + +Example: + +`/var/dump/sonic_dump_sonic_20191118_221625.tar.gz +` +See section 3.6.2.2 for an explanation of the output file name format. + +To view the contents of the file, the user must copy it to a local file in the client file system. If the file is to be extracted within the directory to which it is copied, the directory should have at least 50 MB of available space. To extract the file inside of the directory to which it has been copied while displaying a list of output files, the following command can be used: + +` +tar xvzf filename.tar.gz +` +The files are extracted to a directory tree, organized based on the type of information contained in the files. Example file categories for which sub-directories are provided in the output file tree include: + +- log files ("log" directory ) +- Linux configuration files ("etc" directory) +- generic application "dump" output ("dump" directory) +- network hardware driver information ("sai" directory) +- detailed information on various processes ("proc" directory). + +To extract the file contents to an alternate location, the following form of the "tar" command can be used: + +` + tar xvzf filename.tar.gz -C /path/to/destination/directory +` +Some of the larger "extracted" files will be compressed in gzip format. This includes log files and core files and also includes other files containing a large amount of output (e.g. a dump of all BGP tables). These files have a ".gz" file type. They can be extracted using: + +` +gunzip +` + + +# 3 Design +## 3.1 Overview +The "show techsupport" command causes invocation of an RPC sent from the management framework to a process in the host to cause collection of a list of flexibly defined sets of diagnostic information (information "items"). The collected list of items is stored in a compressed "tar" file with a unique name. The command output provides the location of the resulting compressed tar file. + +The "since" option can be used, if desired, to restrict the time scope for log files and core files to be collected. This option is passed to the host process for use during invocation of the applicable information gathering sub-functions. + +## 3.2 DB Changes +N/A +### 3.2.1 CONFIG DB +### 3.2.2 APP DB +### 3.2.3 STATE DB +### 3.2.4 ASIC DB +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design +N/A +### 3.3.1 Orchestration Agent +### 3.3.2 Other Process +The "show techsupport" feature requires RPC support in a process running within the host context. The host process handling the RPC is responsible for dispatching "show techsupport" requests from the management framework container to trigger allocation of an output file, gathering and packing of the required information into the output file, and sending a response to the management framework RPC agent to specify the name and path of the output file. + +## 3.4 SyncD +N/A + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +The following Sonic Yang model is used for implementation of this feature: + +```module: sonic-show-techsupport + + rpcs: + +---x sonic-show-techsupport-info + +---w input + | +---w date? yang:date-and-time + +--ro output + +--ro output-filename? string +``` + + + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +N/A +#### 3.6.2.2 Show Commands + +Command syntax summary: + +` +show techsupport [since ] +` + +Command Description: + +Gather information for troubleshooting. Display the name of a file containing the resulting group of collected information items in a compressed "tar" file. + + +Syntax Description: + +| Keyword | Description | +|:--------------|:----------- | +| since | This option uses a text string containing the desired starting Date/Time for collected log files and core files. The format of the Date/Time in the string is defined by the Yang/IETF date-and-time specification (REF http://www.netconfcentral.org/modules/ietf-yang-types, based on http://www.ietf.org/rfc/rfc6020.txt). If "since is specified, this value is passed to the host process for use during invocation of the applicable log/core file gathering sub-functions.| + +Command Mode: User EXEC + +Output format example and summary: + +``` +Example: + +Output stored in: /var/dump/sonic_dump_sonic_20191008_082312.tar.gz + +-------------------------------------------------- + +Output file name sub-fields are defined a follows: + +- YYYY = Year +- MM = Month (numeric) +- DD = Day of the Month +- HH = hour of the current time (based on execution of the Linux "**date**" command) at the start of command execution +- MM = minute of the current time (based on execution of the Linux "**date**" command) at the start of command execution +- SS = second of the current time (based on execution of the Linux "**date**" command) at the start of command execution +``` + +Command execution example (basic command): + +``` +sonic# show techsupport + +Output stored in: /var/dump/sonic_dump_sonic_20191008_082312.tar.gz + +``` +Command execution Example (using the "since" keyword/subcommand): + +``` +sonic# show tech-support + since Collect logs and core files since a specified date/time + | Pipe through a command + + +sonic# show tech-support since + String date/time in the format: + + "YYYY-MM-DDTHH:MM:SS[.ddd...]Z" or + "YYYY-MM-DDTHH:MM:SS[.ddd...]+hh:mm" or + "YYYY-MM-DDTHH:MM:SS[.ddd...]-hh:mm" Where: + + YYYY = year, MM = month, DD = day, + T (required before time), + HH = hours, MM = minutes, SS = seconds, + .ddd... = decimal fraction of a second (e.g. ".323") + Z indicates zero offset from local time + +/- hh:mm indicates hour:minute offset from local time + +sonic# show tech-support since 2019-11-27T22:02:00Z +Output stored in: /var/dump/sonic_dump_sonic_20191127_220334.tar.gz +``` +Command execution example invocation via REST API: + +``` +REST request via CURL: + +curl -X POST "https://10.11.68.13/restconf/operations/sonic-show-techsupport:sonic-show-techsupport-info" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{ \"sonic-show-techsupport:input\": { \"date\": \"2019-11-27T22:02:00.314+03:08\" }}" + +Request URL: + +https://10.11.68.13/restconf/operations/sonic-show-techsupport:sonic-show-techsupport-info + +Response Body: + +{ + "sonic-show-techsupport:output": { + "output-filename": "/var/dump/sonic_dump_sonic_20191128_013141.tar.gz" + } +} +``` + +Command execution example invocation via gNOI API: + +``` +root@sonic:/usr/sbin# ./gnoi_client -module Sonic -rpc showtechsupport -jsonin "{\"input\":{\"date\":\"2019-11-27T22:02:00Z\"}}" -insecure +Sonic ShowTechsupport +{"sonic-show-techsupport:output":{"output-filename":"/var/dump/sonic_dump_sonic_20191202_194856.tar.gz"}} +``` + +NOTE: See section 3.6.1 for a description of the limitations of the current implementation. A supplementary capability to transfer the tech support file and other diagnostic information files to the client via the Management Framework interface is highly desirable for a future release. + +#### 3.6.2.3 Debug Commands +N/A +#### 3.6.2.4 IS-CLI Compliance +The current implementation differs from the IS-CLI. + + Instead of dumping the technical support information to the output buffer, this implementation dumps the information to a compressed "tar" file and sends the name of the file to the output buffer. This implementation matches the current SONiC implementation. A supplementary capability to enable transfer of the specified file to the client is highly desirable for full functionality of this command when using a REST API or gNMI/gNOI client interface. Without this capability, it is necessary to open a shell on the host and use the SONiC host CLI interface to transfer the file. + +### 3.6.3 REST API Support +REST API support is provided. The REST API corresponds to the SONiC Yang model described in section 3.6.1. + +# 4 Flow Diagrams + +# 5 Error Handling +N/A + +# 6 Serviceability and Debug +Any errors encountered during execution of the "show tech-support" command that prevent retrieval or saving of information are reported in the command output at completion of the operation. + +# 7 Warm Boot Support +N/A + +# 8 Scalability +Refer to section 1.1.3 + +# 9 Unit Test + +| Case | Trigger | Result | +|:-----------|:--------|:-------| +| Basic command execution | Execute the "show techsupport" command with no parameters. | Confirm that the command is accepted without errors and a "result" file name is returned. Confirm that the result file contains the expected set of items. (Examine/expand the contents of the file to ensure that the top level directory tree is correct and that the number of sub-files within the tar file is correct.)| +"since" option (postive test case) | Execute the command with the "--since" TEXT option with a valid date string specifying a time near the end of one of the unfiltered output from the first test.| Same as the "Basic command execution" case. Additionally, confirm that the expected time filtering has occurred by examining one of the affected sub-files.| +"since" option (negative test case #1)|Execute the command with the "--since" TEXT option with an invalid date string.|Verify that an error is returned.| +"since" option (negative test case #2)|Execute the command with the "--since" TEXT option with no date string.|Verify that an error is returned.|Execute the command with the "--since" option with no date string.| Verify that an error is returned.| + + + + + +# 10 Internal Design Information +N/A diff --git a/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md b/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md new file mode 100644 index 0000000000..b2bfff5f0e --- /dev/null +++ b/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md @@ -0,0 +1,250 @@ +# Image Management +Software upgrade and image installation support for SONiC +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +| :--: | :--------: | :---------------: | ------------------ | +| 0.1 | 09/13/2019 | Arunsundar Kannan | Initial version | + +# About this Manual +This document provides general information about the Image management and installation feature implementation in SONiC. +# Scope +Covers northbound interface for the Image Management feature, as well as unit test cases. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +| ---------------- | ------------------------------ | +| Image Management | Image installation, validation | + +# 1 Feature Overview + +Provide management framework capabilities to handle: + +- Image Installation +- List available images +- Removal of an available image +- Set default image for next boot + + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to Image Management + +### 1.1.2 Configuration and Management Requirements +- CLI configuration and show commands +- REST API support +- gNMI Support + +### 1.1.3 Scalability Requirements +N/A +### 1.1.4 Warm Boot Requirements +N/A +## 1.2 Design Overview +### 1.2.1 Basic Approach +Implement Image Management support using translib in sonic-mgmt-framework. +### 1.2.2 Container +Management Container + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +**image install** + +This command is used to install a new image on the alternate image partition. This command takes a path to an installable SONiC image or URL and installs the image. + +**show image-list** + +This command displays information about currently installed images. It displays a list of installed images, currently running image and image set to be loaded in next reboot. + +**image set-default** + +This command is be used to change the image which can be loaded by default in all the subsequent reboots. + +**image remove** + +This command is used to remove the unused SONiC image from the disk. Note that it's *not* allowed to remove currently running image. + +## 2.2 Functional Description +After recieving the request from the client, via an RPC, the REST server will transfer the control to processAction method in the app module(inside trasformer). This method will parse the target uri path and will branch to the corresponding function. These functions will call the python scripts in the host to perform image management related actions, like install, remove ..etc. The response from the output of the script is propagated back to processAction method and is converted to json. The json message is sent back to the client. + +# 3 Design +## 3.1 Overview +Enhancing the management framework backend code and transformer methods to add support for Image management + +## 3.2 DB Changes +State DB + +### 3.2.1 CONFIG DB + +N/A + +### 3.2.2 APP DB + +N/A + +### 3.2.3 STATE DB +The State DB is populated with details regarding the currently used image, image to be in the next boot and the list of available images. + + +### 3.2.4 ASIC DB + +N/A + +### 3.2.5 COUNTER DB + +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent + +N/A + +### 3.3.2 Other Process +N/A + +## 3.4 SyncD +N/A + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models + +``` +module: sonic-image-management + +--rw sonic-image-management + +--rw IMAGE_GLOBAL + | +--rw IMAGE_GLOBAL_LIST* [img_key] + | +--rw img_key enumeration + | +--rw current? string + | +--rw next-boot? string + +--rw IMAGE_TABLE + +--rw IMAGE_TABLE_LIST* [image] + +--rw image string + + rpcs: + +---x image-install + | +---w input + | | +---w imagename? filename-uri-type + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x image-remove + | +---w input + | | +---w imagename? string + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x image-default + +---w input + | +---w imagename? string + +--ro output + +--ro status? int32 + +--ro status-detail? string + +``` + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands + +**image install** + +``` +sonic# image install https://sonic-jenkins.westus.cloudapp.azure.com/job/xxxx/job/buildimage-xxxx-all/xxx/artifact/target/sonic-xxxx.bin + +Done +``` + +**image set-default** + +``` +sonic# image set-default SONiC-OS-HEAD.XXXX +``` + +**image remove** + +``` +sonic# image remove SONiC-OS-HEAD.YYYY + +Image removed +``` + + + +#### 3.6.2.2 Show Commands + +**show image-list** + +``` +sonic# show image-list +Current: SONiC-OS-HEAD.XXXX +Next: SONiC-OS-HEAD.XXXX +Available: +SONiC-OS-HEAD.XXXX +SONiC-OS-HEAD.YYYY +``` + + +#### 3.6.2.3 Debug Commands + +N/A + +#### 3.6.2.4 IS-CLI Compliance + +N/A + +### 3.6.3 REST API Support +* get_sonic_image_management_sonic_image_management +* rpc_sonic_image_management_image_install +* rpc_sonic_image_management_image_remove +* rpc_sonic_image_management_image_default + +# 4 Flow Diagrams +N/A + +# 5 Error Handling + +TBD + +# 6 Serviceability and Debug + +TBD + +# 7 Warm Boot Support + +TBD + +# 8 Scalability +N/A + +# 9 Unit Test +List unit test cases added for this feature including warm boot. + +| Test Name | Test Description | +| :------ | :----- | +| Image install | Image installed successfully and an entry is added in grub.cfg | +| Image remove | Image is removed and the corresponding entry is removed from grub.cfg | +| Image set default | Image is set as zeroth entry(entry for the default image) in grub.cfg | +| Show image list | Image list shows all the entries present in grub.cfg | +# 10 Internal Design Information + diff --git a/doc/mgmt/SONiC_DNS_Support.md b/doc/mgmt/SONiC_DNS_Support.md new file mode 100644 index 0000000000..37e6b3d99b --- /dev/null +++ b/doc/mgmt/SONiC_DNS_Support.md @@ -0,0 +1,433 @@ +# Feature Name +Domain Name System (DNS) support + +# High Level Design Document + +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 05/05 | Venkatesan Mahalingam | Initial version | +| 0.2 | 05/11 | Venkatesan Mahalingam | Added source interface support and addressed comments | + + +# About this Manual + +This document introduces the support of DNS in SONiC. + +# Scope + +This document covers the following, +1) Click commands to configure DNS server & Source interface +2) Click command to show DNS hosts +3) New tables in config DB to handle DNS configs +4) DNS table configuration and show based on OpenConfig YANG model +5) KLISH commands to configure DNS server & Source interface +6) KLISH commands to show DNS hosts +7) Backend support to add DNS configs in "/etc/resolv.conf" file & iptable rules to change the source-IP of the DNS query packets +8) Unit Testcases + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| DNS | Domain Name System | + + + +# 1 Feature Overview + +DNS stands for Domain Name System. It is used to translate human readable domain names into machine readable IP addresses. + +With this feature, users will be able to configure DNS servers and source interface using SONiC Click commands and north bound interface (KLISH/REST/gNMI) provided by management framework module using OpenConfig models. + +Also, backend support to handle new config DB table events to populate nameservers in "/etc/resolv.conf" and add iptable rules to change the source IP (present on the source interface) of the DNS query that is being sent to the DNS server. + +## 1.1 Requirements + +### 1.1.1 Front end configuration and get capabilities + +#### 1.1.1.1 add/delete DNS server +This requirement is to add/delete DNS name server information in the Redis ConfigDB (DNS_SERVER) using Click and mgmt-framework. +The DNS server can be IPv4/IPv6 ipaddress and multiple DNS name servers can be configured. + +#### 1.1.1.2 add/delete DNS source interface +This requirement is to add/delete the global DNS source interface information in the Redis ConfigDB (DNS) using Click and mgmt-framework. + +Only one DNS source interface can be configured in the global DNS table. A new source interface will override the existing DNS source interface. + +#### 1.1.1.4 add/delete VRF name +No special handling is required in "/etc/resolv.conf" file to work on the management VRF (for this release). + +#### 1.1.1.5 Get DNS hosts +This displays the output of the DNS source interface and nameservers. + +### 1.1.2 Backend mechanisms to support configuration and get + +#### 1.1.2.1 add/delete DNS server +The creates or deletes a DNS server entry in the Redis ConfigDB. + +``` + "DNS_SERVER|10.11.0.1": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + "DNS_SERVER|2001:aa:aa::a": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + } +``` + +A change in the DNS_SERVER entry triggers hostcfgd to start the DNS configuration script, which in turn writes each DNS server to the resolv.conf and then restart the dns-config service. + +#### 1.1.2.2 add/delete DNS source interface + +This creates or deletes a global DNS source interface entry in the Redis ConfigDB. + +``` + "DNS|global": { + "type": "hash", + "value": { + "src_intf": "Loopback0" + } + } +``` + +A change in this entry triggers hostcfgd to add iptables SNAT rule incase of IPv4 address and ip6tables SNAT rule in IPv6 source address. + +Only one global DNS source interface is allowed. + +#### 1.1.2.4 get DNS hosts + +Transformer function issues get on DNS tables, parses the response and maps the outputs to the OpenConfig DNS states incase of mgmt-framework interface. + +Incase of Click commands, show commands directly fetches the information from DNS & DNS_SERVER tables and display to the user. + +### 1.1.3 Functional Requirements + +Provide management framework support to +- configure DNS name server +- configure DNS source interface + +### 1.1.4 Configuration and Management Requirements +- Click configuration and show commands +- CLI style configuration and show commands +- REST API support +- gNMI Support + +Details described in Section 3. + +### 1.1.6 Scalability Requirements + +### 1.1.7 Warm Boot Requirements + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +- Implement Click commands using SONiC utilities. +- Implement DNS support using transformer in sonic-mgmt-framework. + +### 1.2.2 Container +The front end code change will be done in management-framework container including: +- XML file for the CLI +- Python script to handle CLI request (actioner) +- Jinja template to render CLI output (renderer) +- OpenConfig YANG model for DNS openconfig-system.yang +- SONiC DNS model for DNS based on Redis DB schema +- transformer functions to + * convert OpenConfig YANG model to SONiC YANG model for DNS related configurations + +### 1.2.3 SAI Overview + +# 2 Functionality + +## 2.1 Target Deployment Use Cases +Manage/configure DNS configurations via gNMI, REST and CLI interfaces + +## 2.2 Functional Description +Provide CLI, gNMI and REST support for DNS configurations. + +## 2.3 Backend change to support new configurations +Provide change in management framework and hostcfgd modules. + +# 3 Design + +## 3.1 Overview + +Enhancing the management framework backend code and transformer methods to add support for DNS. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB +``` +; DNS configuration attributes global to the system. Only one instance of the table exists in the system. +; Key +global_key = “global” ; DNS global configuration +; Attributes +src_intf = ifname ; source interface address for the outgoing DNS packets + +``` + +``` +; DNS name server configuration in the system. +; Key +server_key = IPAddress ; DNS server’s address +; Attributes +No attributes are introduced as part of this design. +``` + +### 3.2.2 APP DB + +### 3.2.3 STATE DB + +### 3.2.4 ASIC DB + +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent + +### 3.3.2 Other Process + +## 3.4 SyncD + +## 3.5 SAI + +## 3.6 User Interface + +### 3.6.1 Data Models + +YANG models needed for DNS handling in the management framework: +1. **openconfig-system.yang** + +2. **sonic-system-dns.yang** + +Supported yang objects and attributes: +```diff + +module: openconfig-system + +--rw system + + +--rw dns + | +--rw config + | | +--rw search* oc-inet:domain-name + | | +--rw oc-sys-ext:source-intf? -> /oc-if:interfaces/interface/name + | +--ro state + | | +--ro search* oc-inet:domain-name + | | +--ro oc-sys-ext:source-intf? -> /oc-if:interfaces/interface/name + | +--rw servers + | | +--rw server* [address] + | | +--rw address -> ../config/address + | | +--rw config + | | | +--rw address? oc-inet:ip-address + | | | +--rw port? oc-inet:port-number + | | +--ro state + | | +--ro address? oc-inet:ip-address + | | +--ro port? oc-inet:port-number + +The above "source-intf" & "address" fields are supported in the scope of this work. + +module: sonic-system-dns + +--rw sonic-system-dns + +--rw DNS_SERVER + | +--rw DNS_SERVER_LIST* [ipaddress] + | +--rw ipaddress inet:ip-address + +--rw DNS + +--rw DNS_LIST* [type] + +--rw type enumeration ('global') + +--rw src_intf? union + +``` + +### 3.6.2 CLI + + +#### 3.6.2.1 Configuration Commands +All commands are executed in `configuration-view`: +``` +sonic# configure terminal +sonic(config)# +``` + +##### 3.6.2.1.1 Configure DNS server & source IP +``` +sonic(config)# ip name-server + source-intf Configure source interface to pick the source IP, used for the DNS query + A.B.C.D/A::B Domain name server + +sonic(config)# ip name-server source-intf + Ethernet Ethernet interface + Loopback Loopback interface + Management Management interface + PortChannel PortChannel interface + Vlan Vlan interface + +sonic(config)# ip name-server 10.11.0.1 + +sonic(config)# ip name-server 2001:aa:aa::a + +sonic(config)# ip name-server source-intf Loopback 0 + +``` + +##### 3.6.2.1.2 Delete DNS server & source IP + +``` +sonic(config)# no ip name-server + source-intf Configure source interface to pick the source IP, used for the DNS query + A.B.C.D/A::B Domain name server + +sonic(config)# no ip name-server 10.11.0.1 + +sonic(config)# no ip name-server 2001:aa:aa::a + +sonic(config)# no ip name-server source-intf + +``` +#### 3.6.2.2 Show dns hosts + +``` +sonic# show hosts +Source Interface : Loopback0 +Name servers are : 10.11.0.1, 2001:aa:aa::a +sonic# +``` + +#### 3.6.2.3 Debug Commands + +#### 3.6.2.4 IS-CLI Compliance + +### 3.6.3 REST API Support +``` +GET - Get existing DNS configuration information from CONFIG DB. + Get DNS peer states +POST - Add DNS configuration into CONFIG DB. +PATCH - Update existing DNS Configuration information in CONFIG DB. +DELETE - Delete a existing DNS configuration from CONFIG DB. +``` +### 3.6.4 Click command support +#### 3.6.4.1 Configuration support +Below click commands are supported for DNS configurations +oot@sonic:~# config dns +Usage: config dns [OPTIONS] COMMAND [ARGS]... + + DNS command line + +Options: + -?, -h, --help Show this message and exit. + +Commands: + add Specify a DNS name server + delete Delete a DNS name server + source_intf DNS source interface configurations +root@sonic:~# +root@sonic:~# config dns source_intf +Usage: config dns source_intf [OPTIONS] COMMAND [ARGS]... + + DNS source interface configurations + +Options: + -?, -h, --help Show this message and exit. + +Commands: + add DNS source interface add configuration + delete DNS source interface delete configuration +root@sonic:~# +``` +root@sonic:~# ping www.yahoo.com +ping: www.yahoo.com: Temporary failure in name resolution + +root@sonic:~# config dns add 10.11.0.1 + +root@sonic:~# ping www.yahoo.com +PING new-fp-shed.wg1.b.yahoo.com (98.137.246.8) 56(84) bytes of data. +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=1 ttl=52 time=24.2 ms +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=2 ttl=52 time=23.9 ms +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=3 ttl=52 time=23.9 ms +^C +--- new-fp-shed.wg1.b.yahoo.com ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2001ms +rtt min/avg/max/mdev = 23.910/24.033/24.262/0.241 ms +root@sonic:~# +``` +#### 3.6.4.2 Show support +Below Click command is used to dump the DNS configurations +``` +root@sonic:~# show hosts +Source Interface : Loopback0 +Name servers are : 10.11.0.1, 2001:aa:aa::a +root@sonic:~# +``` +# 4 Flow Diagrams + +# 5 Error Handling + +# 6 Serviceability and Debug + +# 7 Warm Boot Support +This support is added in the hostcfgd and hence no explicit handling is needed. + +# 8 Scalability + +# 9 Unit Test + +The unit-test for this feature will include: +#### Configuration via CLI + +| Test Name | Test Description | +| :-------- | :----- | +| Configure DNS server | Verify DNS servers are present in the DNS_SERVER table (configDB) and the same is reflected in the resolv.conf file | +| Delete DNS server | Verify DNS servers are not present in the DNS_SERVER table (configDB) and the same is reflected in the resolv.conf file | +| Verify max DNS servers | Verify more than 6 servers are not allowed | +| Configure DNS source interface | Verify source interface is present in the DNS table (configDB) and DNS query packets are transmitted with the source IP present in source interface | +| Delete DNS source interface | Verify source interface is not present in the DNS table (configDB) and DNS query packets are not transmitted with the source IP present in source interface | +| Configure mgmt VRF | Verify that DNS query packets (from mgmt VRF) are transmitted based on nameservers present in resolv.conf file | +| Verify IPv4 DNS query is sent for ping to domain name | Verify whether DNS query is sent based on IPv4 source IP (based on source interface) & destination IP | +| Verify IPv6 DNS query is sent for ping to domain name | Verify whether DNS query is sent based on IPv6 source IP (based on source interface) & destination IP | +| show hosts | Verify source interface and nameservers are displayed correctly | + +#### Configuration via gNMI + +Same test as CLI configuration Test but using gNMI request. +Additional tests will be done to set DNS configuration at different levels of Yang models. + +#### Get configuration via gNMI + +Same as CLI show test but with gNMI request, will verify the JSON response is correct. +Additional tests will be done to get DNS configuration and DNS states at different levels of Yang models. + +#### Configuration via REST (POST/PUT/PATCH) + +Same test as CLI configuration Test but using REST POST request +Additional tests will be done to set DNS configuration at different levels of Yang models. + +**URIs for REST configurations:** + +Source interface - /restconf/data/openconfig-system:system/dns/config/openconfig-system-ext:source-intf +Name server - /restconf/data/openconfig-system:system/dns/servers/server[8.8.8.8]/config + +#### Get configuration via REST (GET) + +Same as CLI show test but with REST GET request, will verify the JSON response is correct. +Additional tests will be done to get DNS configuration and DNS states at different levels of Yang models. + + +# 10 Internal Design Information diff --git a/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md b/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md new file mode 100644 index 0000000000..ff620b4d3c --- /dev/null +++ b/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md @@ -0,0 +1,1069 @@ +# SONiC FRR-BGP Extended Unified Configuration Management Framework +## High Level Design Document +### Rev 0.4 + +## Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + * [Table 1: Abbreviations](#table-1-abbreviations) + * [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.1.1 Functional Requirements](#111-functional-requirements) + * [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + * [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.1.4 Warm Boot Requirements](#114-warmboot-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) + * [2 Functionality](#2-functionality) + * [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + * [3.3 SwSS Design](#33-swss-design) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 User Interface](#36-user-interface) + * [3.6.1 Data Models](#361-data-models) + * [3.6.2 CLI](#362-cli) + * [3.6.2.1 Configuration Commands](#3621-configuration-command) + * [3.6.2.2 Show Commands](#3622-show-command) + * [3.6.2.3 Debug Commands](#3623-debug-command) + * [3.6.2.4 IS-CLI Compliance](#3624-is-cli-compliance) + * [3.6.3 REST API Support](#363-rest-api-support) + * [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 Configuration Sequence](#41-configuration-sequence) + * [4.2 CLI Show Command Sequence](#42-cli-show-command-sequence) + * [4.2.1 CLI Show Sequence - config only](#421-cli-show-sequence-config-only) + * [4.2.2 CLI Show Sequence - State/Statistics](#422-cli-show-sequence-state-statistics) + * [4.3 REST Get Sequence](#43-rest-get-sequence) + * [4.3.1 REST Get Sequence - config only](#431-rest-get-sequence-config-only) + * [4.3.2 REST Get Sequence - State/Statistics](#432-rest-get-sequence-state-statistics) + * [5 Error Handling](#6-error-handling) + * [6 Serviceability and Debug](#7-serviceability-and-debug) + * [7 Warm Boot Support](#8-warm-boot-support) + * [8 Scalability](#9-scalability) + * [9 Unit Test](#10-unit-test) + * [APPENDIX](#APPENDIX) + + +## List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 09/25/2019 | Karthikeyan Arumugam | Initial version | +| 0.2 | 10/30/2019 | Venkatesan Mahalingam | Config DB schema changes | +| 0.3 | 11/20/2019 | Venkatesan Mahalingam | Added various fields in config DB | +| 0.4 | 12/18/2019 | Venkatesan Mahalingam | Addressed the comments on error handling and method of testing | + +## About this Manual +This document provides general information about the implementation of Extended Unified Configuration and Management framework support for FRR-BGP feature in SONiC. +## Scope +This document describes the high level design of FRR-BGP Extended Unified Configuration and Management feature. + +## Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| FRR | Free Range Routing Stack | +| CVL | Config Validation Library | +| VRF | Virtual routing forwarding | +| RIB | Routing Information Base | +| PBR | Policy based routing | +| NBI | North Bound Interface | + + + +# 1 Feature Overview + +This feature extends and provides unified configuration and management capability for FRR-BGP features used in SONiC. This allows the user to configure & manage FRR-BGP using SONiC Management Framework with Open Config data models via REST, gNMI and also provides access via SONiC Management Framework CLI as well. + + +## 1.1 Requirements + + +### 1.1.1 Functional Requirements + + 1. Extend Unified mode for full FRR-BGP config and management in SONiC + 2. Extend sonic-cfggen, bgpcfgd and integrate with FRR-BGP for features supported in SONiC + 3. Support for retrieval of FRR-BGP state and statistics information + 4. For backward compatibility retain access to FRR UI (vtysh) for managing features that are NOT in conflict with SONiC features + +### 1.1.2 Configuration and Management Requirements + +1. Support Open Config data models for BGP config and Management +2. Provide IS-CLI/gNMI/REST support for config and management of FRR-BGP features used in SONIC +3. Enhance with Custom YANG models for features used in BGP that are not supported via Open Config data model +4. Define ABNF schema for BGP features used in SONiC + +### 1.1.3 Scalability Requirements +N/A + +### 1.1.4 Warm Boot Requirements +As state and statistics information is retrieved from FRR-BGP on demand there is no Warm Boot specific requirements for this feature. + +## 1.2 Design Overview +SONiC FRR-BGP Extended Unified config and management capability makes use of Management framework to implement the backend and transformer methods to support Open Config data models for BGP and route policy feature. The backend converts the incoming request to Redis ABNF schema format and writes the configuration to Redis DB. Then from DB events, bgpcfgd will configure FRR-BGP using FRR CLI commands. +It also uses management framework's transformer methods to do syntactic and semantic validation of the requests using ABNF JSON before writing them into the Redis DB. + +### 1.2.1 Basic Approach + +* This enhancement takes comprehensive approach to support BGP features used in SONiC: + * Standard based YANG models and custom YANG models + * Open API spec + * Industry standard CLI + * Config Validation +* REST server, gNMI server, Transformer methods - all in Go +* Marshalling and unmarshalling using YGOT +* Redis updated using CAS(Check-and-Set) trans. (No locking, No rollback) +* Config Validation by using YANG model from ABNF schema + +### 1.2.2 Container + +There will be changes in following containers, +* sonic-mgmt-framework +* sonic-frr + +### 1.2.3 SAI Overview +N/A - software feature + + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Configure and manage FRR-BGP via gNMI, REST and CLI interfaces using SONiC Management Framework. + +## 2.2 Functional Description +Provide GNMI and REST support for config get/set, state get and statistics get, CLI config and show commands for FRR-BGP features used in SONiC. + + +# 3 Design +## 3.1 Overview +The extended unified config and management framework for FRR-BGP in SONiC is represented in below diagram. + +![FRR-BGP Unified Mgmt Framework](images/FRR-BGP-Unified-mgmt-frmwrk.png) + +1. Transformer common app owns the Open config data models related to BGP (which means no separate app module required for handling BGP open-config and augmented YANG objects). + +2. Provide annotations for required objects so that transformer core and common app will take care of handling them. + +3. Provide transformer methods as per the annotations defined to take care of model specific logics and validations. + +4. Define SONiC YANG and Redis ABNF schema for the supported Open Config BGP models & objects. + +5. KLISH CLI and REST clients provide extensive BGP configurations and hence there should not be any need for BGP configurations via vtysh. + +6. In bgpcfgd register for Redis DB events for the BGP and other related objects, so as to translate the Redis DB events to FRR-BGP CLI commands to configure FRR-BGP, similarly, separate config daemons can be present to configure individual features like OSPF, BFD..etc + +7. Update /usr/share/sonic/templates/bgpd.conf.j2 template for new FRR-BGP configurations supported in SONiC which will be used by sonic-cfggen to generate /etc/frr/bgpd.conf file. + +## 3.2 DB Changes +Following section describes the changes to DB. + +### 3.2.1 CONFIG DB + +Added new tables to configure following information: + + * BGP router & address family configurations + * BGP neighbor & address family configurations + * BGP peer group & address family configurations + * BGP listen prefix configuration for peer group + * Route map configurations + * Route redistribute configurations + * Route policy sets (prefix list/community/ext-community/as-path/neighbor-set/tag-set configurations + +Enhanced following table to configure additional attributes: + + * BGP Neighbor table + +#### 3.2.1.1 BGP_GLOBALS +```JSON +;Defines BGP globals table +; +;Status: stable + +key = BGP_GLOBALS|vrf ; +vrf = 1\*15VCHAR ; VRF name +local_asn = 1*10DIGIT ; Local ASN for the BGP instance +router_id = \*IPv4prefix ; Router ID IPv4 address +load_balance_mp_relax = "true" / "false" ; +grace_restart = "true" / "false" ; +always_compare_med = "true" / "false" ; +load_balance_mp_relax = "true" / "false" ; +graceful_restart_enable = "true" / "false" ; +gr_restart_time = 1*4DIGIT ; {1..3600 }; +gr_stale_routes_time = 1*4DIGIT ; {1..3600 }; +ebgp_route_distance = 1*2DIGIT ; {1..255 }; +ibgp_route_distance = 1*2DIGIT ; {1..255 }; +external_compare_router_id = "true" / "false" ; +ignore_as_path_length = "true" / "false" ; +log_nbr_state_changes = "true" / "false" ; +rr_cluster_id = 1*64VCHAR ; Route reflector cluster id +rr_allow_out_policy = "true" / "false" ; Router reflector allow outbound Policy +disable_ebgp_connected_rt_check = "true" / "false" ; +fast_external_failover = "true" / "false" ; +network_import_check = "true" / "false" ; +graceful_shutdown = "true" / "false" ; +route_flap_dampen = "true" / "false" ; +route_flap_dampen_half_life = 1*2DIGIT; {1..45} +route_flap_dampen_reuse_threshold = 1*5DIGIT; {1..20000} +route_flap_dampen_suppress_threshold = 1*5DIGIT; {1..20000} +route_flap_dampen_max_suppress = 1*3DIGIT; {1..255} +rr_clnt_to_clnt_reflection = "true" / "false" ; +max_dynamic_neighbors = 1*4DIGIT; {1..5000} +read_quanta = 1*2DIGIT; {1..10} Indicates how many packets to read from peer socket per I/O cycle +write_quanta = 1*2DIGIT; {1..10} Indicates how many packets to write to peer socket per run +coalesce_time = 1*10DIGIT; Subgroup coalesce timer value in milli-sec +route_map_process_delay = 1*3DIGIT; { 0..600} +deterministic_med = "true" / "false" ; +med_confed = "true" / "false" ; Compare MED among confederation paths when set to true +med_missing_as_worst = "true" / "false" ; Treat missing MED as the least preferred one when set to true +compare_confed_as_path = "true" / "false" ; Compare path lengths including confederation sets & sequences in selecting a route +as_path_mp_as_set = "true" / "false" ; Generate an AS_SET +default_ipv4_unicast = "true" / "false" ; Activate ipv4-unicast for a peer by default +default_local_preference = "true" / "false" ; Configure default local preference value +default_show_hostname = "true" / "false" ; Show hostname in BGP dumps +default_shutdown = "true" / "false" ; Apply administrative shutdown to newly configured peers +default_subgroup_pkt_queue_max = 1*2DIGIT; {20..100} Configure subgroup packet queue max +max_med_time = 1*5DIGIT; {5..86400} Time (seconds) period for max-med +max_med_val = 1*10DIGIT; Max MED value to be used +max_delay = 1*4DIGIT; {0..3600} Maximum delay for best path calculation +establish_wait = 1*5DIGIT; {Maximum delay for updates} +``` +#### 3.2.1.2 BGP_GLOBALS_AF +```JSON +;Defines BGP Address family table +; +;Status: stable + +key = BGP_GLOBALS_AF|vrf|af_name ; +vrf = 1\*15VCHAR ; VRF name +af_name = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +max_ebgp_paths = 1*3DIGIT ; {1..256} +max_ibgp_paths = 1*3DIGIT ; {1..256} +aggregate_prefix = IPv4Prefix / IPv6prefix ; +aggregate_as_set = "true" / "false" ; +aggregate_summary_only = "true" / "false" ; +network_prefix = IPv4Prefix / IPv6prefix ; +network_policy = 1*64VCHAR ; +network_backdoor = "true" / "false" ; +route_download_filter = 1*64VCHAR ; +ebgp_route_distance = 1*3DIGIT; { 1.255} +ibgp_route_distance = 1*3DIGIT; { 1.255} +ibgp_equal_cluster_length = "true" / "false" ; + +``` + +#### 3.2.1.3 BGP_LISTEN_PREFIX +```JSON +;Defines BGP Listen Prefix table +; +;Status: stable + +key = BGP_GLOBALS_LISTEN_PREFIX|vrf|IPprefix ; +vrf = 1\*15VCHAR ; VRF name +IPprefix = IPv4Prefix / IPv6prefix ; +peer_group_name = 1*64VCHAR ; Peer group this listen prefix is associated with +``` + +#### 3.2.1.4 BGP_NEIGHBOR + +```JSON +;Defines BGP neighbor table +; +;Status: stable + +key = BGP_NEIGHBOR|vrf|neighbor_ip ; +vrf = 1\*15VCHAR ; VRF name +neighbor_ip = IPv4Prefix / IPv6prefix ; +local_asn = 1*10DIGIT ; Local ASN for the BGP neighbor +name = 1*64VCHAR ; BGP neighbor description +asn = 1*10DIGIT; Remote ASN value +ebgp_multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +ebgp_multihop_ttl = 1*3DIGIT ; {1..255} EBGP multihop count +auth_password = STRING ; Set a password +enabled = "true" / "false" ; Neighbor admin status +keepalive_intvl = 1*4DIGIT ; {1..3600} keepalive interval +hold_time = 1*4DIGIT ; {1..3600} hold time +local_address = IPprefix ; local IP address +peer_group_name = 1*64VCHAR ; peer group name +peer_type = "internal" / "external" Internal/External BGP peer +conn_retry = 1*5DIGIT ; {1..65535} Connection retry timer +min_adv_interval = 1*3DIGIT ; {1..600} Minimum interval between sending BGP routing updates +passive_mode = "true" / "false" ; Don't send open messages +capability_ext_nexthop = "true" / "false" ; Advertise extended next-hop capability +disable_ebgp_connected_route_check = "true" / "false" ; one-hop away EBGP peer using loopback address +enforce_first_as = "true" / "false" ; Enforce the first AS for EBGP route +solo_peer = "true" / "false" ; Solo peer - part of its own update group +ttl_security_hops = 1*3DIGIT ; {1.254} BGP ttl-security parameters +bfd = "true" / "false" ; Enable BFD support +capability-dynamic = "true" / "false" ; Advertise dynamic capability +dont-negotiate-capability = "true" / "false" ; Do not perform capability negotiation +enforce-multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +override-capability = "true" / "false" ; Override capability negotiation result +peer-port = 1*5DIGIT ; {0..65535} Neighbor's BGP port +shutdown-message = "true" / "false" ; Add a shutdown message +strict-capability-match = "true" / "false" ; Strict capability negotiation match + +``` +#### 3.2.1.5 BGP_NEIGHBOR_AF +```JSON +;Defines BGP Neighbor table at an address family level +; +;Status: stable + +key = BGP_NEIGHBOR_AF|vrf|neighbor_ip|af_name ; +vrf = 1\*15VCHAR ; VRF name +neighbor_ip = IPv4Prefix / IPv6prefix ; +af_name = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +enabled = "true" / "false" ; Neighbor admin status +send_default_route = "true" / "false" ; +default_rmap = 1*64VCHAR ; Filter sending default routes bsaed on this route map. +max_prefix_limit = 1*10DIGIT; Maximum number of prefixes to accept from this peer +max_prefix_warning_only = "true" / "false" ; Only give warning message when limit is exceeded +max_prefix_warning_threshold = 1*3DIGIT; Threshold value (%) at which to generate a warning msg +max_prefix_restart_interval = 1*5DIGIT; Restart bgp connection after limit is exceeded +route_map_in = 1*64VCHAR ; Apply route map on incoming routes from neighbor +route_map_out = 1*64VCHAR ; Apply route map on outgoing routes to neighbor +soft_reconfiguration_in = "true" / "false" ; Per neighbor soft reconfiguration +unsuppress_map_name = 1*64VCHAR ; Route-map to selectively un-suppress suppressed routes +route_reflector_client = "true" / "false" ; Configure a neighbor as Route Reflector client +weight = 1*5DIGIT ; {0..65535} Set default weight for routes from this neighbor +as_override = "true" / "false" ; Override ASNs in outbound updates if aspath equals remote-as +send_community = "standard" / "extended" / "both" / "none" ; Send Community attribute to this neighbor +add_path_tx_all = "true" / "false" ; +add_path_tx_bestpath = "true" / "false" ; +unchanged_as_path = "true" / "false" ; +unchanged_med = "true" / "false" ; +unchanged_nexthop = "true" / "false" ; +filter_list_name = 1*64VCHAR ; +filter_list_direction = "inbound" / "outbound"; +nexthop_self_enabled = "true" / "false" ; +nexthop_self_force = "true" / "false" ; +prefix_list_name = 1*64VCHAR ; +prefix_list_direction = "inbound" / "outbound"; +remove_private_as_enabled = "true" / "false" ; +replace_private_as = "true" / "false" ; +remove_private_as_all = "true" / "false" ; +allow_as_count = 1*3DIGIT ; Number of occurences of AS number +allow_asin = "true" / "false" ; Accept as-path with my AS present in it +allow_as_origin = "true" / "false" ; Only accept my AS in the as-path if the route was originated in my AS +capability_orf_send = "true" / "false" ; Capability to receive the outbound route filtering from this neighbor +capability_orf_receive = "true" / "false" ; Capability to send the outbound route filtering to this neighbor +capability_orf_both = "true" / "false" ; Capability to send and receive the outbound route filtering to/from this neighbor +route-server-client = "true" / "false" ; Configure a neighbor as Route Server client +``` + +#### 3.2.1.6 BGP_PEER_GROUP +The existing BGP_PEER_RANGE (peer group) table does not have vrf-name as the key (but added as the field in the table as per VRF HLD) but FRR and open config models have the peer-group under VRF context, so, the new table BGP_PEER_GROUP has been introduced for configurations from management framework (This is not a backward compatible change, expecting the user to migrate to this table in the near future). + +```JSON +;Defines BGP peer group table +; +;Status: stable + +key = BGP_PEER_GROUP|vrf|pgrp_name ; +vrf = 1\*15VCHAR ; VRF name +pgrp_name = 1*64VCHAR ; alias name for the peer group , must be unique +local_asn = 1*10DIGIT ; Local ASN for the BGP neighbor +name = 1*64VCHAR ; BGP neighbor description +asn = 1*10DIGIT; Remote ASN value +ebgp_multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +ebgp_multihop_ttl = 1*3DIGIT ; {1..255} EBGP multihop count +auth_password = STRING ; Set a password +enabled = "true" / "false" ; Neighbor admin status +keepalive_intvl = 1*4DIGIT ; {1..3600} keepalive interval +hold_time = 1*4DIGIT ; {1..3600} hold time +local_address = IPprefix ; local IP address +peer_group_name = 1*64VCHAR ; peer group name +peer_type = "internal" / "external" Internal/External BGP peer +conn_retry = 1*5DIGIT ; {1..65535} Connection retry timer +min_adv_interval = 1*3DIGIT ; {1..600} Minimum interval between sending BGP routing updates +passive_mode = "true" / "false" ; Don't send open messages +capability_ext_nexthop = "true" / "false" ; Advertise extended next-hop capability +disable_ebgp_connected_route_check = "true" / "false" ; one-hop away EBGP peer using loopback address +enforce_first_as = "true" / "false" ; Enforce the first AS for EBGP route +solo_peer = "true" / "false" ; Solo peer - part of its own update group +ttl_security_hops = 1*3DIGIT ; {1.254} BGP ttl-security parameters +bfd = "true" / "false" ; Enable BFD support +capability-dynamic = "true" / "false" ; Advertise dynamic capability +dont-negotiate-capability = "true" / "false" ; Do not perform capability negotiation +enforce-multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +override-capability = "true" / "false" ; Override capability negotiation result +peer-port = 1*5DIGIT ; {0..65535} Neighbor's BGP port +shutdown-message = "true" / "false" ; Add a shutdown message +strict-capability-match = "true" / "false" ; Strict capability negotiation match +``` +#### 3.2.1.7 BGP_PEER_GROUP_AF + +```JSON +;Defines BGP per address family peer group table +; +;Status: stable + +key = BGP_PEER_GROUP_AF|vrf|pgrp_name|af_name" ; +vrf = 1\*15VCHAR ; VRF name +af_name = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +pgrp_name = 1*64VCHAR ; alias name for the peer group template, must be unique +af_name = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +enabled = "true" / "false" ; Neighbor admin status +send_default_route = "true" / "false" ; +default_rmap = 1*64VCHAR ; Filter sending default routes bsaed on this route map. +max_prefix_limit = 1*10DIGIT; Maximum number of prefixes to accept from this peer +max_prefix_warning_only = "true" / "false" ; Only give warning message when limit is exceeded +max_prefix_warning_threshold = 1*3DIGIT; Threshold value (%) at which to generate a warning msg +max_prefix_restart_interval = 1*5DIGIT; Restart bgp connection after limit is exceeded +route_map_in = 1*64VCHAR ; Apply route map on incoming routes from neighbor +route_map_out = 1*64VCHAR ; Apply route map on outgoing routes to neighbor +soft_reconfiguration_in = "true" / "false" ; Per neighbor soft reconfiguration +unsuppress_map_name = 1*64VCHAR ; Route-map to selectively un-suppress suppressed routes +route_reflector_client = "true" / "false" ; Configure a neighbor as Route Reflector client +weight = 1*5DIGIT ; {0..65535} Set default weight for routes from this neighbor +as_override = "true" / "false" ; Override ASNs in outbound updates if aspath equals remote-as +send_community = "standard" / "extended" / "both" / "none" ; Send Community attribute to this neighbor +add_path_tx_all = "true" / "false" ; +add_path_tx_bestpath = "true" / "false" ; +unchanged_as_path = "true" / "false" ; +unchanged_med = "true" / "false" ; +unchanged_nexthop = "true" / "false" ; +filter_list_name = 1*64VCHAR ; +filter_list_direction = "inbound" / "outbound"; +nexthop_self_enabled = "true" / "false" ; +nexthop_self_force = "true" / "false" ; +prefix_list_name = 1*64VCHAR ; +prefix_list_direction = "inbound" / "outbound"; +remove_private_as_enabled = "true" / "false" ; +replace_private_as = "true" / "false" ; +remove_private_as_all = "true" / "false" ; +allow_as_count = 1*3DIGIT ; Number of occurences of AS number +allow_asin = "true" / "false" ; Accept as-path with my AS present in it +allow_as_origin = "true" / "false" ; Only accept my AS in the as-path if the route was originated in my AS +capability_orf_send = "true" / "false" ; Capability to receive the outbound route filtering from this neighbor +capability_orf_receive = "true" / "false" ; Capability to send the outbound route filtering to this neighbor +capability_orf_both = "true" / "false" ; Capability to send and receive the outbound route filtering to/from this neighbor +route-server-client = "true" / "false" ; Configure a neighbor as Route Server client +``` +#### 3.2.1.8 ROUTE_MAP +```JSON +;Defines route map table +; +;Status: stable + +key = ROUTE_MAP|route_map_name|stmt_name ; +route_map_name = 1*64VCHAR ; route map name +stmt_name = 1*64VCHAR ; statment name +route_operation = "ACCEPT" / "REJECT" +match_interface = 1*64VCHAR ; Match interface name +match_prefix_set = 1*64VCHAR ; Match prefix sets +match_med = 1*10DIGIT ; Match metric of route +match_origin = 1*64VCHAR ; Match BGP origin code +match_local_pref = 1*64VCHAR ; Match local-preference of route +match_community = 1*64VCHAR ; Match BGP community list +match_ext_community = 1*64VCHAR ; Match BGP/VPN extended community list +match_as_path = 1*64VCHAR ; Match BGP AS path list +call_route_map = 1*64VCHAR ; Jump to another Route-Map after match+set + +set_origin = 1*64VCHAR ; Set BGP origin code +set_local_pref = 1*64VCHAR ; Set BGP local preference path attribute +set_next_hop = 1*64VCHAR ; Set IP address of next hop +set_med = 1*64VCHAR ; Set Metric value for destination routing protocol +set_repeat_asn = 1*3DIGIT ; NO of times the set_asn number to be repeated +set_asn = 1*10DIGIT ; Set ASN number +set_community_inline = 1*64VCHAR ; Set BGP community attribute inline +set_community_ref = 1*64VCHAR ; Refer BGP community attribute from community table +set_ext_community_inline = 1*64VCHAR ; Set BGP extended community attribute inline +set_ext_community_ref = 1*64VCHAR ; Refer BGP extended community attribute from extended community table +``` +#### 3.2.1.8 ROUTE_REDISTRIBUTE +```JSON +;Defines route redistribution table +; +;Status: stable + +key = ROUTE_REDISTRIBUTE|vrf|src_protocol|dst_protocol|addr_family ; +vrf = 1\*15VCHAR ; VRF name +src_protocol = "connected" / "static" / "ospf" / "ospf3" +dst_protocol = "bgp" +addr_family = "ipv4" / "ipv6" +route_map = 1*64VCHAR ; route map filter to apply for redistribution +``` + +### 3.2.1.9 IP_PREFIX_SET +```JSON +;Defines prefix set table +; +;Status: stable + +key = IP_PREFIX_SET:name ; prefix_set_name must be unique +name = 1*255VCHAR ; community set name +mode = "IPv4"/"IPv6" ; mode of prefix set. + +```` +#### 3.2.1.9.1 IP_PREFIX +```JSON +;Defines prefix table +; +;Status: stable +key = IP_PREFIX:set_name:ip_prefix:masklength_range; an instance of this key will be repeated for each prefix + ; an instance of this key/value pair will be repeated for each prefix +set_name = 1*255VCHAR ; community set name +ip_prefix = IPv4prefix / IPv6prefix ; prefix, example 1.1.1.1/32 +masklength_range = 1*255VCHAR ; exact or (masklength_range..low-masklength_range_high). example 8..16 or exact +``` +### 3.2.1.10 BGP_COMMUNITY_SET +```JSON +;Defines community table +; +;Status: stable +key = BGP_COMMUNITY_SET|name ; name must be unique +set_type = "STANDARD"/"EXPANDED" +match_action = "ANY/ALL" +community_member = string list ; community member list + ; Acceptable List of communities as ("AA:NN","local-AS", "no-advertise", "no-export" | regex) + +```` +### 3.2.1.11 BGP_EXT_COMMUNITY_SET +```JSON +;Defines extended community table +; +;Status: stable +key = BGP_EXT_COMMUNITY_SET|name ; name must be unique +set_type = "STANDARD"/"EXPANDED" +match_action = "ANY/ALL" +community_member = string list; community member list + ; Acceptable List of communities as ("route-target/route-origin:AA:NN" or "IP_Address" or regex) +```` +### 3.2.1.12 BGP_AS_PATH_SET +```JSON +;Defines extended community table +; +;Status: stable +key = AS_PATH_SET|name ; name must be unique + +as_path_member = string list; AS path list + ;Acceptable List of as paths "string, string" +```` + +### 3.2.2 APP DB +N/A + +### 3.2.3 STATE DB +No changes to State DB, State and statistics information will be retrieved directly from FRR-BGP. + +### 3.2.4 ASIC DB +N/A + +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent +No changes to Orch agent. + +### 3.3.2 Other Process + +#### 3.3.2.1 FRR Template Changes + +FRR template must be enhanced to contain FRR-BGP related configuration that are supported via FRR-BGP extended unified (Config DB is propagated to FRR config at startup) with non-integrated mode (FRR configuration is saved in individual files: “bgpd.conf”, “zebra.conf” and ospfd.conf....etc) config management framework. + +On startup, sonic-cfggen will use "/usr/share/sonic/templates/bgpd.conf.j2" to generate "/etc/frr/bgpd.conf". + +## 3.4 SyncD +No changes to SyncD + +## 3.5 SAI +No changes to SAI APIs. + +## 3.6 User Interface +### 3.6.1 Data Models +List of Open-config YANG models required for FRR-BGP Unified Configuration and Management are, + + 1) openconfig-network-instance.yang + + 2) openconfig-routing-policy.yang + +BGP and "routing policy" related augmented and not-supported fields are available in openconfig-bgp-ext.yang and openconfig-routing-policy-ext.yang files respectively. +Supported YANG containers: +``` +module: openconfig-network-instance + +--rw network-instances + +--rw network-instance* [name] + +--rw table-connections + | ... + +--rw protocols + +--rw protocol* [identifier name] + +--rw bgp + +--rw global + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--rw neighbors + | +--rw neighbor* [neighbor-address] + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--rw peer-groups + | +---rw peer-group* [peer-group-name] + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--ro rib + +--ro afi-safis + ... + +module: openconfig-routing-policy + +--rw routing-policy + +--rw defined-sets + | ... + +--rw policy-definitions + +--rw policy-definition* [name] + ... + +``` +### 3.6.2 CLI + 1. For all configuration commands, the CLI request is converted to a corresponding REST client SDK request based on the Open Config data model that was generated by the Swagger generator, and is given to the REST server. + + 2. From there on it will follow the same path as a REST config request for create, update and delete operations. + + 3. The Swagger generated REST server handles all the REST requests from the client SDK and invokes a common handler for all the create, update, replace, delete and get operations along with path and payload. This common handler converts all the requests into Transformer arguments and invokes the corresponding Transformer APIs. + + 4. For show commands, the CLI request is converted to a corresponding REST client SDK get request based on Open Config data model's config or state object on a case by case basis. + + 5. For show commands that requires retrieval of the data that doesn't contain any state information (information only based on the configuration), the backend callback will fetch the data from CONFIG_DB. + + 6. For show commands that requires retrieval of state or statistics information the backend, managemnt framework executes the FRR CLI using "docker exec bgp vtysh -c " to fetch the data in JSON format from FRR-BGP. + + 7. At transformer the JSON output (retrived from FRR BGP) is converted back to corresponding open config objects and returned to the caller. + + 8. For CLI show, the output returned in object format is then translated back to CLI Jinga template for output display in CLI. + + +#### 3.6.2.1 Configuration Commands + +##### 3.6.2.1.1 BGP Router mode commands + +|Command Description |CLI Command | +|:-----------------|:---------------| +|Enable BGP routing instance |sonic(config)# router bgp \ [vrf \] | +|Override configured BGP router-id |sonic(config-router-bgp)# router-id \ | +|Configure default best path selection |sonic(config-router-bgp)# bestpath {as-path { confed \| ignore \| multipath-relax \[as-set] \| med { confed \| missing-as-worst } } | +|Configure graceful restart capability params |sonic(config-router-bgp)# graceful-restart preserve-fw-state
sonic(config-router-bgp)# graceful-restart restart-time <1-3600>
sonic(config-router-bgp)# graceful-restart stalepath-time <1-3600>| +|Configure BGP IPv4/IPv6 neighbor |sonic(config-router-bgp)# neighbor { \ \| \ } | +|Configure BGP peer group |sonic(config-router-bgp)# peer-group \| +|Enter address family command mode|sonic(config-router-bgp)# address-family { ipv4 unicast \| ipv6 unicast \| l2vpn evpn} | +| Subgroup coalesce timer | sonic(config-router-bgp)# coalesce-time \ | +| How many packets to read from peer socket per I/O cycle | sonic(config-router-bgp)# read-quanta \ | +| How many packets to write to peer socket per run | sonic(config-router-bgp)# write-quanta \ | +| Configure client to client route reflection | sonic(config-router-bgp)# client-to-client reflection | +| Configure Route-Reflector Cluster-id | sonic(config-router-bgp)# cluster-id { <32-bit-val> \| } | +| Log neighbor up/down and reset reason(default) | sonic(config-router-bgp)# log-neighbor-changes | +| Pick the best-MED path among paths advertised from the neighboring AS | sonic(config-router-bgp)# deterministic-med | +| Enable route-flap dampening | sonic(config-router-bgp)# dampening \ \ \ | +| Disable checking if nexthop is connected on ebgp sessions | sonic(config-router-bgp)# disable-ebgp-connected-route-check | +| Graceful shutdown parameters | sonic(config-router-bgp)# graceful-shutdown | +| Configure BGP defaults | sonic(config-router-bgp)# bgp listen \{ limit \ \| range \ } | +| Advertise routes with max-med | sonic(config-router-bgp)# max-med on-startup [\