Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECMP feature support #500

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

ECMP feature support #500

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0fb5134
feat(evpn): support ecmp feature in vendor plugin
atulpatel261194 Oct 7, 2024
ea574f1
feat(evpn): remove prints
Inbanoth Oct 10, 2024
ffad720
fix: fix build issues
atulpatel261194 Feb 23, 2025
aafee56
fix: report error in comp details
atulpatel261194 Sep 30, 2024
4541eaa
docs: add redis and jaeger env vars
artek-koltun Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The following variables are used throughout this document:

| Variable | Description |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| BRIDGE_IP | opi-intel-bridge gRPC/HTTP listening IP address e.g. 10.10.10.10 or localhost |
| BRIDGE_IP | opi-intel-bridge gRPC/HTTP listening IP address e.g. 10.10.10.10 or localhost |
| BRIDGE_PORT | opi-intel-bridge gRPC listening port e.g. 50051 |
| BRIDGE_ADDR | BRIDGE_IP:BRIDGE_PORT |
| BRIDGE_HTTP_PORT | opi-intel-bridge http gateway port e.g. 8082 |
Expand All @@ -58,6 +58,8 @@ The following variables are used throughout this document:
| BLK_PF_BDF | physical function PCI address e.g. 0000:af:01.0 for virtio-blk |
| TARGET_IP | storage target ip address |
| TARGET_PORT | storage target port |
| JAEGER_PORT | Jaeger port |
| REDIS_PORT | Redis port |

### Build and import

Expand Down Expand Up @@ -86,7 +88,7 @@ Make sure `/var/tmp/spdk.sock` is created.
On xPU run

```bash
$ docker run --rm -it -v /var/tmp/:/var/tmp/ -p $BRIDGE_PORT:$BRIDGE_PORT -p $BRIDGE_HTTP_PORT:$BRIDGE_HTTP_PORT ghcr.io/opiproject/opi-intel-bridge:main /opi-intel-bridge-storage -grpc_port=$BRIDGE_PORT -http_port=$BRIDGE_HTTP_PORT
$ docker run --rm -it -v /var/tmp/:/var/tmp/ -p $BRIDGE_PORT:$BRIDGE_PORT -p $BRIDGE_HTTP_PORT:$BRIDGE_HTTP_PORT -p $JAEGER_PORT:$JAEGER_PORT -p $REDIS_PORT:$REDIS_PORT ghcr.io/opiproject/opi-intel-bridge:main /opi-intel-bridge-storage -grpc_port=$BRIDGE_PORT -http_port=$BRIDGE_HTTP_PORT

2023/09/12 20:29:05 TLS files are not specified. Use insecure connection.
2023/09/12 20:29:05 Connection to SPDK will be via: unix detected from /var/tmp/spdk.sock
Expand Down
31 changes: 15 additions & 16 deletions config-intel-e2000.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,32 +25,31 @@ grpc:
server_port: 51703
num_threads: 10
static_external_macs: []
interfaces:
phyports:
- rep: "enp0s1f0d1"
vsi: 0
- rep: "enp0s1f0d3"
vsi: 1
grpcacc: "enp0s1f0d2"
grpchost: "00:0d:00:03:09:64"
vrfmux: "enp0s1f0d4"
portmux: "enp0s1f0d5"
p4:
enabled: true
representors:
port_mux: "port-mux"
vrf_mux: "vrf-mux"
grpc_acc: "host"
grpc_host: "00:20:00:00:14:48"
phy0_rep: "port0"
phy1_rep: "port1"
config:
p4infofile: /root/p4files/opi_ln.p4info.txt
binfile: /root/p4files/opi_ln.pb.bin
p4infofile: /root/networking.ethernet.acceleration.mev.infra.joint/gw_integration/p4files/evpn_gw.p4info.txt
binfile: /root/networking.ethernet.acceleration.mev.infra.joint/gw_integration/p4files/evpn_gw.pb.bin
linuxfrr:
enabled: true
defaultvtep: "vxlan-vtep"
portmux: "enp0s1f0d5"
vrfmux: "enp0s1f0d4"
ipmtu: 2962
localas: 65011
netlink:
enabled: true
pollinterval: 1
phyports:
- name: "enp0s1f0d1"
vsi: 0
- name: "enp0s1f0d2"
vsi: 1
grddefaultroute: false
enableecmp: true
loglevel:
db: INFO
grpc: INFO
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ require (
github.com/onsi/ginkgo/v2 v2.14.0
github.com/opiproject/gospdk v0.0.0-20240415072512-98d71122a73b
github.com/opiproject/opi-api v0.0.0-20240415072823-bb755a5f6ecc
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20240902142753-8f51de196b54
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20250207120615-90ff64f06ea5
github.com/opiproject/opi-smbios-bridge v0.1.3-0.20240113044816-4401aa6a3d1a
github.com/opiproject/opi-spdk-bridge v0.1.2-0.20240417152307-a0f9ef0e5260
github.com/opiproject/opi-strongswan-bridge v0.1.2-0.20231211064623-e4ef0e4fa95f
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,6 +461,8 @@ github.com/opiproject/opi-api v0.0.0-20240415072823-bb755a5f6ecc h1:iBcdnHiFFCIK
github.com/opiproject/opi-api v0.0.0-20240415072823-bb755a5f6ecc/go.mod h1:92pv4ulvvPMuxCJ9ND3aYbmBfEMLx0VCjpkiR7ZTqPY=
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20240902142753-8f51de196b54 h1:dBiCOF+XA5thXXULV5/UYma6CS3Q1VqvEF4OKsCmbrA=
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20240902142753-8f51de196b54/go.mod h1:0SsCMInH7SAg4ML4aI/Ts8jt7iXyMmcqxBPUD3y4EgQ=
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20250207120615-90ff64f06ea5 h1:30KcLfInrypEYbCcufnZk1M2OFSxesWqachVTWSiLcw=
github.com/opiproject/opi-evpn-bridge v0.2.1-0.20250207120615-90ff64f06ea5/go.mod h1:0SsCMInH7SAg4ML4aI/Ts8jt7iXyMmcqxBPUD3y4EgQ=
github.com/opiproject/opi-smbios-bridge v0.1.3-0.20240113044816-4401aa6a3d1a h1:JHNZJxcoWvrcoxyPRXLKEciaEZ5Dfsd6RQYd9km9MSM=
github.com/opiproject/opi-smbios-bridge v0.1.3-0.20240113044816-4401aa6a3d1a/go.mod h1:QhKBKdPcS25fDW89Rcsw9N1c07hdcj8OXsr2c+PgeIg=
github.com/opiproject/opi-spdk-bridge v0.1.2-0.20240417152307-a0f9ef0e5260 h1:bUzUhKxcC8k6MaApN2XSeYB6frZ0uubPcZjPvLBRG1E=
Expand Down
103 changes: 57 additions & 46 deletions pkg/evpn/LinuxVendorModule/intele2000/intelE2000.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ func handlebp(objectData *eventbus.ObjectData) {
log.Printf("LVM : GetBP error: %s\n", err)
comp.Name = lvmComp
comp.CompStatus = common.ComponentStatusError
comp.Details = fmt.Sprintf("LVM : GetBP error: %s\n", err)
if comp.Timer == 0 {
comp.Timer = 2 * time.Second
} else {
Expand All @@ -94,6 +95,7 @@ func handlebp(objectData *eventbus.ObjectData) {
log.Printf("LVM: Mismatch in resoruce version %+v\n and bp resource version %+v\n", objectData.ResourceVersion, bp.ResourceVersion)
comp.Name = lvmComp
comp.CompStatus = common.ComponentStatusError
comp.Details = fmt.Sprintf("LVM: Mismatch in resoruce version %+v\n and bp resource version %+v\n", objectData.ResourceVersion, bp.ResourceVersion)
if comp.Timer == 0 {
comp.Timer = 2 * time.Second
} else {
Expand All @@ -113,10 +115,10 @@ func handlebp(objectData *eventbus.ObjectData) {
}
}
if bp.Status.BPOperStatus != infradb.BridgePortOperStatusToBeDeleted {
status := setUpBp(bp)
details, status := setUpBp(bp)
comp.Name = lvmComp
comp.Details = details
if status {
comp.Details = ""
comp.CompStatus = common.ComponentStatusSuccess
comp.Timer = 0
} else {
Expand All @@ -133,8 +135,9 @@ func handlebp(objectData *eventbus.ObjectData) {
log.Printf("error updaing bp status %s\n", err)
}
} else {
status := tearDownBp(bp)
details, status := tearDownBp(bp)
comp.Name = lvmComp
comp.Details = details
if status {
comp.CompStatus = common.ComponentStatusSuccess
comp.Timer = 0
Expand Down Expand Up @@ -162,7 +165,7 @@ func MactoVport(mac *net.HardwareAddr) int {
}

// setUpBp sets up a bridge port
func setUpBp(bp *infradb.BridgePort) bool {
func setUpBp(bp *infradb.BridgePort) (string, bool) {
MacAddress := fmt.Sprintf("%+v", *bp.Spec.MacAddress)
vportID := MactoVport(bp.Spec.MacAddress)
link := fmt.Sprintf("vport-%+v", vportID)
Expand All @@ -171,73 +174,73 @@ func setUpBp(bp *infradb.BridgePort) bool {
muxIntf, err := nlink.LinkByName(ctx, portMux)
if err != nil {
log.Printf("Failed to get link information for %s, error is %v\n", portMux, err)
return false
return fmt.Sprintf("Failed to get link information for %s, error is %v\n", portMux, err), false
}
vlanLink := &netlink.Vlan{LinkAttrs: netlink.LinkAttrs{Name: link, ParentIndex: muxIntf.Attrs().Index}, VlanId: vportID, VlanProtocol: netlink.VLAN_PROTOCOL_8021AD}
if err = nlink.LinkAdd(ctx, vlanLink); err != nil {
log.Printf("Failed to add VLAN sub-interface %s: %v\n", link, err)
return false
return fmt.Sprintf("Failed to add VLAN sub-interface %s: %v\n", link, err), false
}
log.Printf("LVM: Executed ip link add link %s name %s type vlan protocol 802.1ad id %s\n", portMux, link, vport)
brIntf, err := nlink.LinkByName(ctx, brTenant)
if err != nil {
log.Printf("Failed to get link information for %s: %v\n", brTenant, err)
return false
return fmt.Sprintf("Failed to get link information for %s: %v\n", brTenant, err), false
}
if err = nlink.LinkSetMaster(ctx, vlanLink, brIntf); err != nil {
log.Printf("Failed to set master for %s: %v\n", brIntf, err)
return false
return fmt.Sprintf("Failed to get link information for %s: %v\n", brTenant, err), false
}
if err = nlink.LinkSetUp(ctx, vlanLink); err != nil {
log.Printf("Failed to set up link for %v: %s\n", vlanLink, err)
return false
return fmt.Sprintf("Failed to set up link for %v: %s\n", vlanLink, err), false
}
if err = nlink.LinkSetMTU(ctx, vlanLink, ipMtu); err != nil {
log.Printf("Failed to set MTU for %v: %s\n", vlanLink, err)
return false
return fmt.Sprintf("Failed to set MTU for %v: %s\n", vlanLink, err), false
}
log.Printf("LVM: Executed ip link set %s master %s up mtu %d \n", link, brTenant, ipMtu)
for _, vlan := range bp.Spec.LogicalBridges {
BrObj, err := infradb.GetLB(vlan)
if err != nil {
log.Printf("LVM: unable to find key %s and error is %v", vlan, err)
return false
return fmt.Sprintf("LVM: unable to find key %s and error is %v", vlan, err), false
}
if BrObj.Spec.VlanID > math.MaxUint16 {
log.Printf("LVM : VlanID %v value passed in Logical Bridge create is greater than 16 bit value\n", BrObj.Spec.VlanID)
return false
return fmt.Sprintf("LVM : VlanID %v value passed in Logical Bridge create is greater than 16 bit value\n", BrObj.Spec.VlanID), false
}
//TODO: Update opi-api to change vlanid to uint16 in LogiclaBridge
vid := uint16(BrObj.Spec.VlanID)
if err = nlink.BridgeVlanAdd(ctx, vlanLink, vid, false, false, false, false); err != nil {
log.Printf("Failed to add VLAN %d to bridge interface %s: %v\n", vportID, link, err)
return false
return fmt.Sprintf("Failed to add VLAN %d to bridge interface %s: %v\n", vportID, link, err), false
}
log.Printf("LVM: Executed bridge vlan add dev %s vid %d \n", link, vid)
}
if err = nlink.BridgeFdbAdd(ctx, link, MacAddress); err != nil {
log.Printf("LVM: Error in executing command %s %s with error %s\n", "bridge fdb add", link, err)
return false
return fmt.Sprintf("LVM: Error in executing command %s %s with error %s\n", "bridge fdb add", link, err), false
}
log.Printf("LVM: Executed bridge fdb add %s dev %s master static extern_learn\n", MacAddress, link)
return true
return "", true
}

// tearDownBp tears down the bridge port
func tearDownBp(bp *infradb.BridgePort) bool {
func tearDownBp(bp *infradb.BridgePort) (string, bool) {
vportID := MactoVport(bp.Spec.MacAddress)
link := fmt.Sprintf("vport-%+v", vportID)
Intf, err := nlink.LinkByName(ctx, link)
if err != nil {
log.Printf("Failed to get link %v: %s\n", link, err)
return true
return fmt.Sprintf("Failed to get link %v: %s\n", link, err), true
}
if err = nlink.LinkDel(ctx, Intf); err != nil {
log.Printf("Failed to delete link %v: %s\n", link, err)
return false
return fmt.Sprintf("Failed to delete link %v: %s\n", link, err), false
}
log.Printf(" LVM: Executed ip link delete %v\n", link)
return true
return "", true
}

// handlevrf handles the vrf functionality
Expand All @@ -250,6 +253,7 @@ func handlevrf(objectData *eventbus.ObjectData) {
log.Printf("LVM : GetVrf error: %s\n", err)
comp.Name = lvmComp
comp.CompStatus = common.ComponentStatusError
comp.Details = fmt.Sprintf("LVM : GetVrf error: %s\n", err)
if comp.Timer == 0 { // wait timer is 2 powerof natural numbers ex : 1,2,3...
comp.Timer = 2 * time.Second
} else {
Expand All @@ -265,6 +269,7 @@ func handlevrf(objectData *eventbus.ObjectData) {
log.Printf("LVM: Mismatch in resoruce version %+v\n and vrf resource version %+v\n", objectData.ResourceVersion, vrf.ResourceVersion)
comp.Name = lvmComp
comp.CompStatus = common.ComponentStatusError
comp.Details = fmt.Sprintf("LVM: Mismatch in resoruce version %+v\n and vrf resource version %+v\n", objectData.ResourceVersion, vrf.ResourceVersion)
if comp.Timer == 0 { // wait timer is 2 powerof natural numbers ex : 1,2,3...
comp.Timer = 2 * time.Second
} else {
Expand All @@ -284,10 +289,10 @@ func handlevrf(objectData *eventbus.ObjectData) {
}
}
if vrf.Status.VrfOperStatus != infradb.VrfOperStatusToBeDeleted {
statusUpdate := setUpVrf(vrf)
details, status := setUpVrf(vrf)
comp.Name = lvmComp
if statusUpdate {
comp.Details = ""
comp.Details = details
if status {
comp.CompStatus = common.ComponentStatusSuccess
comp.Timer = 0
} else {
Expand All @@ -303,8 +308,10 @@ func handlevrf(objectData *eventbus.ObjectData) {
log.Printf("error updaing vrf status %s\n", err)
}
} else {
details, statusUpdate := tearDownVrf(vrf)
comp.Name = lvmComp
if tearDownVrf(vrf) {
comp.Details = details
if statusUpdate {
comp.CompStatus = common.ComponentStatusSuccess
} else {
if comp.Timer == 0 {
Expand All @@ -325,81 +332,85 @@ func handlevrf(objectData *eventbus.ObjectData) {
func disableRpFilter(iface string) {
// Work-around for the observation that sometimes the sysctl -w command did not take effect.
rpFilterDisabled := false
for i := 0; i < 3; i++ {
for i := 0; i < 5; i++ {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's 5 here? create a named const?

rpDisable := fmt.Sprintf("net.ipv4.conf.%s.rp_filter=0", iface)
run([]string{"sysctl", "-w", rpDisable}, false)
time.Sleep(2 * time.Millisecond)
output, errCode := run([]string{"sysctl", "-w", rpDisable}, false)
if errCode != 0 {
log.Printf("Error setting rp_filter: %s\n", output)
continue
}
time.Sleep(200 * time.Millisecond)
rpDisable = fmt.Sprintf("net.ipv4.conf.%s.rp_filter", iface)
CP, err := run([]string{"sysctl", "-n", rpDisable}, false)
if err == 0 && strings.HasPrefix(CP, "0") {
output, errCode = run([]string{"sysctl", "-n", rpDisable}, false)
if errCode == 0 && strings.HasPrefix(output, "0") {
rpFilterDisabled = true
log.Printf("LVM: rpFilterDisabled: %+v\n", rpFilterDisabled)
log.Printf("RP filter disabled on interface %s\n", iface)
break
}
}
if !rpFilterDisabled {
log.Printf("Failed to disable rp_filtering on interface %s\n", iface)
log.Printf("Failed to disable rp_filter on interface %s\n", iface)
}
}

// setUpVrf sets up a vrf
func setUpVrf(vrf *infradb.Vrf) bool {
func setUpVrf(vrf *infradb.Vrf) (string, bool) {
log.Printf("LVM configure linux function \n")
vlanIntf := fmt.Sprintf("rep-%+v", path.Base(vrf.Name))
if path.Base(vrf.Name) == "GRD" {
disableRpFilter("rep-" + path.Base(vrf.Name))
return true
return "", true
}
muxIntf, err := nlink.LinkByName(ctx, vrfMux)
if err != nil {
log.Printf("Failed to get link information for %s, error is %v\n", vrfMux, err)
return false
return fmt.Sprintf("Failed to get link information for %s, error is %v\n", vrfMux, err), false
}
vlanLink := &netlink.Vlan{LinkAttrs: netlink.LinkAttrs{Name: vlanIntf, ParentIndex: muxIntf.Attrs().Index}, VlanId: int(*vrf.Metadata.RoutingTable[0])}
if err = nlink.LinkAdd(ctx, vlanLink); err != nil {
log.Printf("Failed to add VLAN sub-interface %s: %v\n", vlanIntf, err)
return false
return fmt.Sprintf("Failed to add VLAN sub-interface %s: %v\n", vlanIntf, err), false
}
log.Printf(" LVM: Executed ip link add link %s name rep-%s type vlan id %s\n", vrfMux, path.Base(vrf.Name), strconv.Itoa(int(*vrf.Metadata.RoutingTable[0])))
vrfIntf, err := nlink.LinkByName(ctx, path.Base(vrf.Name))
if err != nil {
log.Printf("Failed to get link information for %s: %v\n", path.Base(vrf.Name), err)
return false
return fmt.Sprintf("Failed to get link information for %s: %v\n", path.Base(vrf.Name), err), false
}
if err = nlink.LinkSetMaster(ctx, vlanLink, vrfIntf); err != nil {
log.Printf("Failed to set master for %v: %s\n", vlanIntf, err)
return false
return fmt.Sprintf("Failed to set master for %v: %s\n", vlanIntf, err), false
}
if err = nlink.LinkSetUp(ctx, vlanLink); err != nil {
log.Printf("Failed to set up link for %v: %s\n", vlanLink, err)
return false
return fmt.Sprintf("Failed to set up link for %v: %s\n", vlanLink, err), false
}
if err = nlink.LinkSetMTU(ctx, vlanLink, ipMtu); err != nil {
log.Printf("Failed to set MTU for %v: %s\n", vlanLink, err)
return false
return fmt.Sprintf("Failed to set MTU for %v: %s\n", vlanLink, err), false
}
log.Printf(" LVM: Executed ip link set rep-%s master %s up mtu %d\n", path.Base(vrf.Name), path.Base(vrf.Name), ipMtu)
disableRpFilter("rep-" + path.Base(vrf.Name))
return true
return "", true
}

// tearDownVrf tears down a vrf
func tearDownVrf(vrf *infradb.Vrf) bool {
func tearDownVrf(vrf *infradb.Vrf) (string, bool) {
vlanIntf := fmt.Sprintf("rep-%+v", path.Base(vrf.Name))
if path.Base(vrf.Name) == "GRD" {
return true
return "", true
}
Intf, err := nlink.LinkByName(ctx, vlanIntf)
if err != nil {
log.Printf("Failed to get link %v: %s\n", vlanIntf, err)
return false
return fmt.Sprintf("Failed to get link %v: %s\n", vlanIntf, err), false
}
if err = nlink.LinkDel(ctx, Intf); err != nil {
log.Printf("Failed to delete link %v: %s\n", vlanIntf, err)
return false
return fmt.Sprintf("Failed to delete link %v: %s\n", vlanIntf, err), false
}
log.Printf(" LVM: Executed ip link delete rep-%s\n", path.Base(vrf.Name))
return true
return "", true
}

var ipMtu int
Expand All @@ -417,8 +428,8 @@ func Initialize() {
}
}
}
portMux = config.GlobalConfig.LinuxFrr.PortMux
vrfMux = config.GlobalConfig.LinuxFrr.VrfMux
portMux = config.GlobalConfig.Interfaces.PortMux
vrfMux = config.GlobalConfig.Interfaces.VrfMux
ipMtu = config.GlobalConfig.LinuxFrr.IPMtu
brTenant = "br-tenant"
ctx = context.Background()
Expand Down
Loading
Loading