Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove MBG (partially) from code #168

Merged
merged 32 commits into from
Nov 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
46adcca
Remove MBG from code...
michalmalka Nov 6, 2023
c059941
Remove MBG from code...
michalmalka Nov 6, 2023
cfd51de
Update events.go
michalmalka Nov 6, 2023
3f32021
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 6, 2023
48931f5
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 6, 2023
136468a
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 6, 2023
02879bd
Update events.go
michalmalka Nov 7, 2023
5f84592
Merge branch 'clusterlink-net:main' into remove_mbg
michalmalka Nov 7, 2023
6f2e6af
Merge branch 'clusterlink-net:main' into remove_mbg
michalmalka Nov 7, 2023
f3b0cd2
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 7, 2023
3c8ccdb
cmd/cl-adm: Add log-level flag (#167)
kfirtoledo Nov 7, 2023
a25b2cd
Add `-v` / `--version` to executables (#171)
elevran Nov 8, 2023
1002150
bring back manifests
michalmalka Nov 8, 2023
78dae84
remove MBG
michalmalka Nov 8, 2023
d65c30d
bootstrap/platform/k8s: Switch CA on gwctl pod
orozery Nov 8, 2023
2bb50b0
tests/k8s.sh: Change logs output directory
orozery Nov 8, 2023
3180ef8
Remove MBG from code...
michalmalka Nov 6, 2023
287053a
Update events.go
michalmalka Nov 7, 2023
de2ec1a
build(deps): bump the cli group with 1 update (#165)
dependabot[bot] Nov 6, 2023
20ccbb7
build(deps): bump github.com/bombsimon/logrusr/v4 from 4.0.0 to 4.1.0…
dependabot[bot] Nov 6, 2023
e5043a4
bring back manifests
michalmalka Nov 8, 2023
27f68c2
remove MBG
michalmalka Nov 8, 2023
3b116d6
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 9, 2023
181ebe5
Remove MBG from code...
michalmalka Nov 6, 2023
9354b4b
Update events.go
michalmalka Nov 6, 2023
5e7b418
Remove MBG from code...
michalmalka Nov 6, 2023
cd6c854
Update events.go
michalmalka Nov 7, 2023
eac1d97
bring back manifests
michalmalka Nov 8, 2023
ff2fe6f
remove MBG
michalmalka Nov 8, 2023
3164cf5
Remove MBG from code...
michalmalka Nov 6, 2023
1739967
bring back manifests
michalmalka Nov 8, 2023
28e9565
Merge branch 'remove_mbg' of github.com:michalmalka/clusterlink into …
michalmalka Nov 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
SW_VERSION ?= latest
IMAGE_ORG ?= mcnet

IMAGE_TAG_BASE ?= quay.io/$(IMAGE_ORG)/mbg
IMAGE_TAG_BASE ?= quay.io/$(IMAGE_ORG)/clusterlink
IMG ?= $(IMAGE_TAG_BASE):$(SW_VERSION)
#-----------------------------------------------------------------------------
# Target: clean
Expand Down Expand Up @@ -67,7 +67,7 @@ GOFLAGS :=

build:
@echo "Start go build phase"
$(GO) build -o ./bin/gwctl ./cmd/gwctl/main.go
$(GO) build -o ./bin/gwctl ./cmd/gwctl
$(GO) build -o ./bin/cl-controlplane ./cmd/cl-controlplane
$(GO) build -o ./bin/cl-dataplane ./cmd/cl-dataplane
$(GO) build -o ./bin/cl-go-dataplane ./cmd/cl-go-dataplane
Expand All @@ -89,8 +89,8 @@ install:
cp ./bin/gwctl /usr/local/bin/

clean-tests:
kind delete cluster --name=mbg1
kind delete cluster --name=mbg2
kind delete cluster --name=peer1
kind delete cluster --name=peer2

#------------------------------------------------------
# Run Targets
Expand Down
5 changes: 0 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,6 @@ This is an incomplete work in progress, provided in the interest of sharing expe
The code is pre-alpha quality right now. This means that it shouldn't be used in
production at all.

Before settling on the `ClusterLink` name. the code base was previously developed as part
of a "Multi-Border Gateway" concept exploration. You may still find various places in
code and documentation referring to the old name (directly or using the `MBG` acronym).
We're in the process of changing all such references.

## What Is ClusterLink?

The ClusterLink project simplifies the connection between application services that are
Expand Down
7 changes: 6 additions & 1 deletion cmd/cl-adm/cmd/create/create_peer.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,17 @@ type PeerOptions struct {
Dataplanes uint16
// DataplaneType is the type of dataplane to create (envoy or go-based)
DataplaneType string
// LogLevel is the log level.
LogLevel string
}

// AddFlags adds flags to fs and binds them to options.
func (o *PeerOptions) AddFlags(fs *pflag.FlagSet) {
fs.StringVar(&o.Name, "name", "", "Peer name.")
fs.Uint16Var(&o.Dataplanes, "dataplanes", 1, "Number of dataplanes.")
fs.StringVar(&o.DataplaneType, "dataplane-type", "envoy", "Type of dataplane, Supported values: \"envoy\" (default), \"go\"")
fs.StringVar(&o.DataplaneType, "dataplane-type", platform.DataplaneTypeEnvoy, "Type of dataplane, Supported values: \"envoy\" (default), \"go\"")
fs.StringVar(&o.LogLevel, "log-level", "info",
"The log level. One of fatal, error, warn, info, debug.")
}

// RequiredFlags are the names of flags that must be explicitly specified.
Expand Down Expand Up @@ -185,6 +189,7 @@ func (o *PeerOptions) Run() error {
GWCTLCertificate: gwctlCert,
Dataplanes: o.Dataplanes,
DataplaneType: o.DataplaneType,
LogLevel: o.LogLevel,
})
if err != nil {
return err
Expand Down
2 changes: 2 additions & 0 deletions cmd/cl-controlplane/cl-controlplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ import (
"os"

"github.com/clusterlink-net/clusterlink/cmd/cl-controlplane/app"
"github.com/clusterlink-net/clusterlink/pkg/versioninfo"
)

func main() {
command := app.NewCLControlplaneCommand()
command.Version = versioninfo.Short()
if err := command.Execute(); err != nil {
os.Exit(1)
}
Expand Down
2 changes: 2 additions & 0 deletions cmd/cl-dataplane/cl-dataplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ import (
"os"

"github.com/clusterlink-net/clusterlink/cmd/cl-dataplane/app"
"github.com/clusterlink-net/clusterlink/pkg/versioninfo"
)

func main() {
command := app.NewCLDataplaneCommand()
command.Version = versioninfo.Short()
if err := command.Execute(); err != nil {
os.Exit(1)
}
Expand Down
2 changes: 2 additions & 0 deletions cmd/cl-go-dataplane/cl-go-dataplane.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ import (
"os"

"github.com/clusterlink-net/clusterlink/cmd/cl-go-dataplane/app"
"github.com/clusterlink-net/clusterlink/pkg/versioninfo"
)

func main() {
command := app.NewCLGoDataplaneCommand()
command.Version = versioninfo.Short()
if err := command.Execute(); err != nil {
os.Exit(1)
}
Expand Down
3 changes: 3 additions & 0 deletions cmd/gwctl/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"github.com/spf13/cobra"

"github.com/clusterlink-net/clusterlink/cmd/gwctl/subcommand"
"github.com/clusterlink-net/clusterlink/pkg/versioninfo"
)

func main() {
Expand All @@ -28,7 +29,9 @@ func main() {
Short: "gwctl is a CLI that sends a control message (REST API) to the gateway",
Long: `gwctl CLI is part of the multi-cloud network project,
that allow sending control messages (REST API) to publish, connect and update policies for services`,
Version: versioninfo.Short(),
}

// Add all commands
rootCmd.AddCommand(subcommand.InitCmd()) // init command of Gwctl
rootCmd.AddCommand(createCmd())
Expand Down
10 changes: 5 additions & 5 deletions docs/Policy.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@

## This Document lays out the design of policy engine of MBG
## This Document lays out the design of policy engine of ClusterLink


<img src="policy-arch.png" width="600">

The MBG's control plane and data plane have hookpoints to publish events to another component called event manager. A hookpoint is a point in code where *a new connection request has arrived* or *a service listing request arrives*. The event manager receives an event-type, and standard attributes which are associated with the event-type. The event manager then publishes the event to Policy Dispatcher which maintains a list of policy agents (ACL, LB, Limiter, Monitor, etc) which have subscribed to a policy with a certain priority in which they need to executed. For example, *a new connection request* event could be subscribed by ACL, LB and Monitor with a priority of 0, 1 and 2 which means ACL will be the first policy agent and Monitor the last. This also means that Policy Dispatcher maintains the states and handles the stitching of multiple policy agents for an event and handles the movement of shared states between the policy agents.
The ClusterLink's control plane and data plane have hookpoints to publish events to another component called event manager. A hookpoint is a point in code where *a new connection request has arrived* or *a service listing request arrives*. The event manager receives an event-type, and standard attributes which are associated with the event-type. The event manager then publishes the event to Policy Dispatcher which maintains a list of policy agents (ACL, LB, Limiter, Monitor, etc) which have subscribed to a policy with a certain priority in which they need to executed. For example, *a new connection request* event could be subscribed by ACL, LB and Monitor with a priority of 0, 1 and 2 which means ACL will be the first policy agent and Monitor the last. This also means that Policy Dispatcher maintains the states and handles the stitching of multiple policy agents for an event and handles the movement of shared states between the policy agents.

Policy engine is kept separate to allow extensibility. We could consider dynamically loadable policy agents in javascript web assembly (WASM). However, initially we will start with a set of well known policy agents manually written in golang. The Operator should be able to send commands to the policy dispatcher to program it (add policy, remove policy, subscribe policy event, etc)

### Events + Attributes + Policy Agents Interested + Responses

| Event |Attributes |Policy Agents |Response |
|----------------|-------------------------------|-----------------------------|----------|
|New connection|Src Service, Dst Service, Direction, *Optional Src MBG* |ACL, LB, Connection Limiter, Rate Limiter, Monitor | Allow/Deny, Target MBG, Optional Rate-limit |
|Expose Service | Service ID | ACL | Target MBGs|
|Service List Request | Src MBG |ACL|Services|
|New connection|Src Service, Dst Service, Direction, *Optional Src Peer* |ACL, LB, Connection Limiter, Rate Limiter, Monitor | Allow/Deny, Target Peer, Optional Rate-limit |
|Expose Service | Service ID | ACL | Target Peers|
|Service List Request | Src Peer |ACL|Services|

More such events should be added and their attributes clearly attributed, This is just a first step towards that.
3 changes: 3 additions & 0 deletions pkg/bootstrap/platform/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ type Config struct {
Dataplanes uint16
// DataplaneType is the type of dataplane to create (envoy or go-based)
DataplaneType string

// LogLevel is the log level.
LogLevel string
}

const (
Expand Down
7 changes: 4 additions & 3 deletions pkg/bootstrap/platform/k8s.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ spec:
containers:
- name: cl-controlplane
image: cl-controlplane
args: ["--log-level", "info", "--platform", "k8s"]
args: ["--log-level", "{{.logLevel}}", "--platform", "k8s"]
imagePullPolicy: IfNotPresent
ports:
- containerPort: {{.controlplanePort}}
Expand Down Expand Up @@ -157,7 +157,7 @@ spec:
containers:
- name: dataplane
{{ if (eq .dataplaneType .dataplaneTypeEnvoy) }}image: cl-dataplane{{ else }}image: cl-go-dataplane{{ end }}
args: ["--log-level", "info", "--controlplane-host", "cl-controlplane"]
args: ["--log-level", "{{.logLevel}}", "--controlplane-host", "cl-controlplane"]
imagePullPolicy: IfNotPresent
ports:
- containerPort: {{.dataplanePort}}
Expand Down Expand Up @@ -185,7 +185,7 @@ spec:
volumes:
- name: ca
secret:
secretName: cl-fabric
secretName: cl-peer
- name: tls
secret:
secretName: gwctl
Expand Down Expand Up @@ -273,6 +273,7 @@ func K8SConfig(config *Config) ([]byte, error) {
"peer": config.Peer,
"dataplanes": config.Dataplanes,
"dataplaneType": config.DataplaneType,
"logLevel": config.LogLevel,

"dataplaneTypeEnvoy": DataplaneTypeEnvoy,

Expand Down
4 changes: 2 additions & 2 deletions pkg/controlplane/authz.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ func (cp *Instance) AuthorizeEgress(req *EgressAuthorizationRequest) (*EgressAut
return &EgressAuthorizationResponse{Allowed: false}, nil
}

target := authResp.TargetMbg
target := authResp.TargetPeer
peer := cp.GetPeer(target)
if peer == nil {
return nil, fmt.Errorf("peer '%s' does not exist", target)
Expand Down Expand Up @@ -140,7 +140,7 @@ func (cp *Instance) AuthorizeIngress(req *IngressAuthorizationRequest, peer stri

resp.ServiceExists = true

connReq := eventmanager.ConnectionRequestAttr{DstService: req.Service, Direction: eventmanager.Incoming, OtherMbg: peer}
connReq := eventmanager.ConnectionRequestAttr{DstService: req.Service, Direction: eventmanager.Incoming, OtherPeer: peer}
authResp, err := cp.policyDecider.AuthorizeAndRouteConnection(&connReq)
if err != nil {
return nil, err
Expand Down
24 changes: 12 additions & 12 deletions pkg/controlplane/eventmanager/events.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,13 @@ type ConnectionRequestAttr struct {
SrcService string
DstService string
Direction Direction
OtherMbg string //Optional: Would not be set if its an outgoing connection
OtherPeer string //Optional: Would not be set if its an outgoing connection
}

type ConnectionRequestResp struct {
Action Action
TargetMbg string
BitRate int // Mbps
Action Action
TargetPeer string
BitRate int // Mbps
}

type ConnectionStatusAttr struct {
Expand All @@ -88,12 +88,12 @@ type ConnectionStatusAttr struct {

type NewRemoteServiceAttr struct {
Service string
Mbg string
Peer string
}

type RemoveRemoteServiceAttr struct {
Service string
Mbg string
Peer string
}

type NewRemoteServiceResp struct {
Expand All @@ -105,24 +105,24 @@ type ExposeRequestAttr struct {
}

type ExposeRequestResp struct {
Action Action
TargetMbgs []string
Action Action
TargetPeers []string
}

type AddPeerAttr struct {
PeerMbg string
Peer string
}

type AddPeerResp struct {
Action Action
}

type RemovePeerAttr struct {
PeerMbg string
Peer string
}

type ServiceListRequestAttr struct {
SrcMbg string
SrcPeer string
}

type ServiceListRequestResp struct {
Expand All @@ -131,7 +131,7 @@ type ServiceListRequestResp struct {
}

type ServiceRequestAttr struct {
SrcMbg string
SrcPeer string
}

type ServiceRequestResp struct {
Expand Down
6 changes: 3 additions & 3 deletions pkg/k8s/kubernetes/kubeinformer.go
Original file line number Diff line number Diff line change
Expand Up @@ -391,10 +391,10 @@ func (k *kubeData) initInformers(client kubernetes.Interface) error {
}

// CreateService Add support to create a service/NodePort for a target port
func (k *kubeData) CreateService(serviceName string, port int, targetPort int, namespace, mbgAppName string) error {
func (k *kubeData) CreateService(serviceName string, port int, targetPort int, namespace, svcAppName string) error {
var selectorMap map[string]string
if mbgAppName != "" {
selectorMap = map[string]string{"app": mbgAppName}
if svcAppName != "" {
selectorMap = map[string]string{"app": svcAppName}
}
serviceSpec := &v1.Service{
ObjectMeta: metav1.ObjectMeta{Name: serviceName},
Expand Down
4 changes: 2 additions & 2 deletions pkg/policyengine/PolicyDispatcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ func (pH *PolicyHandler) filterOutDisabledPeers(peers []string) []string {
}

func (pH *PolicyHandler) decideIncomingConnection(requestAttr *event.ConnectionRequestAttr) (event.ConnectionRequestResp, error) {
src := getServiceAttrs(requestAttr.SrcService, requestAttr.OtherMbg)
src := getServiceAttrs(requestAttr.SrcService, requestAttr.OtherPeer)
dest := getServiceAttrs(requestAttr.DstService, "")
decisions, err := pH.connectivityPDP.Decide(src, []policytypes.WorkloadAttrs{dest})
if err != nil {
Expand Down Expand Up @@ -155,7 +155,7 @@ func (pH *PolicyHandler) decideOutgoingConnection(requestAttr *event.ConnectionR
if err != nil {
return event.ConnectionRequestResp{Action: event.Deny}, err
}
return event.ConnectionRequestResp{Action: event.Allow, TargetMbg: targetPeer}, nil
return event.ConnectionRequestResp{Action: event.Allow, TargetPeer: targetPeer}, nil
}

func (pH *PolicyHandler) AuthorizeAndRouteConnection(connReq *event.ConnectionRequestAttr) (event.ConnectionRequestResp, error) {
Expand Down
18 changes: 9 additions & 9 deletions pkg/policyengine/PolicyDispatcher_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ func TestOutgoingConnectionRequests(t *testing.T) {
requestAttr := event.ConnectionRequestAttr{SrcService: svcName, DstService: svcName, Direction: event.Outgoing}
connReqResp, err := ph.AuthorizeAndRouteConnection(&requestAttr)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer2, connReqResp.TargetMbg)
require.Equal(t, peer2, connReqResp.TargetPeer)
require.Nil(t, err)

// Src service does not match the spec of the single access policy
Expand Down Expand Up @@ -160,26 +160,26 @@ func TestLoadBalancer(t *testing.T) {
connReqResp, err := ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer1, connReqResp.TargetMbg) // LB policy requires this request to be served by peer1
require.Equal(t, peer1, connReqResp.TargetPeer) // LB policy requires this request to be served by peer1

err = ph.DeleteLBPolicy(&apiLBPolicy) // LB policy is deleted - the random default policy now takes effect
require.Nil(t, err)
connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Contains(t, []string{peer1, peer2}, connReqResp.TargetMbg)
require.Contains(t, []string{peer1, peer2}, connReqResp.TargetPeer)

ph.DeletePeer(peer1) // peer1 is deleted, so all requests should go to peer2
connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer2, connReqResp.TargetMbg)
require.Equal(t, peer2, connReqResp.TargetPeer)

ph.DeletePeer(peer1) // deleting peer1 again should make no change
connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer2, connReqResp.TargetMbg)
require.Equal(t, peer2, connReqResp.TargetPeer)

ph.DeletePeer(peer2) // deleting peer2 should result in an deny, as there are no available peers left
connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
Expand Down Expand Up @@ -216,29 +216,29 @@ func TestDisableEnablePeers(t *testing.T) {
connReqResp, err := ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer1, connReqResp.TargetMbg) // LB policy defaults this request to be served by peer1
require.Equal(t, peer1, connReqResp.TargetPeer) // LB policy defaults this request to be served by peer1

ph.DisablePeer(peer1)

connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer2, connReqResp.TargetMbg) // peer1 is now disabled, so peer2 must be used
require.Equal(t, peer2, connReqResp.TargetPeer) // peer1 is now disabled, so peer2 must be used

ph.DisablePeer(peer2)

connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Deny, connReqResp.Action) // no enabled peers - a Deny is returned
require.Equal(t, "", connReqResp.TargetMbg)
require.Equal(t, "", connReqResp.TargetPeer)

ph.EnablePeer(peer1)
ph.EnablePeer(peer2)

connReqResp, err = ph.AuthorizeAndRouteConnection(&requestAttr)
require.Nil(t, err)
require.Equal(t, event.Allow, connReqResp.Action)
require.Equal(t, peer1, connReqResp.TargetMbg) // peer1 was re-enabled, so it is now chosen again
require.Equal(t, peer1, connReqResp.TargetPeer) // peer1 was re-enabled, so it is now chosen again
}

func addRemoteSvc(t *testing.T, svc, peer string, ph policyengine.PolicyDecider) {
Expand Down
Loading