From 5e56a669b03509949111a5b7f0bafbe7142cea8a Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Thu, 16 May 2024 05:57:00 -0400 Subject: [PATCH 01/53] Update operator.md (#588) Signed-off-by: welisheva22 --- website/content/en/docs/main/tasks/operator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/en/docs/main/tasks/operator.md b/website/content/en/docs/main/tasks/operator.md index 77a085ea4..3ae53e41e 100644 --- a/website/content/en/docs/main/tasks/operator.md +++ b/website/content/en/docs/main/tasks/operator.md @@ -10,7 +10,7 @@ which automatically deploys both the ClusterLink operator and ClusterLink compon However, it's important to note that ClusterLink deployment necessitates peer certificates for proper functioning. Detailed instructions for creating these peer certificates can be found in the [user guide][]. -## The common use-case +## The common use case The common use case for deploying ClusterLink on a cloud based K8s cluster (i.e., EKS, GKE, IKS, etc.) is using the CLI command: From 775bb85585daa1f9893cdcba8042a8690c228440 Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Sun, 19 May 2024 10:03:49 +0300 Subject: [PATCH 02/53] minor edits from @welisheva (#584) Signed-off-by: Etai Lev Ran --- .../content/en/docs/main/concepts/_index.md | 2 +- .../content/en/docs/main/concepts/fabric.md | 4 +-- .../content/en/docs/main/concepts/peers.md | 34 +++++++++++-------- .../docs/main/getting-started/developers.md | 2 +- 4 files changed, 23 insertions(+), 19 deletions(-) diff --git a/website/content/en/docs/main/concepts/_index.md b/website/content/en/docs/main/concepts/_index.md index 15139cbb9..5ea1086e1 100644 --- a/website/content/en/docs/main/concepts/_index.md +++ b/website/content/en/docs/main/concepts/_index.md @@ -1,5 +1,5 @@ --- title: Core Concepts -description: Core Concepts of the ClusterLink system. +description: Core Concepts of the ClusterLink system weight: 30 --- diff --git a/website/content/en/docs/main/concepts/fabric.md b/website/content/en/docs/main/concepts/fabric.md index d0b9ad062..881a8878d 100644 --- a/website/content/en/docs/main/concepts/fabric.md +++ b/website/content/en/docs/main/concepts/fabric.md @@ -7,7 +7,7 @@ weight: 10 The concept of a *Fabric* encapsulates a set of cooperating [peers][]. All peers in a fabric can communicate and may share [services][] between them, with access governed by [policies][]. - The Fabric acts as a root of trust for peer to peer communications (i.e., + The Fabric acts as a root of trust for peer-to-peer communications (i.e., it functions as the certificate authority enabling mutual authentication between peers). @@ -24,7 +24,7 @@ Currently, the concept of a Fabric is just that - a concept. It is not represent ### Prerequisites -The following assume that you have access to the `clusterlink` CLI and one or more +The following sections assume that you have access to the `clusterlink` CLI and one or more peers (i.e., clusters) where you'll deploy ClusterLink. The CLI can be downloaded from the ClusterLink [releases page on GitHub][]. diff --git a/website/content/en/docs/main/concepts/peers.md b/website/content/en/docs/main/concepts/peers.md index 9cde662a4..ff1767f88 100644 --- a/website/content/en/docs/main/concepts/peers.md +++ b/website/content/en/docs/main/concepts/peers.md @@ -6,23 +6,23 @@ weight: 20 A *Peer* represents a location, such as a Kubernetes cluster, participating in a [fabric][]. Each peer may host one or more [services][] - it wishes to share with other peers. A peer is managed by a peer administrator, + that it may wish to share with other peers. A peer is managed by a peer administrator, which is responsible for running the ClusterLink control and data planes. The administrator will typically deploy the ClusterLink components by configuring - the [deployment CR][]. They may also wish to provide - (often) coarse-grained access policies in accordance with high level corporate + the [deployment Custom Resource (CR)][operator-cr]. The administrator may also wish + to provide coarse-grained access policies (and often do) in accordance with high level corporate policies (e.g., "production peers should only communicate with other production peers"). Once a peer has been added to a fabric, it can communicate with any other peer belonging to it. All configuration relating to service sharing (e.g., the exporting and importing of services, and the setting of fine grained application policies) can be done with lowered privileges (e.g., by users, such as application owners). Remote peers are - represented by the Peer Custom Resource Definition (CRD). Each Peer CR instance + represented by peer Custom Resources (CRs). Each Peer CR instance defines a remote cluster and the network endpoints of its ClusterLink gateways. ## Prerequisites -The following assume that you have access to the `clusterlink` CLI and one or more +The following sections assume that you have access to the `clusterlink` CLI and one or more peers (i.e., clusters) where you'll deploy ClusterLink. The CLI can be downloaded from the ClusterLink [releases page on GitHub][]. It also assumes that you have access to the [previously created fabric][] @@ -37,8 +37,9 @@ Creating a new peer is a **fabric administrator** level operation and should be ### Create a new peer certificate -To create a new peer certificate belonging to a fabric, confirm that the fabric CA files - are available in the current working directory, and then execute the following CLI command: +To create a new peer certificate belonging to a fabric, confirm that the fabric + Certificate Authority (CA) files are available in the current working directory, + and then execute the following CLI command: ```sh clusterlink create peer-cert --name --fabric @@ -55,9 +56,9 @@ This will create the certificate and private key files (`cert.pem` and You can override the default by setting the `--output ` option. {{< notice info >}} -You will need the CA certificate (but **not** the CA private key) and the peer certificate - and private in the next step. They can be provided out of band (e.g., over email) to the - peer administrator. +You will need the CA certificate (but **not** the CA private key) and the peer's certificate + and private key pair in the next step. They can be provided out of band (e.g., over email) to the + peer administrator or by any other means for secure transfer of sensitive data. {{< /notice >}} ## Deploy ClusterLink to a new peer @@ -67,9 +68,12 @@ This operation is typically done by a local **peer administrator**, usually diff than the **fabric administrator**. {{< /notice >}} -Before proceeding, ensure that the CA certificate (the CA private key is not needed), - and the peer certificate and key files which were created in the previous step are - in the current working directory. +Before proceeding, ensure that the following files (created in the previous step) are + available in the current working directory: + + 1. CA certificate; + 1. peer certificate; and + 1. peer private key. ### Install the ClusterLink deployment operator @@ -180,7 +184,7 @@ There are two fundamental attributes in the peer CRD: the peer name and the list during connection establishment. The name is used by importers in referencing an export (see [services][] for details). -Gateway endpoint would typically be a implemented via a `NodePort` or `LoadBalancer` +Gateway endpoint would typically be implemented via a `NodePort` or `LoadBalancer` K8s service. A `NodePort` service would typically be used in local deployments (e.g., when running in kind clusters during development) and a `LoadBalancer` service would be used in cloud based deployments. These can be automatically configured and @@ -197,7 +201,7 @@ Gateway endpoint would typically be a implemented via a `NodePort` or `LoadBalan Once a peer has been created and initialized with the ClusterLink control and data planes as well as one or more remote peers, you can proceed with configuring [services][] and [policies][]. - For a complete end to end use case, refer to the [iperf tutorial][]. + For a complete end-to-end use case, refer to the [iperf tutorial][]. [fabric]: {{< relref "fabric" >}} [previously created fabric]: {{< relref "fabric#create-a-new-fabric-ca" >}} diff --git a/website/content/en/docs/main/getting-started/developers.md b/website/content/en/docs/main/getting-started/developers.md index 29a451620..729bc8e28 100644 --- a/website/content/en/docs/main/getting-started/developers.md +++ b/website/content/en/docs/main/getting-started/developers.md @@ -29,7 +29,7 @@ Here are the key steps for setting up your developer environment, making a chang [contribution guide][]. - We follow [GitHub's Standard Fork & Pull Request Workflow][]. -All contributed code should should pass precommit checks such as linting and tests. These +All contributed code should pass precommit checks such as linting and other tests. These are run automatically as part of the CI process on every pull request. You may wish to run these locally, before initiating a PR: From 3f9e02c19771a6ed50dc13b41083b256021fb1f8 Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Sun, 19 May 2024 10:53:40 +0300 Subject: [PATCH 03/53] tests/e2e/k8s: Fix race in peer status check (#585) This commit fixes a race where a peer status is updated after being fetched for comparison. Signed-off-by: Or Ozeri --- tests/e2e/k8s/test_basic.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/e2e/k8s/test_basic.go b/tests/e2e/k8s/test_basic.go index e49dee97f..30d6a5798 100644 --- a/tests/e2e/k8s/test_basic.go +++ b/tests/e2e/k8s/test_basic.go @@ -16,6 +16,7 @@ package k8s import ( "fmt" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -161,6 +162,11 @@ func (s *TestSuite) TestControlplaneCRUD() { // list peers objects, err = client0.Peers.List() require.Nil(s.T(), err) + if !assert.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Peer), []v1alpha1.Peer{peerFromServer}) { + objects, err = client0.Peers.Get(peer.Name) + require.Nil(s.T(), err) + peerFromServer = *objects.(*v1alpha1.Peer) + } require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Peer), []v1alpha1.Peer{peerFromServer}) // add another peer (for upcoming load-balancing test) From 41944cd4074d0c1cff496b47cf7045872b2780bb Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Sun, 19 May 2024 15:30:14 +0300 Subject: [PATCH 04/53] copyright: update all copyright lines and remove year (#592) * remove copyright years from text * add copyright years to license --------- Signed-off-by: Etai Lev Ran --- .devcontainer/dev/post-create.sh | 2 +- .devcontainer/website/post-create.sh | 2 +- .licenserc.yaml | 3 +- LICENSE | 41 +++++++------------ cmd/cl-controlplane/app/server.go | 2 +- cmd/cl-controlplane/cl-controlplane.go | 2 +- cmd/cl-dataplane/app/envoy.go | 2 +- cmd/cl-dataplane/app/envoyconf.go | 2 +- cmd/cl-dataplane/app/server.go | 2 +- cmd/cl-dataplane/cl-dataplane.go | 2 +- cmd/cl-go-dataplane/app/server.go | 2 +- cmd/cl-go-dataplane/cl-go-dataplane.go | 2 +- cmd/cl-operator/main.go | 2 +- cmd/clusterlink/cl-adm.go | 2 +- cmd/clusterlink/cmd/cmd.go | 2 +- cmd/clusterlink/cmd/create/create.go | 2 +- cmd/clusterlink/cmd/create/create_fabric.go | 2 +- cmd/clusterlink/cmd/create/create_peer.go | 2 +- cmd/clusterlink/cmd/delete/delete.go | 2 +- cmd/clusterlink/cmd/delete/delete_peer.go | 2 +- cmd/clusterlink/cmd/deploy/deploy.go | 2 +- cmd/clusterlink/cmd/deploy/deploy_peer.go | 2 +- cmd/clusterlink/config/config.go | 2 +- cmd/gwctl/config/config.go | 2 +- cmd/gwctl/main.go | 2 +- cmd/gwctl/subcommand/config.go | 2 +- cmd/gwctl/subcommand/export.go | 2 +- cmd/gwctl/subcommand/gwctl.go | 2 +- cmd/gwctl/subcommand/import.go | 2 +- cmd/gwctl/subcommand/metrics.go | 2 +- cmd/gwctl/subcommand/peer.go | 2 +- cmd/gwctl/subcommand/policy.go | 2 +- cmd/util/util.go | 2 +- config/config.go | 2 +- demos/bookinfo/cloud/apply_lb.py | 2 +- demos/bookinfo/cloud/gw_failover.py | 2 +- demos/bookinfo/cloud/test.py | 2 +- demos/bookinfo/kind/apply_lb.py | 2 +- demos/bookinfo/kind/gw_failover.py | 2 +- demos/bookinfo/kind/test.py | 2 +- demos/bookinfo/test.py | 2 +- demos/iperf3/cloud/test.py | 2 +- demos/iperf3/kind/iperf3_client_start.py | 2 +- demos/iperf3/kind/test.py | 2 +- demos/iperf3/test.py | 2 +- demos/qotd/kind/test.py | 2 +- demos/speedtest/kind/apply_policy.py | 2 +- demos/speedtest/kind/service_import.py | 2 +- demos/speedtest/kind/test.py | 2 +- demos/utils/__init__.py | 2 +- demos/utils/cloud.py | 2 +- demos/utils/clusterlink.py | 2 +- demos/utils/common.py | 2 +- demos/utils/k8s.py | 2 +- demos/utils/kind.py | 2 +- .../kind/flannel/create_cni_bridge.py | 2 +- hack/boilerplate.go.txt | 2 +- .../clusterlink.net/v1alpha1/accesspolicy.go | 2 +- .../v1alpha1/accesspolicy_test.go | 2 +- pkg/apis/clusterlink.net/v1alpha1/export.go | 2 +- .../v1alpha1/groupversion_info.go | 2 +- pkg/apis/clusterlink.net/v1alpha1/import.go | 2 +- .../v1alpha1/instance_types.go | 2 +- pkg/apis/clusterlink.net/v1alpha1/peer.go | 2 +- .../v1alpha1/zz_generated.deepcopy.go | 2 +- pkg/bootstrap/cert.go | 2 +- pkg/bootstrap/crypt.go | 2 +- pkg/bootstrap/platform/config.go | 2 +- pkg/bootstrap/platform/k8s.go | 2 +- pkg/client/client.go | 2 +- pkg/controlplane/api/authz.go | 2 +- pkg/controlplane/api/heartbeat.go | 2 +- pkg/controlplane/api/servername.go | 2 +- pkg/controlplane/api/xds.go | 2 +- .../authz/connectivitypdp/accesspolicy.go | 2 +- .../authz/connectivitypdp/connectivity_pdp.go | 2 +- .../connectivitypdp/connectivity_pdp_test.go | 2 +- pkg/controlplane/authz/controllers.go | 2 +- pkg/controlplane/authz/loadbalancer.go | 2 +- pkg/controlplane/authz/manager.go | 2 +- pkg/controlplane/authz/server.go | 2 +- pkg/controlplane/control/controllers.go | 2 +- pkg/controlplane/control/manager.go | 2 +- pkg/controlplane/control/peer.go | 2 +- pkg/controlplane/control/port.go | 2 +- pkg/controlplane/eventmanager/events.go | 2 +- pkg/controlplane/peer/client.go | 2 +- pkg/controlplane/rest/accesspolicy.go | 2 +- pkg/controlplane/rest/export.go | 2 +- pkg/controlplane/rest/import.go | 2 +- pkg/controlplane/rest/manager.go | 2 +- pkg/controlplane/rest/peer.go | 2 +- pkg/controlplane/rest/server.go | 2 +- pkg/controlplane/store/accesspolicies.go | 2 +- pkg/controlplane/store/exports.go | 2 +- pkg/controlplane/store/imports.go | 2 +- pkg/controlplane/store/peers.go | 2 +- pkg/controlplane/store/types.go | 2 +- pkg/controlplane/xds/controllers.go | 2 +- pkg/controlplane/xds/manager.go | 2 +- pkg/controlplane/xds/server.go | 2 +- pkg/dataplane/api/servername.go | 2 +- pkg/dataplane/client/fetcher.go | 2 +- pkg/dataplane/client/xds.go | 2 +- pkg/dataplane/server/dataplane.go | 2 +- pkg/dataplane/server/forwarder.go | 2 +- pkg/dataplane/server/listener.go | 2 +- pkg/dataplane/server/server.go | 2 +- pkg/k8s/kubernetes/kube.go | 2 +- pkg/k8s/kubernetes/kubeinformer.go | 2 +- pkg/metrics/metrics.go | 2 +- .../controller/instance_controller.go | 2 +- .../controller/instance_controller_test.go | 2 +- pkg/store/kv/bolt/store.go | 2 +- pkg/store/kv/manager.go | 2 +- pkg/store/kv/store.go | 2 +- pkg/store/kv/types.go | 2 +- pkg/store/types.go | 2 +- pkg/util/controller/controller.go | 2 +- pkg/util/controller/manager.go | 2 +- pkg/util/grpc/server.go | 2 +- pkg/util/http/server.go | 2 +- pkg/util/jsonapi/client.go | 2 +- pkg/util/log/util.go | 2 +- pkg/util/net/util.go | 2 +- pkg/util/rest/client.go | 2 +- pkg/util/rest/server.go | 2 +- pkg/util/runnable/manager.go | 2 +- pkg/util/sniproxy/server.go | 2 +- pkg/util/tcp/listener.go | 2 +- pkg/util/tls/util.go | 2 +- pkg/versioninfo/variables.go | 2 +- pkg/versioninfo/version.go | 2 +- tests/e2e/k8s/k8s_test.go | 2 +- tests/e2e/k8s/services/errors.go | 2 +- tests/e2e/k8s/services/httpecho/client.go | 2 +- tests/e2e/k8s/services/httpecho/server.go | 2 +- tests/e2e/k8s/services/iperf3/client.go | 2 +- tests/e2e/k8s/services/iperf3/server.go | 2 +- tests/e2e/k8s/suite.go | 2 +- tests/e2e/k8s/test_basic.go | 2 +- tests/e2e/k8s/test_export.go | 2 +- tests/e2e/k8s/test_import.go | 2 +- tests/e2e/k8s/test_loadbalancing.go | 2 +- tests/e2e/k8s/test_operator.go | 2 +- tests/e2e/k8s/test_peer.go | 2 +- tests/e2e/k8s/test_performance.go | 2 +- tests/e2e/k8s/test_policy.go | 2 +- tests/e2e/k8s/util/async.go | 2 +- tests/e2e/k8s/util/clusterlink.go | 2 +- tests/e2e/k8s/util/fabric.go | 2 +- tests/e2e/k8s/util/k8s_yaml.go | 2 +- tests/e2e/k8s/util/kind.go | 2 +- tests/e2e/k8s/util/policies.go | 2 +- tests/k8s.sh | 2 +- 155 files changed, 168 insertions(+), 182 deletions(-) diff --git a/.devcontainer/dev/post-create.sh b/.devcontainer/dev/post-create.sh index 2ef5f6a55..7d2b70dc0 100755 --- a/.devcontainer/dev/post-create.sh +++ b/.devcontainer/dev/post-create.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/.devcontainer/website/post-create.sh b/.devcontainer/website/post-create.sh index 8894228cf..9a88a5755 100755 --- a/.devcontainer/website/post-create.sh +++ b/.devcontainer/website/post-create.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/.licenserc.yaml b/.licenserc.yaml index 7572e624b..f296a5e4b 100644 --- a/.licenserc.yaml +++ b/.licenserc.yaml @@ -2,10 +2,9 @@ header: license: spdx-id: Apache-2.0 copyright-owner: Apache Software Foundation - copyright-year: "2023" software-name: ClusterLink content: | - Copyright 2023 The ClusterLink Authors. + Copyright (c) The ClusterLink Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at diff --git a/LICENSE b/LICENSE index 261eeb9e9..6c2f3ad9b 100644 --- a/LICENSE +++ b/LICENSE @@ -1,3 +1,17 @@ +Copyright 2022-present The ClusterLink Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use these files except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ @@ -172,30 +186,3 @@ defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/cmd/cl-controlplane/app/server.go b/cmd/cl-controlplane/app/server.go index 9f6793dff..d20af45a1 100644 --- a/cmd/cl-controlplane/app/server.go +++ b/cmd/cl-controlplane/app/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-controlplane/cl-controlplane.go b/cmd/cl-controlplane/cl-controlplane.go index 2c1855d99..ab5d6d47b 100644 --- a/cmd/cl-controlplane/cl-controlplane.go +++ b/cmd/cl-controlplane/cl-controlplane.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-dataplane/app/envoy.go b/cmd/cl-dataplane/app/envoy.go index 421fc61af..6944c6aff 100644 --- a/cmd/cl-dataplane/app/envoy.go +++ b/cmd/cl-dataplane/app/envoy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-dataplane/app/envoyconf.go b/cmd/cl-dataplane/app/envoyconf.go index a0b23d835..417c0d561 100644 --- a/cmd/cl-dataplane/app/envoyconf.go +++ b/cmd/cl-dataplane/app/envoyconf.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-dataplane/app/server.go b/cmd/cl-dataplane/app/server.go index e4559a334..1477ab761 100644 --- a/cmd/cl-dataplane/app/server.go +++ b/cmd/cl-dataplane/app/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-dataplane/cl-dataplane.go b/cmd/cl-dataplane/cl-dataplane.go index ea73d2536..536eca941 100644 --- a/cmd/cl-dataplane/cl-dataplane.go +++ b/cmd/cl-dataplane/cl-dataplane.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-go-dataplane/app/server.go b/cmd/cl-go-dataplane/app/server.go index 21da55dee..763361013 100644 --- a/cmd/cl-go-dataplane/app/server.go +++ b/cmd/cl-go-dataplane/app/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-go-dataplane/cl-go-dataplane.go b/cmd/cl-go-dataplane/cl-go-dataplane.go index 120ec9f40..cc8e078c7 100644 --- a/cmd/cl-go-dataplane/cl-go-dataplane.go +++ b/cmd/cl-go-dataplane/cl-go-dataplane.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/cl-operator/main.go b/cmd/cl-operator/main.go index 7660940ad..e581c9041 100644 --- a/cmd/cl-operator/main.go +++ b/cmd/cl-operator/main.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cl-adm.go b/cmd/clusterlink/cl-adm.go index 2e6190986..934fd7051 100644 --- a/cmd/clusterlink/cl-adm.go +++ b/cmd/clusterlink/cl-adm.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/cmd.go b/cmd/clusterlink/cmd/cmd.go index c756f30bb..d9425e156 100644 --- a/cmd/clusterlink/cmd/cmd.go +++ b/cmd/clusterlink/cmd/cmd.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/create/create.go b/cmd/clusterlink/cmd/create/create.go index 851b0af4c..eb51c315c 100644 --- a/cmd/clusterlink/cmd/create/create.go +++ b/cmd/clusterlink/cmd/create/create.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/create/create_fabric.go b/cmd/clusterlink/cmd/create/create_fabric.go index 3965e95d2..e58be0b7d 100644 --- a/cmd/clusterlink/cmd/create/create_fabric.go +++ b/cmd/clusterlink/cmd/create/create_fabric.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/create/create_peer.go b/cmd/clusterlink/cmd/create/create_peer.go index a1addb95b..d29a9e5f2 100644 --- a/cmd/clusterlink/cmd/create/create_peer.go +++ b/cmd/clusterlink/cmd/create/create_peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/delete/delete.go b/cmd/clusterlink/cmd/delete/delete.go index 6707c44ce..f4b295d94 100644 --- a/cmd/clusterlink/cmd/delete/delete.go +++ b/cmd/clusterlink/cmd/delete/delete.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/delete/delete_peer.go b/cmd/clusterlink/cmd/delete/delete_peer.go index e163ccdbe..05010b7ef 100644 --- a/cmd/clusterlink/cmd/delete/delete_peer.go +++ b/cmd/clusterlink/cmd/delete/delete_peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/deploy/deploy.go b/cmd/clusterlink/cmd/deploy/deploy.go index dfe312f14..0644d90c9 100644 --- a/cmd/clusterlink/cmd/deploy/deploy.go +++ b/cmd/clusterlink/cmd/deploy/deploy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/cmd/deploy/deploy_peer.go b/cmd/clusterlink/cmd/deploy/deploy_peer.go index ff100c858..50a3fee0f 100644 --- a/cmd/clusterlink/cmd/deploy/deploy_peer.go +++ b/cmd/clusterlink/cmd/deploy/deploy_peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/clusterlink/config/config.go b/cmd/clusterlink/config/config.go index 0de4e9bcb..46cd07a6f 100644 --- a/cmd/clusterlink/config/config.go +++ b/cmd/clusterlink/config/config.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/config/config.go b/cmd/gwctl/config/config.go index 98af141b4..8c16c51e9 100644 --- a/cmd/gwctl/config/config.go +++ b/cmd/gwctl/config/config.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/main.go b/cmd/gwctl/main.go index a520301af..575e9e94f 100644 --- a/cmd/gwctl/main.go +++ b/cmd/gwctl/main.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/config.go b/cmd/gwctl/subcommand/config.go index 49e6c417f..87505e653 100644 --- a/cmd/gwctl/subcommand/config.go +++ b/cmd/gwctl/subcommand/config.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/export.go b/cmd/gwctl/subcommand/export.go index a8623f902..d328f12cb 100644 --- a/cmd/gwctl/subcommand/export.go +++ b/cmd/gwctl/subcommand/export.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/gwctl.go b/cmd/gwctl/subcommand/gwctl.go index 2a93c05fe..c58518256 100644 --- a/cmd/gwctl/subcommand/gwctl.go +++ b/cmd/gwctl/subcommand/gwctl.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/import.go b/cmd/gwctl/subcommand/import.go index 1ccfd6347..dfde1f675 100644 --- a/cmd/gwctl/subcommand/import.go +++ b/cmd/gwctl/subcommand/import.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/metrics.go b/cmd/gwctl/subcommand/metrics.go index 509e4415d..70d5b793e 100644 --- a/cmd/gwctl/subcommand/metrics.go +++ b/cmd/gwctl/subcommand/metrics.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/peer.go b/cmd/gwctl/subcommand/peer.go index 4cc087dfa..b1dcfbed9 100644 --- a/cmd/gwctl/subcommand/peer.go +++ b/cmd/gwctl/subcommand/peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/gwctl/subcommand/policy.go b/cmd/gwctl/subcommand/policy.go index 96b6bd669..e8ce7eba6 100644 --- a/cmd/gwctl/subcommand/policy.go +++ b/cmd/gwctl/subcommand/policy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/cmd/util/util.go b/cmd/util/util.go index f8546ca81..91b8e6873 100644 --- a/cmd/util/util.go +++ b/cmd/util/util.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/config/config.go b/config/config.go index 921ca7f79..da5b92083 100644 --- a/config/config.go +++ b/config/config.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/demos/bookinfo/cloud/apply_lb.py b/demos/bookinfo/cloud/apply_lb.py index d73d9015a..5d8131a5d 100755 --- a/demos/bookinfo/cloud/apply_lb.py +++ b/demos/bookinfo/cloud/apply_lb.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/cloud/gw_failover.py b/demos/bookinfo/cloud/gw_failover.py index 87ceec853..623d9485e 100755 --- a/demos/bookinfo/cloud/gw_failover.py +++ b/demos/bookinfo/cloud/gw_failover.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/cloud/test.py b/demos/bookinfo/cloud/test.py index 2648c7a9b..9569441c7 100755 --- a/demos/bookinfo/cloud/test.py +++ b/demos/bookinfo/cloud/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/kind/apply_lb.py b/demos/bookinfo/kind/apply_lb.py index d078436fd..11c01dbed 100755 --- a/demos/bookinfo/kind/apply_lb.py +++ b/demos/bookinfo/kind/apply_lb.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/kind/gw_failover.py b/demos/bookinfo/kind/gw_failover.py index f2d6b0324..17fd9a21d 100755 --- a/demos/bookinfo/kind/gw_failover.py +++ b/demos/bookinfo/kind/gw_failover.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/kind/test.py b/demos/bookinfo/kind/test.py index c4b015677..507f09fef 100755 --- a/demos/bookinfo/kind/test.py +++ b/demos/bookinfo/kind/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/bookinfo/test.py b/demos/bookinfo/test.py index 68c28c020..69f16d880 100644 --- a/demos/bookinfo/test.py +++ b/demos/bookinfo/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/iperf3/cloud/test.py b/demos/iperf3/cloud/test.py index 1687bd153..c9dcef569 100755 --- a/demos/iperf3/cloud/test.py +++ b/demos/iperf3/cloud/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/iperf3/kind/iperf3_client_start.py b/demos/iperf3/kind/iperf3_client_start.py index 6e7b59ffc..e924af1ec 100755 --- a/demos/iperf3/kind/iperf3_client_start.py +++ b/demos/iperf3/kind/iperf3_client_start.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/iperf3/kind/test.py b/demos/iperf3/kind/test.py index 5754f66fb..e5736199b 100755 --- a/demos/iperf3/kind/test.py +++ b/demos/iperf3/kind/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/iperf3/test.py b/demos/iperf3/test.py index 3fd07af4d..0ce6b1034 100644 --- a/demos/iperf3/test.py +++ b/demos/iperf3/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/qotd/kind/test.py b/demos/qotd/kind/test.py index 07f28d0f3..5a32a44b5 100755 --- a/demos/qotd/kind/test.py +++ b/demos/qotd/kind/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/speedtest/kind/apply_policy.py b/demos/speedtest/kind/apply_policy.py index 0130f9df6..8f8d65935 100755 --- a/demos/speedtest/kind/apply_policy.py +++ b/demos/speedtest/kind/apply_policy.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/speedtest/kind/service_import.py b/demos/speedtest/kind/service_import.py index 0745fcc7d..92bde5551 100755 --- a/demos/speedtest/kind/service_import.py +++ b/demos/speedtest/kind/service_import.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/speedtest/kind/test.py b/demos/speedtest/kind/test.py index d1c6d87f6..16414dc69 100755 --- a/demos/speedtest/kind/test.py +++ b/demos/speedtest/kind/test.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/__init__.py b/demos/utils/__init__.py index 40143186e..a51a17d6f 100644 --- a/demos/utils/__init__.py +++ b/demos/utils/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/cloud.py b/demos/utils/cloud.py index 381acf1c1..01e4b1f1e 100644 --- a/demos/utils/cloud.py +++ b/demos/utils/cloud.py @@ -1,5 +1,5 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/clusterlink.py b/demos/utils/clusterlink.py index 095231cc3..b8ec96005 100644 --- a/demos/utils/clusterlink.py +++ b/demos/utils/clusterlink.py @@ -1,4 +1,4 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/common.py b/demos/utils/common.py index 9322d63c4..bfeaa1f2e 100644 --- a/demos/utils/common.py +++ b/demos/utils/common.py @@ -1,4 +1,4 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/k8s.py b/demos/utils/k8s.py index 4baf693cb..66a7e5d96 100644 --- a/demos/utils/k8s.py +++ b/demos/utils/k8s.py @@ -1,5 +1,5 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/kind.py b/demos/utils/kind.py index 13709a4c2..1c9bcf675 100644 --- a/demos/utils/kind.py +++ b/demos/utils/kind.py @@ -1,4 +1,4 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/demos/utils/manifests/kind/flannel/create_cni_bridge.py b/demos/utils/manifests/kind/flannel/create_cni_bridge.py index 709b11126..663eb3f85 100644 --- a/demos/utils/manifests/kind/flannel/create_cni_bridge.py +++ b/demos/utils/manifests/kind/flannel/create_cni_bridge.py @@ -1,4 +1,4 @@ -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at diff --git a/hack/boilerplate.go.txt b/hack/boilerplate.go.txt index 7f6952f84..0990c0fd7 100644 --- a/hack/boilerplate.go.txt +++ b/hack/boilerplate.go.txt @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/accesspolicy.go b/pkg/apis/clusterlink.net/v1alpha1/accesspolicy.go index 66c1935f7..77ef77f7e 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/accesspolicy.go +++ b/pkg/apis/clusterlink.net/v1alpha1/accesspolicy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/accesspolicy_test.go b/pkg/apis/clusterlink.net/v1alpha1/accesspolicy_test.go index 43bbd3dd6..09802e733 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/accesspolicy_test.go +++ b/pkg/apis/clusterlink.net/v1alpha1/accesspolicy_test.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/export.go b/pkg/apis/clusterlink.net/v1alpha1/export.go index 42c741d6f..6f1871e3c 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/export.go +++ b/pkg/apis/clusterlink.net/v1alpha1/export.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/groupversion_info.go b/pkg/apis/clusterlink.net/v1alpha1/groupversion_info.go index 79ec8de8b..72c135506 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/groupversion_info.go +++ b/pkg/apis/clusterlink.net/v1alpha1/groupversion_info.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/import.go b/pkg/apis/clusterlink.net/v1alpha1/import.go index abd8e2dc6..c4d628fcb 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/import.go +++ b/pkg/apis/clusterlink.net/v1alpha1/import.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/instance_types.go b/pkg/apis/clusterlink.net/v1alpha1/instance_types.go index 826efd8b9..4889d035c 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/instance_types.go +++ b/pkg/apis/clusterlink.net/v1alpha1/instance_types.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/peer.go b/pkg/apis/clusterlink.net/v1alpha1/peer.go index 74df49000..263a1750e 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/peer.go +++ b/pkg/apis/clusterlink.net/v1alpha1/peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/apis/clusterlink.net/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/clusterlink.net/v1alpha1/zz_generated.deepcopy.go index c82fb7344..e98d84035 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/zz_generated.deepcopy.go +++ b/pkg/apis/clusterlink.net/v1alpha1/zz_generated.deepcopy.go @@ -1,6 +1,6 @@ //go:build !ignore_autogenerated -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/bootstrap/cert.go b/pkg/bootstrap/cert.go index b60ea0a10..742f97ebc 100644 --- a/pkg/bootstrap/cert.go +++ b/pkg/bootstrap/cert.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/bootstrap/crypt.go b/pkg/bootstrap/crypt.go index 90dbba640..202a8a54f 100644 --- a/pkg/bootstrap/crypt.go +++ b/pkg/bootstrap/crypt.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/bootstrap/platform/config.go b/pkg/bootstrap/platform/config.go index 90a1336ae..c226769b5 100644 --- a/pkg/bootstrap/platform/config.go +++ b/pkg/bootstrap/platform/config.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/bootstrap/platform/k8s.go b/pkg/bootstrap/platform/k8s.go index 30e8cb807..8d2c1f0f4 100644 --- a/pkg/bootstrap/platform/k8s.go +++ b/pkg/bootstrap/platform/k8s.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/client/client.go b/pkg/client/client.go index 0c175fca1..60a4f388d 100644 --- a/pkg/client/client.go +++ b/pkg/client/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/api/authz.go b/pkg/controlplane/api/authz.go index d8e9554a1..ff5594d2b 100644 --- a/pkg/controlplane/api/authz.go +++ b/pkg/controlplane/api/authz.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/api/heartbeat.go b/pkg/controlplane/api/heartbeat.go index 314fd8e2a..976c7013e 100644 --- a/pkg/controlplane/api/heartbeat.go +++ b/pkg/controlplane/api/heartbeat.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/api/servername.go b/pkg/controlplane/api/servername.go index 99d0d1d35..1c454cae0 100644 --- a/pkg/controlplane/api/servername.go +++ b/pkg/controlplane/api/servername.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/api/xds.go b/pkg/controlplane/api/xds.go index d28c1dff7..5825ac375 100644 --- a/pkg/controlplane/api/xds.go +++ b/pkg/controlplane/api/xds.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/connectivitypdp/accesspolicy.go b/pkg/controlplane/authz/connectivitypdp/accesspolicy.go index b19147b13..94fdec07c 100644 --- a/pkg/controlplane/authz/connectivitypdp/accesspolicy.go +++ b/pkg/controlplane/authz/connectivitypdp/accesspolicy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/connectivitypdp/connectivity_pdp.go b/pkg/controlplane/authz/connectivitypdp/connectivity_pdp.go index db944f067..111caad87 100644 --- a/pkg/controlplane/authz/connectivitypdp/connectivity_pdp.go +++ b/pkg/controlplane/authz/connectivitypdp/connectivity_pdp.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/connectivitypdp/connectivity_pdp_test.go b/pkg/controlplane/authz/connectivitypdp/connectivity_pdp_test.go index 138c848c4..4c8ffac77 100644 --- a/pkg/controlplane/authz/connectivitypdp/connectivity_pdp_test.go +++ b/pkg/controlplane/authz/connectivitypdp/connectivity_pdp_test.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/controllers.go b/pkg/controlplane/authz/controllers.go index b5adfde14..280cae367 100644 --- a/pkg/controlplane/authz/controllers.go +++ b/pkg/controlplane/authz/controllers.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/loadbalancer.go b/pkg/controlplane/authz/loadbalancer.go index bfec8018b..1229aed5b 100644 --- a/pkg/controlplane/authz/loadbalancer.go +++ b/pkg/controlplane/authz/loadbalancer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index 93c3dee95..381b2be7e 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/authz/server.go b/pkg/controlplane/authz/server.go index 4891d62e9..b91667f6b 100644 --- a/pkg/controlplane/authz/server.go +++ b/pkg/controlplane/authz/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/control/controllers.go b/pkg/controlplane/control/controllers.go index c295ea2e0..d544fcacd 100644 --- a/pkg/controlplane/control/controllers.go +++ b/pkg/controlplane/control/controllers.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/control/manager.go b/pkg/controlplane/control/manager.go index cee8f4fb5..22a69de42 100644 --- a/pkg/controlplane/control/manager.go +++ b/pkg/controlplane/control/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/control/peer.go b/pkg/controlplane/control/peer.go index bb02c87fb..da336ae5e 100644 --- a/pkg/controlplane/control/peer.go +++ b/pkg/controlplane/control/peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/control/port.go b/pkg/controlplane/control/port.go index a48648808..c075a6d51 100644 --- a/pkg/controlplane/control/port.go +++ b/pkg/controlplane/control/port.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/eventmanager/events.go b/pkg/controlplane/eventmanager/events.go index 15e3a75ac..b74e0c52b 100644 --- a/pkg/controlplane/eventmanager/events.go +++ b/pkg/controlplane/eventmanager/events.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/peer/client.go b/pkg/controlplane/peer/client.go index f74dddbd2..070bef829 100644 --- a/pkg/controlplane/peer/client.go +++ b/pkg/controlplane/peer/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/accesspolicy.go b/pkg/controlplane/rest/accesspolicy.go index a301e2a24..6175a4265 100644 --- a/pkg/controlplane/rest/accesspolicy.go +++ b/pkg/controlplane/rest/accesspolicy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/export.go b/pkg/controlplane/rest/export.go index 6c29dc753..f4d93702e 100644 --- a/pkg/controlplane/rest/export.go +++ b/pkg/controlplane/rest/export.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/import.go b/pkg/controlplane/rest/import.go index 5daf1ff93..d7832af05 100644 --- a/pkg/controlplane/rest/import.go +++ b/pkg/controlplane/rest/import.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/manager.go b/pkg/controlplane/rest/manager.go index 0f8acd3fa..d210e5e02 100644 --- a/pkg/controlplane/rest/manager.go +++ b/pkg/controlplane/rest/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/peer.go b/pkg/controlplane/rest/peer.go index 9a8a71662..7bce24c55 100644 --- a/pkg/controlplane/rest/peer.go +++ b/pkg/controlplane/rest/peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/rest/server.go b/pkg/controlplane/rest/server.go index 651dc6de8..e2306f663 100644 --- a/pkg/controlplane/rest/server.go +++ b/pkg/controlplane/rest/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/store/accesspolicies.go b/pkg/controlplane/store/accesspolicies.go index 4e29b1466..62338117f 100644 --- a/pkg/controlplane/store/accesspolicies.go +++ b/pkg/controlplane/store/accesspolicies.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/store/exports.go b/pkg/controlplane/store/exports.go index 681b87e76..7abca5de3 100644 --- a/pkg/controlplane/store/exports.go +++ b/pkg/controlplane/store/exports.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/store/imports.go b/pkg/controlplane/store/imports.go index a0a4967b7..97c42dedd 100644 --- a/pkg/controlplane/store/imports.go +++ b/pkg/controlplane/store/imports.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/store/peers.go b/pkg/controlplane/store/peers.go index 14d819bec..57adc2e9d 100644 --- a/pkg/controlplane/store/peers.go +++ b/pkg/controlplane/store/peers.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/store/types.go b/pkg/controlplane/store/types.go index d9babb9a3..c12c8fd1e 100644 --- a/pkg/controlplane/store/types.go +++ b/pkg/controlplane/store/types.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/xds/controllers.go b/pkg/controlplane/xds/controllers.go index 7c860557f..03abcf56b 100644 --- a/pkg/controlplane/xds/controllers.go +++ b/pkg/controlplane/xds/controllers.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/xds/manager.go b/pkg/controlplane/xds/manager.go index 65fdcbfd6..e1147a5cd 100644 --- a/pkg/controlplane/xds/manager.go +++ b/pkg/controlplane/xds/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/controlplane/xds/server.go b/pkg/controlplane/xds/server.go index 723088145..5317da488 100644 --- a/pkg/controlplane/xds/server.go +++ b/pkg/controlplane/xds/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/api/servername.go b/pkg/dataplane/api/servername.go index eaa6f66b0..943541575 100644 --- a/pkg/dataplane/api/servername.go +++ b/pkg/dataplane/api/servername.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/client/fetcher.go b/pkg/dataplane/client/fetcher.go index fb73c28e0..6a721aed8 100644 --- a/pkg/dataplane/client/fetcher.go +++ b/pkg/dataplane/client/fetcher.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/client/xds.go b/pkg/dataplane/client/xds.go index fbb91edcc..84c70e67f 100644 --- a/pkg/dataplane/client/xds.go +++ b/pkg/dataplane/client/xds.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/server/dataplane.go b/pkg/dataplane/server/dataplane.go index e8a53837d..6d380fae7 100644 --- a/pkg/dataplane/server/dataplane.go +++ b/pkg/dataplane/server/dataplane.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/server/forwarder.go b/pkg/dataplane/server/forwarder.go index e30c24f6e..39e89f257 100644 --- a/pkg/dataplane/server/forwarder.go +++ b/pkg/dataplane/server/forwarder.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/server/listener.go b/pkg/dataplane/server/listener.go index f914f1016..6865dde05 100644 --- a/pkg/dataplane/server/listener.go +++ b/pkg/dataplane/server/listener.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/dataplane/server/server.go b/pkg/dataplane/server/server.go index 1a091b2a0..4b259b620 100644 --- a/pkg/dataplane/server/server.go +++ b/pkg/dataplane/server/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/k8s/kubernetes/kube.go b/pkg/k8s/kubernetes/kube.go index a4ae7775c..9c0016c61 100644 --- a/pkg/k8s/kubernetes/kube.go +++ b/pkg/k8s/kubernetes/kube.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/k8s/kubernetes/kubeinformer.go b/pkg/k8s/kubernetes/kubeinformer.go index 513c59c3c..abfb7c819 100644 --- a/pkg/k8s/kubernetes/kubeinformer.go +++ b/pkg/k8s/kubernetes/kubeinformer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go index 38c8a0521..4d98adec9 100644 --- a/pkg/metrics/metrics.go +++ b/pkg/metrics/metrics.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/operator/controller/instance_controller.go b/pkg/operator/controller/instance_controller.go index 7434adf0c..3629f3423 100644 --- a/pkg/operator/controller/instance_controller.go +++ b/pkg/operator/controller/instance_controller.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/operator/controller/instance_controller_test.go b/pkg/operator/controller/instance_controller_test.go index 148afcb6b..523219e5e 100644 --- a/pkg/operator/controller/instance_controller_test.go +++ b/pkg/operator/controller/instance_controller_test.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/store/kv/bolt/store.go b/pkg/store/kv/bolt/store.go index 3c2e85702..3d976d1e6 100644 --- a/pkg/store/kv/bolt/store.go +++ b/pkg/store/kv/bolt/store.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/store/kv/manager.go b/pkg/store/kv/manager.go index c7c9fb9d5..cd94f0125 100644 --- a/pkg/store/kv/manager.go +++ b/pkg/store/kv/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/store/kv/store.go b/pkg/store/kv/store.go index 74a568ec4..2034212a1 100644 --- a/pkg/store/kv/store.go +++ b/pkg/store/kv/store.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/store/kv/types.go b/pkg/store/kv/types.go index bf5e2859d..09729b2af 100644 --- a/pkg/store/kv/types.go +++ b/pkg/store/kv/types.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/store/types.go b/pkg/store/types.go index ce1c72880..c096a0a80 100644 --- a/pkg/store/types.go +++ b/pkg/store/types.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/controller/controller.go b/pkg/util/controller/controller.go index b2d5a4cf9..8658bb4d3 100644 --- a/pkg/util/controller/controller.go +++ b/pkg/util/controller/controller.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/controller/manager.go b/pkg/util/controller/manager.go index dd9042745..5be6384eb 100644 --- a/pkg/util/controller/manager.go +++ b/pkg/util/controller/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/grpc/server.go b/pkg/util/grpc/server.go index 4a16f46b8..1323b1a5a 100644 --- a/pkg/util/grpc/server.go +++ b/pkg/util/grpc/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/http/server.go b/pkg/util/http/server.go index fed6cc0e6..579118dcd 100644 --- a/pkg/util/http/server.go +++ b/pkg/util/http/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/jsonapi/client.go b/pkg/util/jsonapi/client.go index 7e93fe4cf..48612a395 100644 --- a/pkg/util/jsonapi/client.go +++ b/pkg/util/jsonapi/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/log/util.go b/pkg/util/log/util.go index d6f15235e..a456ebac0 100644 --- a/pkg/util/log/util.go +++ b/pkg/util/log/util.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/net/util.go b/pkg/util/net/util.go index 22bd8817a..06a212df3 100644 --- a/pkg/util/net/util.go +++ b/pkg/util/net/util.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/rest/client.go b/pkg/util/rest/client.go index 3f072637e..3ecdb2861 100644 --- a/pkg/util/rest/client.go +++ b/pkg/util/rest/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/rest/server.go b/pkg/util/rest/server.go index 012fa5805..abf02b61c 100644 --- a/pkg/util/rest/server.go +++ b/pkg/util/rest/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/runnable/manager.go b/pkg/util/runnable/manager.go index c71404db0..433275c5e 100644 --- a/pkg/util/runnable/manager.go +++ b/pkg/util/runnable/manager.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/sniproxy/server.go b/pkg/util/sniproxy/server.go index c2696281b..01ed6ad47 100644 --- a/pkg/util/sniproxy/server.go +++ b/pkg/util/sniproxy/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/tcp/listener.go b/pkg/util/tcp/listener.go index 9c0a8476d..7b91953cb 100644 --- a/pkg/util/tcp/listener.go +++ b/pkg/util/tcp/listener.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/util/tls/util.go b/pkg/util/tls/util.go index ef5199810..c39d8cc4b 100644 --- a/pkg/util/tls/util.go +++ b/pkg/util/tls/util.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/versioninfo/variables.go b/pkg/versioninfo/variables.go index 0e0e583e7..a97a70299 100644 --- a/pkg/versioninfo/variables.go +++ b/pkg/versioninfo/variables.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/pkg/versioninfo/version.go b/pkg/versioninfo/version.go index 9dbaf35b5..27539b4c7 100644 --- a/pkg/versioninfo/version.go +++ b/pkg/versioninfo/version.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/k8s_test.go b/tests/e2e/k8s/k8s_test.go index b1aa5bb13..5efc31c5e 100644 --- a/tests/e2e/k8s/k8s_test.go +++ b/tests/e2e/k8s/k8s_test.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/services/errors.go b/tests/e2e/k8s/services/errors.go index c6f5c05d8..07b95f36d 100644 --- a/tests/e2e/k8s/services/errors.go +++ b/tests/e2e/k8s/services/errors.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/services/httpecho/client.go b/tests/e2e/k8s/services/httpecho/client.go index 7669411e3..5fcf777f8 100644 --- a/tests/e2e/k8s/services/httpecho/client.go +++ b/tests/e2e/k8s/services/httpecho/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/services/httpecho/server.go b/tests/e2e/k8s/services/httpecho/server.go index ef83b6349..e3fca0a8e 100644 --- a/tests/e2e/k8s/services/httpecho/server.go +++ b/tests/e2e/k8s/services/httpecho/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/services/iperf3/client.go b/tests/e2e/k8s/services/iperf3/client.go index 357d50ec7..0ed147aa3 100644 --- a/tests/e2e/k8s/services/iperf3/client.go +++ b/tests/e2e/k8s/services/iperf3/client.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/services/iperf3/server.go b/tests/e2e/k8s/services/iperf3/server.go index a7c574e56..6fc25355b 100644 --- a/tests/e2e/k8s/services/iperf3/server.go +++ b/tests/e2e/k8s/services/iperf3/server.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/suite.go b/tests/e2e/k8s/suite.go index f94e2d453..d9064d26c 100644 --- a/tests/e2e/k8s/suite.go +++ b/tests/e2e/k8s/suite.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_basic.go b/tests/e2e/k8s/test_basic.go index 30d6a5798..946a28d14 100644 --- a/tests/e2e/k8s/test_basic.go +++ b/tests/e2e/k8s/test_basic.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_export.go b/tests/e2e/k8s/test_export.go index 33880453b..71dbb02af 100644 --- a/tests/e2e/k8s/test_export.go +++ b/tests/e2e/k8s/test_export.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_import.go b/tests/e2e/k8s/test_import.go index 8d8205d7c..244c49c38 100644 --- a/tests/e2e/k8s/test_import.go +++ b/tests/e2e/k8s/test_import.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_loadbalancing.go b/tests/e2e/k8s/test_loadbalancing.go index 28d8bf22a..96d2d2964 100644 --- a/tests/e2e/k8s/test_loadbalancing.go +++ b/tests/e2e/k8s/test_loadbalancing.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_operator.go b/tests/e2e/k8s/test_operator.go index 7d9aa7dde..f1125c7ce 100644 --- a/tests/e2e/k8s/test_operator.go +++ b/tests/e2e/k8s/test_operator.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_peer.go b/tests/e2e/k8s/test_peer.go index d526ecb1f..94a24877c 100644 --- a/tests/e2e/k8s/test_peer.go +++ b/tests/e2e/k8s/test_peer.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_performance.go b/tests/e2e/k8s/test_performance.go index 283d36676..a7c8ee97d 100644 --- a/tests/e2e/k8s/test_performance.go +++ b/tests/e2e/k8s/test_performance.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/test_policy.go b/tests/e2e/k8s/test_policy.go index 17e29b970..f9b0429ac 100644 --- a/tests/e2e/k8s/test_policy.go +++ b/tests/e2e/k8s/test_policy.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/async.go b/tests/e2e/k8s/util/async.go index 2de5582b0..45f48ef2a 100644 --- a/tests/e2e/k8s/util/async.go +++ b/tests/e2e/k8s/util/async.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/clusterlink.go b/tests/e2e/k8s/util/clusterlink.go index 50cd54303..fb5d3eaca 100644 --- a/tests/e2e/k8s/util/clusterlink.go +++ b/tests/e2e/k8s/util/clusterlink.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/fabric.go b/tests/e2e/k8s/util/fabric.go index 076544b6a..de7c29783 100644 --- a/tests/e2e/k8s/util/fabric.go +++ b/tests/e2e/k8s/util/fabric.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/k8s_yaml.go b/tests/e2e/k8s/util/k8s_yaml.go index ad3ab142f..bc7b484aa 100644 --- a/tests/e2e/k8s/util/k8s_yaml.go +++ b/tests/e2e/k8s/util/k8s_yaml.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/kind.go b/tests/e2e/k8s/util/kind.go index b5ee0ab0e..43fea25b2 100644 --- a/tests/e2e/k8s/util/kind.go +++ b/tests/e2e/k8s/util/kind.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/e2e/k8s/util/policies.go b/tests/e2e/k8s/util/policies.go index 9c2d08f0e..d4b36ba4a 100644 --- a/tests/e2e/k8s/util/policies.go +++ b/tests/e2e/k8s/util/policies.go @@ -1,4 +1,4 @@ -// Copyright 2023 The ClusterLink Authors. +// Copyright (c) The ClusterLink Authors. // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at diff --git a/tests/k8s.sh b/tests/k8s.sh index 506cbfb1e..194057471 100755 --- a/tests/k8s.sh +++ b/tests/k8s.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Copyright 2023 The ClusterLink Authors. +# Copyright (c) The ClusterLink Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at From fbdf4fcbc3203aa4dbcdf57fbc2a1494b992f76a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 May 2024 09:04:03 +0300 Subject: [PATCH 05/53] build(deps): bump the k8s group with 3 updates (#593) Bumps the k8s group with 3 updates: [k8s.io/api](https://github.com/kubernetes/api), [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.30.0 to 0.30.1 - [Commits](https://github.com/kubernetes/api/compare/v0.30.0...v0.30.1) Updates `k8s.io/apimachinery` from 0.30.0 to 0.30.1 - [Commits](https://github.com/kubernetes/apimachinery/compare/v0.30.0...v0.30.1) Updates `k8s.io/client-go` from 0.30.0 to 0.30.1 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/kubernetes/client-go/compare/v0.30.0...v0.30.1) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s - dependency-name: k8s.io/apimachinery dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 6 +++--- go.sum | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 641238bee..032e19c39 100644 --- a/go.mod +++ b/go.mod @@ -19,9 +19,9 @@ require ( golang.org/x/net v0.25.0 google.golang.org/grpc v1.63.2 google.golang.org/protobuf v1.34.1 - k8s.io/api v0.30.0 - k8s.io/apimachinery v0.30.0 - k8s.io/client-go v0.30.0 + k8s.io/api v0.30.1 + k8s.io/apimachinery v0.30.1 + k8s.io/client-go v0.30.1 k8s.io/utils v0.0.0-20230726121419-3b25d923346b sigs.k8s.io/controller-runtime v0.18.2 sigs.k8s.io/e2e-framework v0.3.0 diff --git a/go.sum b/go.sum index a55a17cdd..12babae4e 100644 --- a/go.sum +++ b/go.sum @@ -275,14 +275,14 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/api v0.30.0 h1:siWhRq7cNjy2iHssOB9SCGNCl2spiF1dO3dABqZ8niA= -k8s.io/api v0.30.0/go.mod h1:OPlaYhoHs8EQ1ql0R/TsUgaRPhpKNxIMrKQfWUp8QSE= +k8s.io/api v0.30.1 h1:kCm/6mADMdbAxmIh0LBjS54nQBE+U4KmbCfIkF5CpJY= +k8s.io/api v0.30.1/go.mod h1:ddbN2C0+0DIiPntan/bye3SW3PdwLa11/0yqwvuRrJM= k8s.io/apiextensions-apiserver v0.30.0 h1:jcZFKMqnICJfRxTgnC4E+Hpcq8UEhT8B2lhBcQ+6uAs= k8s.io/apiextensions-apiserver v0.30.0/go.mod h1:N9ogQFGcrbWqAY9p2mUAL5mGxsLqwgtUce127VtRX5Y= -k8s.io/apimachinery v0.30.0 h1:qxVPsyDM5XS96NIh9Oj6LavoVFYff/Pon9cZeDIkHHA= -k8s.io/apimachinery v0.30.0/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= -k8s.io/client-go v0.30.0 h1:sB1AGGlhY/o7KCyCEQ0bPWzYDL0pwOZO4vAtTSh/gJQ= -k8s.io/client-go v0.30.0/go.mod h1:g7li5O5256qe6TYdAMyX/otJqMhIiGgTapdLchhmOaY= +k8s.io/apimachinery v0.30.1 h1:ZQStsEfo4n65yAdlGTfP/uSHMQSoYzU/oeEbkmF7P2U= +k8s.io/apimachinery v0.30.1/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= +k8s.io/client-go v0.30.1 h1:uC/Ir6A3R46wdkgCV3vbLyNOYyCJ8oZnjtJGKfytl/Q= +k8s.io/client-go v0.30.1/go.mod h1:wrAqLNs2trwiCH/wxxmT/x3hKVH9PuV0GGW0oDoHVqc= k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= From 323f5a936379d2e18721795f5e5afeaa664f1cf7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 21 May 2024 09:25:47 +0300 Subject: [PATCH 06/53] build(deps): bump google.golang.org/grpc in the grpc-go group (#594) Bumps the grpc-go group with 1 update: [google.golang.org/grpc](https://github.com/grpc/grpc-go). Updates `google.golang.org/grpc` from 1.63.2 to 1.64.0 - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.63.2...v1.64.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor dependency-group: grpc-go ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 11 +++++------ go.sum | 22 ++++++++++------------ 2 files changed, 15 insertions(+), 18 deletions(-) diff --git a/go.mod b/go.mod index 032e19c39..28e73f355 100644 --- a/go.mod +++ b/go.mod @@ -17,7 +17,7 @@ require ( github.com/stretchr/testify v1.9.0 go.etcd.io/bbolt v1.3.10 golang.org/x/net v0.25.0 - google.golang.org/grpc v1.63.2 + google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.1 k8s.io/api v0.30.1 k8s.io/apimachinery v0.30.1 @@ -31,7 +31,7 @@ require ( github.com/beorn7/perks v1.0.1 // indirect github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect github.com/cespare/xxhash/v2 v2.2.0 // indirect - github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa // indirect + github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect github.com/emicklei/go-restful/v3 v3.11.0 // indirect @@ -79,16 +79,15 @@ require ( go.uber.org/zap v1.26.0 // indirect golang.org/x/crypto v0.23.0 // indirect golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect - golang.org/x/oauth2 v0.17.0 // indirect + golang.org/x/oauth2 v0.18.0 // indirect golang.org/x/sys v0.20.0 // indirect golang.org/x/term v0.20.0 // indirect golang.org/x/text v0.15.0 // indirect golang.org/x/time v0.3.0 // indirect gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect google.golang.org/appengine v1.6.8 // indirect - google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20240227224415-6ceb2ff114de // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect diff --git a/go.sum b/go.sum index 12babae4e..2931c5ddc 100644 --- a/go.sum +++ b/go.sum @@ -9,8 +9,8 @@ github.com/census-instrumentation/opencensus-proto v0.4.1 h1:iKLQ0xPNFxR/2hzXZMr github.com/census-instrumentation/opencensus-proto v0.4.1/go.mod h1:4T9NM4+4Vw91VeyqjLS6ao50K5bOcLKN6Q42XnYaRYw= github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa h1:jQCWAUqqlij9Pgj2i/PB79y4KOPYVyFYdROxgaCwdTQ= -github.com/cncf/xds/go v0.0.0-20231128003011-0fa0005c9caa/go.mod h1:x/1Gn8zydmfq8dk6e9PdstVsDgu9RuyIIJqAaF//0IM= +github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50 h1:DBmgJDC9dTfkVyGgipamEh2BpGYxScCH1TOF1LL1cXc= +github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50/go.mod h1:5e1+Vvlzido69INQaVO6d87Qn543Xr6nooe9Kz7oBFM= github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -194,8 +194,8 @@ golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac= golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= -golang.org/x/oauth2 v0.17.0 h1:6m3ZPmLEFdVxKKWnKq4VqZ60gutO35zm+zrAHVmHyDQ= -golang.org/x/oauth2 v0.17.0/go.mod h1:OzPDGQiuQMguemayvdylqddI7qcD9lnSDb+1FiwQ5HA= +golang.org/x/oauth2 v0.18.0 h1:09qnuIAgzdx1XplqJvW6CQqMCtGZykZWcXzPMPUusvI= +golang.org/x/oauth2 v0.18.0/go.mod h1:Wf7knwG0MPoWIMMBgFlEaSUDaKskp0dCfrlJRJXbBi8= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -252,14 +252,12 @@ gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY= google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM= google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds= -google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de h1:F6qOa9AZTYJXOUEr4jDysRDLrm4PHePlge4v4TGAlxY= -google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de/go.mod h1:VUhTRKeHn9wwcdrk73nvdC9gF178Tzhmt/qyaFcPLSo= -google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de h1:jFNzHPIeuzhdRwVhbZdiym9q0ory/xY3sA+v2wPg8I0= -google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de/go.mod h1:5iCWqnniDlqZHrd3neWVTOwvh/v6s3232omMecelax8= -google.golang.org/genproto/googleapis/rpc v0.0.0-20240227224415-6ceb2ff114de h1:cZGRis4/ot9uVm639a+rHCUaG0JJHEsdyzSQTMX+suY= -google.golang.org/genproto/googleapis/rpc v0.0.0-20240227224415-6ceb2ff114de/go.mod h1:H4O17MA/PE9BsGx3w+a+W2VOLLD1Qf7oJneAoU6WktY= -google.golang.org/grpc v1.63.2 h1:MUeiw1B2maTVZthpU5xvASfTh3LDbxHd6IJ6QQVU+xM= -google.golang.org/grpc v1.63.2/go.mod h1:WAX/8DgncnokcFUldAxq7GeB5DXHDbMF+lLvDomNkRA= +google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 h1:RFiFrvy37/mpSpdySBDrUdipW/dHwsRwh3J3+A9VgT4= +google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237/go.mod h1:Z5Iiy3jtmioajWHDGFk7CeugTyHtPvMHA4UTmUkyalE= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 h1:NnYq6UN9ReLM9/Y01KWNOWyI5xQ9kbIms5GGJVwS/Yc= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY= +google.golang.org/grpc v1.64.0 h1:KH3VH9y/MgNQg1dE7b3XfVK0GsPSIzJwdF617gUSbvY= +google.golang.org/grpc v1.64.0/go.mod h1:oxjF8E3FBnjp+/gVFYdWacaLDx9na1aqy9oovLpxQYg= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg= From 590873768ced7f97fe2e3f701434b458b352c717 Mon Sep 17 00:00:00 2001 From: Ziv Nevo <79099626+zivnevo@users.noreply.github.com> Date: Tue, 21 May 2024 09:41:57 +0300 Subject: [PATCH 07/53] add local gateway name as attribute (#595) Signed-off-by: Ziv Nevo --- pkg/controlplane/authz/manager.go | 10 +++++++++- tests/e2e/k8s/test_policy.go | 12 ++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index 381b2be7e..6abfa21a9 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -97,6 +97,7 @@ type Manager struct { loadBalancer *LoadBalancer connectivityPDP *connectivitypdp.PDP + peerName string peerTLS *tls.ParsedCertData peerLock sync.RWMutex peerClient map[string]*peer.Client @@ -206,7 +207,7 @@ func (m *Manager) getPodInfoByIP(ip string) *podInfo { func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationRequest) (*egressAuthorizationResponse, error) { m.logger.Infof("Received egress authorization request: %v.", req) - srcAttributes := connectivitypdp.WorkloadAttrs{} + srcAttributes := connectivitypdp.WorkloadAttrs{GatewayNameLabel: m.peerName} podInfo := m.getPodInfoByIP(req.IP) if podInfo != nil { srcAttributes[ServiceNamespaceLabel] = podInfo.namespace @@ -362,6 +363,7 @@ func (m *Manager) authorizeIngress( dstAttributes := connectivitypdp.WorkloadAttrs{ ServiceNameLabel: req.ServiceName.Name, ServiceNamespaceLabel: req.ServiceName.Namespace, + GatewayNameLabel: m.peerName, } decision, err := m.connectivityPDP.Decide(srcAttributes, dstAttributes, req.ServiceName.Namespace) if err != nil { @@ -442,11 +444,17 @@ func NewManager(peerTLS *tls.ParsedCertData, cl client.Client, namespace string) return nil, fmt.Errorf("unable to create JWK verifing key: %w", err) } + dnsNames := peerTLS.DNSNames() + if len(dnsNames) == 0 { + return nil, fmt.Errorf("expected peer certificate to contain at least one DNS name") + } + return &Manager{ client: cl, namespace: namespace, connectivityPDP: connectivitypdp.NewPDP(), loadBalancer: NewLoadBalancer(), + peerName: dnsNames[0], peerTLS: peerTLS, peerClient: make(map[string]*peer.Client), jwkSignKey: jwkSignKey, diff --git a/tests/e2e/k8s/test_policy.go b/tests/e2e/k8s/test_policy.go index f9b0429ac..bd97bbc69 100644 --- a/tests/e2e/k8s/test_policy.go +++ b/tests/e2e/k8s/test_policy.go @@ -40,15 +40,19 @@ func (s *TestSuite) TestPolicyLabels() { // 1. Create a policy that allows traffic only to the echo service at cl[0] - apply in cl[1] (on egress) // In addition, create a policy to only allow traffic from cl[1] - apply in cl[0] (on ingress) allowEchoPolicyName := "allow-access-to-echo-svc" - dstLabels := map[string]string{ + srcLabels := map[string]string{ // allow traffic only from cl1 + authz.GatewayNameLabel: cl[1].Name(), + } + dstLabels := map[string]string{ // allow traffic only to echo in cl1 authz.ServiceNameLabel: httpEchoService.Name, authz.GatewayNameLabel: cl[0].Name(), } - allowEchoPolicy := util.NewPolicy(allowEchoPolicyName, v1alpha1.AccessPolicyActionAllow, nil, dstLabels) + allowEchoPolicy := util.NewPolicy(allowEchoPolicyName, v1alpha1.AccessPolicyActionAllow, srcLabels, dstLabels) require.Nil(s.T(), cl[1].CreatePolicy(allowEchoPolicy)) - srcLabels := map[string]string{authz.GatewayNameLabel: cl[1].Name()} - specificSrcPeerPolicy := util.NewPolicy("specific-peer", v1alpha1.AccessPolicyActionAllow, srcLabels, nil) + srcLabels = map[string]string{authz.GatewayNameLabel: cl[1].Name()} // allow traffic only from cl1 + dstLabels = map[string]string{authz.GatewayNameLabel: cl[0].Name()} // allow traffic only to cl0 + specificSrcPeerPolicy := util.NewPolicy("specific-peer", v1alpha1.AccessPolicyActionAllow, srcLabels, dstLabels) require.Nil(s.T(), cl[0].CreatePolicy(specificSrcPeerPolicy)) data, err := cl[1].AccessService(httpecho.GetEchoValue, importedService, true, nil) From 59220f1e2a32dd854e5ecaf3b093a350294cb962 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Tue, 21 May 2024 04:22:07 -0400 Subject: [PATCH 08/53] Update index.md (#589) Signed-off-by: welisheva22 Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- .../en/blog/clusterlink-intro/index.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/website/content/en/blog/clusterlink-intro/index.md b/website/content/en/blog/clusterlink-intro/index.md index c3cf1a902..9b27f9cc8 100644 --- a/website/content/en/blog/clusterlink-intro/index.md +++ b/website/content/en/blog/clusterlink-intro/index.md @@ -10,7 +10,7 @@ type: blog Deploying microservice applications across multiple clusters offers many benefits. These include, for example, improved fault tolerance, increased scalability, performance - or regulatory compliance. In addition, use of multiple locations may be required to + and regulatory compliance. In addition, use of multiple locations may be required to access specialized hardware, services or data sources available only in those locations. Kubernetes SIG multicluster [lists][SIG-MC] additional reasons for using multiple clusters. @@ -28,13 +28,13 @@ In order to facilitate cross cluster communications, you could use Kubernetes as [Istio][], [Skupper][] and [Submariner][]. By and large, these solutions attempt to conjoin the multiple clusters, flattening the isolated networks into a single flat "mesh" shared between the connected clusters. - The goal is, to the extent possible, extend the Kubernetes single cluster network - abstraction to multicluster use cases. + The goal is, to the extent possible, to extend the Kubernetes single cluster network + abstraction into multicluster use cases. The creation of a shared mesh is not always desirable and may place additional constraints on administrators, developers and the workloads they manage. For example, - it might make assumptions on objects, such as Services, being the "same" based - on the objects sharing a name (i.e., namespace sameness as defined [by Istio][] + it might make assumptions that Kubernetes objects, such as Services, are the "same" based + on the objects sharing a name (i.e., "namespace sameness" as defined [by Istio][] and [by SIG-MC][]), or require planning IP addresses assignment across independent clusters. This post introduces [ClusterLink][], an [open source][] project that @@ -71,7 +71,7 @@ ClusterLink consists of several main components that work together to securely ClusterLink uses the Kubernetes API servers for its configuration store. The **control plane** is responsible for watching for changes in relevant built-in - and custom resources and configuring the data plane Pod using [Envoy's xDS][] protocol. + and custom resources and configuring the data plane Pods using [Envoy's xDS][] protocol. The control plane is also responsible for managing local Kubernetes services and endpoints corresponding to imported remote services. By using standard Kubernetes services, ClusterLink integrates seamlessly with the Kubernetes @@ -82,7 +82,7 @@ The local service endpoints refer to **data plane** Pods, responsible for workload-to-service secure tunnels to other clusters. The data plane uses [HTTP CONNECT][] with [mutual TLS][] for security. The use of HTTPS over tcp/443 removes the need for VPNs and special firewall - configurations. Certificate based mTLS guarantees in-transit data + configurations. Certificate-based mTLS guarantees in-transit data encryption and limits allowed connections to other fabric peers only. In addition, all data plane connections between clusters are explicitly approved by the control plane and must pass independent egress and ingress access policies @@ -96,18 +96,18 @@ In addition to the typical multicluster networking use cases, such as significant benefits which are not well served by other solutions. Specifically, ClusterLink can address requirements of use cases where: -- **clusters are aligned with organizational units** and sharing *internal* +- **clusters are aligned with organizational units** - sharing *internal* microservices with other clusters and namespaces should be limited to those belonging to the same unit, while also communicating with *exposed* services belonging to other units. - services are owned by **different administrative domains** (e.g., different - development teams) and thus judicious sharing and more stringent access + development teams) - thus judicious sharing and more stringent access controls across clusters are needed. - it is desirable to **increase scalability and limit information sharing** by - minimizing information exchanged between clusters. With ClusterLink, each + minimizing information exchanged between clusters - with ClusterLink, each cluster manages its own naming and load balancing, requiring considerable less cross-cluster metadata for its communication. -- there is a need for **separation of concerns** between network administrators +- there is a need for **separation of concerns** - that is, between network administrators and application owners. ## Getting started with ClusterLink From 5f5d151a342d1373a45ccc9f06273e20b5622623 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Tue, 21 May 2024 05:04:00 -0400 Subject: [PATCH 09/53] Documentation tweaks (#587) Signed-off-by: welisheva22 Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- .../en/blog/clusterlink-intro/index.md | 2 +- .../content/en/docs/main/concepts/fabric.md | 2 +- .../content/en/docs/main/concepts/peers.md | 8 ++--- .../content/en/docs/main/concepts/policies.md | 18 +++++----- .../en/docs/main/doc-contribution/_index.md | 2 +- .../en/docs/main/getting-started/_index.md | 4 +-- .../docs/main/getting-started/developers.md | 2 +- .../en/docs/main/getting-started/users.md | 20 ++++++----- .../content/en/docs/main/overview/_index.md | 4 +-- website/content/en/docs/main/tasks/_index.md | 2 +- .../content/en/docs/main/tasks/operator.md | 18 +++++----- .../en/docs/main/tutorials/bookinfo/index.md | 33 +++++++++---------- .../en/docs/main/tutorials/iperf/index.md | 6 ++-- .../docs/v0.1/getting-started/developers.md | 4 +-- .../docs/v0.2/getting-started/developers.md | 4 +-- 15 files changed, 66 insertions(+), 63 deletions(-) diff --git a/website/content/en/blog/clusterlink-intro/index.md b/website/content/en/blog/clusterlink-intro/index.md index 9b27f9cc8..2d27b8715 100644 --- a/website/content/en/blog/clusterlink-intro/index.md +++ b/website/content/en/blog/clusterlink-intro/index.md @@ -57,7 +57,7 @@ ClusterLink offers a secure and performant solution to interconnect services - **Export**/**Import**: services must be explicitly shared by clusters before they can be used. A service can be imported by any number of peers. To increase availability or performance, a service can be exported by more than one peer. -- **Access policies**: ClusterLink supports fine grained segmentation with a +- **Access policies**: ClusterLink supports fine-grained segmentation with a "default deny" policy, adhering to "zero trust" principles. Access policies are used to explicitly allow and deny communications. Affected workloads are defined in terms of their attributes (such as location, environment, namespace or even diff --git a/website/content/en/docs/main/concepts/fabric.md b/website/content/en/docs/main/concepts/fabric.md index 881a8878d..80f1df002 100644 --- a/website/content/en/docs/main/concepts/fabric.md +++ b/website/content/en/docs/main/concepts/fabric.md @@ -44,7 +44,7 @@ This command will create the CA files `cert.pem` and `key.pem` in a directory na ## Related tasks Once a Fabric has been created and initialized, you can proceed with configuring - [peers][]. For a complete, end to end, use case please refer to the + [peers][]. For a complete, end-to-end use case, please refer to the [iperf tutorial][]. [peers]: {{< relref "peers" >}} diff --git a/website/content/en/docs/main/concepts/peers.md b/website/content/en/docs/main/concepts/peers.md index ff1767f88..81595f047 100644 --- a/website/content/en/docs/main/concepts/peers.md +++ b/website/content/en/docs/main/concepts/peers.md @@ -9,15 +9,15 @@ A *Peer* represents a location, such as a Kubernetes cluster, participating in a that it may wish to share with other peers. A peer is managed by a peer administrator, which is responsible for running the ClusterLink control and data planes. The administrator will typically deploy the ClusterLink components by configuring - the [deployment Custom Resource (CR)][operator-cr]. The administrator may also wish - to provide coarse-grained access policies (and often do) in accordance with high level corporate + the [Deployment Custom Resource (CR)][operator-cr]. They may also wish to define + coarse-grained access policies, in accordance with high level corporate policies (e.g., "production peers should only communicate with other production peers"). Once a peer has been added to a fabric, it can communicate with any other peer belonging to it. All configuration relating to service sharing (e.g., the exporting and importing of services, and the setting of fine grained application policies) can be done with lowered privileges (e.g., by users, such as application owners). Remote peers are - represented by peer Custom Resources (CRs). Each Peer CR instance + represented by the Peer Custom Resources (CRs). Each peer CR instance defines a remote cluster and the network endpoints of its ClusterLink gateways. ## Prerequisites @@ -187,7 +187,7 @@ There are two fundamental attributes in the peer CRD: the peer name and the list Gateway endpoint would typically be implemented via a `NodePort` or `LoadBalancer` K8s service. A `NodePort` service would typically be used in local deployments (e.g., when running in kind clusters during development) and a `LoadBalancer` service - would be used in cloud based deployments. These can be automatically configured and + would be used in cloud-based deployments. These can be automatically configured and created via the [ClusterLink CR][]. The peer's status section includes a `Reachable` condition indicating whether the peer is currently reachable, and in case it is not reachable, the last time it was. diff --git a/website/content/en/docs/main/concepts/policies.md b/website/content/en/docs/main/concepts/policies.md index c9297476d..e7e03816d 100644 --- a/website/content/en/docs/main/concepts/policies.md +++ b/website/content/en/docs/main/concepts/policies.md @@ -7,14 +7,14 @@ weight: 40 Access policies allow users and administrators fine-grained control over which client workloads may access which service. This is an important security mechanism for applying [micro-segmentation][], which is a basic requirement of [zero-trust][] - systems. Another zero-trust principle, "Deny by default / Allow by exception", is also - addressed by ClusterLink's access policies: a connection without an explicit policy allowing it, + systems. Another zero-trust principle, "Deny by default / Allow by exception," is also + addressed by ClusterLink's access policies: a connection without an explicit policy allowing it will be dropped. Access policies can also be used for enforcing corporate security rules, as well as segmenting the fabric into trust zones. ClusterLink's access policies are based on attributes that are attached to [peers][], [services][] and client workloads. - Each attribute is a key:value pair, similar to how [labels][] + Each attribute is a key/value pair, similar to how [labels][] are used in Kubernetes. This approach, called ABAC (Attribute Based Access Control), allows referring to a set of entities in a single policy, rather than listing individual entity names. Using attributes is safer, more resilient to changes, and easier to @@ -28,16 +28,16 @@ Destinations are defined in terms of the attributes attached to the target servi Both client workloads and target services may inherit some attributes from their hosting peer. There are two tiers of access policies in ClusterLink. The high-priority tier - is intended for cluster/Peer administrators to set access rules which cannot be + is intended for cluster/peer administrators to set access rules which cannot be overridden by cluster users. High-priority policies are controlled by the - `PrivilegedAccessPolicy` CRD, and are cluster-scoped (i.e., have no namespace). + `PrivilegedAccessPolicy` CRD, and are cluster scoped (i.e., have no namespace). Regular policies are intended for cluster users, such as application developers and owners, and are controlled by the `AccessPolicy` CRD. Regular policies are namespaced, and have an effect in their namespace only. That is, they do not affect connections to/from other namespaces. For a connection to be established, both the ClusterLink gateway on the client - side and the ClusterLink gateway on the Service side must allow the connection. + side and the ClusterLink gateway on the service side must allow the connection. Each gateway (independently) follows these steps to decide if the connection is allowed: 1. All instances of `PrivilegedAccessPolicy` in the cluster with `deny` action are considered. @@ -65,7 +65,7 @@ Recall that a connection is dropped if it does not match any access policy. Hence, for a connection to be allowed, an access policy with an `allow` action must be created on both sides of the connection. Creating an access policy is accomplished by creating an `AccessPolicy` CR in - the relevant namespace (see Note above). + the relevant namespace (see note above). Creating a high-priority access policy is accomplished by creating a `PrivilegedAccessPolicy` CR. Instances of `PrivilegedAccessPolicy` have no namespace and affect the entire cluster. @@ -114,9 +114,9 @@ The `AccessPolicySpec` defines the following fields: - **Action** (string, required): whether the policy allows or denies the specified connection. Value must be either `allow` or `deny`. - **From** (WorkloadSetOrSelector array, required): specifies connection sources. - A connection's source must match one of the specified sources to be matched by the policy + A connection's source must match one of the specified sources to be matched by the policy. - **To** (WorkloadSetOrSelectorList array, required): specifies connection destinations. - A connection's destination must match one of the specified destinations to be matched by the policy + A connection's destination must match one of the specified destinations to be matched by the policy. A `WorkloadSetOrSelector` object has two fields; exactly one of them must be specified. diff --git a/website/content/en/docs/main/doc-contribution/_index.md b/website/content/en/docs/main/doc-contribution/_index.md index 91d4a6e09..b59dba19f 100644 --- a/website/content/en/docs/main/doc-contribution/_index.md +++ b/website/content/en/docs/main/doc-contribution/_index.md @@ -8,7 +8,7 @@ We use [Hugo][] to format and generate our [website][], the [Docsy][] theme for styling and site structure, and [Netlify][] to manage the deployment of the site. Hugo is an open-source static site generator that provides us with templates, content organization in a standard directory structure, and a website generation - engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps + engine. We write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website. All submissions, including submissions by project members, require review. We diff --git a/website/content/en/docs/main/getting-started/_index.md b/website/content/en/docs/main/getting-started/_index.md index d49d66781..cb828f264 100644 --- a/website/content/en/docs/main/getting-started/_index.md +++ b/website/content/en/docs/main/getting-started/_index.md @@ -4,9 +4,9 @@ description: Getting started guides for users and developers weight: 20 --- -The following provide quick start guides for [users][] and [developers][]. +The following sections provide quick start guides for [users][] and [developers][]. -If you're a content author, wishing to contribute additional documentation or guides, +If you're a content author who wishes to contribute additional documentation or guides, please refer to the [contribution guidelines][]. [users]: {{< relref "users" >}} diff --git a/website/content/en/docs/main/getting-started/developers.md b/website/content/en/docs/main/getting-started/developers.md index 729bc8e28..3722293d5 100644 --- a/website/content/en/docs/main/getting-started/developers.md +++ b/website/content/en/docs/main/getting-started/developers.md @@ -49,7 +49,7 @@ $ go test -v ./tests/e2e/k8s -testify.m TestConnectivity ### Tests in CICD All pull requests undergo automated testing before being merged. This includes, for example, - linting, end-to-end tests and DCO validation. Logs in CICD default to `info` lavel, and + linting, end-to-end tests and DCO validation. Logs in CICD default to `info` level, and can be increased to `debug` by setting environment variable `DEBUG=1`. You can also enable debug logging from the UI when re-running a CICD job, by selecting "enable debug logging". diff --git a/website/content/en/docs/main/getting-started/users.md b/website/content/en/docs/main/getting-started/users.md index 99318f7cc..c83833a2a 100644 --- a/website/content/en/docs/main/getting-started/users.md +++ b/website/content/en/docs/main/getting-started/users.md @@ -4,7 +4,7 @@ description: Installing and configuring a basic ClusterLink deployment weight: 22 --- -This guide will give you a quick start on installing and setting up the ClusterLink on a Kubernetes cluster. +This guide will give you a quick start on installing and setting up ClusterLink on a Kubernetes cluster. ## Prerequisites @@ -49,11 +49,9 @@ To set up ClusterLink on a Kubernetes cluster, follow these steps: This command will create the certificate files `cert.pem` and `key.pem` in a directory named ``/``. The `--path ` flag can be used to change the directory location. - The `--name` option is optional, and by default, "default_fabric" will be used. + Here too, the `--name` option is optional, and by default, "default_fabric" will be used. -{{< notice note >}} -All the peer certificates in the fabric should be created from the same fabric CA files in step 1. -{{< /notice >}} +**All the peer certificates in the fabric should be created from the same fabric CA files in step 1.** 1. {{< anchor install-cl-operator >}}Install ClusterLink deployment: @@ -79,7 +77,7 @@ Deploy ClusterLink in a console with access to the cluster (step 3). ## Try it out -Check out the [ClusterLink Tutorials][] for setting up multi-cluster connectivity +Check out the [ClusterLink tutorials][] for setting up multi-cluster connectivity for applications using two or more clusters. ## Uninstall ClusterLink @@ -108,6 +106,12 @@ This command using the current `kubectl` context. rm `which clusterlink` ``` -[kind]: https://kind.sigs.k8s.io/) -[ClusterLink deployment operator]: {{< relref "../tasks/operator" >}} +## Links for further information + +* [Kind](https://kind.sigs.k8s.io/) +* [ClusterLink deployment operator][] +* [ClusterLink tutorials][] + +[Kind]: https://kind.sigs.k8s.io/docs/user/quick-start/ +[ClusterLink deployment operator]: {{< relref "../tasks/operator/" >}} [ClusterLink tutorials]: {{< relref "../tutorials/" >}} diff --git a/website/content/en/docs/main/overview/_index.md b/website/content/en/docs/main/overview/_index.md index 8721a9205..b2fc22859 100644 --- a/website/content/en/docs/main/overview/_index.md +++ b/website/content/en/docs/main/overview/_index.md @@ -37,10 +37,10 @@ ClusterLink uses a set of unprivileged gateways serving connections to and from ## Why is it unique? -The distributed control plane and the fine grained connection establishment control are the main +The distributed control plane and the fine-grained connection establishment control are the main advantages of ClusterLink over some of its competitors. Performance evaluation on clusters deployed in the same Google Cloud zone shows that ClusterLink can outperform some existing solutions by almost 2× while providing - fine grained authorization on a per connection basis. + fine-grained authorization on a per connection basis. ## Where should I go next? diff --git a/website/content/en/docs/main/tasks/_index.md b/website/content/en/docs/main/tasks/_index.md index 9355d1b07..0271126bf 100644 --- a/website/content/en/docs/main/tasks/_index.md +++ b/website/content/en/docs/main/tasks/_index.md @@ -1,5 +1,5 @@ --- title: Tasks -description: How to do single specific targeted activities with ClusterLink. +description: How to do single specific targeted activities with ClusterLink weight: 35 --- diff --git a/website/content/en/docs/main/tasks/operator.md b/website/content/en/docs/main/tasks/operator.md index 3ae53e41e..637ac7d80 100644 --- a/website/content/en/docs/main/tasks/operator.md +++ b/website/content/en/docs/main/tasks/operator.md @@ -1,6 +1,6 @@ --- title: Deployment Operator -description: Usage and configuration of the ClusterLink deployment operator. +description: Usage and configuration of the ClusterLink deployment operator weight: 50 --- @@ -12,14 +12,14 @@ Detailed instructions for creating these peer certificates can be found in the [ ## The common use case -The common use case for deploying ClusterLink on a cloud based K8s cluster (i.e., EKS, GKE, IKS, etc.) is using the CLI command: +The common use case for deploying ClusterLink on a cloud-based K8s cluster (i.e., EKS, GKE, IKS, etc.) is using the CLI command: ```sh clusterlink deploy peer --name --fabric ``` The command assumes that `kubectl` is configured to access the correct peer (K8s cluster) -and that certificates files are placed in the current working directory. +and that certificate files are placed in the current working directory. If they are not, use the flag `--path ` to reference the directory where certificate files are stored. The command deploys the ClusterLink operator in the `clusterlink-operator` namespace and converts the peer certificates to secrets in the `clusterlink-system` namespace, where ClusterLink components will be installed. @@ -103,7 +103,7 @@ The `deploy peer` {{< anchor commandline-flags >}} command has the following fla By default, it uses port `443` for the `LoadBalancer` ingress type. For the `NodePort` ingress type, the port number will be allocated by Kubernetes. In case the user changes the default value, it is the user's responsibility to ensure the port number is valid and available for use. - - **ingress-annotations:** This field add annotations to the ingress service. + - **ingress-annotations:** This field adds annotations to the ingress service. The flag can be repeated to add several annotations. For example: `--ingress-annotations load-balancer-type=nlb --ingress-annotations load-balancer-name=cl-nlb`. - **log-level:** This field determines the severity log level for all the components (controlplane and dataplane). By default, it uses `info` log level. @@ -113,12 +113,12 @@ The `deploy peer` {{< anchor commandline-flags >}} command has the following fla 2. General deployment flags: - **start:** Determines which components to deploy and start in the cluster. - `all` (defualt) starts the clusterlink operator, converts the peer certificates to secrets, + `all` (default) starts the clusterlink operator, converts the peer certificates to secrets, and deploys the operator ClusterLink custom resource to create the ClusterLink components. - `operator`, deploys only the `ClusterLink` operator and convert the peer certificates to secrets. - `none`, doesn't deploy anything but creates ClusterLink custom resource YAML. - - **path**: represents the path where the peer and fabric certificates are stored, - By default is the working current working directory. + `operator` deploys only the `ClusterLink` operator and convert the peer certificates to secrets. + `none` doesn't deploy anything but creates the ClusterLink custom resource YAML. + - **path**: Represents the path where the peer and fabric certificates are stored, + by default is the working current working directory. ## Manual Deployment without CLI diff --git a/website/content/en/docs/main/tutorials/bookinfo/index.md b/website/content/en/docs/main/tutorials/bookinfo/index.md index a2c34e75e..2f00fa50a 100644 --- a/website/content/en/docs/main/tutorials/bookinfo/index.md +++ b/website/content/en/docs/main/tutorials/bookinfo/index.md @@ -1,18 +1,18 @@ --- title: BookInfo -description: Running BookInfo application with different policies. +description: Running BookInfo application with different policies --- -The tutorial sets up [Istio BookInfo application][] in different clusters. +The tutorial sets up the [Istio BookInfo application][] in different clusters. The tutorial demonstrates the use of AccessPolicy and PrivilegedAccessPolicy custom resources. -The tutorial shows different load-balancing policies like: random, round-robin or static destination. +The tutorial shows different load-balancing policies like: random, round robin or static destination. For more details, see the [policies documentation][]. This test creates three kind clusters: -* Two Product-Page microservice (application frontend) and details microservice run on the first cluster. -* The Reviews-V2 (display rating with black stars) and Rating microservices run on the second cluster. -* The Reviews-V3 (display rating with red stars) and Rating microservices run on the third cluster. +* Two productpage microservices (application frontend) and a details microservice run on the first cluster. +* The reviews-v2 (display rating with black stars) and rating microservices run on the second cluster. +* The reviews-v3 (display rating with red stars) and rating microservices run on the third cluster. System illustration: @@ -38,14 +38,14 @@ In this tutorial we set up a local environment using [kind][]. To setup three kind clusters: -1. Install kind using [kind installation guide][]. +1. Install kind using the [kind installation guide][]. 2. Create a directory for all the tutorial files: ```sh mkdir bookinfo-tutorial && cd bookinfo-tutorial ``` -3. create three kind clusters: +3. Create three kind clusters: ```sh kind create cluster --name=client @@ -75,7 +75,6 @@ kubectl apply -f $BOOKINFO_FILES/review/rating.yaml kubectl config use-context kind-server2 kubectl apply -f $BOOKINFO_FILES/review/review-v3.yaml kubectl apply -f $BOOKINFO_FILES/review/rating.yaml - ``` ## Deploy ClusterLink @@ -106,7 +105,7 @@ kubectl apply -f $BOOKINFO_FILES/review/rating.yaml which is more suitable for Kubernetes clusters running in the cloud. {{< /notice >}} -2. Verify that ClusterLink control and data plane components are running: +2. Verify that the ClusterLink control and data plane components are running. It may take a few seconds for the deployments to be successfully created. @@ -127,7 +126,7 @@ kubectl apply -f $BOOKINFO_FILES/review/rating.yaml ## Enable cross-cluster access In this step, we enable connectivity access for the BookInfo application - by connecting the ProductPage service (client) to the reviews-v2 service (server1) + by connecting the productpage service (client) to the reviews-v2 service (server1) and reviews-v3 (server2). We establish connections between the peers, export the reviews service on the server side, import the reviews service on the client side, and create a policy to allow the connection. @@ -169,7 +168,7 @@ brew link --force gettext ## BookInfo test -To run the BookInfo application use a Firefox web browser to connect the ProductPage microservice: +To run the BookInfo application use a Firefox web browser to connect the productpage microservice: ```sh kubectl config use-context kind-client @@ -178,14 +177,14 @@ To run the BookInfo application use a Firefox web browser to connect the Product ``` {{< notice note >}} -By default, a round-robin policy is set. +By default, a round robin policy is set. {{< /notice >}} ## Apply privileged access policy In the previous steps, an unprivileged access policy was set to allow connectivity. -To enforce high-priority policy use the `PrivilegedAccessPolicy` CRD. -In this example, we enforce that the ProductPage service can access only reviews-v3 from server2, +To enforce high-priority policy use the `PrivilegedAccessPolicy` CR. +In this example, we enforce that the productpage service can access only reviews-v3 from server2, and deny all services from server1: {{< tabpane text=true >}} @@ -335,9 +334,9 @@ spec: {{% /tab %}} {{< /tabpane >}} -## Apply round-robin load-balancing policy +## Apply round robin load-balancing policy -To apply a round-robin load-balancing policy (which is used by default) to the connection to reviews import: +To apply a round robin load-balancing policy (which is used by default) to the connection to reviews import: {{< tabpane text=true >}} {{% tab header="File" %}} diff --git a/website/content/en/docs/main/tutorials/iperf/index.md b/website/content/en/docs/main/tutorials/iperf/index.md index ebcf6e50f..99aae9890 100644 --- a/website/content/en/docs/main/tutorials/iperf/index.md +++ b/website/content/en/docs/main/tutorials/iperf/index.md @@ -1,9 +1,9 @@ --- title: iPerf3 -description: Running basic connectivity between iPerf3 applications across two sites using ClusterLink. +description: Running basic connectivity between iPerf3 applications across two sites using ClusterLink --- -In this tutorial we'll establish iPerf3 connectivity between two kind cluster using ClusterLink. +In this tutorial we'll establish iPerf3 connectivity between two kind clusters using ClusterLink. The tutorial uses two kind clusters: 1) Client cluster - runs ClusterLink along with an iPerf3 client. @@ -463,4 +463,4 @@ iperf Done. [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start [core concepts]: {{< relref "../../concepts/_index.md" >}} -[policies]: {{< relref "../../concepts/policies/_index.md" >}} \ No newline at end of file +[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/content/en/docs/v0.1/getting-started/developers.md b/website/content/en/docs/v0.1/getting-started/developers.md index d4051d2c2..0350b10db 100644 --- a/website/content/en/docs/v0.1/getting-started/developers.md +++ b/website/content/en/docs/v0.1/getting-started/developers.md @@ -29,7 +29,7 @@ Here are the key steps for setting up your developer environment, making a chang [contribution guide](https://github.com/clusterlink-net/clusterlink/blob/main/CONTRIBUTING.md). - We follow [GitHub's Standard Fork & Pull Request Workflow](https://gist.github.com/Chaser324/ce0505fbed06b947d962) -All contributed code should should pass precommit checks such as linting and tests. These +All contributed code should pass precommit checks such as linting and tests. These are run automatically as part of the CI process on every pull request. You may wish to run these locally, before initiating a PR: @@ -49,7 +49,7 @@ $ go test -v ./tests/e2e/k8s -testify.m TestConnectivity ### Tests in CICD All pull requests undergo automated testing before being merged. This includes, for example, - linting, end-to-end tests and DCO validation. Logs in CICD default to `info` lavel, and + linting, end-to-end tests and DCO validation. Logs in CICD default to `info` level, and can be increased to `debug` by setting environment variable `DEBUG=1`. You can also enable debug logging from the UI when re-running a CICD job, by selecting "enable debug logging". diff --git a/website/content/en/docs/v0.2/getting-started/developers.md b/website/content/en/docs/v0.2/getting-started/developers.md index 8f595e92b..a1e81950f 100644 --- a/website/content/en/docs/v0.2/getting-started/developers.md +++ b/website/content/en/docs/v0.2/getting-started/developers.md @@ -29,7 +29,7 @@ Here are the key steps for setting up your developer environment, making a chang [contribution guide](https://github.com/clusterlink-net/clusterlink/blob/main/CONTRIBUTING.md). - We follow [GitHub's Standard Fork & Pull Request Workflow](https://gist.github.com/Chaser324/ce0505fbed06b947d962) -All contributed code should should pass precommit checks such as linting and tests. These +All contributed code should pass precommit checks such as linting and tests. These are run automatically as part of the CI process on every pull request. You may wish to run these locally, before initiating a PR: @@ -49,7 +49,7 @@ $ go test -v ./tests/e2e/k8s -testify.m TestConnectivity ### Tests in CICD All pull requests undergo automated testing before being merged. This includes, for example, - linting, end-to-end tests and DCO validation. Logs in CICD default to `info` lavel, and + linting, end-to-end tests and DCO validation. Logs in CICD default to `info` level, and can be increased to `debug` by setting environment variable `DEBUG=1`. You can also enable debug logging from the UI when re-running a CICD job, by selecting "enable debug logging". From 638ef0d7d265dd81e96934e84cb9a915b2c1b48c Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Tue, 21 May 2024 14:29:44 +0300 Subject: [PATCH 10/53] Update linkcheck.yml to only run on main repo, not on lforks (#590) Signed-off-by: Etai Lev Ran --- .github/workflows/linkcheck.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/linkcheck.yml b/.github/workflows/linkcheck.yml index e5036fc83..c136eb286 100644 --- a/.github/workflows/linkcheck.yml +++ b/.github/workflows/linkcheck.yml @@ -7,6 +7,7 @@ on: jobs: broken-link-checker: + if: github.repository_owner == 'clusterlink-net' # do not run on forks name: Check broken links runs-on: ubuntu-latest steps: @@ -14,4 +15,4 @@ jobs: uses: ruzickap/action-my-broken-link-checker@v2 with: url: https://clusterlink.net - cmd_params: '--buffer-size=65536 --max-connections=16 --rate-limit=16 --timeout=20' # muffet parameters + cmd_params: '--buffer-size=65536 --max-connections=2 --rate-limit=4 --timeout=20' # muffet parameters From f26011426d9804dbc29366b6cd4e6970734805e6 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Tue, 21 May 2024 12:02:43 -0400 Subject: [PATCH 11/53] One more commit - and have the other commits been applied? If not, what to do? (#598) Signed-off-by: welisheva22 Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- website/content/en/docs/main/getting-started/users.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/en/docs/main/getting-started/users.md b/website/content/en/docs/main/getting-started/users.md index c83833a2a..a378ec561 100644 --- a/website/content/en/docs/main/getting-started/users.md +++ b/website/content/en/docs/main/getting-started/users.md @@ -62,7 +62,7 @@ To set up ClusterLink on a Kubernetes cluster, follow these steps: This command will deploy the ClusterLink operator on the `clusterlink-operator` namespace and convert the peer certificates to secrets in the namespace where ClusterLink components will be installed. By default, the `clusterlink-system` namespace is used. - in addition it will create a ClusterLink instance custom resource object and deploy it to the operator. + In addition, it will create a ClusterLink instance custom resource object and deploy it to the operator. The operator will then create the ClusterLink components in the `clusterlink-system` namespace and enable ClusterLink in the cluster. The command assumes that `kubectl` is set to the correct peer (K8s cluster) and that the certificates were created by running the previous command on the same working directory. From 20a682fe2a06adf008a7bd4011b4c932546a49f5 Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Tue, 21 May 2024 15:38:20 +0300 Subject: [PATCH 12/53] remove controlplane CRUD mode This commit removes the legacy CRUD-based controlplane (i.e. the non-CRD mode). Signed-off-by: Or Ozeri --- .github/dependabot.yml | 4 - Makefile | 4 - cmd/cl-controlplane/app/server.go | 79 +-- cmd/clusterlink/cmd/create/create_peer.go | 22 - cmd/clusterlink/cmd/deploy/deploy_peer.go | 12 - cmd/clusterlink/config/config.go | 15 - cmd/gwctl/Dockerfile | 8 - cmd/gwctl/config/config.go | 239 --------- cmd/gwctl/main.go | 113 ----- cmd/gwctl/subcommand/config.go | 112 ---- cmd/gwctl/subcommand/export.go | 216 -------- cmd/gwctl/subcommand/gwctl.go | 187 ------- cmd/gwctl/subcommand/import.go | 236 --------- cmd/gwctl/subcommand/metrics.go | 69 --- cmd/gwctl/subcommand/peer.go | 215 -------- cmd/gwctl/subcommand/policy.go | 210 -------- cmd/util/util.go | 31 -- demos/bookinfo/kind/test.py | 8 +- demos/bookinfo/test.py | 8 +- demos/iperf3/kind/README.md | 190 ------- .../manifests/iperf3-client/iperf3-svc.yaml | 18 - demos/qotd/kind/test.py | 10 +- demos/speedtest/kind/README.md | 6 +- demos/utils/clusterlink.py | 3 +- docs/installation.md | 27 +- go.mod | 1 - go.sum | 4 - hack/install_clusterlink.sh | 2 +- pkg/bootstrap/cert.go | 14 - pkg/bootstrap/platform/config.go | 4 - pkg/bootstrap/platform/k8s.go | 100 +--- pkg/client/client.go | 84 --- pkg/controlplane/authz/controllers.go | 140 +++-- pkg/controlplane/authz/manager.go | 57 +-- pkg/controlplane/control/controllers.go | 106 ++-- pkg/controlplane/control/manager.go | 95 +--- pkg/controlplane/control/peer.go | 22 +- pkg/controlplane/eventmanager/events.go | 49 -- pkg/controlplane/rest/accesspolicy.go | 163 ------ pkg/controlplane/rest/export.go | 207 -------- pkg/controlplane/rest/import.go | 245 --------- pkg/controlplane/rest/manager.go | 134 ----- pkg/controlplane/rest/peer.go | 210 -------- pkg/controlplane/rest/server.go | 45 -- pkg/controlplane/store/accesspolicies.go | 165 ------ pkg/controlplane/store/exports.go | 166 ------ pkg/controlplane/store/imports.go | 165 ------ pkg/controlplane/store/peers.go | 165 ------ pkg/controlplane/store/types.go | 111 ---- pkg/controlplane/xds/manager.go | 7 +- pkg/k8s/kubernetes/kube.go | 23 - pkg/k8s/kubernetes/kubeinformer.go | 477 ------------------ pkg/metrics/metrics.go | 90 ---- .../controller/instance_controller.go | 2 +- pkg/store/kv/bolt/store.go | 125 ----- pkg/store/kv/manager.go | 31 -- pkg/store/kv/store.go | 138 ----- pkg/store/kv/types.go | 44 -- pkg/store/types.go | 48 -- pkg/util/http/server.go | 4 +- pkg/util/net/util.go | 39 -- pkg/util/rest/client.go | 163 ------ pkg/util/rest/server.go | 289 ----------- tests/e2e/k8s/test_basic.go | 404 +-------------- tests/e2e/k8s/test_import.go | 214 ++++---- tests/e2e/k8s/test_operator.go | 1 - tests/e2e/k8s/util/clusterlink.go | 228 +-------- tests/e2e/k8s/util/fabric.go | 45 -- tests/e2e/k8s/util/k8s_yaml.go | 92 ---- tests/k8s.sh | 102 ---- .../content/en/docs/main/tasks/operator.md | 1 - .../content/en/docs/v0.2/tasks/operator.md | 1 - 72 files changed, 311 insertions(+), 6753 deletions(-) delete mode 100644 cmd/gwctl/Dockerfile delete mode 100644 cmd/gwctl/config/config.go delete mode 100644 cmd/gwctl/main.go delete mode 100644 cmd/gwctl/subcommand/config.go delete mode 100644 cmd/gwctl/subcommand/export.go delete mode 100644 cmd/gwctl/subcommand/gwctl.go delete mode 100644 cmd/gwctl/subcommand/import.go delete mode 100644 cmd/gwctl/subcommand/metrics.go delete mode 100644 cmd/gwctl/subcommand/peer.go delete mode 100644 cmd/gwctl/subcommand/policy.go delete mode 100644 cmd/util/util.go delete mode 100644 demos/iperf3/kind/README.md delete mode 100755 demos/iperf3/testdata/manifests/iperf3-client/iperf3-svc.yaml delete mode 100644 pkg/client/client.go delete mode 100644 pkg/controlplane/eventmanager/events.go delete mode 100644 pkg/controlplane/rest/accesspolicy.go delete mode 100644 pkg/controlplane/rest/export.go delete mode 100644 pkg/controlplane/rest/import.go delete mode 100644 pkg/controlplane/rest/manager.go delete mode 100644 pkg/controlplane/rest/peer.go delete mode 100644 pkg/controlplane/rest/server.go delete mode 100644 pkg/controlplane/store/accesspolicies.go delete mode 100644 pkg/controlplane/store/exports.go delete mode 100644 pkg/controlplane/store/imports.go delete mode 100644 pkg/controlplane/store/peers.go delete mode 100644 pkg/controlplane/store/types.go delete mode 100644 pkg/k8s/kubernetes/kube.go delete mode 100644 pkg/k8s/kubernetes/kubeinformer.go delete mode 100644 pkg/metrics/metrics.go delete mode 100644 pkg/store/kv/bolt/store.go delete mode 100644 pkg/store/kv/manager.go delete mode 100644 pkg/store/kv/store.go delete mode 100644 pkg/store/kv/types.go delete mode 100644 pkg/store/types.go delete mode 100644 pkg/util/net/util.go delete mode 100644 pkg/util/rest/client.go delete mode 100644 pkg/util/rest/server.go delete mode 100755 tests/k8s.sh diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 265a079f8..ccf1d91d9 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -32,10 +32,6 @@ updates: directory: "cmd/cl-go-dataplane" schedule: interval: "monthly" - - package-ecosystem: "docker" - directory: "cmd/gwctl" - schedule: - interval: "monthly" - package-ecosystem: "docker" directory: "cmd/cl-dataplane" schedule: diff --git a/Makefile b/Makefile index f48eecdf9..7c25f218b 100755 --- a/Makefile +++ b/Makefile @@ -116,7 +116,6 @@ codegen: controller-gen ## Generate ClusterRole, CRDs and DeepCopyObject. cli-build: @echo "Start go build phase" - $(GO) build -o $(BIN_DIR)/gwctl $(LD_FLAGS) ./cmd/gwctl $(GO) build -o $(BIN_DIR)/clusterlink $(LD_FLAGS) ./cmd/clusterlink build: cli-build @@ -129,7 +128,6 @@ docker-build: build docker build --platform $(PLATFORMS) --progress=plain --rm --tag cl-controlplane -f ./cmd/cl-controlplane/Dockerfile . docker build --platform $(PLATFORMS) --progress=plain --rm --tag cl-dataplane -f ./cmd/cl-dataplane/Dockerfile . docker build --platform $(PLATFORMS) --progress=plain --rm --tag cl-go-dataplane -f ./cmd/cl-go-dataplane/Dockerfile . - docker build --platform $(PLATFORMS) --progress=plain --rm --tag gwctl -f ./cmd/gwctl/Dockerfile . docker build --platform $(PLATFORMS) --progress=plain --rm --tag cl-operator -f ./cmd/cl-operator/Dockerfile . push-image: build @@ -137,11 +135,9 @@ push-image: build docker buildx build --platform $(PLATFORMS) --progress=plain --rm --tag $(IMAGE_BASE)/cl-go-dataplane:$(IMAGE_VERSION) --push -f ./cmd/cl-go-dataplane/Dockerfile . docker buildx build --platform $(PLATFORMS) --progress=plain --rm --tag $(IMAGE_BASE)/cl-dataplane:$(IMAGE_VERSION) --push -f ./cmd/cl-dataplane/Dockerfile . docker buildx build --platform $(PLATFORMS) --progress=plain --rm --tag $(IMAGE_BASE)/cl-operator:$(IMAGE_VERSION) --push -f ./cmd/cl-operator/Dockerfile . - docker buildx build --platform $(PLATFORMS) --progress=plain --rm --tag $(IMAGE_BASE)/gwctl:$(IMAGE_VERSION) --push -f ./cmd/gwctl/Dockerfile . install: mkdir -p ~/.local/bin - cp ./bin/gwctl ~/.local/bin/ cp ./bin/clusterlink ~/.local/bin/ clean-tests: diff --git a/cmd/cl-controlplane/app/server.go b/cmd/cl-controlplane/app/server.go index d20af45a1..459a02268 100644 --- a/cmd/cl-controlplane/app/server.go +++ b/cmd/cl-controlplane/app/server.go @@ -34,14 +34,11 @@ import ( "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" "github.com/clusterlink-net/clusterlink/pkg/controlplane/authz" "github.com/clusterlink-net/clusterlink/pkg/controlplane/control" - cprest "github.com/clusterlink-net/clusterlink/pkg/controlplane/rest" "github.com/clusterlink-net/clusterlink/pkg/controlplane/xds" - "github.com/clusterlink-net/clusterlink/pkg/store/kv" - "github.com/clusterlink-net/clusterlink/pkg/store/kv/bolt" "github.com/clusterlink-net/clusterlink/pkg/util/controller" "github.com/clusterlink-net/clusterlink/pkg/util/grpc" + "github.com/clusterlink-net/clusterlink/pkg/util/http" "github.com/clusterlink-net/clusterlink/pkg/util/log" - utilrest "github.com/clusterlink-net/clusterlink/pkg/util/rest" "github.com/clusterlink-net/clusterlink/pkg/util/runnable" "github.com/clusterlink-net/clusterlink/pkg/util/sniproxy" "github.com/clusterlink-net/clusterlink/pkg/util/tls" @@ -80,9 +77,6 @@ type Options struct { LogFile string // LogLevel is the log level. LogLevel string - // CRDMode indicates a k8s CRD-based controlplane. - // This flag will be removed once the CRD-based controlplane feature is complete and stable. - CRDMode bool } // AddFlags adds flags to fs and binds them to options. @@ -91,7 +85,6 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) { "Path to a file where logs will be written. If not specified, logs will be printed to stderr.") fs.StringVar(&o.LogLevel, "log-level", logLevel, "The log level. One of fatal, error, warn, info, debug.") - fs.BoolVar(&o.CRDMode, "crd-mode", false, "Run a CRD-based controlplane.") } // Run the various controlplane servers. @@ -160,20 +153,17 @@ func (o *Options) Run() error { managerOptions := manager.Options{ Cache: cache.Options{ - ByObject: make(map[client.Object]cache.ByObject), + ByObject: map[client.Object]cache.ByObject{ + &v1alpha1.Peer{}: { + Namespaces: map[string]cache.Config{ + namespace: {}, + }, + }, + }, }, Scheme: scheme, } - // limit watch for v1alpha1.Peer and EndpointSlice to the namespace given by 'namespace' - if o.CRDMode { - managerOptions.Cache.ByObject[&v1alpha1.Peer{}] = cache.ByObject{ - Namespaces: map[string]cache.Config{ - namespace: {}, - }, - } - } - mgr, err := manager.New(config, managerOptions) if err != nil { return fmt.Errorf( @@ -186,7 +176,7 @@ func (o *Options) Run() error { grpcServerName: grpcServerAddress, }) - httpServer := utilrest.NewServer("controlplane-http", parsedCertData.ServerConfig()) + httpServer := http.NewServer("controlplane-http", parsedCertData.ServerConfig()) grpcServer := grpc.NewServer("controlplane-grpc", parsedCertData.ServerConfig()) authzManager, err := authz.NewManager(parsedCertData, mgr.GetClient(), namespace) @@ -194,63 +184,26 @@ func (o *Options) Run() error { return fmt.Errorf("cannot create authorization manager: %w", err) } - err = authz.CreateControllers(authzManager, mgr, o.CRDMode) + err = authz.CreateControllers(authzManager, mgr) if err != nil { return fmt.Errorf("cannot create authz controllers: %w", err) } - authz.RegisterHandlers(authzManager, &httpServer.Server) + authz.RegisterHandlers(authzManager, httpServer) - controlManager := control.NewManager(mgr.GetClient(), parsedCertData, namespace, o.CRDMode) + controlManager := control.NewManager(mgr.GetClient(), parsedCertData, namespace) - err = control.CreateControllers(controlManager, mgr, o.CRDMode) + err = control.CreateControllers(controlManager, mgr) if err != nil { return fmt.Errorf("cannot create control controllers: %w", err) } - xdsManager := xds.NewManager(o.CRDMode) + xdsManager := xds.NewManager() xds.RegisterService( context.Background(), xdsManager, grpcServer.GetGRPCServer()) - if o.CRDMode { - err := xds.CreateControllers(xdsManager, mgr) - if err != nil { - return fmt.Errorf("cannot create xDS controllers: %w", err) - } - } else { - // open store - kvStore, err := bolt.Open(StoreFile) - if err != nil { - return err - } - - defer func() { - if err := kvStore.Close(); err != nil { - logrus.Warnf("Cannot close store: %v.", err) - } - }() - - storeManager := kv.NewManager(kvStore) - - restManager, err := cprest.NewManager( - namespace, storeManager, xdsManager, authzManager, controlManager) - if err != nil { - return err - } - - cprest.RegisterHandlers(restManager, httpServer) - - authzManager.SetGetImportCallback(restManager.GetK8sImport) - authzManager.SetGetExportCallback(restManager.GetK8sExport) - authzManager.SetGetPeerCallback(restManager.GetK8sPeer) - controlManager.SetGetImportCallback(restManager.GetK8sImport) - controlManager.SetGetMergeImportListCallback(restManager.GetMergeImportList) - controlManager.SetPeerStatusCallback(func(pr *v1alpha1.Peer) { - restManager.UpdatePeerStatus(pr.Name, &pr.Status) - }) - controlManager.SetExportStatusCallback(func(export *v1alpha1.Export) { - restManager.UpdateExportStatus(export.Name, &export.Status) - }) + if err := xds.CreateControllers(xdsManager, mgr); err != nil { + return fmt.Errorf("cannot create xDS controllers: %w", err) } runnableManager := runnable.NewManager() diff --git a/cmd/clusterlink/cmd/create/create_peer.go b/cmd/clusterlink/cmd/create/create_peer.go index d29a9e5f2..ba7b15369 100644 --- a/cmd/clusterlink/cmd/create/create_peer.go +++ b/cmd/clusterlink/cmd/create/create_peer.go @@ -95,24 +95,6 @@ func (o *PeerOptions) createDataplane(peerCert *bootstrap.Certificate) (*bootstr return cert, nil } -func (o *PeerOptions) createGWCTL(peerCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateGWCTLCertificate(peerCert) - if err != nil { - return nil, err - } - - outDirectory := config.GWCTLDirectory(o.Name, o.Fabric, o.Path) - if err := os.Mkdir(outDirectory, 0o755); err != nil { - return nil, err - } - - if err := o.saveCertificate(cert, outDirectory); err != nil { - return nil, err - } - - return cert, nil -} - // Run the 'create peer-cert' subcommand. func (o *PeerOptions) Run() error { if _, err := idna.Lookup.ToASCII(o.Name); err != nil { @@ -151,10 +133,6 @@ func (o *PeerOptions) Run() error { return err } - if _, err := o.createGWCTL(peerCertificate); err != nil { - return err - } - return nil } diff --git a/cmd/clusterlink/cmd/deploy/deploy_peer.go b/cmd/clusterlink/cmd/deploy/deploy_peer.go index 50a3fee0f..9bb5fc3a8 100644 --- a/cmd/clusterlink/cmd/deploy/deploy_peer.go +++ b/cmd/clusterlink/cmd/deploy/deploy_peer.go @@ -78,9 +78,6 @@ type PeerOptions struct { DataplaneType string // LogLevel is the log level. LogLevel string - // CRDMode indicates whether to run a k8s CRD-based controlplane. - // This flag will be removed once the CRD-based controlplane feature is complete and stable. - CRDMode bool } // NewCmdDeployPeer returns a cobra.Command to run the 'deploy peer' subcommand. @@ -135,7 +132,6 @@ func (o *PeerOptions) AddFlags(fs *pflag.FlagSet) { fs.Uint16Var(&o.DataplaneReplicas, "dataplane-replicas", 1, "Number of dataplanes.") fs.StringVar(&o.LogLevel, "log-level", "info", "The log level. One of fatal, error, warn, info, debug.") - fs.BoolVar(&o.CRDMode, "crd-mode", false, "Run a CRD-based controlplane.") } // RequiredFlags are the names of flags that must be explicitly specified. @@ -181,12 +177,6 @@ func (o *PeerOptions) Run() error { return err } - gwctlCert, err := bootstrap.ReadCertificates( - config.GWCTLDirectory(o.Name, o.Fabric, o.Path), true) - if err != nil { - return err - } - // Create k8s deployment YAML platformCfg := &platform.Config{ Peer: o.Name, @@ -194,12 +184,10 @@ func (o *PeerOptions) Run() error { PeerCertificate: peerCertificate, ControlplaneCertificate: controlplaneCert, DataplaneCertificate: dataplaneCert, - GWCTLCertificate: gwctlCert, Dataplanes: o.DataplaneReplicas, DataplaneType: o.DataplaneType, LogLevel: o.LogLevel, ContainerRegistry: o.ContainerRegistry, - CRDMode: o.CRDMode, Namespace: o.Namespace, IngressType: o.Ingress, IngressAnnotations: o.IngressAnnotations, diff --git a/cmd/clusterlink/config/config.go b/cmd/clusterlink/config/config.go index 46cd07a6f..759b5d3b4 100644 --- a/cmd/clusterlink/config/config.go +++ b/cmd/clusterlink/config/config.go @@ -24,25 +24,15 @@ const ( CertificateFileName = "cert.pem" // DefaultFabric is the default fabric name. DefaultFabric = "default_fabric" - // DockerRunFile is the filename of the docker-run script. - DockerRunFile = "docker-run.sh" - // GWCTLInitFile is the filename of the gwctl-init script. - GWCTLInitFile = "gwctl-init.sh" // K8SYAMLFile is the filename of the kubernetes deployment yaml file. K8SYAMLFile = "k8s.yaml" - // K8SSecretYAMLFile is the filename of the kubernetes secrets yaml file. - K8SSecretYAMLFile = "cl-secret.yaml" //nolint:gosec // G101(Potential hardcoded credentials): Enable secret usage in filenames. // K8SClusterLinkInstanceYAMLFile is the filename of the ClusterLink instance CRD file that will use by the operator. K8SClusterLinkInstanceYAMLFile = "cl-instance.yaml" - // PersistencyDirectoryName is the directory name containing container persisted files. - PersistencyDirectoryName = "persist" // ControlplaneDirectoryName is the directory name containing controlplane server configuration. ControlplaneDirectoryName = "controlplane" // DataplaneDirectoryName is the directory name containing dataplane server configuration. DataplaneDirectoryName = "dataplane" - // GWCTLDirectoryName is the directory name containing gwctl certificates. - GWCTLDirectoryName = "gwctl" // GHCR is the path to the GitHub container registry. GHCR = "ghcr.io/clusterlink-net" @@ -70,11 +60,6 @@ func DataplaneDirectory(peer, fabric, path string) string { return filepath.Join(PeerDirectory(peer, fabric, path), DataplaneDirectoryName) } -// GWCTLDirectory returns the path for a gwctl instance. -func GWCTLDirectory(peer, fabric, path string) string { - return filepath.Join(PeerDirectory(peer, fabric, path), GWCTLDirectoryName) -} - // FabricCertificate returns the fabric certificate name. func FabricCertificate(name, path string) string { return filepath.Join(FabricDirectory(name, path), CertificateFileName) diff --git a/cmd/gwctl/Dockerfile b/cmd/gwctl/Dockerfile deleted file mode 100644 index 865c81895..000000000 --- a/cmd/gwctl/Dockerfile +++ /dev/null @@ -1,8 +0,0 @@ -FROM alpine:3.19 - -# Copy binary -RUN mkdir -p /usr/local/bin -COPY ./bin/gwctl /usr/local/bin/gwctl - -# Install bash -RUN apk add bash diff --git a/cmd/gwctl/config/config.go b/cmd/gwctl/config/config.go deleted file mode 100644 index 8c16c51e9..000000000 --- a/cmd/gwctl/config/config.go +++ /dev/null @@ -1,239 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package config - -import ( - "encoding/json" - "errors" - "os" - "os/user" - "path" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/client" - "github.com/clusterlink-net/clusterlink/pkg/util/tls" -) - -const ( - projectFolder = "/.gw/" - configFile = "gwctl" -) - -// ClientConfig contain all Client configuration to send requests to the GW. -type ClientConfig struct { - GwIP string `json:"gwIp"` - GwPort uint16 `json:"gwPort"` - ID string `json:"id"` - CaFile string `json:"caFile"` - CertFile string `json:"certFile"` - KeyFile string `json:"keyFile"` - Dataplane string `json:"dataplane"` - PolicyEngineIP string `json:"policyEngineIp"` - MetricsManagerIP string `json:"metricsManagerIp"` - logger *logrus.Entry `json:"-"` -} - -// NewClientConfig create config file with all the configuration of the Client. -func NewClientConfig(cfg *ClientConfig) (*ClientConfig, error) { - clientCfg := ClientConfig{ - ID: cfg.ID, - GwIP: cfg.GwIP, - GwPort: cfg.GwPort, - Dataplane: cfg.Dataplane, - CertFile: cfg.CertFile, - KeyFile: cfg.KeyFile, - CaFile: cfg.CaFile, - PolicyEngineIP: cfg.PolicyEngineIP, - MetricsManagerIP: cfg.MetricsManagerIP, - logger: logrus.WithField("component", "gwctl/config"), - } - - _, err := clientCfg.createProjectfolder() - if err != nil { - return nil, err - } - err = clientCfg.createConfigFile() - if err != nil { - return nil, err - } - return &clientCfg, nil -} - -// GetConfigFromID return configuration of Client according to the Client ID. -func GetConfigFromID(id string) (*ClientConfig, error) { - c, err := readConfigFromFile(id) - return &c, err -} - -// GetGwIP return the gw IP that the Client is connected. -func (c *ClientConfig) GetGwIP() string { - return c.GwIP -} - -// GetGwPort return the gw port that the Client is connected. -func (c *ClientConfig) GetGwPort() uint16 { - return c.GwPort -} - -// GetID return the Client ID. -func (c *ClientConfig) GetID() string { - return c.ID -} - -// GetDataplane return the Client dataplane type (MTLS or TCP). -func (c *ClientConfig) GetDataplane() string { - return c.Dataplane -} - -// GetCert return the Client certificate. -func (c *ClientConfig) GetCert() string { - return c.CertFile -} - -// GetCaFile return the Client certificate Authority. -func (c *ClientConfig) GetCaFile() string { - return c.CaFile -} - -// GetKeyFile return the Client key file. -func (c *ClientConfig) GetKeyFile() string { - return c.KeyFile -} - -// GetPolicyEngineIP return the policy server address. -func (c *ClientConfig) GetPolicyEngineIP() string { - return c.PolicyEngineIP -} - -// GetMetricsManagerIP return the metrics manager address. -func (c *ClientConfig) GetMetricsManagerIP() string { - return c.MetricsManagerIP -} - -/********************************/ -/******** Config functions **********/ -/********************************/ - -func (c *ClientConfig) createProjectfolder() (string, error) { - usr, err := user.Current() - if err != nil { - return "", err - } - fol := path.Join(usr.HomeDir, projectFolder) - // Create folder - err = os.MkdirAll(fol, 0o755) - if err != nil { - c.logger.Errorln(err) - return "", err - } - return fol, nil -} - -func (c *ClientConfig) createConfigFile() error { - jsonC, err := json.MarshalIndent(c, "", "\t") - if err != nil { - c.logger.Errorln("Client create config File", err) - return err - } - f, err := ClientPath(c.ID) - if err != nil { - c.logger.Errorln("Client get config file path", err) - return err - } - err = os.WriteFile(f, jsonC, 0o600) // RW by owner only - c.logger.Println("Create Client config File:", f) - if err != nil { - c.logger.Errorln("Creating client config File", err) - return err - } - return c.SetDefaultClient(c.ID) -} - -// SetDefaultClient set the default Client the CLI will use. -func (c *ClientConfig) SetDefaultClient(id string) error { - // Check if the file exist - file, err := ClientPath(id) - if err != nil { - c.logger.Errorf("failed to get client config file path for id %v\n", id) - return err - } - if _, err := os.Stat(file); errors.Is(err, os.ErrNotExist) { - c.logger.Errorf("Client config File with id %v does not exist\n", id) - return err - } - - // Remove if the link exist - link, err := ClientPath("") - if err != nil { - c.logger.Errorf("failed to get client config link path\n") - return err - } - if _, err := os.Lstat(link); err == nil { - os.Remove(link) - } - // Create a link - err = os.Symlink(file, link) - if err != nil { - c.logger.Errorln("Error creating symlink:", err) - return err - } - return nil -} - -func readConfigFromFile(id string) (ClientConfig, error) { - file, err := ClientPath(id) - if err != nil { - return ClientConfig{}, err - } - data, err := os.ReadFile(file) - if err != nil { - return ClientConfig{}, err - } - var s ClientConfig - err = json.Unmarshal(data, &s) - if err != nil { - return ClientConfig{}, err - } - return s, nil -} - -// GetClientFromID loads Client from file according to the id. -func GetClientFromID(id string) (*client.Client, error) { - c, err := GetConfigFromID(id) - if err != nil { - return nil, err - } - - parsedCertData, err := tls.ParseFiles(c.CaFile, c.CertFile, c.KeyFile) - if err != nil { - return nil, err - } - - return client.New(c.GwIP, c.GwPort, parsedCertData.ClientConfig(c.ID)), nil -} - -// ClientPath get CLI config file from id. -func ClientPath(id string) (string, error) { - cfgFile := configFile - if id != "" { - cfgFile += "_" + id - } - // set cfg file in home directory - usr, err := user.Current() - if err != nil { - return "", err - } - return path.Join(usr.HomeDir, projectFolder, cfgFile), nil -} diff --git a/cmd/gwctl/main.go b/cmd/gwctl/main.go deleted file mode 100644 index 575e9e94f..000000000 --- a/cmd/gwctl/main.go +++ /dev/null @@ -1,113 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package main - -import ( - "os" - - "github.com/sirupsen/logrus" - "github.com/spf13/cobra" - - "github.com/clusterlink-net/clusterlink/cmd/gwctl/subcommand" - "github.com/clusterlink-net/clusterlink/pkg/versioninfo" -) - -func main() { - rootCmd := &cobra.Command{ - Use: "gwctl", - Short: "gwctl is a CLI that sends a control message (REST API) to the gateway", - Long: `gwctl CLI is part of the multi-cloud network project, - that allow sending control messages (REST API) to publish, connect and update policies for services`, - Version: versioninfo.Short(), - } - - // Add all commands - rootCmd.AddCommand(subcommand.InitCmd()) // init command of Gwctl - rootCmd.AddCommand(createCmd()) - rootCmd.AddCommand(getCmd()) - rootCmd.AddCommand(deleteCmd()) - rootCmd.AddCommand(updateCmd()) - rootCmd.AddCommand(subcommand.ConfigCmd()) - - logrus.SetLevel(logrus.WarnLevel) - - // Execute runs the cobra command of the gwctl - err := rootCmd.Execute() - if err != nil { - os.Exit(1) - } -} - -// createCmd contains all the create commands of the CLI. -func createCmd() *cobra.Command { - createCmd := &cobra.Command{ - Use: "create", - Short: "Create", - Long: `Create`, - } - // Add all create commands - createCmd.AddCommand(subcommand.PeerCreateCmd()) - createCmd.AddCommand(subcommand.ExportCreateCmd()) - createCmd.AddCommand(subcommand.ImportCreateCmd()) - createCmd.AddCommand(subcommand.PolicyCreateCmd()) - return createCmd -} - -// deleteCmd contains all the delete commands of the CLI. -func deleteCmd() *cobra.Command { - deleteCmd := &cobra.Command{ - Use: "delete", - Short: "Delete", - Long: `Delete`, - } - // Add all delete commands - deleteCmd.AddCommand(subcommand.PeerDeleteCmd()) - deleteCmd.AddCommand(subcommand.ExportDeleteCmd()) - deleteCmd.AddCommand(subcommand.ImportDeleteCmd()) - deleteCmd.AddCommand(subcommand.PolicyDeleteCmd()) - return deleteCmd -} - -// updateCmd contains all the update commands of the CLI. -func updateCmd() *cobra.Command { - updateCmd := &cobra.Command{ - Use: "update", - Short: "Update", - Long: `Update`, - } - // Add all update commands - updateCmd.AddCommand(subcommand.PeerUpdateCmd()) - updateCmd.AddCommand(subcommand.ExportUpdateCmd()) - updateCmd.AddCommand(subcommand.ImportUpdateCmd()) - updateCmd.AddCommand(subcommand.PolicyUpdateCmd()) - return updateCmd -} - -// getCmd contains all the get commands of the CLI. -func getCmd() *cobra.Command { - getCmd := &cobra.Command{ - Use: "get", - Short: "Get", - Long: `Get`, - } - // Add all get commands - getCmd.AddCommand(subcommand.StateGetCmd()) - getCmd.AddCommand(subcommand.PeerGetCmd()) - getCmd.AddCommand(subcommand.ExportGetCmd()) - getCmd.AddCommand(subcommand.ImportGetCmd()) - getCmd.AddCommand(subcommand.PolicyGetCmd()) - getCmd.AddCommand(subcommand.MetricsGetCmd()) - getCmd.AddCommand(subcommand.AllGetCmd()) - return getCmd -} diff --git a/cmd/gwctl/subcommand/config.go b/cmd/gwctl/subcommand/config.go deleted file mode 100644 index 87505e653..000000000 --- a/cmd/gwctl/subcommand/config.go +++ /dev/null @@ -1,112 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package subcommand - -import ( - "encoding/json" - "fmt" - - "github.com/spf13/cobra" - "github.com/spf13/pflag" - - "github.com/clusterlink-net/clusterlink/cmd/gwctl/config" -) - -// ConfigCmd contains all the config commands of the CLI. -func ConfigCmd() *cobra.Command { - configCmd := &cobra.Command{ - Use: "config", - Short: "config", - Long: `config`, - } - - configCmd.AddCommand(currentContextCmd()) - configCmd.AddCommand(useContextCmd()) - return configCmd -} - -// getContextCmd is the command line options for 'config current-context'. -type currentContextOptions struct{} - -// currentContextCmd - get the last gwctl context command to use. -func currentContextCmd() *cobra.Command { - o := currentContextOptions{} - cmd := &cobra.Command{ - Use: "current-context", - Short: "Get gwctl current context.", - Long: `Get gwctl current context.`, - RunE: func(cmd *cobra.Command, args []string) error { - return o.run() - }, - } - - return cmd -} - -// run performs the execution of the 'config current-context' subcommand. -func (o *currentContextOptions) run() error { - s, err := config.GetConfigFromID("") - if err != nil { - return err - } - - sJSON, err := json.MarshalIndent(s, "", " ") - if err != nil { - fmt.Println("gwctl current state\n", string(sJSON)) - } - return err -} - -// useContext is the command line options for 'config use-context'. -type useContextOptions struct { - myID string -} - -// useContextCmd - set gwctl context. -func useContextCmd() *cobra.Command { - o := useContextOptions{} - cmd := &cobra.Command{ - Use: "use-context", - Short: "use gwctl context.", - Long: `use gwctl context.`, - RunE: func(cmd *cobra.Command, args []string) error { - return o.run() - }, - } - - o.addFlags(cmd.Flags()) - - return cmd -} - -// addFlags registers flags for the CLI. -func (o *useContextOptions) addFlags(fs *pflag.FlagSet) { - fs.StringVar(&o.myID, "myid", "", "gwctl ID") -} - -// run performs the execution of the 'config current-context' subcommand. -func (o *useContextOptions) run() error { - c, err := config.GetConfigFromID(o.myID) - if err != nil { - return err - } - - err = c.SetDefaultClient(o.myID) - if err != nil { - return err - } - - fmt.Println("gwctl use context ", o.myID) - return nil -} diff --git a/cmd/gwctl/subcommand/export.go b/cmd/gwctl/subcommand/export.go deleted file mode 100644 index d328f12cb..000000000 --- a/cmd/gwctl/subcommand/export.go +++ /dev/null @@ -1,216 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package subcommand - -import ( - "fmt" - - "github.com/spf13/cobra" - "github.com/spf13/pflag" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - - "github.com/clusterlink-net/clusterlink/cmd/gwctl/config" - cmdutil "github.com/clusterlink-net/clusterlink/cmd/util" - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" -) - -// exportCreateOptions is the command line options for 'create export' or 'update export'. -type exportCreateOptions struct { - myID string - name string - host string - port uint16 - external string -} - -// ExportCreateCmd - Create an exported service. -func ExportCreateCmd() *cobra.Command { - o := exportCreateOptions{} - cmd := &cobra.Command{ - Use: "export", - Short: "Create an exported service", - Long: `Create an exported service that can be accessed by other peers`, - RunE: func(cmd *cobra.Command, args []string) error { - return o.run(false) - }, - } - - o.addFlags(cmd.Flags()) - cmdutil.MarkFlagsRequired(cmd, []string{"name", "port"}) - return cmd -} - -// ExportUpdateCmd - Update an exported service. -func ExportUpdateCmd() *cobra.Command { - o := exportCreateOptions{} - cmd := &cobra.Command{ - Use: "export", - Short: "Update an exported service", - Long: `Update an exported service that can be accessed by other peers`, - RunE: func(cmd *cobra.Command, args []string) error { - return o.run(true) - }, - } - - o.addFlags(cmd.Flags()) - cmdutil.MarkFlagsRequired(cmd, []string{"name", "port"}) - return cmd -} - -// addFlags registers flags for the CLI. -func (o *exportCreateOptions) addFlags(fs *pflag.FlagSet) { - fs.StringVar(&o.myID, "myid", "", "gwctl ID") - fs.StringVar(&o.name, "name", "", "Exported service name") - fs.StringVar(&o.host, "host", "", "Exported service endpoint hostname (IP/DNS), if unspecified, uses the service name") - fs.Uint16Var(&o.port, "port", 0, "Exported service port") - fs.StringVar(&o.external, "external", "", - "External endpoint : -- gwctl - - See example in the next step. - -### Step 4: Peers communication -In this step, we add a peer for each gateway using the gwctl: - - gwctl create peer --myid peer1 --name peer2 --host $PEER2_IP --port 30443 - gwctl create peer --myid peer2 --name peer1 --host $PEER1_IP --port 30443 - -When running Kind cluster on macOS run instead the following: - - kubectl config use-context kind-peer1 - export GWCTL1=`kubectl get pods -l app=gwctl -o custom-columns=:metadata.name --no-headers` - kubectl exec -i $GWCTL1 -- gwctl create peer --myid peer1 --name peer2 --host $PEER2_IP --port 30443 - - kubectl config use-context kind-peer2 - export GWCTL2=`kubectl get pods -l app=gwctl -o custom-columns=:metadata.name --no-headers` - kubectl exec -i $GWCTL2 -- gwctl create peer --myid peer2 --name peer1 --host $PEER1_IP --port 30443 - - -### Step 5: Export a service -In this step, we add the iperf3 server to the cluster as an exported service that can be accessed from remote peers. -Export the iperf3-server service to the Cluster 2 gateway: - - gwctl create export --myid peer2 --name iperf3-server --host iperf3-server --port 5000 - -When running Kind cluster on macOS run instead the following: - - kubectl exec -i $GWCTL2 -- gwctl create export --myid peer2 --name iperf3-server --host iperf3-server --port 5000 - -Note: iperf3-client doesn't need to be added since it is not exported. - -### Step 6: import iperf3 server service from Cluster 2 -In this step, we import the iperf3-server service from Cluster 2 gateway to Cluster 1 gateway -First, we specify which service we want to import and specify the local k8s endpoint (host:port) that will create for this service: - - gwctl create import --myid peer1 --name iperf3-server --host iperf3-server --port 5000 - -When running Kind cluster on macOS run instead the following: - - kubectl config use-context kind-peer1 - kubectl exec -i $GWCTL1-- gwctl create import --myid peer1 --name iperf3-server --port 5000 - -Second, we specify the peer we want to import the service: - - gwctl create binding --myid peer1 --import iperf3-server --peer peer2 - -When running Kind cluster on macOS run instead the following: - - kubectl config use-context kind-peer1 - kubectl exec -i $GWCTL1 -- gwctl create binding --myid peer1 --import iperf3-server --peer peer2 - -### Step 7: Create access policy -In this step, we create a policy that allow to all traffic from peer1 and peer2: - - gwctl --myid peer1 create policy --type access --policyFile $PROJECT_DIR/examples/policies/allowAll.json - gwctl --myid peer2 create policy --type access --policyFile $PROJECT_DIR/examples/policies/allowAll.json - -When running Kind cluster on macOS run instead the following: - - kubectl config use-context kind-peer1 - kubectl cp $PROJECT_DIR/examples/policies/allowAll.json gwctl:/tmp/allowAll.json - kubectl exec -i $GWCTL1 -- gwctl create policy --type access --policyFile /tmp/allowAll.json - kubectl config use-context kind-peer2 - kubectl cp $PROJECT_DIR/examples/policies//allowAll.json gwctl:/tmp/allowAll.json - kubectl exec -i $GWCTL2 -- gwctl create policy --type access --policyFile /tmp/allowAll.json - -### Final Step : Test Service connectivity -Start the iperf3 test from cluster 1: - - kubectl config use-context kind-peer1 - export IPERF3CLIENT=`kubectl get pods -l app=iperf3-client -o custom-columns=:metadata.name --no-headers` - kubectl exec -i $IPERF3CLIENT -- iperf3 -c iperf3-server --port 5000 - -### Cleanup -Delete all Kind clusters: - - kind delete cluster --name=peer1 - kind delete cluster --name=peer2 diff --git a/demos/iperf3/testdata/manifests/iperf3-client/iperf3-svc.yaml b/demos/iperf3/testdata/manifests/iperf3-client/iperf3-svc.yaml deleted file mode 100755 index 885728b0f..000000000 --- a/demos/iperf3/testdata/manifests/iperf3-client/iperf3-svc.yaml +++ /dev/null @@ -1,18 +0,0 @@ -################################################################ -#Name: cluster-iperf3-service -#Desc: service file for connecting iperf3 service -# port 5000 -################################################################ -apiVersion: v1 -kind: Service -metadata: - name: gwctl-iperf3-service -spec: - type: ClusterIP - selector: - app: gwctl - ports: - - protocol: TCP - port: 5000 - targetPort: 5000 - \ No newline at end of file diff --git a/demos/qotd/kind/test.py b/demos/qotd/kind/test.py index 5a32a44b5..39c66b215 100755 --- a/demos/qotd/kind/test.py +++ b/demos/qotd/kind/test.py @@ -14,11 +14,11 @@ ############################################################################################## # Name: quote of today -# Info: support qotd application with gwctl inside the clusters +# Info: support qotd application inside the clusters # In this we create three kind clusters -# 1) cluster1- contain gw, gwctl,webApp and engravingApp microservices (qotd services) -# 2) cluster2- contain gw, gwctl, quoteApp, authorApp, imageApp, dbApp microservices (qotd services) -# 3) cluster3- contain gw, gwctl, pdfApp and ratingApp microservices (qotd services) +# 1) cluster1- contain gw, webApp and engravingApp microservices (qotd services) +# 2) cluster2- contain gw, quoteApp, authorApp, imageApp, dbApp microservices (qotd services) +# 3) cluster3- contain gw, pdfApp and ratingApp microservices (qotd services) ############################################################################################## import os import sys @@ -106,7 +106,7 @@ cl2.exports.create(dbApp.name, dbApp.namespace, dbApp.port) cl2.exports.create(imageApp.name, imageApp.namespace, imageApp.port) - ###Set gwctl3 + ###Set cl3 service cl3.useCluster() cl3.loadService(ratingApp.name, "registry.gitlab.com/quote-of-the-day/qotd-ratings-service/v4.0.0:latest", f"{qotdFol}/qotd_rating.yaml", namespace= ratingApp.namespace) diff --git a/demos/speedtest/kind/README.md b/demos/speedtest/kind/README.md index 3365889e4..ab1a1be30 100644 --- a/demos/speedtest/kind/README.md +++ b/demos/speedtest/kind/README.md @@ -2,9 +2,9 @@ In this demo we use OpenSpeedTest application for checking connectivity between different kind clusters using the Clusterlink components. This demo shows different access policies defined by various attributes such as source service, destination service, and destination gateway. This setup uses three Kind clusters- -1. Cluster 1- contains GW, gwctl (GW CLI component), and firefox client. -2. Cluster 2- contains GW, gwctl (GW CLI component), and OpenSpeedTest server. -3. cluster 3- contains GW, gwctl (GW CLI component), and two firefox clients. +1. Cluster 1- contains GW, and firefox client. +2. Cluster 2- contains GW, and OpenSpeedTest server. +3. cluster 3- contains GW, and two firefox clients. System illustration: diff --git a/demos/utils/clusterlink.py b/demos/utils/clusterlink.py index b8ec96005..a928d3ad0 100644 --- a/demos/utils/clusterlink.py +++ b/demos/utils/clusterlink.py @@ -242,11 +242,10 @@ def create_peer_cert(self, name, dir): runcmdDir(f"{CL_CLI} create peer-cert --name {name}",dir) # deploy_peer deploys clusterlink to the cluster using ClusterLink CLI. - def deploy_peer(self, name, dir, logLevel="info", dataplane="envoy", container_reg="", CRDMode=True, ingress_type="", ingress_port=0): + def deploy_peer(self, name, dir, logLevel="info", dataplane="envoy", container_reg="", ingress_type="", ingress_port=0): flag = f"--container-registry={container_reg} " if container_reg != "" else "" flag += f"--ingress={ingress_type} " if ingress_type != "" else "" flag += f"--ingress-port={ingress_port} " if ingress_port != 0 else "" - flag += "--crd-mode=true " if CRDMode else "" runcmdDir(f"{CL_CLI} deploy peer --name {name} --log-level {logLevel} --dataplane {dataplane} {flag}",dir) waitPod("cl-controlplane", CLUSTELINK_NS) waitPod("cl-dataplane", CLUSTELINK_NS) diff --git a/docs/installation.md b/docs/installation.md index d609b05a5..cbd03374f 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,6 +1,6 @@ # Installation guide for ClusterLink -The ClusterLink gateway contains three main components: the control-plane, data-plane, and gwctl (more details can be found [here](../README.md#what-is-clusterlink)). +The ClusterLink gateway contains two main components: the control-plane and the data-plane (more details can be found [here](../README.md#what-is-clusterlink)). ClusterLink can be deployed in any K8s based cluster (e.g., google GKE, amazon EKS, IBM IKS, KIND etc.). @@ -8,7 +8,6 @@ To deploy the ClusterLink gateway in a K8s cluster, follow these steps: 1. Create and deploy certificates and the ClusterLink deployment YAML file. 2. Expose the ClusterLink deployment to a public IP. -3. Optionally, create a central gwctl component to control one or more gateways. Before you begin, please build the project according to the [instructions](../README.md#building-clustelink). @@ -39,17 +38,15 @@ In this step, we generate the ClusterLink certificates and deployment files. kubectl apply -f $DEPLOY_DIR/peer1/k8s.yaml -5) Verify that all components (cl-controlplane, cl-dataplane, gwctl) are set up and ready. +5) Verify that all components (cl-controlplane, cl-dataplane) are set up and ready. kubectl rollout status deployment cl-controlplane kubectl rollout status deployment cl-dataplane - kubectl wait --for=condition=ready pod -l app=gwctl Expected output: deployment "cl-controlplane" successfully rolled out deployment "cl-dataplane" successfully rolled out - pod/gwctl condition met ## 2. Expose the ClusterLink deployment to a public IP @@ -89,26 +86,6 @@ export PEER1_PORT=30443 Now, the ClusterLink gateway can be accessed through `$PEER1_IP` at port `$PEER1_PORT`. -## 3. Create a central gwctl (optional) -By default for each K8s cluster a gwctl pod is created that use REST APIs to send control messages to the -ClusterLink gateway, using the command: - - kubectl exec -i -- gwctl - -To create a single gwctl that controls one or more ClusterLink gateways, follow these steps: - -1. Install the local control (gwctl): - - sudo make install - -2. Initialize the gwctl CLI for the cluster (e.g., peer1): - - gwctl init --id peer1 --gwIP $PEER1_IP --gwPort $PEER1_PORT --certca $DEPLOY_DIR/cert.pem --cert $DEPLOY_DIR/peer1/gwctl/cert.pem --key $DEPLOY_DIR/peer1/gwctl/key.pem - -3. To run gwctl command: - - gwctl --myid peer1 - ## Additional setup modes ### Debug mode diff --git a/go.mod b/go.mod index 28e73f355..289e0d2ae 100644 --- a/go.mod +++ b/go.mod @@ -15,7 +15,6 @@ require ( github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 - go.etcd.io/bbolt v1.3.10 golang.org/x/net v0.25.0 google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.1 diff --git a/go.sum b/go.sum index 2931c5ddc..abd8cbea9 100644 --- a/go.sum +++ b/go.sum @@ -161,8 +161,6 @@ github.com/vladimirvivien/gexe v0.2.0/go.mod h1:LHQL00w/7gDUKIak24n801ABp8C+ni6e github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -go.etcd.io/bbolt v1.3.10 h1:+BqfJTcCzTItrop8mq/lbzL8wSGtj94UO/3U31shqG0= -go.etcd.io/bbolt v1.3.10/go.mod h1:bK3UQLPJZly7IlNmV7uVHJDxfe5aK9Ll93e/74Y9oEQ= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= @@ -201,8 +199,6 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ= -golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= diff --git a/hack/install_clusterlink.sh b/hack/install_clusterlink.sh index 74d3218b8..060a002ed 100755 --- a/hack/install_clusterlink.sh +++ b/hack/install_clusterlink.sh @@ -73,7 +73,7 @@ printf "\n\n" printf "%s has been successfully downloaded.\n" "$filename" printf "\n" -printf "ClusterLink CLI (gwctl and clusterlink) has been installed in the following directory:\n" +printf "ClusterLink CLI (clusterlink) has been installed in the following directory:\n" printf "\n" printf "\t\e[1;33m%s\n\e[0m" "$install_path" printf "\n" diff --git a/pkg/bootstrap/cert.go b/pkg/bootstrap/cert.go index 742f97ebc..86bd87ee7 100644 --- a/pkg/bootstrap/cert.go +++ b/pkg/bootstrap/cert.go @@ -101,20 +101,6 @@ func CreateDataplaneCertificate(peer string, peerCert *Certificate) (*Certificat return &Certificate{cert: cert}, nil } -// CreatePeerCertificate creates a gwctl certificate. -func CreateGWCTLCertificate(peerCert *Certificate) (*Certificate, error) { - cert, err := createCertificate(&certificateConfig{ - Parent: peerCert.cert, - Name: "gwctl", - IsClient: true, - }) - if err != nil { - return nil, err - } - - return &Certificate{cert: cert}, nil -} - // CertificateFromRaw initializes a certificate from raw data. func CertificateFromRaw(rawCert, rawKey []byte) (*Certificate, error) { cert, err := certificateFromRaw(rawCert, rawKey) diff --git a/pkg/bootstrap/platform/config.go b/pkg/bootstrap/platform/config.go index c226769b5..f02cc292a 100644 --- a/pkg/bootstrap/platform/config.go +++ b/pkg/bootstrap/platform/config.go @@ -32,8 +32,6 @@ type Config struct { ControlplaneCertificate *bootstrap.Certificate // DataplaneCertificate is the dataplane certificate. DataplaneCertificate *bootstrap.Certificate - // GWCTLCertificate is the gwctl certificate. - GWCTLCertificate *bootstrap.Certificate // Dataplanes is the number of dataplane servers to run. Dataplanes uint16 @@ -52,8 +50,6 @@ type Config struct { IngressPort uint16 // IngressAnnotations is the annotations added to the ingress service. IngressAnnotations map[string]string - // CRDMode indicates a CRD-based controlplane. - CRDMode bool } const ( diff --git a/pkg/bootstrap/platform/k8s.go b/pkg/bootstrap/platform/k8s.go index 8d2c1f0f4..811245f5a 100644 --- a/pkg/bootstrap/platform/k8s.go +++ b/pkg/bootstrap/platform/k8s.go @@ -54,41 +54,9 @@ metadata: data: cert: {{.dataplaneCert}} key: {{.dataplaneKey}} -{{ if not .crdMode }} ---- -apiVersion: v1 -kind: Secret -metadata: - name: cl-peer - namespace: {{.namespace}} -data: - ca: {{.peerCA}} ---- -apiVersion: v1 -kind: Secret -metadata: - name: gwctl - namespace: {{.namespace}} -data: - cert: {{.gwctlCert}} - key: {{.gwctlKey}} -{{ end }} ` - k8sTemplate = `{{ if not .crdMode }}--- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: cl-controlplane - namespace: {{.namespace}} -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 100Mi -{{ end }} ---- + k8sTemplate = `--- apiVersion: apps/v1 kind: Deployment metadata: @@ -113,15 +81,10 @@ spec: - name: tls secret: secretName: cl-controlplane -{{ if not .crdMode }} - - name: cl-controlplane - persistentVolumeClaim: - claimName: cl-controlplane -{{ end }} containers: - name: cl-controlplane image: {{.containerRegistry}}cl-controlplane:{{.tag}} - args: ["--log-level", "{{.logLevel}}"{{if .crdMode }}, "--crd-mode"{{ end }}] + args: ["--log-level", "{{.logLevel}}"] imagePullPolicy: IfNotPresent ports: - containerPort: {{.controlplanePort}} @@ -138,10 +101,6 @@ spec: mountPath: {{.controlplaneKeyMountPath}} subPath: "key" readOnly: true -{{ if not .crdMode }} - - name: cl-controlplane - mountPath: {{.persistencyDirectoryMountPath}} -{{ end }} env: - name: {{ .namespaceEnvVariable }} valueFrom: @@ -194,53 +153,6 @@ spec: mountPath: {{.dataplaneKeyMountPath}} subPath: "key" readOnly: true -{{ if not .crdMode }} ---- -apiVersion: v1 -kind: Pod -metadata: - name: gwctl - namespace: {{.namespace}} - labels: - app: gwctl -spec: - volumes: - - name: ca - secret: - secretName: cl-peer - - name: tls - secret: - secretName: gwctl - containers: - - name: gwctl - image: {{.containerRegistry}}gwctl:{{.tag}} - imagePullPolicy: IfNotPresent - command: ["/bin/sh"] - args: - - -c - - >- - gwctl init --id {{.peer}} \ - --gwIP cl-dataplane \ - --gwPort {{.dataplanePort}} \ - --certca /root/ca.pem \ - --cert /root/cert.pem \ - --key /root/key.pem && - gwctl config use-context --myid {{.peer}} && - sleep infinity - volumeMounts: - - name: ca - mountPath: /root/ca.pem - subPath: "ca" - readOnly: true - - name: tls - mountPath: /root/cert.pem - subPath: "cert" - readOnly: true - - name: tls - mountPath: /root/key.pem - subPath: "key" - readOnly: true -{{ end }} --- apiVersion: v1 kind: Service @@ -280,7 +192,6 @@ rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] -{{ if .crdMode }} - apiGroups: ["clusterlink.net"] resources: ["exports", "peers", "accesspolicies", "privilegedaccesspolicies"] verbs: ["get", "list", "watch"] @@ -290,7 +201,6 @@ rules: - apiGroups: ["clusterlink.net"] resources: ["imports/status", "exports/status", "peers/status"] verbs: ["update"] -{{ end }} --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding @@ -363,8 +273,6 @@ func K8SConfig(config *Config) ([]byte, error) { "controlplanePort": cpapi.ListenPort, "dataplanePort": dpapi.ListenPort, - - "crdMode": config.CRDMode, } var k8sConfig bytes.Buffer @@ -392,8 +300,6 @@ func K8SCertificateConfig(config *Config) ([]byte, error) { "controlplaneKey": base64.StdEncoding.EncodeToString(config.ControlplaneCertificate.RawKey()), "dataplaneCert": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawCert()), "dataplaneKey": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawKey()), - "gwctlCert": base64.StdEncoding.EncodeToString(config.GWCTLCertificate.RawCert()), - "gwctlKey": base64.StdEncoding.EncodeToString(config.GWCTLCertificate.RawKey()), "namespace": config.Namespace, } @@ -456,8 +362,6 @@ func K8SEmptyCertificateConfig(config *Config) ([]byte, error) { "controlplaneKey": "", "dataplaneCert": "", "dataplaneKey": "", - "gwctlCert": "", - "gwctlKey": "", "namespace": config.Namespace, } diff --git a/pkg/client/client.go b/pkg/client/client.go deleted file mode 100644 index 60a4f388d..000000000 --- a/pkg/client/client.go +++ /dev/null @@ -1,84 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package client - -import ( - "crypto/tls" - "encoding/json" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - event "github.com/clusterlink-net/clusterlink/pkg/controlplane/eventmanager" - "github.com/clusterlink-net/clusterlink/pkg/util/jsonapi" - "github.com/clusterlink-net/clusterlink/pkg/util/rest" -) - -// Client for accessing the API. -type Client struct { - client *jsonapi.Client - - // Peers client. - Peers *rest.Client - // Exports client. - Exports *rest.Client - // Imports client. - Imports *rest.Client - // Access policies client. - AccessPolicies *rest.Client -} - -// New returns a new client. -func New(host string, port uint16, tlsConfig *tls.Config) *Client { - client := jsonapi.NewClient(host, port, tlsConfig) - return &Client{ - client: client, - Peers: rest.NewClient(&rest.Config{ - Client: client, - BasePath: "/peers", - SampleObject: v1alpha1.Peer{}, - SampleList: []v1alpha1.Peer{}, - }), - Exports: rest.NewClient(&rest.Config{ - Client: client, - BasePath: "/exports", - SampleObject: v1alpha1.Export{}, - SampleList: []v1alpha1.Export{}, - }), - Imports: rest.NewClient(&rest.Config{ - Client: client, - BasePath: "/imports", - SampleObject: v1alpha1.Import{}, - SampleList: []v1alpha1.Import{}, - }), - AccessPolicies: rest.NewClient(&rest.Config{ - Client: client, - BasePath: "/policies", - SampleObject: v1alpha1.AccessPolicy{}, - SampleList: []v1alpha1.AccessPolicy{}, - }), - } -} - -func (c *Client) GetMetrics() (map[string]event.ConnectionStatusAttr, error) { - var connections map[string]event.ConnectionStatusAttr - path := "/metrics/" + event.ConnectionStatus - resp, err := c.client.Get(path) - if err != nil { - return make(map[string]event.ConnectionStatusAttr), err - } - - if err := json.Unmarshal(resp.Body, &connections); err != nil { - return make(map[string]event.ConnectionStatusAttr), err - } - return connections, nil -} diff --git a/pkg/controlplane/authz/controllers.go b/pkg/controlplane/authz/controllers.go index 280cae367..1e972acec 100644 --- a/pkg/controlplane/authz/controllers.go +++ b/pkg/controlplane/authz/controllers.go @@ -26,81 +26,79 @@ import ( ) // CreateControllers creates the various k8s controllers used to update the xDS manager. -func CreateControllers(mgr *Manager, controllerManager ctrl.Manager, crdMode bool) error { - if crdMode { - err := controller.AddToManager(controllerManager, &controller.Spec{ - Name: "authz.access-policy", - Object: &v1alpha1.AccessPolicy{}, - AddHandler: func(ctx context.Context, object any) error { - accPolicy := connectivitypdp.PolicyFromCR(object.(*v1alpha1.AccessPolicy)) - return mgr.AddAccessPolicy(accPolicy) - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - return mgr.DeleteAccessPolicy(name, false) - }, - }) - if err != nil { - return err - } +func CreateControllers(mgr *Manager, controllerManager ctrl.Manager) error { + err := controller.AddToManager(controllerManager, &controller.Spec{ + Name: "authz.access-policy", + Object: &v1alpha1.AccessPolicy{}, + AddHandler: func(ctx context.Context, object any) error { + accPolicy := connectivitypdp.PolicyFromCR(object.(*v1alpha1.AccessPolicy)) + return mgr.AddAccessPolicy(accPolicy) + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + return mgr.DeleteAccessPolicy(name, false) + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "authz.privileged-access-policy", - Object: &v1alpha1.PrivilegedAccessPolicy{}, - AddHandler: func(_ context.Context, object any) error { - accPolicy := connectivitypdp.PolicyFromPrivilegedCR(object.(*v1alpha1.PrivilegedAccessPolicy)) - return mgr.AddAccessPolicy(accPolicy) - }, - DeleteHandler: func(_ context.Context, name types.NamespacedName) error { - return mgr.DeleteAccessPolicy(name, true) - }, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "authz.privileged-access-policy", + Object: &v1alpha1.PrivilegedAccessPolicy{}, + AddHandler: func(_ context.Context, object any) error { + accPolicy := connectivitypdp.PolicyFromPrivilegedCR(object.(*v1alpha1.PrivilegedAccessPolicy)) + return mgr.AddAccessPolicy(accPolicy) + }, + DeleteHandler: func(_ context.Context, name types.NamespacedName) error { + return mgr.DeleteAccessPolicy(name, true) + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "authz.peer", - Object: &v1alpha1.Peer{}, - AddHandler: func(ctx context.Context, object any) error { - mgr.AddPeer(object.(*v1alpha1.Peer)) - return nil - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - mgr.DeletePeer(name.Name) - return nil - }, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "authz.peer", + Object: &v1alpha1.Peer{}, + AddHandler: func(ctx context.Context, object any) error { + mgr.AddPeer(object.(*v1alpha1.Peer)) + return nil + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + mgr.DeletePeer(name.Name) + return nil + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "authz.import", - Object: &v1alpha1.Import{}, - AddHandler: func(ctx context.Context, object any) error { - return nil - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - return nil - }, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "authz.import", + Object: &v1alpha1.Import{}, + AddHandler: func(ctx context.Context, object any) error { + return nil + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + return nil + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "authz.export", - Object: &v1alpha1.Export{}, - AddHandler: func(ctx context.Context, object any) error { - return nil - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - return nil - }, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "authz.export", + Object: &v1alpha1.Export{}, + AddHandler: func(ctx context.Context, object any) error { + return nil + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + return nil + }, + }) + if err != nil { + return err } return controller.AddToManager(controllerManager, &controller.Spec{ diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index 6abfa21a9..fceb9f18d 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -109,28 +109,9 @@ type Manager struct { jwkSignKey jwk.Key jwkVerifyKey jwk.Key - // callback for getting an import (for non-CRD mode) - getImportCallback func(name string, imp *v1alpha1.Import) error - // callback for getting an export (for non-CRD mode) - getExportCallback func(name string, imp *v1alpha1.Export) error - // callback for getting a peer (for non-CRD mode) - getPeerCallback func(name string, pr *v1alpha1.Peer) error - logger *logrus.Entry } -func (m *Manager) SetGetImportCallback(callback func(name string, imp *v1alpha1.Import) error) { - m.getImportCallback = callback -} - -func (m *Manager) SetGetExportCallback(callback func(name string, imp *v1alpha1.Export) error) { - m.getExportCallback = callback -} - -func (m *Manager) SetGetPeerCallback(callback func(name string, pr *v1alpha1.Peer) error) { - m.getPeerCallback = callback -} - // AddPeer defines a new route target for egress dataplane connections. func (m *Manager) AddPeer(pr *v1alpha1.Peer) { m.logger.Infof("Adding peer '%s'.", pr.Name) @@ -219,7 +200,7 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR } var imp v1alpha1.Import - if err := m.getImport(ctx, req.ImportName, &imp); err != nil { + if err := m.client.Get(ctx, req.ImportName, &imp); err != nil { return nil, fmt.Errorf("cannot get import %v: %w", req.ImportName, err) } @@ -230,9 +211,13 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR } importSource := lbResult.Get() + peerName := types.NamespacedName{ + Name: importSource.Peer, + Namespace: m.namespace, + } var pr v1alpha1.Peer - if err := m.getPeer(ctx, importSource.Peer, &pr); err != nil { + if err := m.client.Get(ctx, peerName, &pr); err != nil { return nil, fmt.Errorf("cannot get peer '%s': %w", importSource.Peer, err) } @@ -349,7 +334,7 @@ func (m *Manager) authorizeIngress( Name: req.ServiceName.Name, } var export v1alpha1.Export - if err := m.getExport(ctx, exportName, &export); err != nil { + if err := m.client.Get(ctx, exportName, &export); err != nil { if errors.IsNotFound(err) || !meta.IsStatusConditionTrue(export.Status.Conditions, v1alpha1.ExportValid) { return resp, nil } @@ -397,34 +382,6 @@ func (m *Manager) authorizeIngress( return resp, nil } -func (m *Manager) getImport(ctx context.Context, name types.NamespacedName, imp *v1alpha1.Import) error { - if m.getImportCallback != nil { - return m.getImportCallback(name.Name, imp) - } - - return m.client.Get(ctx, name, imp) -} - -func (m *Manager) getExport(ctx context.Context, name types.NamespacedName, export *v1alpha1.Export) error { - if m.getExportCallback != nil { - return m.getExportCallback(name.Name, export) - } - - return m.client.Get(ctx, name, export) -} - -func (m *Manager) getPeer(ctx context.Context, name string, pr *v1alpha1.Peer) error { - if m.getPeerCallback != nil { - return m.getPeerCallback(name, pr) - } - - peerName := types.NamespacedName{ - Name: name, - Namespace: m.namespace, - } - return m.client.Get(ctx, peerName, pr) -} - // NewManager returns a new authorization manager. func NewManager(peerTLS *tls.ParsedCertData, cl client.Client, namespace string) (*Manager, error) { // generate RSA key-pair for JWT signing diff --git a/pkg/controlplane/control/controllers.go b/pkg/controlplane/control/controllers.go index d544fcacd..282a706fc 100644 --- a/pkg/controlplane/control/controllers.go +++ b/pkg/controlplane/control/controllers.go @@ -27,62 +27,60 @@ import ( ) // CreateControllers creates the various k8s controllers used to update the control manager. -func CreateControllers(mgr *Manager, controllerManager ctrl.Manager, crdMode bool) error { - if crdMode { - err := controller.AddToManager(controllerManager, &controller.Spec{ - Name: "control.peer", - Object: &v1alpha1.Peer{}, - AddHandler: func(ctx context.Context, object any) error { - mgr.AddPeer(object.(*v1alpha1.Peer)) - return nil - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - mgr.DeletePeer(name.Name) - return nil - }, - }) - if err != nil { - return err - } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "control.service", - Object: &v1.Service{}, - AddHandler: func(ctx context.Context, object any) error { - return mgr.addService(ctx, object.(*v1.Service)) - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - return mgr.deleteService(ctx, name) - }, - }) - if err != nil { - return err - } +func CreateControllers(mgr *Manager, controllerManager ctrl.Manager) error { + err := controller.AddToManager(controllerManager, &controller.Spec{ + Name: "control.peer", + Object: &v1alpha1.Peer{}, + AddHandler: func(ctx context.Context, object any) error { + mgr.AddPeer(object.(*v1alpha1.Peer)) + return nil + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + mgr.DeletePeer(name.Name) + return nil + }, + }) + if err != nil { + return err + } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "control.service", + Object: &v1.Service{}, + AddHandler: func(ctx context.Context, object any) error { + return mgr.addService(ctx, object.(*v1.Service)) + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + return mgr.deleteService(ctx, name) + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "control.export", - Object: &v1alpha1.Export{}, - AddHandler: func(ctx context.Context, object any) error { - return mgr.AddExport(ctx, object.(*v1alpha1.Export)) - }, - DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { - return nil - }, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "control.export", + Object: &v1alpha1.Export{}, + AddHandler: func(ctx context.Context, object any) error { + return mgr.AddExport(ctx, object.(*v1alpha1.Export)) + }, + DeleteHandler: func(ctx context.Context, name types.NamespacedName) error { + return nil + }, + }) + if err != nil { + return err + } - err = controller.AddToManager(controllerManager, &controller.Spec{ - Name: "control.import", - Object: &v1alpha1.Import{}, - AddHandler: func(ctx context.Context, object any) error { - return mgr.AddImport(ctx, object.(*v1alpha1.Import)) - }, - DeleteHandler: mgr.DeleteImport, - }) - if err != nil { - return err - } + err = controller.AddToManager(controllerManager, &controller.Spec{ + Name: "control.import", + Object: &v1alpha1.Import{}, + AddHandler: func(ctx context.Context, object any) error { + return mgr.AddImport(ctx, object.(*v1alpha1.Import)) + }, + DeleteHandler: mgr.DeleteImport, + }) + if err != nil { + return err } return controller.AddToManager(controllerManager, &controller.Spec{ diff --git a/pkg/controlplane/control/manager.go b/pkg/controlplane/control/manager.go index 22a69de42..4f88fb5fb 100644 --- a/pkg/controlplane/control/manager.go +++ b/pkg/controlplane/control/manager.go @@ -137,30 +137,14 @@ type Manager struct { client client.Client namespace string - crdMode bool ports *portManager lock sync.Mutex serviceToImport map[string]types.NamespacedName - // callback for getting all merge imports (for non-CRD mode) - getMergeImportListCallback func() *v1alpha1.ImportList - // callback for getting an import (for non-CRD mode) - getImportCallback func(name string, imp *v1alpha1.Import) error - // callback for setting the status of an export (for non-CRD mode) - exportStatusCallback func(*v1alpha1.Export) - logger *logrus.Entry } -func (m *Manager) SetGetMergeImportListCallback(callback func() *v1alpha1.ImportList) { - m.getMergeImportListCallback = callback -} - -func (m *Manager) SetGetImportCallback(callback func(name string, imp *v1alpha1.Import) error) { - m.getImportCallback = callback -} - // AddImport adds a listening socket for an imported remote service. func (m *Manager) AddImport(ctx context.Context, imp *v1alpha1.Import) (err error) { m.logger.Infof("Adding import '%s/%s'.", imp.Namespace, imp.Name) @@ -171,10 +155,6 @@ func (m *Manager) AddImport(ctx context.Context, imp *v1alpha1.Import) (err erro } defer func() { - if !m.crdMode { - return - } - serviceValidCond := &metav1.Condition{ Type: v1alpha1.ImportServiceValid, Status: metav1.ConditionTrue, @@ -308,10 +288,6 @@ func (m *Manager) DeleteImport(ctx context.Context, name types.NamespacedName) e return errors.Join(errs...) } -func (m *Manager) SetExportStatusCallback(callback func(*v1alpha1.Export)) { - m.exportStatusCallback = callback -} - // AddExport defines a new route target for ingress dataplane connections. func (m *Manager) AddExport(ctx context.Context, export *v1alpha1.Export) (err error) { m.logger.Infof("Adding export '%s/%s'.", export.Namespace, export.Name) @@ -336,20 +312,15 @@ func (m *Manager) AddExport(ctx context.Context, export *v1alpha1.Export) (err e m.logger.Infof( "Updating export '%s/%s' status: %v.", export.Namespace, export.Name, *conditions) - if m.exportStatusCallback != nil { - m.exportStatusCallback(export) - } else { - // CRD-mode - statusError := m.client.Status().Update(ctx, export) - if statusError != nil { - if err == nil { - err = statusError - return - } - - m.logger.Warnf("Error updating export status: %v.", statusError) + statusError := m.client.Status().Update(ctx, export) + if statusError != nil { + if err == nil { + err = statusError return } + + m.logger.Warnf("Error updating export status: %v.", statusError) + return } } @@ -407,8 +378,9 @@ func (m *Manager) addEndpointSlice(ctx context.Context, endpointSlice *discv1.En if endpointSlice.Labels[discv1.LabelServiceName] == dpapp.Name && endpointSlice.Namespace == m.namespace { m.logger.Infof("Adding a dataplane endpoint slice: %s", endpointSlice.Name) - mergeImportList, err := m.getMergeImportList(ctx) - if err != nil { + mergeImportList := v1alpha1.ImportList{} + labelSelector := client.MatchingLabels{v1alpha1.LabelImportMerge: "true"} + if err := m.client.List(ctx, &mergeImportList, labelSelector); err != nil { return err } @@ -465,10 +437,6 @@ func (m *Manager) deleteEndpointSlice(ctx context.Context, name types.Namespaced } func (m *Manager) checkExportService(ctx context.Context, name types.NamespacedName) error { - if !m.crdMode { - return nil - } - var export v1alpha1.Export if err := m.client.Get(ctx, name, &export); err != nil { if !k8serrors.IsNotFound(err) { @@ -483,7 +451,7 @@ func (m *Manager) checkExportService(ctx context.Context, name types.NamespacedN func (m *Manager) checkImportService(ctx context.Context, name types.NamespacedName) error { var imp v1alpha1.Import - if err := m.getImport(ctx, name, &imp); err != nil { + if err := m.client.Get(ctx, name, &imp); err != nil { if !k8serrors.IsNotFound(err) { return err } @@ -505,7 +473,7 @@ func (m *Manager) checkImportService(ctx context.Context, name types.NamespacedN return nil } - if err := m.getImport(ctx, name, &imp); err != nil { + if err := m.client.Get(ctx, name, &imp); err != nil { if !k8serrors.IsNotFound(err) { return err } @@ -530,12 +498,10 @@ func (m *Manager) allocateTargetPort(ctx context.Context, imp *v1alpha1.Import) if imp.Spec.TargetPort == 0 { imp.Spec.TargetPort = leasedPort - if m.crdMode { - m.logger.Infof("Updating target port for import %v.", name) - if err := m.client.Update(ctx, imp); err != nil { - m.ports.Release(name) - return err - } + m.logger.Infof("Updating target port for import %v.", name) + if err := m.client.Update(ctx, imp); err != nil { + m.ports.Release(name) + return err } } @@ -638,7 +604,7 @@ func (m *Manager) checkEndpointSlice(ctx context.Context, namespace, endpointSli var imp v1alpha1.Import var dataplaneEndpointSlice discv1.EndpointSlice shouldDelete := false - if err := m.getImport(ctx, types.NamespacedName{ + if err := m.client.Get(ctx, types.NamespacedName{ Namespace: namespace, Name: parsed.importName, }, &imp); err != nil { @@ -756,7 +722,7 @@ func (m *Manager) addImportEndpointSlices(ctx context.Context, imp *v1alpha1.Imp } if err := m.client.Get(ctx, importName, &v1.Service{}); err != nil { - if k8serrors.IsNotFound(err) && m.crdMode { + if k8serrors.IsNotFound(err) { return &importServiceNotExistError{name: importName} } @@ -808,28 +774,6 @@ func (m *Manager) deleteImportEndpointSlices(ctx context.Context, imp types.Name return nil } -func (m *Manager) getImport(ctx context.Context, name types.NamespacedName, imp *v1alpha1.Import) error { - if m.getImportCallback != nil { - return m.getImportCallback(name.Name, imp) - } - - return m.client.Get(ctx, name, imp) -} - -func (m *Manager) getMergeImportList(ctx context.Context) (*v1alpha1.ImportList, error) { - if m.getMergeImportListCallback != nil { - return m.getMergeImportListCallback(), nil - } - - mergeImportList := v1alpha1.ImportList{} - labelSelector := client.MatchingLabels{v1alpha1.LabelImportMerge: "true"} - if err := m.client.List(ctx, &mergeImportList, labelSelector); err != nil { - return nil, err - } - - return &mergeImportList, nil -} - func checkServiceLabels(service *v1.Service, importName types.NamespacedName) bool { if managedBy, ok := service.Labels[LabelManagedBy]; !ok || managedBy != AppName { return false @@ -945,14 +889,13 @@ func endpointSliceChanged(endpointSlice1, endpointSlice2 *discv1.EndpointSlice) } // NewManager returns a new control manager. -func NewManager(cl client.Client, peerTLS *tls.ParsedCertData, namespace string, crdMode bool) *Manager { +func NewManager(cl client.Client, peerTLS *tls.ParsedCertData, namespace string) *Manager { logger := logrus.WithField("component", "controlplane.control.manager") return &Manager{ peerManager: newPeerManager(cl, peerTLS), client: cl, namespace: namespace, - crdMode: crdMode, ports: newPortManager(), serviceToImport: make(map[string]types.NamespacedName), logger: logger, diff --git a/pkg/controlplane/control/peer.go b/pkg/controlplane/control/peer.go index da336ae5e..708edcae3 100644 --- a/pkg/controlplane/control/peer.go +++ b/pkg/controlplane/control/peer.go @@ -54,9 +54,8 @@ type peerMonitor struct { // peerManager manages peers status. type peerManager struct { - client client.Client - peerTLS *tls.ParsedCertData - peerStatusCallback func(*v1alpha1.Peer) + client client.Client + peerTLS *tls.ParsedCertData lock sync.Mutex monitors map[string]*peerMonitor @@ -190,10 +189,6 @@ func (m *peerManager) Name() string { return "peerManager" } -func (m *peerManager) SetPeerStatusCallback(callback func(*v1alpha1.Peer)) { - m.peerStatusCallback = callback -} - // Start the peer manager. func (m *peerManager) Start() error { m.updaterWG.Add(1) @@ -216,15 +211,10 @@ func (m *peerManager) Start() error { currPeer := monitor.Peer() - if m.peerStatusCallback != nil { - m.peerStatusCallback(&currPeer) - } else { - // CRD-mode - err := m.client.Status().Update(context.Background(), &currPeer) - if err != nil { - m.logger.Warnf("Cannot update peer '%s' status: %v", pr.Name, err) - continue - } + err := m.client.Status().Update(context.Background(), &currPeer) + if err != nil { + m.logger.Warnf("Cannot update peer '%s' status: %v", pr.Name, err) + continue } break diff --git a/pkg/controlplane/eventmanager/events.go b/pkg/controlplane/eventmanager/events.go deleted file mode 100644 index b74e0c52b..000000000 --- a/pkg/controlplane/eventmanager/events.go +++ /dev/null @@ -1,49 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package eventmanager - -import "time" - -type Direction int - -const ( - Incoming Direction = iota - Outgoing -) - -type ConnectionState int - -const ( - Ongoing ConnectionState = iota - Complete - Denied - PeerDenied -) - -const ( - ConnectionStatus = "ConnectionStatus" -) - -type ConnectionStatusAttr struct { - ConnectionID string // Unique ID to track a connection from start to end within the gateway - SrcService string // Source application/service initiating the connection - DstService string // Destination application/service receiving the connection - IncomingBytes int - OutgoingBytes int - DestinationPeer string // The peer(gateway) where the destination/source service is located depending on the Direction - StartTstamp time.Time - LastTstamp time.Time - Direction Direction // Incoming/Outgoing - State ConnectionState -} diff --git a/pkg/controlplane/rest/accesspolicy.go b/pkg/controlplane/rest/accesspolicy.go deleted file mode 100644 index 6175a4265..000000000 --- a/pkg/controlplane/rest/accesspolicy.go +++ /dev/null @@ -1,163 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "encoding/json" - "fmt" - - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/types" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/authz/connectivitypdp" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/store" -) - -type accessPolicyHandler struct { - manager *Manager -} - -func toPDPPolicy(policy *store.AccessPolicy, namespace string) *connectivitypdp.AccessPolicy { - return connectivitypdp.PolicyFromCR(&v1alpha1.AccessPolicy{ - ObjectMeta: metav1.ObjectMeta{ - Name: policy.Name, - Namespace: namespace, - }, - Spec: policy.AccessPolicySpec, - }) -} - -func accessPolicyToAPI(policy *store.AccessPolicy) *v1alpha1.AccessPolicy { - if policy == nil { - return nil - } - - return &v1alpha1.AccessPolicy{ - ObjectMeta: metav1.ObjectMeta{ - Name: policy.Name, - }, - Spec: policy.AccessPolicySpec, - } -} - -// CreateAccessPolicy creates an access policy to allow/deny specific connections. -func (m *Manager) CreateAccessPolicy(policy *store.AccessPolicy) error { - m.logger.Infof("Creating access policy '%s'.", policy.Name) - - if m.initialized { - if err := m.acPolicies.Create(policy); err != nil { - return err - } - } - - return m.authzManager.AddAccessPolicy(toPDPPolicy(policy, m.namespace)) -} - -// UpdateAccessPolicy updates an access policy to allow/deny specific connections. -func (m *Manager) UpdateAccessPolicy(policy *store.AccessPolicy) error { - m.logger.Infof("Updating access policy '%s'.", policy.Name) - - err := m.acPolicies.Update(policy.Name, func(old *store.AccessPolicy) *store.AccessPolicy { - return policy - }) - if err != nil { - return err - } - - return m.authzManager.AddAccessPolicy(toPDPPolicy(policy, m.namespace)) -} - -// DeleteAccessPolicy removes an access policy to allow/deny specific connections. -func (m *Manager) DeleteAccessPolicy(name string) (*store.AccessPolicy, error) { - m.logger.Infof("Deleting access policy '%s'.", name) - - policy, err := m.acPolicies.Delete(name) - if err != nil { - return nil, err - } - if policy == nil { - return nil, nil - } - - namespacedName := types.NamespacedName{ - Name: name, - Namespace: m.namespace, - } - if err := m.authzManager.DeleteAccessPolicy(namespacedName, false); err != nil { - return nil, err - } - - return policy, err -} - -// GetAccessPolicy returns an access policy with the given name. -func (m *Manager) GetAccessPolicy(name string) *store.AccessPolicy { - m.logger.Infof("Getting access policy '%s'.", name) - return m.acPolicies.Get(name) -} - -// GetAllAccessPolicies returns the list of all AccessPolicies. -func (m *Manager) GetAllAccessPolicies() []*store.AccessPolicy { - m.logger.Info("Listing all access policies.") - return m.acPolicies.GetAll() -} - -// Decode an access policy. -func (h *accessPolicyHandler) Decode(data []byte) (any, error) { - var policy v1alpha1.AccessPolicy - if err := json.Unmarshal(data, &policy); err != nil { - return nil, fmt.Errorf("cannot decode access policy: %w", err) - } - - if err := policy.Spec.Validate(); err != nil { - return nil, err - } - - return store.NewAccessPolicy(&policy), nil -} - -// Create an access policy. -func (h *accessPolicyHandler) Create(object any) error { - return h.manager.CreateAccessPolicy(object.(*store.AccessPolicy)) -} - -// Update an access policy. -func (h *accessPolicyHandler) Update(object any) error { - return h.manager.UpdateAccessPolicy(object.(*store.AccessPolicy)) -} - -// Delete an access policy. -func (h *accessPolicyHandler) Delete(name any) (any, error) { - return h.manager.DeleteAccessPolicy(name.(string)) -} - -// Get an access policy. -func (h *accessPolicyHandler) Get(name string) (any, error) { - policy := h.manager.GetAccessPolicy(name) - if policy == nil { - return nil, nil - } - return accessPolicyToAPI(policy), nil -} - -// List all access policies. -func (h *accessPolicyHandler) List() (any, error) { - policies := h.manager.GetAllAccessPolicies() - apiPolicies := make([]*v1alpha1.AccessPolicy, len(policies)) - for i, policy := range policies { - apiPolicies[i] = accessPolicyToAPI(policy) - } - return apiPolicies, nil -} diff --git a/pkg/controlplane/rest/export.go b/pkg/controlplane/rest/export.go deleted file mode 100644 index f4d93702e..000000000 --- a/pkg/controlplane/rest/export.go +++ /dev/null @@ -1,207 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "context" - "encoding/json" - "fmt" - - "k8s.io/apimachinery/pkg/api/errors" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/types" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/store" -) - -type exportHandler struct { - manager *Manager -} - -func toK8SExport(export *store.Export, namespace string) *v1alpha1.Export { - return &v1alpha1.Export{ - ObjectMeta: metav1.ObjectMeta{ - Name: export.Name, - Namespace: namespace, - }, - Spec: v1alpha1.ExportSpec{ - Host: export.ExportSpec.Host, - Port: export.ExportSpec.Port, - }, - Status: export.Status, - } -} - -func exportToAPI(export *store.Export) *v1alpha1.Export { - if export == nil { - return nil - } - - return &v1alpha1.Export{ - ObjectMeta: metav1.ObjectMeta{ - Name: export.Name, - }, - Spec: export.ExportSpec, - Status: export.Status, - } -} - -// CreateExport defines a new route target for ingress dataplane connections. -func (m *Manager) CreateExport(export *store.Export) error { - m.logger.Infof("Creating export '%s'.", export.Name) - - if m.initialized { - if err := m.exports.Create(export); err != nil { - return err - } - } - - k8sExport := toK8SExport(export, m.namespace) - if err := m.xdsManager.AddExport(k8sExport); err != nil { - return err - } - return m.controlManager.AddExport(context.Background(), k8sExport) -} - -// UpdateExport updates a new route target for ingress dataplane connections. -func (m *Manager) UpdateExport(export *store.Export) error { - m.logger.Infof("Updating export '%s'.", export.Name) - - err := m.exports.Update(export.Name, func(old *store.Export) *store.Export { - return export - }) - if err != nil { - return err - } - - k8sExport := toK8SExport(export, m.namespace) - if err := m.xdsManager.AddExport(k8sExport); err != nil { - return err - } - return m.controlManager.AddExport(context.Background(), k8sExport) -} - -// UpdateExportStatus updates the status of an existing export. -func (m *Manager) UpdateExportStatus(name string, status *v1alpha1.ExportStatus) { - m.logger.Infof("Updating status of export '%s'.", name) - - err := m.exports.Update(name, func(old *store.Export) *store.Export { - old.Status = *status - return old - }) - if err != nil { - m.logger.Errorf("Error updating status of export '%s': %v", name, err) - } -} - -// GetExport returns an existing export. -func (m *Manager) GetExport(name string) *store.Export { - m.logger.Infof("Getting export '%s'.", name) - return m.exports.Get(name) -} - -// DeleteExport removes the possibility for ingress dataplane connections to access a given service. -func (m *Manager) DeleteExport(name string) (*store.Export, error) { - m.logger.Infof("Deleting export '%s'.", name) - - export, err := m.exports.Delete(name) - if err != nil { - return nil, err - } - if export == nil { - return nil, nil - } - - namespacedName := types.NamespacedName{ - Name: name, - Namespace: m.namespace, - } - err = m.xdsManager.DeleteExport(namespacedName) - if err != nil { - // practically impossible - return export, err - } - - return export, nil -} - -// GetAllExports returns the list of all exports. -func (m *Manager) GetAllExports() []*store.Export { - m.logger.Info("Listing all exports.") - return m.exports.GetAll() -} - -func (m *Manager) GetK8sExport(name string, export *v1alpha1.Export) error { - storeExport := m.exports.Get(name) - if storeExport == nil { - return errors.NewNotFound(schema.GroupResource{}, name) - } - - *export = *toK8SExport(storeExport, m.namespace) - return nil -} - -// Decode an export. -func (h *exportHandler) Decode(data []byte) (any, error) { - var export v1alpha1.Export - if err := json.Unmarshal(data, &export); err != nil { - return nil, fmt.Errorf("cannot decode export: %w", err) - } - - if export.Name == "" { - return nil, fmt.Errorf("empty export name") - } - - if export.Spec.Port == 0 { - return nil, fmt.Errorf("missing service port") - } - - return store.NewExport(&export), nil -} - -// Create an export. -func (h *exportHandler) Create(object any) error { - return h.manager.CreateExport(object.(*store.Export)) -} - -// Update an export. -func (h *exportHandler) Update(object any) error { - return h.manager.UpdateExport(object.(*store.Export)) -} - -// Get an export. -func (h *exportHandler) Get(name string) (any, error) { - export := exportToAPI(h.manager.GetExport(name)) - if export == nil { - return nil, nil - } - return export, nil -} - -// Delete an export. -func (h *exportHandler) Delete(name any) (any, error) { - return h.manager.DeleteExport(name.(string)) -} - -// List all exports. -func (h *exportHandler) List() (any, error) { - exports := h.manager.GetAllExports() - apiExports := make([]*v1alpha1.Export, len(exports)) - for i, export := range exports { - apiExports[i] = exportToAPI(export) - } - return apiExports, nil -} diff --git a/pkg/controlplane/rest/import.go b/pkg/controlplane/rest/import.go deleted file mode 100644 index d7832af05..000000000 --- a/pkg/controlplane/rest/import.go +++ /dev/null @@ -1,245 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "context" - "encoding/json" - "fmt" - - "k8s.io/apimachinery/pkg/api/errors" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/types" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/store" -) - -type importHandler struct { - manager *Manager -} - -func toK8SImport(imp *store.Import, namespace string) *v1alpha1.Import { - return &v1alpha1.Import{ - ObjectMeta: metav1.ObjectMeta{ - Name: imp.Name, - Namespace: namespace, - Labels: imp.Labels, - }, - Spec: imp.ImportSpec, - } -} - -func importToAPI(imp *store.Import) *v1alpha1.Import { - if imp == nil { - return nil - } - - return &v1alpha1.Import{ - ObjectMeta: metav1.ObjectMeta{ - Name: imp.Name, - Labels: imp.Labels, - }, - Spec: imp.ImportSpec, - } -} - -// CreateImport creates a listening socket for an imported remote service. -func (m *Manager) CreateImport(imp *store.Import) error { - m.logger.Infof("Creating import '%s'.", imp.Name) - - k8sImp := toK8SImport(imp, m.namespace) - - if m.initialized { - if err := m.imports.Create(imp); err != nil { - return err - } - - err := m.controlManager.AddImport(context.Background(), k8sImp) - if err != nil { - return err - } - - imp.TargetPort = k8sImp.Spec.TargetPort - - err = m.imports.Update(imp.Name, func(old *store.Import) *store.Import { - return imp - }) - if err != nil { - return err - } - } - - if err := m.xdsManager.AddImport(k8sImp); err != nil { - // practically impossible - return err - } - - return nil -} - -// UpdateImport updates a listening socket for an imported remote service. -func (m *Manager) UpdateImport(imp *store.Import) error { - m.logger.Infof("Updating import '%s'.", imp.Name) - - err := m.imports.Update(imp.Name, func(old *store.Import) *store.Import { - return imp - }) - if err != nil { - return err - } - - k8sImp := toK8SImport(imp, m.namespace) - err = m.controlManager.AddImport(context.Background(), k8sImp) - if err != nil { - return err - } - - imp.TargetPort = k8sImp.Spec.TargetPort - - err = m.imports.Update(imp.Name, func(old *store.Import) *store.Import { - return imp - }) - if err != nil { - return err - } - - if err := m.xdsManager.AddImport(k8sImp); err != nil { - // practically impossible - return err - } - - return nil -} - -// GetImport returns an existing import. -func (m *Manager) GetImport(name string) *store.Import { - m.logger.Infof("Getting import '%s'.", name) - return m.imports.Get(name) -} - -// DeleteImport removes the listening socket of a previously imported service. -func (m *Manager) DeleteImport(name string) (*store.Import, error) { - m.logger.Infof("Deleting import '%s'.", name) - - imp, err := m.imports.Delete(name) - if err != nil { - return nil, err - } - if imp == nil { - return nil, nil - } - - namespacedName := types.NamespacedName{ - Name: name, - Namespace: m.namespace, - } - if err := m.xdsManager.DeleteImport(namespacedName); err != nil { - // practically impossible - return imp, err - } - - err = m.controlManager.DeleteImport(context.Background(), namespacedName) - if err != nil { - return nil, err - } - - return imp, nil -} - -// GetAllImports returns the list of all imports. -func (m *Manager) GetAllImports() []*store.Import { - m.logger.Info("Listing all imports.") - return m.imports.GetAll() -} - -func (m *Manager) GetMergeImportList() *v1alpha1.ImportList { - mergeImportList := v1alpha1.ImportList{} - for _, imp := range m.imports.GetAll() { - if imp.Labels[v1alpha1.LabelImportMerge] != "true" { - continue - } - - mergeImportList.Items = append(mergeImportList.Items, *toK8SImport(imp, m.namespace)) - } - - return &mergeImportList -} - -func (m *Manager) GetK8sImport(name string, imp *v1alpha1.Import) error { - storeImport := m.imports.Get(name) - if storeImport == nil { - return errors.NewNotFound(schema.GroupResource{}, name) - } - - *imp = *toK8SImport(storeImport, m.namespace) - return nil -} - -// Decode an import. -func (h *importHandler) Decode(data []byte) (any, error) { - var imp v1alpha1.Import - if err := json.Unmarshal(data, &imp); err != nil { - return nil, fmt.Errorf("cannot decode import: %w", err) - } - - if imp.Name == "" { - return nil, fmt.Errorf("empty import name") - } - - if imp.Spec.Port == 0 { - return nil, fmt.Errorf("missing service port") - } - - if len(imp.Spec.Sources) == 0 { - return nil, fmt.Errorf("missing sources") - } - - return store.NewImport(&imp), nil -} - -// Create an import. -func (h *importHandler) Create(object any) error { - return h.manager.CreateImport(object.(*store.Import)) -} - -// Update an import. -func (h *importHandler) Update(object any) error { - return h.manager.UpdateImport(object.(*store.Import)) -} - -// Get an import. -func (h *importHandler) Get(name string) (any, error) { - imp := importToAPI(h.manager.GetImport(name)) - if imp == nil { - return nil, nil - } - return imp, nil -} - -// Delete an import. -func (h *importHandler) Delete(name any) (any, error) { - return h.manager.DeleteImport(name.(string)) -} - -// List all imports. -func (h *importHandler) List() (any, error) { - imports := h.manager.GetAllImports() - apiImports := make([]*v1alpha1.Import, len(imports)) - for i, imp := range imports { - apiImports[i] = importToAPI(imp) - } - return apiImports, nil -} diff --git a/pkg/controlplane/rest/manager.go b/pkg/controlplane/rest/manager.go deleted file mode 100644 index d210e5e02..000000000 --- a/pkg/controlplane/rest/manager.go +++ /dev/null @@ -1,134 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "fmt" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/controlplane/authz" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/control" - cpstore "github.com/clusterlink-net/clusterlink/pkg/controlplane/store" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/xds" - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// Manager of a controlplane, where all API servers delegate their requested actions to. -type Manager struct { - namespace string - - peers *cpstore.Peers - exports *cpstore.Exports - imports *cpstore.Imports - acPolicies *cpstore.AccessPolicies - - xdsManager *xds.Manager - authzManager *authz.Manager - controlManager *control.Manager - - initialized bool - - logger *logrus.Entry -} - -// init initializes the controlplane manager. -func (m *Manager) init() error { - // add peers - for _, p := range m.GetAllPeers() { - if err := m.CreatePeer(p); err != nil { - return err - } - } - - // add exports - for _, export := range m.GetAllExports() { - if err := m.CreateExport(export); err != nil { - return err - } - } - - // add exports - for _, imp := range m.GetAllImports() { - if err := m.CreateImport(imp); err != nil { - return err - } - } - - // add access policies - for _, policy := range m.GetAllAccessPolicies() { - if err := m.CreateAccessPolicy(policy); err != nil { - return err - } - } - - m.initialized = true - - return nil -} - -// NewManager returns a new controlplane CRUD manager. -func NewManager( - namespace string, - storeManager store.Manager, - xdsManager *xds.Manager, - authzManager *authz.Manager, - controlManager *control.Manager, -) (*Manager, error) { - logger := logrus.WithField("component", "controlplane.rest.manager") - - peers, err := cpstore.NewPeers(storeManager) - if err != nil { - return nil, fmt.Errorf("cannot load peers from store: %w", err) - } - logger.Infof("Loaded %d peers.", peers.Len()) - - exports, err := cpstore.NewExports(storeManager) - if err != nil { - return nil, fmt.Errorf("cannot load exports from store: %w", err) - } - logger.Infof("Loaded %d exports.", exports.Len()) - - imports, err := cpstore.NewImports(storeManager) - if err != nil { - return nil, fmt.Errorf("cannot load imports from store: %w", err) - } - logger.Infof("Loaded %d imports.", imports.Len()) - - acPolicies, err := cpstore.NewAccessPolicies(storeManager) - if err != nil { - return nil, fmt.Errorf("cannot load access policies from store: %w", err) - } - logger.Infof("Loaded %d access policies.", acPolicies.Len()) - - m := &Manager{ - namespace: namespace, - peers: peers, - exports: exports, - imports: imports, - acPolicies: acPolicies, - xdsManager: xdsManager, - authzManager: authzManager, - controlManager: controlManager, - initialized: false, - logger: logger, - } - - // initialize instance - if err := m.init(); err != nil { - return nil, err - } - - return m, nil -} diff --git a/pkg/controlplane/rest/peer.go b/pkg/controlplane/rest/peer.go deleted file mode 100644 index 7bce24c55..000000000 --- a/pkg/controlplane/rest/peer.go +++ /dev/null @@ -1,210 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "encoding/json" - "fmt" - - "k8s.io/apimachinery/pkg/api/errors" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime/schema" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/store" -) - -type peerHandler struct { - manager *Manager -} - -func toK8SPeer(peer *store.Peer) *v1alpha1.Peer { - k8sPeer := &v1alpha1.Peer{ - ObjectMeta: metav1.ObjectMeta{Name: peer.Name}, - Spec: v1alpha1.PeerSpec{ - Gateways: make([]v1alpha1.Endpoint, len(peer.Gateways)), - }, - Status: peer.Status, - } - - for i, gw := range peer.PeerSpec.Gateways { - k8sPeer.Spec.Gateways[i].Host = gw.Host - k8sPeer.Spec.Gateways[i].Port = gw.Port - } - - return k8sPeer -} - -func peerToAPI(peer *store.Peer) *v1alpha1.Peer { - if peer == nil { - return nil - } - - return &v1alpha1.Peer{ - ObjectMeta: metav1.ObjectMeta{ - Name: peer.Name, - }, - Spec: peer.PeerSpec, - Status: peer.Status, - } -} - -// CreatePeer defines a new route target for egress dataplane connections. -func (m *Manager) CreatePeer(peer *store.Peer) error { - m.logger.Infof("Creating peer '%s'.", peer.Name) - - if m.initialized { - if err := m.peers.Create(peer); err != nil { - return err - } - } - - k8sPeer := toK8SPeer(peer) - m.authzManager.AddPeer(k8sPeer) - m.controlManager.AddPeer(k8sPeer) - return m.xdsManager.AddPeer(k8sPeer) -} - -// UpdatePeer updates new route target for egress dataplane connections. -func (m *Manager) UpdatePeer(peer *store.Peer) error { - m.logger.Infof("Updating peer '%s'.", peer.Name) - - err := m.peers.Update(peer.Name, func(old *store.Peer) *store.Peer { - return peer - }) - if err != nil { - return err - } - - k8sPeer := toK8SPeer(peer) - m.authzManager.AddPeer(k8sPeer) - m.controlManager.AddPeer(k8sPeer) - return m.xdsManager.AddPeer(k8sPeer) -} - -// UpdatePeerStatus updates the status of an existing peer. -func (m *Manager) UpdatePeerStatus(name string, status *v1alpha1.PeerStatus) { - m.logger.Infof("Updating status of peer '%s'.", name) - - err := m.peers.Update(name, func(old *store.Peer) *store.Peer { - old.Status = *status - return old - }) - if err != nil { - m.logger.Errorf("Error updating status of peer '%s': %v", name, err) - } -} - -// GetPeer returns an existing peer. -func (m *Manager) GetPeer(name string) *store.Peer { - m.logger.Infof("Getting peer '%s'.", name) - return m.peers.Get(name) -} - -// DeletePeer removes the possibility for egress dataplane connections to be routed to a given peer. -func (m *Manager) DeletePeer(name string) (*store.Peer, error) { - m.logger.Infof("Deleting peer '%s'.", name) - - pr, err := m.peers.Delete(name) - if err != nil { - return nil, err - } - if pr == nil { - return nil, nil - } - - m.authzManager.DeletePeer(name) - m.controlManager.DeletePeer(name) - - err = m.xdsManager.DeletePeer(name) - if err != nil { - // practically impossible - return nil, err - } - - return pr, nil -} - -// GetAllPeers returns the list of all peers. -func (m *Manager) GetAllPeers() []*store.Peer { - m.logger.Info("Listing all peers.") - return m.peers.GetAll() -} - -func (m *Manager) GetK8sPeer(name string, peer *v1alpha1.Peer) error { - storePeer := m.peers.Get(name) - if storePeer == nil { - return errors.NewNotFound(schema.GroupResource{}, name) - } - - *peer = *toK8SPeer(storePeer) - return nil -} - -// Decode a peer. -func (h *peerHandler) Decode(data []byte) (any, error) { - var peer v1alpha1.Peer - if err := json.Unmarshal(data, &peer); err != nil { - return nil, fmt.Errorf("cannot decode peer: %w", err) - } - - if peer.Name == "" { - return nil, fmt.Errorf("empty peer name") - } - - for i, ep := range peer.Spec.Gateways { - if ep.Host == "" { - return nil, fmt.Errorf("gateway #%d missing host", i) - } - if ep.Port == 0 { - return nil, fmt.Errorf("gateway #%d (host '%s') missing port", i, ep.Host) - } - } - - return store.NewPeer(&peer), nil -} - -// Create a peer. -func (h *peerHandler) Create(object any) error { - return h.manager.CreatePeer(object.(*store.Peer)) -} - -// Update a peer. -func (h *peerHandler) Update(object any) error { - return h.manager.UpdatePeer(object.(*store.Peer)) -} - -// Get a peer. -func (h *peerHandler) Get(name string) (any, error) { - peer := peerToAPI(h.manager.GetPeer(name)) - if peer == nil { - return nil, nil - } - return peer, nil -} - -// Delete a peer. -func (h *peerHandler) Delete(name any) (any, error) { - return h.manager.DeletePeer(name.(string)) -} - -// List all peers. -func (h *peerHandler) List() (any, error) { - peers := h.manager.GetAllPeers() - apiPeers := make([]*v1alpha1.Peer, len(peers)) - for i, peer := range peers { - apiPeers[i] = peerToAPI(peer) - } - return apiPeers, nil -} diff --git a/pkg/controlplane/rest/server.go b/pkg/controlplane/rest/server.go deleted file mode 100644 index e2306f663..000000000 --- a/pkg/controlplane/rest/server.go +++ /dev/null @@ -1,45 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "github.com/clusterlink-net/clusterlink/pkg/util/rest" -) - -// RegisteHandlers registers the HTTP handlers for REST requests. -func RegisterHandlers(manager *Manager, srv *rest.Server) { - srv.AddObjectHandlers(&rest.ServerObjectSpec{ - BasePath: "/peers", - Handler: &peerHandler{manager: manager}, - DeleteByValue: false, - }) - - srv.AddObjectHandlers(&rest.ServerObjectSpec{ - BasePath: "/exports", - Handler: &exportHandler{manager: manager}, - DeleteByValue: false, - }) - - srv.AddObjectHandlers(&rest.ServerObjectSpec{ - BasePath: "/imports", - Handler: &importHandler{manager: manager}, - DeleteByValue: false, - }) - - srv.AddObjectHandlers(&rest.ServerObjectSpec{ - BasePath: "/policies", - Handler: &accessPolicyHandler{manager: manager}, - DeleteByValue: false, - }) -} diff --git a/pkg/controlplane/store/accesspolicies.go b/pkg/controlplane/store/accesspolicies.go deleted file mode 100644 index 62338117f..000000000 --- a/pkg/controlplane/store/accesspolicies.go +++ /dev/null @@ -1,165 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -import ( - "fmt" - "sync" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// AccessPolicies is a cached persistent store of Access Policies. -type AccessPolicies struct { - lock sync.RWMutex - cache map[string]*AccessPolicy - store store.ObjectStore - - logger *logrus.Entry -} - -// Create an AccessPolicy. -func (s *AccessPolicies) Create(policy *AccessPolicy) error { - s.logger.Infof("Creating: '%s'.", policy.Name) - - if policy.Version > accessPolicyStructVersion { - return fmt.Errorf("incompatible access policy version %d, expected: %d", - policy.Version, accessPolicyStructVersion) - } - - // persist to store - if err := s.store.Create(policy.Name, policy); err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[policy.Name] = policy - return nil -} - -// Update an access policy. -func (s *AccessPolicies) Update(name string, mutator func(*AccessPolicy) *AccessPolicy) error { - s.logger.Infof("Updating: '%s'.", name) - - // persist to store - var policy *AccessPolicy - err := s.store.Update(name, func(a any) any { - policy = mutator(a.(*AccessPolicy)) - return policy - }) - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[name] = policy - return nil -} - -// Get an access policy. -func (s *AccessPolicies) Get(name string) *AccessPolicy { - s.logger.Debugf("Getting '%s'.", name) - - s.lock.RLock() - defer s.lock.RUnlock() - return s.cache[name] -} - -// Delete an access policy. -func (s *AccessPolicies) Delete(name string) (*AccessPolicy, error) { - s.logger.Infof("Deleting: '%s'.", name) - - // delete from store - if err := s.store.Delete(name); err != nil { - return nil, err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // delete from cache - val := s.cache[name] - delete(s.cache, name) - return val, nil -} - -// GetAll returns all access policies in the cache. -func (s *AccessPolicies) GetAll() []*AccessPolicy { - s.logger.Debug("Getting all access policies.") - - s.lock.RLock() - defer s.lock.RUnlock() - - policies := make([]*AccessPolicy, 0, len(s.cache)) - for _, policy := range s.cache { - policies = append(policies, policy) - } - - return policies -} - -// Len returns the number of cached access policies. -func (s *AccessPolicies) Len() int { - s.lock.RLock() - defer s.lock.RUnlock() - return len(s.cache) -} - -// init loads the cache with items from the backing store. -func (s *AccessPolicies) init() error { - s.logger.Info("Initializing.") - - // get all policies from backing store - policies, err := s.store.GetAll() - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store all policies to the cache - for _, object := range policies { - if policy, ok := object.(*AccessPolicy); ok { - s.cache[policy.Name] = policy - } - } - - return nil -} - -// NewAccessPolicies returns a new cached store of access policies. -func NewAccessPolicies(manager store.Manager) (*AccessPolicies, error) { - logger := logrus.WithField("component", "controlplane.store.accesspolicies") - - policies := &AccessPolicies{ - cache: make(map[string]*AccessPolicy), - store: manager.GetObjectStore(accessPolicyStoreName, AccessPolicy{}), - logger: logger, - } - - if err := policies.init(); err != nil { - return nil, err - } - - return policies, nil -} diff --git a/pkg/controlplane/store/exports.go b/pkg/controlplane/store/exports.go deleted file mode 100644 index 7abca5de3..000000000 --- a/pkg/controlplane/store/exports.go +++ /dev/null @@ -1,166 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -import ( - "fmt" - "sync" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// Exports is a cached persistent store of exports. -type Exports struct { - lock sync.RWMutex - cache map[string]*Export - store store.ObjectStore - - logger *logrus.Entry -} - -// Create an export. -func (s *Exports) Create(export *Export) error { - s.logger.Infof("Creating: '%s'.", export.Name) - - if export.Version > exportStructVersion { - return fmt.Errorf("incompatible export version %d, expected: %d", - export.Version, exportStructVersion) - } - - // persist to store - if err := s.store.Create(export.Name, export); err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[export.Name] = export - return nil -} - -// Update an export. -func (s *Exports) Update(name string, mutator func(*Export) *Export) error { - s.logger.Infof("Updating: '%s'.", name) - - // persist to store - var export *Export - err := s.store.Update(name, func(a any) any { - export = mutator(a.(*Export)) - return export - }) - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[name] = export - return nil -} - -// Get an export. -func (s *Exports) Get(name string) *Export { - s.logger.Debugf("Getting '%s'.", name) - - s.lock.RLock() - defer s.lock.RUnlock() - - return s.cache[name] -} - -// Delete an export. -func (s *Exports) Delete(name string) (*Export, error) { - s.logger.Infof("Deleting: '%s'.", name) - - // delete from store - if err := s.store.Delete(name); err != nil { - return nil, err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // delete from cache - val := s.cache[name] - delete(s.cache, name) - return val, nil -} - -// GetAll returns all exports in the cache. -func (s *Exports) GetAll() []*Export { - s.logger.Debug("Getting all exports.") - - s.lock.RLock() - defer s.lock.RUnlock() - - exports := make([]*Export, 0, len(s.cache)) - for _, export := range s.cache { - exports = append(exports, export) - } - - return exports -} - -// Len returns the number of cached exports. -func (s *Exports) Len() int { - s.lock.RLock() - defer s.lock.RUnlock() - return len(s.cache) -} - -// init loads the cache with items from the backing store. -func (s *Exports) init() error { - s.logger.Info("Initializing.") - - // get all exports from backing store - exports, err := s.store.GetAll() - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store all exports to the cache - for _, object := range exports { - if export, ok := object.(*Export); ok { - s.cache[export.Name] = export - } - } - - return nil -} - -// NewExports returns a new cached store of exports. -func NewExports(manager store.Manager) (*Exports, error) { - logger := logrus.WithField("component", "controlplane.store.exports") - - exports := &Exports{ - cache: make(map[string]*Export), - store: manager.GetObjectStore(exportStoreName, Export{}), - logger: logger, - } - - if err := exports.init(); err != nil { - return nil, err - } - - return exports, nil -} diff --git a/pkg/controlplane/store/imports.go b/pkg/controlplane/store/imports.go deleted file mode 100644 index 97c42dedd..000000000 --- a/pkg/controlplane/store/imports.go +++ /dev/null @@ -1,165 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -import ( - "fmt" - "sync" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// Imports is a cached persistent store of imports. -type Imports struct { - lock sync.RWMutex - cache map[string]*Import - store store.ObjectStore - - logger *logrus.Entry -} - -// Create an import. -func (s *Imports) Create(imp *Import) error { - s.logger.Infof("Creating: '%s'.", imp.Name) - - if imp.Version > importStructVersion { - return fmt.Errorf("incompatible import version %d, expected: %d", - imp.Version, importStructVersion) - } - - // persist to store - if err := s.store.Create(imp.Name, imp); err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[imp.Name] = imp - return nil -} - -// Update an import. -func (s *Imports) Update(name string, mutator func(*Import) *Import) error { - s.logger.Infof("Updating: '%s'.", name) - - // persist to store - var imp *Import - err := s.store.Update(name, func(a any) any { - imp = mutator(a.(*Import)) - return imp - }) - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[name] = imp - return nil -} - -// Get an import. -func (s *Imports) Get(name string) *Import { - s.logger.Debugf("Getting '%s'.", name) - - s.lock.RLock() - defer s.lock.RUnlock() - return s.cache[name] -} - -// Delete an import. -func (s *Imports) Delete(name string) (*Import, error) { - s.logger.Infof("Deleting: '%s'.", name) - - // delete from store - if err := s.store.Delete(name); err != nil { - return nil, err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // delete from cache - val := s.cache[name] - delete(s.cache, name) - return val, nil -} - -// GetAll returns all imports in the cache. -func (s *Imports) GetAll() []*Import { - s.logger.Debug("Getting all imports.") - - s.lock.RLock() - defer s.lock.RUnlock() - - imports := make([]*Import, 0, len(s.cache)) - for _, imp := range s.cache { - imports = append(imports, imp) - } - - return imports -} - -// Len returns the number of cached imports. -func (s *Imports) Len() int { - s.lock.RLock() - defer s.lock.RUnlock() - return len(s.cache) -} - -// init loads the cache with items from the backing store. -func (s *Imports) init() error { - s.logger.Info("Initializing.") - - // get all imports from backing store - imports, err := s.store.GetAll() - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store all imports to the cache - for _, object := range imports { - if imp, ok := object.(*Import); ok { - s.cache[imp.Name] = imp - } - } - - return nil -} - -// NewImports returns a new cached store of imports. -func NewImports(manager store.Manager) (*Imports, error) { - logger := logrus.WithField("component", "controlplane.store.imports") - - imports := &Imports{ - cache: make(map[string]*Import), - store: manager.GetObjectStore(importStoreName, Import{}), - logger: logger, - } - - if err := imports.init(); err != nil { - return nil, err - } - - return imports, nil -} diff --git a/pkg/controlplane/store/peers.go b/pkg/controlplane/store/peers.go deleted file mode 100644 index 57adc2e9d..000000000 --- a/pkg/controlplane/store/peers.go +++ /dev/null @@ -1,165 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -import ( - "fmt" - "sync" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// Peers is a cached persistent store of peers. -type Peers struct { - lock sync.RWMutex - cache map[string]*Peer - store store.ObjectStore - - logger *logrus.Entry -} - -// Create a peer. -func (s *Peers) Create(peer *Peer) error { - s.logger.Infof("Creating: '%s'.", peer.Name) - - if peer.Version > peerStructVersion { - return fmt.Errorf("incompatible peer version %d, expected: %d", - peer.Version, peerStructVersion) - } - - // persist to store - if err := s.store.Create(peer.Name, peer); err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[peer.Name] = peer - return nil -} - -// Update a peer. -func (s *Peers) Update(name string, mutator func(*Peer) *Peer) error { - s.logger.Infof("Updating: '%s'.", name) - - // persist to store - var peer *Peer - err := s.store.Update(name, func(a any) any { - peer = mutator(a.(*Peer)) - return peer - }) - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store in cache - s.cache[name] = peer - return nil -} - -// Get a peer. -func (s *Peers) Get(name string) *Peer { - s.logger.Debugf("Getting '%s'.", name) - - s.lock.RLock() - defer s.lock.RUnlock() - return s.cache[name] -} - -// Delete a peer. -func (s *Peers) Delete(name string) (*Peer, error) { - s.logger.Infof("Deleting: '%s'.", name) - - // delete from store - if err := s.store.Delete(name); err != nil { - return nil, err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // delete from cache - val := s.cache[name] - delete(s.cache, name) - return val, nil -} - -// GetAll returns all peers in the cache. -func (s *Peers) GetAll() []*Peer { - s.logger.Debug("Getting all peers.") - - s.lock.RLock() - defer s.lock.RUnlock() - - peers := make([]*Peer, 0, len(s.cache)) - for _, peer := range s.cache { - peers = append(peers, peer) - } - - return peers -} - -// Len returns the number of cached peers. -func (s *Peers) Len() int { - s.lock.RLock() - defer s.lock.RUnlock() - return len(s.cache) -} - -// init loads the cache with items from the backing store. -func (s *Peers) init() error { - s.logger.Info("Initializing.") - - // get all peers from backing store - peers, err := s.store.GetAll() - if err != nil { - return err - } - - s.lock.Lock() - defer s.lock.Unlock() - - // store all peers to the cache - for _, object := range peers { - if peer, ok := object.(*Peer); ok { - s.cache[peer.Name] = peer - } - } - - return nil -} - -// NewPeers returns a new cached store of peers. -func NewPeers(manager store.Manager) (*Peers, error) { - logger := logrus.WithField("component", "controlplane.store.peers") - - peers := &Peers{ - cache: make(map[string]*Peer), - store: manager.GetObjectStore(peerStoreName, Peer{}), - logger: logger, - } - - if err := peers.init(); err != nil { - return nil, err - } - - return peers, nil -} diff --git a/pkg/controlplane/store/types.go b/pkg/controlplane/store/types.go deleted file mode 100644 index c12c8fd1e..000000000 --- a/pkg/controlplane/store/types.go +++ /dev/null @@ -1,111 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -import ( - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" -) - -const ( - peerStoreName = "peer" - exportStoreName = "export" - importStoreName = "import" - accessPolicyStoreName = "accessPolicy" - - exportStructVersion = 1 - importStructVersion = 1 - peerStructVersion = 1 - accessPolicyStructVersion = 1 -) - -// Peer represents a remote peer. -type Peer struct { - v1alpha1.PeerSpec - // Name of the peer. - Name string - // Status of the peer. - Status v1alpha1.PeerStatus - // Version of the struct when object was created. - Version uint32 -} - -// NewPeer creates a new peer. -func NewPeer(peer *v1alpha1.Peer) *Peer { - return &Peer{ - PeerSpec: peer.Spec, - Name: peer.Name, - Status: peer.Status, - Version: peerStructVersion, - } -} - -// Export represents a local service that may be exported. -type Export struct { - v1alpha1.ExportSpec - // Name of the export. - Name string - // Status of the export. - Status v1alpha1.ExportStatus - // Version of the struct when object was created. - Version uint32 -} - -// NewExport creates a new export. -func NewExport(export *v1alpha1.Export) *Export { - return &Export{ - ExportSpec: export.Spec, - Name: export.Name, - Status: export.Status, - Version: exportStructVersion, - } -} - -// Import represents an external service that can be bound to (multiple) exported services of remote peers. -type Import struct { - v1alpha1.ImportSpec - // Name of import. - Name string - // Labels defined for the import - Labels map[string]string - // Version of the struct when object was created. - Version uint32 -} - -// NewImport creates a new import. -func NewImport(imp *v1alpha1.Import) *Import { - return &Import{ - ImportSpec: imp.Spec, - Name: imp.Name, - Labels: imp.Labels, - Version: importStructVersion, - } -} - -// AccessPolicy to allow/deny specific connections. -type AccessPolicy struct { - v1alpha1.AccessPolicySpec - // Name of access policy. - Name string - // Version of the struct when object was created. - Version uint32 -} - -// NewAccessPolicy creates a new access policy. -func NewAccessPolicy(policy *v1alpha1.AccessPolicy) *AccessPolicy { - return &AccessPolicy{ - AccessPolicySpec: policy.Spec, - Name: policy.Name, - Version: accessPolicyStructVersion, - } -} diff --git a/pkg/controlplane/xds/manager.go b/pkg/controlplane/xds/manager.go index e1147a5cd..0c316bed5 100644 --- a/pkg/controlplane/xds/manager.go +++ b/pkg/controlplane/xds/manager.go @@ -45,8 +45,6 @@ import ( // - Import -> Listener (whose name starts with a designated prefix) // Note that imported service bindings are handled by the egress authz server. type Manager struct { - crdMode bool - clusters *cache.LinearCache listeners *cache.LinearCache @@ -132,7 +130,7 @@ func (m *Manager) DeleteExport(name types.NamespacedName) error { func (m *Manager) AddImport(imp *v1alpha1.Import) error { m.logger.Infof("Adding import '%s/%s'.", imp.Namespace, imp.Name) - if m.crdMode && !meta.IsStatusConditionTrue(imp.Status.Conditions, v1alpha1.ImportTargetPortValid) { + if !meta.IsStatusConditionTrue(imp.Status.Conditions, v1alpha1.ImportTargetPortValid) { // target port not yet allocated, skip m.logger.Infof("Skipping import with no valid target port '%s/%s'.", imp.Namespace, imp.Name) return nil @@ -283,11 +281,10 @@ func makeTCPProxyFilter(clusterName, statPrefix string, } // NewManager creates an uninitialized, non-registered xDS manager. -func NewManager(crdMode bool) *Manager { +func NewManager() *Manager { logger := logrus.WithField("component", "controlplane.xds.manager") return &Manager{ - crdMode: crdMode, clusters: cache.NewLinearCache(resource.ClusterType, cache.WithLogger(logger)), listeners: cache.NewLinearCache(resource.ListenerType, cache.WithLogger(logger)), logger: logger, diff --git a/pkg/k8s/kubernetes/kube.go b/pkg/k8s/kubernetes/kube.go deleted file mode 100644 index 9c0016c61..000000000 --- a/pkg/k8s/kubernetes/kube.go +++ /dev/null @@ -1,23 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package kubernetes - -// InitializeKubeDeployment initiates the informers to keep a watch on pod/services/replicasets. -func InitializeKubeDeployment(k8sConfigPath string) error { - err := Data.InitFromConfig(k8sConfigPath) - if err != nil { - return err - } - return nil -} diff --git a/pkg/k8s/kubernetes/kubeinformer.go b/pkg/k8s/kubernetes/kubeinformer.go deleted file mode 100644 index abfb7c819..000000000 --- a/pkg/k8s/kubernetes/kubeinformer.go +++ /dev/null @@ -1,477 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package kubernetes - -import ( - "context" - "errors" - "fmt" - "os" - "path" - "time" - - log "github.com/sirupsen/logrus" - appsv1 "k8s.io/api/apps/v1" - v1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/labels" - "k8s.io/apimachinery/pkg/util/intstr" - "k8s.io/client-go/informers" - "k8s.io/client-go/kubernetes" - "k8s.io/client-go/rest" - "k8s.io/client-go/tools/cache" - "k8s.io/client-go/tools/clientcmd" -) - -// Data contain the kube informer datat should be part of the control plane. -var Data kubeDataInterface = &kubeData{} - -const ( - kubeConfigEnvVariable = "KUBECONFIG" - syncTime = 1 * time.Minute - indexIP = "byIP" - typePod = "Pod" - typeService = "Service" - // AppLabel default pod label kubeinformer use. - AppLabel = "app" -) - -type kubeDataInterface interface { - GetInfoIP(string) (*Info, error) - GetInfoApp(string) (*Info, error) - GetLabel(string, string) (string, error) - GetIpFromLabel(string) ([]string, error) - InitFromConfig(string) error - CreateService(string, int, int, string, string) error - CreateEndpoint(string, string, string, int) error - DeleteService(string) error - DeleteEndpoint(string) error - CheckServiceExist(string) bool - CheckEndpointExist(string) bool -} - -// kubeData contain all k8s information of the cluster/namespace. -type kubeData struct { - kubeDataInterface - // pods and services cache the different object types as *Info pointers - pods cache.SharedIndexInformer - services cache.SharedIndexInformer - // replicaSets caches the ReplicaSets as partially-filled *ObjectMeta pointers - replicaSets cache.SharedIndexInformer - kubeClient *kubernetes.Clientset - serviceMap map[string]string - endpointMap map[string]string - stopChan chan struct{} -} - -type owner struct { - Type string - Name string -} - -// Info contains precollected metadata for Pods and Services. -// Not all the fields are populated for all the above types. To save -// memory, we just keep in memory the necessary data for each Type. -// For more information about which fields are set for each type, please -// refer to the instantiation function of the respective informers. -type Info struct { - // Informers need that internal object is an ObjectMeta instance - metav1.ObjectMeta - Type string - Owner owner - HostIP string - ips []string -} - -var commonIndexers = map[string]cache.IndexFunc{ - indexIP: func(obj interface{}) ([]string, error) { - return obj.(*Info).ips, nil - }, -} - -// GetInfoIP Return the pod information according to the pod Ip. -func (k *kubeData) GetInfoIP(ip string) (*Info, error) { - if info, ok := k.fetchInformers(ip); ok { - // Owner data might be discovered after the owned, so we fetch it - // at the last moment - if info.Owner.Name == "" { - info.Owner = k.getOwner(info) - } - return info, nil - } - - return nil, fmt.Errorf("informers can't find IP %s", ip) -} - -// Return pod information according to the application name. -func (k *kubeData) GetInfoApp(app string) (*Info, error) { - podLister := k.pods.GetIndexer() - // Define the label selector - labelSelector := labels.SelectorFromSet(labels.Set{"app": app}) - // List all pods - allPods := podLister.List() - - // Find the first pod that matches the label selector. - for _, pod := range allPods { - if labelSelector.Matches(labels.Set(pod.(*Info).Labels)) { - return pod.(*Info), nil - } - } - return nil, fmt.Errorf("informers can't find App %s", app) -} - -// Get Ip and key(prefix of label) and return pod label. -func (k *kubeData) GetLabel(ip, key string) (string, error) { - if info, ok := k.fetchInformers(ip); ok { - // Owner data might be discovered after the owned, so we fetch it - // at the last moment - if info.Owner.Name == "" { - info.Owner = k.getOwner(info) - } - return info.Labels[key], nil - } - - return "", fmt.Errorf("informers can't find IP %s", ip) -} - -// GetIPFromLabel Get label(prefix of label) and return pod ip. -func (k *kubeData) GetIPFromLabel(label string) ([]string, error) { - namespace := "default" - label = AppLabel + "=" + label - - podList, err := k.kubeClient.CoreV1().Pods(namespace).List(context.TODO(), metav1.ListOptions{ - LabelSelector: label, - }) - if err != nil { - log.Error(err) - return nil, err - } - if len(podList.Items) == 0 { - log.Errorf("No pods found for label selector %v", label) - return nil, fmt.Errorf("no pods found for label selector %q", label) - } - - podIPs := make([]string, 0, len(podList.Items)) - // We assume that the first matching pod is the correct one - for i := range podList.Items { - podIPs = append(podIPs, podList.Items[i].Status.PodIP) - } - - log.Infof("The label %v match to pod ips %v\n", label, podIPs) - return podIPs, nil -} - -func (k *kubeData) fetchInformers(ip string) (*Info, bool) { - if info, ok := infoForIP(k.pods.GetIndexer(), ip); ok { - return info, true - } - if info, ok := infoForIP(k.services.GetIndexer(), ip); ok { - return info, true - } - return nil, false -} - -func infoForIP(idx cache.Indexer, ip string) (*Info, bool) { - objs, err := idx.ByIndex(indexIP, ip) - if err != nil { - log.WithError(err).WithField("ip", ip).Debug("error accessing index. Ignoring") - return nil, false - } - if len(objs) == 0 { - return nil, false - } - return objs[0].(*Info), true -} - -func (k *kubeData) getOwner(info *Info) owner { - self := owner{ // unless we discover otherwise, an object owns itself - Name: info.Name, - Type: info.Type, - } - - if len(info.OwnerReferences) != 0 { - return self - } - - ownerRef := owner{ - Name: info.OwnerReferences[0].Name, - Type: info.OwnerReferences[0].Kind, - } - - if ownerRef.Type != "ReplicaSet" { - return ownerRef - } - - item, ok, err := k.replicaSets.GetIndexer().GetByKey(info.Namespace + "/" + ownerRef.Name) - if err == nil || !ok { // unable to retrieve the ReplicaSet, return what we already have - log.WithError(err).WithField("key", info.Namespace+"/"+ownerRef.Name). - Debug("can't get ReplicaSet info from informer. Ignoring") - return ownerRef - } - - if rsInfo, ok := item.(*metav1.ObjectMeta); ok && len(rsInfo.OwnerReferences) > 0 { - return owner{ // ReplicaSet created in response to another object - Name: rsInfo.OwnerReferences[0].Name, - Type: rsInfo.OwnerReferences[0].Kind, - } - } - return ownerRef -} - -func (k *kubeData) initPodInformer(informerFactory informers.SharedInformerFactory) error { - pods := informerFactory.Core().V1().Pods().Informer() - // Transform any *v1.Pod instance into a *Info instance to save space - // in the informer's cache - if err := pods.SetTransform(func(i interface{}) (interface{}, error) { - pod, ok := i.(*v1.Pod) - if !ok { - return nil, fmt.Errorf("was expecting a Pod. Got: %T", i) - } - ips := make([]string, 0, len(pod.Status.PodIPs)) - for _, ip := range pod.Status.PodIPs { - // ignoring host-networked Pod IPs - if ip.IP != pod.Status.HostIP { - ips = append(ips, ip.IP) - } - } - return &Info{ - ObjectMeta: metav1.ObjectMeta{ - Name: pod.Name, - Namespace: pod.Namespace, - Labels: pod.Labels, - OwnerReferences: pod.OwnerReferences, - }, - Type: typePod, - HostIP: pod.Status.HostIP, - ips: ips, - }, nil - }); err != nil { - return fmt.Errorf("can't set pods transform: %w", err) - } - if err := pods.AddIndexers(commonIndexers); err != nil { - return fmt.Errorf("can't add %s indexer to Pods informer: %w", indexIP, err) - } - - k.pods = pods - return nil -} - -func (k *kubeData) initServiceInformer(informerFactory informers.SharedInformerFactory) error { - services := informerFactory.Core().V1().Services().Informer() - // Transform any *v1.Service instance into a *Info instance to save space - // in the informer's cache - if err := services.SetTransform(func(i interface{}) (interface{}, error) { - svc, ok := i.(*v1.Service) - if !ok { - return nil, fmt.Errorf("was expecting a Service. Got: %T", i) - } - if svc.Spec.ClusterIP == v1.ClusterIPNone { - return nil, errors.New("not indexing service without ClusterIP") - } - return &Info{ - ObjectMeta: metav1.ObjectMeta{ - Name: svc.Name, - Namespace: svc.Namespace, - Labels: svc.Labels, - }, - Type: typeService, - ips: svc.Spec.ClusterIPs, - }, nil - }); err != nil { - return fmt.Errorf("can't set services transform: %w", err) - } - if err := services.AddIndexers(commonIndexers); err != nil { - return fmt.Errorf("can't add %s indexer to Pods informer: %w", indexIP, err) - } - - k.services = services - return nil -} - -func (k *kubeData) initReplicaSetInformer(informerFactory informers.SharedInformerFactory) error { - k.replicaSets = informerFactory.Apps().V1().ReplicaSets().Informer() - // To save space, instead of storing a complete *appvs1.Replicaset instance, the - // informer's cache will store a *metav1.ObjectMeta with the minimal required fields - if err := k.replicaSets.SetTransform(func(i interface{}) (interface{}, error) { - rs, ok := i.(*appsv1.ReplicaSet) - if !ok { - return nil, fmt.Errorf("was expecting a ReplicaSet. Got: %T", i) - } - return &metav1.ObjectMeta{ - Name: rs.Name, - Namespace: rs.Namespace, - OwnerReferences: rs.OwnerReferences, - }, nil - }); err != nil { - return fmt.Errorf("can't set ReplicaSets transform: %w", err) - } - return nil -} - -func (k *kubeData) InitFromConfig(kubeConfigPath string) error { - // Initialization variables - k.stopChan = make(chan struct{}) - - config, err := loadConfig(kubeConfigPath) - if err != nil { - return err - } - - k.kubeClient, err = kubernetes.NewForConfig(config) - if err != nil { - return err - } - - err = k.initInformers(k.kubeClient) - if err != nil { - return err - } - - return nil -} - -func loadConfig(kubeConfigPath string) (*rest.Config, error) { - // if no config path is provided, load it from the env variable - if kubeConfigPath == "" { - kubeConfigPath = os.Getenv(kubeConfigEnvVariable) - } - // otherwise, load it from the $HOME/.kube/config file - if kubeConfigPath == "" { - homeDir, err := os.UserHomeDir() - if err != nil { - return nil, fmt.Errorf("can't get user home dir: %w", err) - } - kubeConfigPath = path.Join(homeDir, ".kube", "config") - } - config, err := clientcmd.BuildConfigFromFlags("", kubeConfigPath) - if err == nil { - return config, nil - } - // fallback: use in-cluster config - config, err = rest.InClusterConfig() - if err != nil { - return nil, fmt.Errorf("can't access kubenetes. Tried using config from: "+ - "config parameter, %s env, homedir and InClusterConfig. Got: %w", - kubeConfigEnvVariable, err) - } - return config, nil -} - -func (k *kubeData) initInformers(client kubernetes.Interface) error { - informerFactory := informers.NewSharedInformerFactory(client, syncTime) - - err := k.initPodInformer(informerFactory) - if err != nil { - return err - } - err = k.initServiceInformer(informerFactory) - if err != nil { - return err - } - err = k.initReplicaSetInformer(informerFactory) - if err != nil { - return err - } - - log.Infof("Starting kubernetes informers, waiting for synchronization") - informerFactory.Start(k.stopChan) - informerFactory.WaitForCacheSync(k.stopChan) - k.serviceMap = make(map[string]string) - k.endpointMap = make(map[string]string) - log.Infof("Kubernetes informers started") - - return nil -} - -// CreateService Add support to create a service/NodePort for a target port. -func (k *kubeData) CreateService(serviceName string, port, targetPort int, namespace, svcAppName string) error { - var selectorMap map[string]string - if svcAppName != "" { - selectorMap = map[string]string{"app": svcAppName} - } - serviceSpec := &v1.Service{ - ObjectMeta: metav1.ObjectMeta{Name: serviceName}, - Spec: v1.ServiceSpec{ - Ports: []v1.ServicePort{ - { - Protocol: v1.ProtocolTCP, - Port: int32(port), - TargetPort: intstr.FromInt(targetPort), - }, - }, - Type: v1.ServiceTypeClusterIP, - Selector: selectorMap, - }, - } - - _, err := k.kubeClient.CoreV1().Services(namespace).Create(context.TODO(), serviceSpec, metav1.CreateOptions{}) - if err != nil { - return err - } - - k.serviceMap[serviceName] = namespace - - return nil -} - -// CreateEndpoint create k8s endpoint. -func (k *kubeData) CreateEndpoint(epName, namespace, targetIP string, targetPort int) error { - endpoint := &v1.Endpoints{ - ObjectMeta: metav1.ObjectMeta{ - Name: epName, - Namespace: namespace, - }, - Subsets: []v1.EndpointSubset{ - { - Addresses: []v1.EndpointAddress{ - { - IP: targetIP, // Replace with the desired IP address of the endpoint. - }, - }, - Ports: []v1.EndpointPort{ - { - Port: int32(targetPort), - }, - }, - }, - }, - } - - _, err := k.kubeClient.CoreV1().Endpoints(namespace).Create(context.TODO(), endpoint, metav1.CreateOptions{}) - if err != nil { - log.Errorf("Error creating endpoint: %s", err) - return err - } - k.serviceMap[epName] = namespace - - return nil -} - -// DeleteService delete k8s service. -func (k *kubeData) DeleteService(serviceName string) error { - if namespace, ok := k.serviceMap[serviceName]; ok { - return k.kubeClient.CoreV1().Services(namespace).Delete(context.TODO(), serviceName, metav1.DeleteOptions{}) - } - - return fmt.Errorf("serviceName: %s is not exists", serviceName) -} - -// DeleteEndpoint delete k8s endpoint. -func (k *kubeData) DeleteEndpoint(epName string) error { - if namespace, ok := k.endpointMap[epName]; ok { - return k.kubeClient.CoreV1().Endpoints(namespace).Delete(context.TODO(), epName, metav1.DeleteOptions{}) - } - - return fmt.Errorf("epName: %s is not exists", epName) -} diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go deleted file mode 100644 index 4d98adec9..000000000 --- a/pkg/metrics/metrics.go +++ /dev/null @@ -1,90 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -/**********************************************************/ -/* Package Metrics provides an exporter of gateway's connection-level metrics -/**********************************************************/ -package metrics - -import ( - "encoding/json" - "net/http" - - "github.com/go-chi/chi" - "github.com/sirupsen/logrus" - - event "github.com/clusterlink-net/clusterlink/pkg/controlplane/eventmanager" -) - -var ( - mlog = logrus.WithField("component", "Metrics") - MyMetricsManager Metrics -) - -type Metrics struct { - ConnectionFlow map[string]*event.ConnectionStatusAttr -} - -func (m *Metrics) Routes(r *chi.Mux) chi.Router { - r.Route("/"+event.ConnectionStatus, func(r chi.Router) { - r.Get("/", m.GetConnectionMetrics) // Get Metrics from the metrics manager - r.Post("/", m.PostConnectionMetrics) // Post Metrics to the metrics manager - }) - // TODO : Add more endpoints to support query - return r -} - -func (m *Metrics) init(router *chi.Mux) { - m.ConnectionFlow = make(map[string]*event.ConnectionStatusAttr) - - routes := m.Routes(router) - - router.Mount("/metrics", routes) -} - -func (m *Metrics) GetConnectionMetrics(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - if err := json.NewEncoder(w).Encode(m.ConnectionFlow); err != nil { - mlog.Errorf("Error happened in JSON encode. Err: %s", err) - } -} - -func (m *Metrics) PostConnectionMetrics(w http.ResponseWriter, r *http.Request) { - var connectionStatus event.ConnectionStatusAttr - err := json.NewDecoder(r.Body).Decode(&connectionStatus) - if err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - // Aggregate Metrics - m.aggregateMetrics(&connectionStatus) -} - -func (m *Metrics) aggregateMetrics(connectionStatus *event.ConnectionStatusAttr) { - if _, exists := m.ConnectionFlow[connectionStatus.ConnectionID]; exists { - // Update existing metrics - flow := m.ConnectionFlow[connectionStatus.ConnectionID] - flow.IncomingBytes += connectionStatus.IncomingBytes - flow.OutgoingBytes += connectionStatus.OutgoingBytes - flow.LastTstamp = connectionStatus.LastTstamp - flow.State = connectionStatus.State - } else { - m.ConnectionFlow[connectionStatus.ConnectionID] = connectionStatus - } -} - -func StartMetricsManager(router *chi.Mux) { - mlog.Infof("Metrics Manager started") - MyMetricsManager.init(router) -} diff --git a/pkg/operator/controller/instance_controller.go b/pkg/operator/controller/instance_controller.go index 3629f3423..b14827dcb 100644 --- a/pkg/operator/controller/instance_controller.go +++ b/pkg/operator/controller/instance_controller.go @@ -241,7 +241,7 @@ func (r *InstanceReconciler) applyControlplane(ctx context.Context, instance *cl Name: ControlPlaneName, Image: instance.Spec.ContainerRegistry + ControlPlaneName + ":" + instance.Spec.Tag, ImagePullPolicy: corev1.PullIfNotPresent, - Args: []string{"--log-level", instance.Spec.LogLevel, "--crd-mode"}, + Args: []string{"--log-level", instance.Spec.LogLevel}, Ports: []corev1.ContainerPort{ { ContainerPort: cpapi.ListenPort, diff --git a/pkg/store/kv/bolt/store.go b/pkg/store/kv/bolt/store.go deleted file mode 100644 index 3d976d1e6..000000000 --- a/pkg/store/kv/bolt/store.go +++ /dev/null @@ -1,125 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package bolt - -import ( - "bytes" - "fmt" - - "github.com/sirupsen/logrus" - "go.etcd.io/bbolt" - - "github.com/clusterlink-net/clusterlink/pkg/store/kv" -) - -const ( - bucketName = "clink" -) - -// Store implements a store backed by Bolt. -type Store struct { - db *bbolt.DB - - logger *logrus.Entry -} - -// Create a (key, value) in the store. -func (s *Store) Create(key, value []byte) error { - s.logger.Debugf("Creating key: %v.", key) - - return s.db.Update(func(tx *bbolt.Tx) error { - bucket := tx.Bucket([]byte(bucketName)) - if bucket.Get(key) != nil { - return &kv.KeyExistsError{} - } - return bucket.Put(key, value) - }) -} - -// Update a (key, value) in the store. -func (s *Store) Update(key []byte, mutator func([]byte) ([]byte, error)) error { - s.logger.Debugf("Updating key: %v.", key) - - return s.db.Update(func(tx *bbolt.Tx) error { - bucket := tx.Bucket([]byte(bucketName)) - - value := bucket.Get(key) - if value == nil { - return &kv.KeyNotFoundError{} - } - - // update value - updated, err := mutator(value) - if err != nil { - return err - } - - return bucket.Put(key, updated) - }) -} - -// Delete a key (with its respective value) from the store. -func (s *Store) Delete(key []byte) error { - s.logger.Debugf("Deleting key: %v.", key) - - return s.db.Update(func(tx *bbolt.Tx) error { - return tx.Bucket([]byte(bucketName)).Delete(key) - }) -} - -// Range calls f sequentially for each (key, value) where key starts with the given prefix. -func (s *Store) Range(prefix []byte, f func(key, value []byte) error) error { - s.logger.Infof("Iterating over all items with key prefix '%s'.", prefix) - - return s.db.View(func(tx *bbolt.Tx) error { - c := tx.Bucket([]byte(bucketName)).Cursor() - for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() { - s.logger.Debugf("Read key from store: %v.", k) - - if err := f(k, v); err != nil { - return err - } - } - return nil - }) -} - -// Close frees all resources (e.g. file handles, network sockets) used by the store. -func (s *Store) Close() error { - s.logger.Info("Closing store.") - return s.db.Close() -} - -// Open a bolt store. -func Open(path string) (*Store, error) { - // open - db, err := bbolt.Open(path, 0o666, nil) - if err != nil { - return nil, fmt.Errorf("unable to open store: %w", err) - } - - // create the single bucket we use (if does not exist) - err = db.Update(func(tx *bbolt.Tx) error { - _, err := tx.CreateBucketIfNotExists([]byte(bucketName)) - return err - }) - if err != nil { - return nil, fmt.Errorf("unable to create bucket: %w", err) - } - - return &Store{ - db, - logrus.WithField("component", "store.kv.bolt"), - }, nil -} diff --git a/pkg/store/kv/manager.go b/pkg/store/kv/manager.go deleted file mode 100644 index cd94f0125..000000000 --- a/pkg/store/kv/manager.go +++ /dev/null @@ -1,31 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package kv - -import "github.com/clusterlink-net/clusterlink/pkg/store" - -// Manager of multiple object stores that are persisted together. -type Manager struct { - store Store -} - -// GetObjectStore returns a store for a specific object type. -func (m *Manager) GetObjectStore(name string, sampleObject any) store.ObjectStore { - return NewObjectStore(name, m.store, sampleObject) -} - -// NewManager returns a new manager. -func NewManager(s Store) *Manager { - return &Manager{store: s} -} diff --git a/pkg/store/kv/store.go b/pkg/store/kv/store.go deleted file mode 100644 index 2034212a1..000000000 --- a/pkg/store/kv/store.go +++ /dev/null @@ -1,138 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package kv - -import ( - "encoding/json" - "errors" - "fmt" - "reflect" - - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" -) - -// ObjectStore represents a persistent store of objects, backed by a KV-store. -// Key format for an object is: .. -type ObjectStore struct { - store Store - - keyPrefix string - objectType reflect.Type - - logger *logrus.Entry -} - -// kvKey encodes object keys to a single key identifying the object in the store. -func (s *ObjectStore) kvKey(name string) []byte { - return []byte(s.keyPrefix + name) -} - -// Create an object. -func (s *ObjectStore) Create(name string, value any) error { - s.logger.Infof("Creating: '%s'.", name) - - // serialize - encoded, err := json.Marshal(value) - if err != nil { - return fmt.Errorf("unable to serialize object: %w", err) - } - - // persist to store - if err := s.store.Create(s.kvKey(name), encoded); err != nil { - var keyExistsError *KeyExistsError - if errors.As(err, &keyExistsError) { - return &store.ObjectExistsError{} - } - return err - } - - return nil -} - -// Update an object. -func (s *ObjectStore) Update(name string, mutator func(any) any) error { - s.logger.Infof("Updating: '%s'.", name) - - // persist to store - err := s.store.Update(s.kvKey(name), func(value []byte) ([]byte, error) { - // de-serialize old value - decoded := reflect.New(s.objectType).Interface() - if err := json.Unmarshal(value, decoded); err != nil { - return nil, fmt.Errorf("unable to decode value for object '%s': %w", name, err) - } - - // serialize mutated value - encoded, err := json.Marshal(mutator(decoded)) - if err != nil { - return nil, fmt.Errorf("unable to serialize mutated object '%s': %w", name, err) - } - - return encoded, nil - }) - if err != nil { - var keyNotFoundError *KeyNotFoundError - if errors.As(err, &keyNotFoundError) { - return &store.ObjectNotFoundError{} - } - return err - } - - return nil -} - -// Delete an object identified by the given name. -func (s *ObjectStore) Delete(name string) error { - s.logger.Infof("Deleting: '%s'.", name) - return s.store.Delete(s.kvKey(name)) -} - -// GetAll returns all of the objects in the store. -func (s *ObjectStore) GetAll() ([]any, error) { - s.logger.Info("Getting all objects.") - - var objects []any - err := s.store.Range([]byte(s.keyPrefix), func(key, value []byte) error { - s.logger.Debugf("De-serializing: %v.", value) - - decoded := reflect.New(s.objectType).Interface() - if err := json.Unmarshal(value, decoded); err != nil { - return fmt.Errorf("unable to decode object for key %v: %w", key, err) - } - - objects = append(objects, decoded) - return nil - }) - if err != nil { - return nil, err - } - - return objects, nil -} - -// NewObjectStore returns a new object store backed by a KV-store. -func NewObjectStore(name string, s Store, sampleObject any) *ObjectStore { - logger := logrus.WithFields(logrus.Fields{ - "component": "store.kv.object-store", - "name": name, - }) - - return &ObjectStore{ - store: s, - keyPrefix: name + ".", - objectType: reflect.TypeOf(sampleObject), - logger: logger, - } -} diff --git a/pkg/store/kv/types.go b/pkg/store/kv/types.go deleted file mode 100644 index 09729b2af..000000000 --- a/pkg/store/kv/types.go +++ /dev/null @@ -1,44 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package kv - -// Store represents a key-value store. -type Store interface { - // Create a (key, value) in the store. - // Returns KeyExistsError if key already exists. - Create(key, value []byte) error - // Update a (key, value) in the store. - // Returns KeyNotFoundError if key does not exist. - Update(key []byte, mutator func([]byte) ([]byte, error)) error - // Delete a key (with its respective value) from the store. - Delete(key []byte) error - // Range calls f sequentially for each (key, value) where key starts with the given prefix. - Range(prefix []byte, f func(key, value []byte) error) error - // Close frees all resources (e.g. file handles, network sockets) used by the Store. - Close() error -} - -// KeyExistsError represents an error caused due to a key which exists. -type KeyExistsError struct{} - -func (e *KeyExistsError) Error() string { - return "key already exists" -} - -// KeyNotFoundError represents an error caused due to a key which does not exist. -type KeyNotFoundError struct{} - -func (e *KeyNotFoundError) Error() string { - return "key not found" -} diff --git a/pkg/store/types.go b/pkg/store/types.go deleted file mode 100644 index c096a0a80..000000000 --- a/pkg/store/types.go +++ /dev/null @@ -1,48 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package store - -// Manager of multiple object stores that are persisted together. -type Manager interface { - // GetObjectStore returns a store for a specific object type. - GetObjectStore(name string, sampleObject any) ObjectStore -} - -// ObjectStore represents a persistent store of objects. -type ObjectStore interface { - // Create an object. - // Returns ObjectExistsError if object already exists. - Create(name string, object any) error - // Update an object. - // Returns ObjectNotFoundError if object does not exist. - Update(name string, mutator func(any) any) error - // Delete an object identified by the given name. - Delete(name string) error - // GetAll returns all of the objects in the store. - GetAll() ([]any, error) -} - -// ObjectExistsError represents an error caused due to an object which exists. -type ObjectExistsError struct{} - -func (e *ObjectExistsError) Error() string { - return "object already exists" -} - -// ObjectNotFoundError represents an error caused due to an object which does not exist. -type ObjectNotFoundError struct{} - -func (e *ObjectNotFoundError) Error() string { - return "object not found" -} diff --git a/pkg/util/http/server.go b/pkg/util/http/server.go index 579118dcd..8aaff2b32 100644 --- a/pkg/util/http/server.go +++ b/pkg/util/http/server.go @@ -73,7 +73,7 @@ func (s *Server) GracefulStop() error { } // NewServer returns a new server. -func NewServer(name string, tlsConfig *tls.Config) Server { +func NewServer(name string, tlsConfig *tls.Config) *Server { logger := logrus.WithFields(logrus.Fields{ "component": "http-server", "name": name, @@ -89,7 +89,7 @@ func NewServer(name string, tlsConfig *tls.Config) Server { } router.Use(middleware.Recoverer) - return Server{ + return &Server{ Listener: tcp.NewListener(name), name: name, router: router, diff --git a/pkg/util/net/util.go b/pkg/util/net/util.go deleted file mode 100644 index 06a212df3..000000000 --- a/pkg/util/net/util.go +++ /dev/null @@ -1,39 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -/**********************************************************/ -/* Package netutils contain helper functions for network -/* connection -/**********************************************************/ - -package net - -import ( - "net" - "regexp" -) - -var ( - dnsPattern = `^[a-zA-Z0-9-]{1,63}(\.[a-zA-Z0-9-]{1,63})*$` - dnsRegex = regexp.MustCompile(dnsPattern) -) - -// IsIP returns true if the input is valid IPv4 or IPv6. -func IsIP(str string) bool { - return net.ParseIP(str) != nil -} - -// IsDNS returns true if the input is valid DNS. -func IsDNS(s string) bool { - return dnsRegex.MatchString(s) -} diff --git a/pkg/util/rest/client.go b/pkg/util/rest/client.go deleted file mode 100644 index 3ecdb2861..000000000 --- a/pkg/util/rest/client.go +++ /dev/null @@ -1,163 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "encoding/json" - "fmt" - "net/http" - "reflect" - - "github.com/clusterlink-net/clusterlink/pkg/util/jsonapi" -) - -// Config specifies a client configuration. -type Config struct { - // Client is the underlying HTTP client. - Client *jsonapi.Client - // BasePath is the server HTTP path for manipulating a specific type of objects. - BasePath string - // SampleObject is an instance representing the type returned when getting an object. - SampleObject any - // SampleList is an instance representing the type returned when listing objects. - SampleList any -} - -// Client for issuing REST-JSON requests for a specific type of objects. -type Client struct { - client *jsonapi.Client - basePath string - objectType reflect.Type - listType reflect.Type -} - -// Create an object. -func (c *Client) Create(object any) error { - encoded, err := json.Marshal(object) - if err != nil { - return fmt.Errorf("unable to encode object: %w", err) - } - - resp, err := c.client.Post(c.basePath, encoded) - if err != nil { - return fmt.Errorf("unable to create object: %w", err) - } - - if resp.Status != http.StatusCreated { - return fmt.Errorf("unable to create object (%d), server returned: %s", - resp.Status, resp.Body) - } - - return nil -} - -// Update an object. -func (c *Client) Update(object any) error { - encoded, err := json.Marshal(object) - if err != nil { - return fmt.Errorf("unable to encode object: %w", err) - } - - resp, err := c.client.Put(c.basePath, encoded) - if err != nil { - return fmt.Errorf("unable to update object: %w", err) - } - - if resp.Status != http.StatusNoContent { - return fmt.Errorf("unable to update object (%d), server returned: %s", - resp.Status, resp.Body) - } - - return nil -} - -// Get an object. -func (c *Client) Get(name string) (any, error) { - resp, err := c.client.Get(c.basePath + "/" + name) - if err != nil { - return nil, fmt.Errorf("unable to get object: %w", err) - } - - if resp.Status != http.StatusOK { - return nil, fmt.Errorf("unable to get object (%d), server returned: %s", - resp.Status, resp.Body) - } - - decoded := reflect.New(c.objectType).Interface() - if err := json.Unmarshal(resp.Body, decoded); err != nil { - return nil, fmt.Errorf("unable to decode object %v: %w", decoded, err) - } - - return decoded, nil -} - -// Delete an object, either by name or the object itself. -func (c *Client) Delete(object any) error { - var body []byte - path := c.basePath - - if name, ok := object.(string); ok { - // delete by name - path = path + "/" + name - } else { - // delete by object - encoded, err := json.Marshal(object) - if err != nil { - return fmt.Errorf("cannot encode object: %w", err) - } - body = encoded - } - - resp, err := c.client.Delete(path, body) - if err != nil { - return fmt.Errorf("unable to delete object: %w", err) - } - - if resp.Status != http.StatusNoContent { - return fmt.Errorf("unable to delete object (%d), server returned: %s", - resp.Status, resp.Body) - } - - return nil -} - -// List all objects. -func (c *Client) List() (any, error) { - resp, err := c.client.Get(c.basePath) - if err != nil { - return nil, fmt.Errorf("unable to list objects: %w", err) - } - - if resp.Status != http.StatusOK { - return nil, fmt.Errorf("unable to list objects (%d), server returned: %s", - resp.Status, resp.Body) - } - - decoded := reflect.New(c.listType).Interface() - if err := json.Unmarshal(resp.Body, decoded); err != nil { - return nil, fmt.Errorf("unable to decode object list %v: %w", decoded, err) - } - - return decoded, nil -} - -// NewClient returns a new REST-JSON client. -func NewClient(config *Config) *Client { - return &Client{ - client: config.Client, - basePath: config.BasePath, - objectType: reflect.TypeOf(config.SampleObject), - listType: reflect.TypeOf(config.SampleList), - } -} diff --git a/pkg/util/rest/server.go b/pkg/util/rest/server.go deleted file mode 100644 index abf02b61c..000000000 --- a/pkg/util/rest/server.go +++ /dev/null @@ -1,289 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package rest - -import ( - "crypto/tls" - "encoding/json" - "errors" - "io" - "net/http" - "reflect" - - "github.com/go-chi/chi" - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/store" - utilhttp "github.com/clusterlink-net/clusterlink/pkg/util/http" -) - -// Server for handling REST-JSON requests. -type Server struct { - utilhttp.Server - - logger *logrus.Entry -} - -// Handler for object operations. -type Handler interface { - // Decode and validate an object. - Decode(data []byte) (any, error) - // Create an object. - Create(object any) error - // Update an object. - Update(object any) error - // Get an object. - Get(name string) (any, error) - // Delete an object. - Delete(object any) (any, error) - // List all objects. - List() (any, error) -} - -// ServerObjectSpec specifies a set of server handlers for a specific object type. -type ServerObjectSpec struct { - // BasePath is the server HTTP path for manipulating a specific type of objects. - BasePath string - // Handler interface for object operations. - Handler Handler - // DeleteByValue is true for object types which are deletable by sending their value, instead of their name. - DeleteByValue bool -} - -func (s *Server) create(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "create", "path": r.URL.Path}) - requestLogger.WithField("body-length", r.ContentLength).Infof("Handling request.") - - body, err := io.ReadAll(r.Body) - if err != nil { - requestLogger.Errorf("Cannot read request body: %v.", err) - return - } - - requestLogger.Debugf("Body: %v.", body) - - object, err := spec.Handler.Decode(body) - if err != nil { - requestLogger.Errorf("Cannot decode object: %v.", err) - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - - if err := spec.Handler.Create(object); err != nil { - var objectExistsErr *store.ObjectExistsError - if errors.As(err, &objectExistsErr) { - requestLogger.Errorf("Object already exists.") - http.Error(w, "object already exists", http.StatusBadRequest) - return - } - - requestLogger.Errorf("Cannot create object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - w.WriteHeader(http.StatusCreated) - w.Header().Set("Location", r.URL.String()) -} - -func (s *Server) update(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "update", "path": r.URL.Path}) - requestLogger.WithField("body-length", r.ContentLength).Infof("Handling request.") - - body, err := io.ReadAll(r.Body) - if err != nil { - requestLogger.Errorf("Cannot read request body: %v.", err) - return - } - - requestLogger.Debugf("Body: %v.", body) - - object, err := spec.Handler.Decode(body) - if err != nil { - requestLogger.Errorf("Cannot decode object: %v.", err) - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - - if err := spec.Handler.Update(object); err != nil { - var objectNotFoundError *store.ObjectNotFoundError - if errors.As(err, &objectNotFoundError) { - requestLogger.Errorf("Object not found.") - http.Error(w, "object not found", http.StatusNotFound) - return - } - - requestLogger.Errorf("Cannot update object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - w.WriteHeader(http.StatusNoContent) - w.Header().Set("Location", r.URL.String()) -} - -func (s *Server) get(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "get", "path": r.URL.Path}) - requestLogger.Infof("Handling request.") - - name := chi.URLParam(r, "name") - - result, err := spec.Handler.Get(name) - if err != nil { - requestLogger.Errorf("Cannot get object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - if result == nil { - w.WriteHeader(http.StatusNotFound) - return - } - - encoded, err := json.Marshal(result) - if err != nil { - requestLogger.Errorf("Cannot encode object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - w.WriteHeader(http.StatusOK) - w.Header().Set("Content-Type", "application/json; charset=utf-8") - - if _, err := w.Write(encoded); err != nil { - s.logger.Errorf("Cannot write http response: %v.", err) - } -} - -func (s *Server) deleteObject(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "deleteObject", "path": r.URL.Path}) - requestLogger.WithField("body-length", r.ContentLength).Infof("Handling request.") - - body, err := io.ReadAll(r.Body) - if err != nil { - requestLogger.Errorf("Cannot read request body: %v.", err) - return - } - - requestLogger.Debugf("Body: %v.", body) - - object, err := spec.Handler.Decode(body) - if err != nil { - requestLogger.Errorf("Cannot decode object: %v.", err) - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - - result, err := spec.Handler.Delete(object) - if err != nil { - requestLogger.Errorf("Cannot delete object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - if reflect.ValueOf(result).IsNil() { - w.WriteHeader(http.StatusNotFound) - return - } - - w.WriteHeader(http.StatusNoContent) -} - -func (s *Server) delete(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "delete", "path": r.URL.Path}) - requestLogger.Infof("Handling request.") - - name := chi.URLParam(r, "name") - - result, err := spec.Handler.Delete(name) - if err != nil { - requestLogger.Errorf("Cannot delete object: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - if reflect.ValueOf(result).IsNil() { - w.WriteHeader(http.StatusNotFound) - return - } - - w.WriteHeader(http.StatusNoContent) -} - -func (s *Server) list(spec *ServerObjectSpec, w http.ResponseWriter, r *http.Request) { - requestLogger := s.logger.WithFields(logrus.Fields{"method": "list", "path": r.URL.Path}) - requestLogger.Infof("Handling request.") - - result, err := spec.Handler.List() - if err != nil { - requestLogger.Errorf("Cannot list objects: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - encoded, err := json.Marshal(result) - if err != nil { - requestLogger.Errorf("Cannot encode objects: %v.", err) - http.Error(w, err.Error(), http.StatusInternalServerError) - return - } - - w.WriteHeader(http.StatusOK) - w.Header().Set("Content-Type", "application/json; charset=utf-8") - - if _, err := w.Write(encoded); err != nil { - s.logger.Errorf("Cannot write http response: %v.", err) - } -} - -// AddObjectHandlers adds the server a handlers for managing a specific object type. -func (s *Server) AddObjectHandlers(spec *ServerObjectSpec) { - router := s.Router() - - router.Route(spec.BasePath, func(cr chi.Router) { - cr.Post("/", func(w http.ResponseWriter, r *http.Request) { - s.create(spec, w, r) - }) - cr.Put("/", func(w http.ResponseWriter, r *http.Request) { - s.update(spec, w, r) - }) - cr.Get("/{name}", func(w http.ResponseWriter, r *http.Request) { - s.get(spec, w, r) - }) - cr.Get("/", func(w http.ResponseWriter, r *http.Request) { - s.list(spec, w, r) - }) - - if spec.DeleteByValue { - cr.Delete("/", func(w http.ResponseWriter, r *http.Request) { - s.deleteObject(spec, w, r) - }) - } else { - cr.Delete("/{name}", func(w http.ResponseWriter, r *http.Request) { - s.delete(spec, w, r) - }) - } - }) -} - -// NewServer returns a new empty REST-JSON server. -func NewServer(name string, tlsConfig *tls.Config) *Server { - return &Server{ - Server: utilhttp.NewServer(name, tlsConfig), - logger: logrus.WithFields(logrus.Fields{ - "component": "rest-server", - "name": name, - }), - } -} diff --git a/tests/e2e/k8s/test_basic.go b/tests/e2e/k8s/test_basic.go index 946a28d14..6686222d8 100644 --- a/tests/e2e/k8s/test_basic.go +++ b/tests/e2e/k8s/test_basic.go @@ -14,16 +14,9 @@ package k8s import ( - "fmt" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - - "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services/httpecho" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/util" + "github.com/stretchr/testify/require" ) func (s *TestSuite) TestConnectivity() { @@ -49,398 +42,3 @@ func (s *TestSuite) TestConnectivity() { require.Equal(s.T(), cl[0].Name(), data) }) } - -func (s *TestSuite) TestControlplaneCRUD() { - s.RunOnAllDataplaneTypes(func(cfg *util.PeerConfig) { - cfg.ControlplanePersistency = true - cfg.CRUDMode = true - cl, err := s.fabric.DeployClusterlinks(3, cfg) - require.Nil(s.T(), err) - - client0 := cl[0].Client() - client1 := cl[1].Client() - - // test import API - imp := v1alpha1.Import{ - ObjectMeta: metav1.ObjectMeta{ - Name: httpEchoService.Name, - }, - Spec: v1alpha1.ImportSpec{ - Port: 1234, - Sources: []v1alpha1.ImportSource{{Peer: cl[1].Name(), ExportName: httpEchoService.Name, ExportNamespace: cl[1].Namespace()}}, - }, - } - - // list imports when empty - objects, err := client0.Imports.List() - require.Nil(s.T(), err) - require.Empty(s.T(), objects.(*[]v1alpha1.Import)) - - // get non-existing import - _, err = client0.Imports.Get(imp.Name) - require.NotNil(s.T(), err) - - // delete non-existing import - require.NotNil(s.T(), client0.Imports.Delete(imp.Name)) - // update non-existing import - require.NotNil(s.T(), client0.Imports.Update(&imp)) - // create import - require.Nil(s.T(), client0.Imports.Create(&imp)) - // create import when it already exists - require.NotNil(s.T(), client0.Imports.Create(&imp)) - - importedService := &util.Service{ - Name: imp.Name, - Port: imp.Spec.Port, - } - - accessService := func(allowRetry bool, expectedError error) (string, error) { - return cl[0].AccessService( - httpecho.GetEchoValue, importedService, allowRetry, expectedError) - } - - // verify import listener is up - _, err = accessService(true, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - - // get import - objects, err = client0.Imports.Get(imp.Name) - require.Nil(s.T(), err) - importFromServer := *objects.(*v1alpha1.Import) - require.Equal(s.T(), importFromServer.Name, imp.Name) - require.Equal(s.T(), importFromServer.Spec.Port, imp.Spec.Port) - require.Equal(s.T(), importFromServer.Spec.Sources, imp.Spec.Sources) - require.NotZero(s.T(), importFromServer.Spec.TargetPort) - - // list imports - objects, err = client0.Imports.List() - require.Nil(s.T(), err) - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Import), []v1alpha1.Import{importFromServer}) - - // test peer API - peer := v1alpha1.Peer{ - ObjectMeta: metav1.ObjectMeta{ - Name: cl[1].Name(), - }, - Spec: v1alpha1.PeerSpec{ - Gateways: []v1alpha1.Endpoint{{ - Host: cl[1].IP(), - Port: cl[1].Port(), - }}, - }, - } - - // list peers when empty - objects, err = client0.Peers.List() - require.Nil(s.T(), err) - require.Empty(s.T(), objects.(*[]v1alpha1.Peer)) - - // get non-existing peer - _, err = client0.Peers.Get(peer.Name) - require.NotNil(s.T(), err) - - // delete non-existing peer - require.NotNil(s.T(), client0.Peers.Delete(peer.Name)) - // update non-existing peer - require.NotNil(s.T(), client0.Peers.Update(&peer)) - // create peer - require.Nil(s.T(), client0.Peers.Create(&peer)) - // create peer which already exists - require.NotNil(s.T(), client0.Peers.Create(&peer)) - - // verify no access - _, err = accessService(false, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - - // get peer - objects, err = client0.Peers.Get(peer.Name) - require.Nil(s.T(), err) - peerFromServer := *objects.(*v1alpha1.Peer) - require.Equal(s.T(), peerFromServer.Name, peer.Name) - require.Equal(s.T(), peerFromServer.Spec, peer.Spec) - - // list peers - objects, err = client0.Peers.List() - require.Nil(s.T(), err) - if !assert.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Peer), []v1alpha1.Peer{peerFromServer}) { - objects, err = client0.Peers.Get(peer.Name) - require.Nil(s.T(), err) - peerFromServer = *objects.(*v1alpha1.Peer) - } - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Peer), []v1alpha1.Peer{peerFromServer}) - - // add another peer (for upcoming load-balancing test) - peer2 := v1alpha1.Peer{ - ObjectMeta: metav1.ObjectMeta{ - Name: cl[2].Name(), - }, - Spec: v1alpha1.PeerSpec{ - Gateways: []v1alpha1.Endpoint{{ - Host: cl[2].IP(), - Port: cl[2].Port(), - }}, - }, - } - require.Nil(s.T(), client0.Peers.Create(&peer2)) - - // test access policy API - policy := *util.PolicyAllowAll - - // list access policies when empty - objects, err = client0.AccessPolicies.List() - require.Nil(s.T(), err) - require.Empty(s.T(), objects.(*[]v1alpha1.AccessPolicy)) - - // get non-existing access policy - _, err = client0.AccessPolicies.Get(policy.Name) - require.NotNil(s.T(), err) - - // delete non-existing access policy - require.NotNil(s.T(), client0.AccessPolicies.Delete(policy.Name)) - // update non-existing access policy - require.NotNil(s.T(), client0.AccessPolicies.Update(&policy)) - // create access policy - require.Nil(s.T(), client0.AccessPolicies.Create(&policy)) - // create access policy which already exists - require.NotNil(s.T(), client0.AccessPolicies.Create(&policy)) - - // verify no access - _, err = accessService(false, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - - // get access policy - objects, err = client0.AccessPolicies.Get(policy.Name) - require.Nil(s.T(), err) - require.Equal(s.T(), *objects.(*v1alpha1.AccessPolicy), policy) - - // list access policies - objects, err = client0.AccessPolicies.List() - require.Nil(s.T(), err) - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.AccessPolicy), []v1alpha1.AccessPolicy{policy}) - - // test export API - export := v1alpha1.Export{ - ObjectMeta: metav1.ObjectMeta{ - Name: imp.Name, - }, - Spec: v1alpha1.ExportSpec{ - Host: fmt.Sprintf( - "%s.%s.svc.cluster.local", - httpEchoService.Name, httpEchoService.Namespace), - Port: httpEchoService.Port, - }, - } - - // list exports when empty - objects, err = client1.Exports.List() - require.Nil(s.T(), err) - require.Empty(s.T(), objects.(*[]v1alpha1.Export)) - - // get non-existing export - _, err = client1.Exports.Get(export.Name) - require.NotNil(s.T(), err) - - // delete non-existing export - require.NotNil(s.T(), client1.Exports.Delete(export.Name)) - // update non-existing export - require.NotNil(s.T(), client1.Exports.Update(&export)) - // create export - require.Nil(s.T(), client1.Exports.Create(&export)) - // create export which already exists - require.NotNil(s.T(), client1.Exports.Create(&export)) - - // verify no access - _, err = accessService(false, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - - // get export - objects, err = client1.Exports.Get(export.Name) - require.Nil(s.T(), err) - exportFromServer := *objects.(*v1alpha1.Export) - require.Equal(s.T(), export.Name, exportFromServer.Name) - require.Equal(s.T(), export.Spec, exportFromServer.Spec) - - // list exports - objects, err = client1.Exports.List() - require.Nil(s.T(), err) - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Export), []v1alpha1.Export{exportFromServer}) - - // allow export to be accessed - require.Nil(s.T(), client1.AccessPolicies.Create(&policy)) - // verify access - str, err := accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // create false binding to verify LB policy - imp.Spec.Sources = append(imp.Spec.Sources, - v1alpha1.ImportSource{Peer: cl[2].Name(), ExportName: httpEchoService.Name, ExportNamespace: cl[2].Namespace()}) - imp.Spec.LBScheme = v1alpha1.LBSchemeStatic - require.Nil(s.T(), client0.Imports.Update(&imp)) - - // verify access - str, err = accessService(false, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // update import port - imp.Spec.Port++ - require.Nil(s.T(), client0.Imports.Update(&imp)) - // verify no access to previous port - _, err = accessService(true, &services.ConnectionRefusedError{}) - require.ErrorIs(s.T(), err, &services.ConnectionRefusedError{}) - // verify access to new port - importedService.Port++ - _, err = accessService(true, nil) - require.Nil(s.T(), err) - - // update peer - peer.Spec.Gateways[0].Port++ - require.Nil(s.T(), client0.Peers.Update(&peer)) - // get peer after update - objects, err = client0.Peers.Get(peer.Name) - require.Nil(s.T(), err) - require.Equal(s.T(), objects.(*v1alpha1.Peer).Spec, peer.Spec) - // verify no access after update - _, err = accessService(true, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // update peer back - peer.Spec.Gateways[0].Port-- - require.Nil(s.T(), client0.Peers.Update(&peer)) - // verify access after update back - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // update access policy - policy2 := *util.PolicyAllowAll - policy2.Spec.Action = v1alpha1.AccessPolicyActionDeny - require.Nil(s.T(), client0.AccessPolicies.Update(&policy2)) - // get access policy after update - objects, err = client0.AccessPolicies.Get(policy2.Name) - require.Nil(s.T(), err) - require.Equal(s.T(), objects.(*v1alpha1.AccessPolicy).Spec, policy2.Spec) - // verify no access after update - _, err = accessService(false, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // update access policy back - require.Nil(s.T(), client0.AccessPolicies.Update(&policy)) - // verify access after update back - str, err = accessService(false, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // update export - export.Spec.Port++ - require.Nil(s.T(), client1.Exports.Update(&export)) - // get export after update - objects, err = client1.Exports.Get(export.Name) - require.Nil(s.T(), err) - require.Equal(s.T(), objects.(*v1alpha1.Export).Spec, export.Spec) - // verify no access after update - _, err = accessService(true, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // update export back - export.Spec.Port-- - require.Nil(s.T(), client1.Exports.Update(&export)) - // verify access after update back - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // delete import - require.Nil(s.T(), client0.Imports.Delete(imp.Name)) - // get import after delete - _, err = client0.Imports.Get(imp.Name) - require.NotNil(s.T(), err) - // verify no access after delete - _, err = accessService(true, &services.ServiceNotFoundError{}) - require.ErrorIs(s.T(), err, &services.ServiceNotFoundError{}) - // re-create import - require.Nil(s.T(), client0.Imports.Create(&imp)) - // re-get import from server - objects, err = client0.Imports.Get(imp.Name) - require.Nil(s.T(), err) - importFromServer = *objects.(*v1alpha1.Import) - // verify access after re-create - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // delete peer - require.Nil(s.T(), client0.Peers.Delete(peer.Name)) - // get peer after delete - _, err = client0.Peers.Get(peer.Name) - require.NotNil(s.T(), err) - // verify no access after delete - _, err = accessService(true, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // re-create peer - require.Nil(s.T(), client0.Peers.Create(&peer)) - // verify access after re-create - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // delete access policy - require.Nil(s.T(), client0.AccessPolicies.Delete(policy.Name)) - // get access policy after delete - _, err = client0.AccessPolicies.Get(policy.Name) - require.NotNil(s.T(), err) - // verify no access after delete - _, err = accessService(false, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // re-create access policy - require.Nil(s.T(), client0.AccessPolicies.Create(&policy)) - // verify access after re-create - str, err = accessService(false, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // delete export - require.Nil(s.T(), client1.Exports.Delete(export.Name)) - // get export after delete - _, err = client1.Exports.Get(export.Name) - require.NotNil(s.T(), err) - // verify no access after delete - _, err = accessService(true, &services.ConnectionResetError{}) - require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) - // re-create export - require.Nil(s.T(), client1.Exports.Create(&export)) - // verify access after re-create - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - - // restart controlplanes - runner := util.AsyncRunner{} - runner.Run(cl[0].RestartControlplane) - runner.Run(cl[1].RestartControlplane) - require.Nil(s.T(), runner.Wait()) - - // verify imports after restart - objects, err = client0.Imports.List() - require.Nil(s.T(), err) - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.Import), []v1alpha1.Import{importFromServer}) - - // verify 2 peers after restart - objects, err = client0.Peers.List() - require.Nil(s.T(), err) - require.Equal(s.T(), len(*objects.(*[]v1alpha1.Peer)), 2) - - // verify access policies after restart - objects, err = client0.AccessPolicies.List() - require.Nil(s.T(), err) - require.ElementsMatch(s.T(), *objects.(*[]v1alpha1.AccessPolicy), []v1alpha1.AccessPolicy{policy}) - - // verify exports after restart - objects, err = client1.Exports.List() - require.Nil(s.T(), err) - require.Equal(s.T(), len(*objects.(*[]v1alpha1.Export)), 1) - - // verify access after restart - str, err = accessService(true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), str, cl[1].Name()) - }) -} diff --git a/tests/e2e/k8s/test_import.go b/tests/e2e/k8s/test_import.go index 244c49c38..5bad2b92c 100644 --- a/tests/e2e/k8s/test_import.go +++ b/tests/e2e/k8s/test_import.go @@ -15,7 +15,6 @@ package k8s import ( "context" - "strconv" "strings" "github.com/stretchr/testify/require" @@ -25,7 +24,6 @@ import ( "k8s.io/apimachinery/pkg/types" "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/bootstrap/platform" "github.com/clusterlink-net/clusterlink/pkg/controlplane/control" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services/httpecho" @@ -338,128 +336,100 @@ func (s *TestSuite) TestImportInvalidName() { } func (s *TestSuite) TestImportMerge() { - testFunc := func(crdMode bool) { - cfg := &util.PeerConfig{ - CRUDMode: !crdMode, - DataplaneType: platform.DataplaneTypeEnvoy, - Dataplanes: 1, - } - - cl, err := s.fabric.DeployClusterlinks(1, cfg) - require.Nil(s.T(), err) - - // create export, peer, and allow-all policy - require.Nil(s.T(), cl[0].CreateService(&httpEchoService)) - require.Nil(s.T(), cl[0].CreateExport(&httpEchoService)) - require.Nil(s.T(), cl[0].CreatePolicy(util.PolicyAllowAll)) - require.Nil(s.T(), cl[0].CreatePeer(cl[0])) - - importedService := &util.Service{ - Name: "imported", - Port: 80, - } - - // create merge import - imp := &v1alpha1.Import{ - ObjectMeta: metav1.ObjectMeta{ - Name: importedService.Name, - Namespace: cl[0].Namespace(), - Labels: map[string]string{ - v1alpha1.LabelImportMerge: "true", - }, - }, - Spec: v1alpha1.ImportSpec{ - Port: importedService.Port, - Sources: []v1alpha1.ImportSource{{ - Peer: cl[0].Name(), - ExportName: httpEchoService.Name, - ExportNamespace: cl[0].Namespace(), - }}, - }, - } - if crdMode { - require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), imp)) - - // verify status is bad, since imported service should be pre-created for a merge import - require.Nil(s.T(), cl[0].WaitForImportCondition(imp, v1alpha1.ImportServiceValid, false)) - } else { - // CRUD mode will fail since service does not exist, and operation is not async - require.NotNil(s.T(), cl[0].Client().Imports.Create(imp)) - } - - // create the import service - service := &v1.Service{ - ObjectMeta: metav1.ObjectMeta{ - Name: importedService.Name, - Namespace: cl[0].Namespace(), - }, - Spec: v1.ServiceSpec{ - Ports: []v1.ServicePort{{ - Port: int32(importedService.Port), - }}, + cl, err := s.fabric.DeployClusterlinks(1, nil) + require.Nil(s.T(), err) + + // create export, peer, and allow-all policy + require.Nil(s.T(), cl[0].CreateService(&httpEchoService)) + require.Nil(s.T(), cl[0].CreateExport(&httpEchoService)) + require.Nil(s.T(), cl[0].CreatePolicy(util.PolicyAllowAll)) + require.Nil(s.T(), cl[0].CreatePeer(cl[0])) + + importedService := &util.Service{ + Name: "imported", + Port: 80, + } + + // create merge import + imp := &v1alpha1.Import{ + ObjectMeta: metav1.ObjectMeta{ + Name: importedService.Name, + Namespace: cl[0].Namespace(), + Labels: map[string]string{ + v1alpha1.LabelImportMerge: "true", }, - } - require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), service)) - - if crdMode { - // verify status becomes good - require.Nil(s.T(), cl[0].WaitForImportCondition(imp, v1alpha1.ImportServiceValid, true)) - } else { - // update import to re-try endpoint slice creation - require.Nil(s.T(), cl[0].Client().Imports.Update(imp)) - } - - // verify service access - data, err := cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) - require.Nil(s.T(), err) - require.Equal(s.T(), cl[0].Name(), data) - - // update dataplane endpoint slice via scaling - require.Nil(s.T(), cl[0].ScaleDataplane(0)) - require.Nil(s.T(), cl[0].ScaleDataplane(1)) - - // verify service access - _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) - require.Nil(s.T(), err) - - // delete dataplane endpoint slice by deleting the dataplane service - var dataplaneService v1.Service - require.Nil(s.T(), cl[0].Cluster().Resources().Get( - context.Background(), "cl-dataplane", cl[0].Namespace(), &dataplaneService)) - require.Nil(s.T(), cl[0].Cluster().Resources().Delete( - context.Background(), &dataplaneService)) - - // verify no access - _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, &util.PodFailedError{}) - require.ErrorIs(s.T(), err, &util.PodFailedError{}) - - // create dataplane endpoint slice - dataplaneService.ResourceVersion = "" - require.Nil(s.T(), cl[0].Cluster().Resources().Create( - context.Background(), &dataplaneService)) - - // verify access is back - _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) - require.Nil(s.T(), err) - - // remove merge property of import - delete(imp.Labels, v1alpha1.LabelImportMerge) - if crdMode { - require.Nil(s.T(), cl[0].Cluster().Resources().Update(context.Background(), imp)) - } else { - // CRUD mode will fail since service does not exist, and operation is not async - require.NotNil(s.T(), cl[0].Client().Imports.Update(imp)) - } - - // verify no access - _, err = cl[0].AccessService( - httpecho.RunClientInPod, importedService, true, &util.PodFailedError{}) - require.ErrorIs(s.T(), err, &util.PodFailedError{}) + }, + Spec: v1alpha1.ImportSpec{ + Port: importedService.Port, + Sources: []v1alpha1.ImportSource{{ + Peer: cl[0].Name(), + ExportName: httpEchoService.Name, + ExportNamespace: cl[0].Namespace(), + }}, + }, } - // run test on both CRDMode = {true, false} - for _, crdMode := range []bool{true, false} { - testName := "CRDMode" + strings.ToUpper(strconv.FormatBool(crdMode)) - s.RunSubTest(testName, func() { testFunc(crdMode) }) + require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), imp)) + + // verify status is bad, since imported service should be pre-created for a merge import + require.Nil(s.T(), cl[0].WaitForImportCondition(imp, v1alpha1.ImportServiceValid, false)) + + // create the import service + service := &v1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: importedService.Name, + Namespace: cl[0].Namespace(), + }, + Spec: v1.ServiceSpec{ + Ports: []v1.ServicePort{{ + Port: int32(importedService.Port), + }}, + }, } + require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), service)) + + // verify status becomes good + require.Nil(s.T(), cl[0].WaitForImportCondition(imp, v1alpha1.ImportServiceValid, true)) + + // verify service access + data, err := cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) + require.Nil(s.T(), err) + require.Equal(s.T(), cl[0].Name(), data) + + // update dataplane endpoint slice via scaling + require.Nil(s.T(), cl[0].ScaleDataplane(0)) + require.Nil(s.T(), cl[0].ScaleDataplane(1)) + + // verify service access + _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) + require.Nil(s.T(), err) + + // delete dataplane endpoint slice by deleting the dataplane service + var dataplaneService v1.Service + require.Nil(s.T(), cl[0].Cluster().Resources().Get( + context.Background(), "cl-dataplane", cl[0].Namespace(), &dataplaneService)) + require.Nil(s.T(), cl[0].Cluster().Resources().Delete( + context.Background(), &dataplaneService)) + + // verify no access + _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, &util.PodFailedError{}) + require.ErrorIs(s.T(), err, &util.PodFailedError{}) + + // create dataplane endpoint slice + dataplaneService.ResourceVersion = "" + require.Nil(s.T(), cl[0].Cluster().Resources().Create( + context.Background(), &dataplaneService)) + + // verify access is back + _, err = cl[0].AccessService(httpecho.RunClientInPod, importedService, true, nil) + require.Nil(s.T(), err) + + // remove merge property of import + delete(imp.Labels, v1alpha1.LabelImportMerge) + require.Nil(s.T(), cl[0].Cluster().Resources().Update(context.Background(), imp)) + + // verify no access + _, err = cl[0].AccessService( + httpecho.RunClientInPod, importedService, true, &util.PodFailedError{}) + require.ErrorIs(s.T(), err, &util.PodFailedError{}) } diff --git a/tests/e2e/k8s/test_operator.go b/tests/e2e/k8s/test_operator.go index f1125c7ce..ee3a75bbb 100644 --- a/tests/e2e/k8s/test_operator.go +++ b/tests/e2e/k8s/test_operator.go @@ -34,7 +34,6 @@ import ( func (s *TestSuite) TestOperator() { // Deploy ClusterLink with operator cfg := &util.PeerConfig{ - CRUDMode: false, DataplaneType: platform.DataplaneTypeEnvoy, Dataplanes: 1, DeployWithOperator: true, diff --git a/tests/e2e/k8s/util/clusterlink.go b/tests/e2e/k8s/util/clusterlink.go index fb5d3eaca..236eb7c71 100644 --- a/tests/e2e/k8s/util/clusterlink.go +++ b/tests/e2e/k8s/util/clusterlink.go @@ -17,16 +17,12 @@ import ( "context" "errors" "fmt" - "io" - "net/url" - "syscall" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/client" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services" ) @@ -34,9 +30,7 @@ import ( type ClusterLink struct { cluster *KindCluster namespace string - client *client.Client port uint16 - crdMode bool } // Name returns the peer name. @@ -64,36 +58,6 @@ func (c *ClusterLink) Cluster() *KindCluster { return c.cluster } -// Client returns a controlplane API client for this cluster. -func (c *ClusterLink) Client() *client.Client { - return c.client -} - -// WaitForControlplaneAPI waits until the controlplane API server is up. -func (c *ClusterLink) WaitForControlplaneAPI() error { - var err error - for t := time.Now(); time.Since(t) < time.Second*60; time.Sleep(time.Millisecond * 100) { - var uerr *url.Error - _, err = c.client.Peers.List() - switch { - case err == nil: - return nil - case errors.Is(err, syscall.ECONNREFUSED): - continue - case errors.Is(err, syscall.ECONNRESET): - continue - case errors.Is(err, io.EOF): - continue - case errors.As(err, &uerr) && uerr.Timeout(): - continue - } - - return err - } - - return err -} - // ScaleControlplane scales the controlplane deployment. func (c *ClusterLink) ScaleControlplane(replicas int32) error { return c.cluster.ScaleDeployment("cl-controlplane", c.namespace, replicas) @@ -104,10 +68,7 @@ func (c *ClusterLink) RestartControlplane() error { if err := c.ScaleControlplane(0); err != nil { return err } - if err := c.ScaleControlplane(1); err != nil { - return err - } - return c.WaitForControlplaneAPI() + return c.ScaleControlplane(1) } // ScaleDataplane scales the dataplane deployment. @@ -173,48 +134,7 @@ func (c *ClusterLink) CreatePeer(peer *ClusterLink) error { }, } - if c.crdMode { - return c.cluster.Resources().Create(context.Background(), pr) - } - - return c.client.Peers.Create(pr) -} - -func (c *ClusterLink) UpdatePeer(peer *ClusterLink) error { - return c.client.Peers.Update(&v1alpha1.Peer{ - ObjectMeta: metav1.ObjectMeta{ - Name: peer.Name(), - Namespace: c.namespace, - }, - Spec: v1alpha1.PeerSpec{ - Gateways: []v1alpha1.Endpoint{{ - Host: peer.IP(), - Port: peer.Port(), - }}, - }, - }) -} - -func (c *ClusterLink) GetPeer(peer *ClusterLink) (*v1alpha1.Peer, error) { - res, err := c.client.Peers.Get(peer.Name()) - if err != nil { - return nil, err - } - - return res.(*v1alpha1.Peer), nil -} - -func (c *ClusterLink) GetAllPeers() (*[]v1alpha1.Peer, error) { - res, err := c.client.Peers.List() - if err != nil { - return nil, err - } - - return res.(*[]v1alpha1.Peer), nil -} - -func (c *ClusterLink) DeletePeer(peer *ClusterLink) error { - return c.client.Peers.Delete(peer.Name()) + return c.cluster.Resources().Create(context.Background(), pr) } func (c *ClusterLink) CreateService(service *Service) error { @@ -254,55 +174,18 @@ func (c *ClusterLink) CreateExport(service *Service) error { }, } - if c.crdMode { - return c.cluster.Resources().Create(context.Background(), export) - } - - return c.client.Exports.Create(export) -} - -func (c *ClusterLink) UpdateExport(name string, service *Service) error { - return c.client.Exports.Update(&v1alpha1.Export{ - ObjectMeta: metav1.ObjectMeta{ - Name: name, - }, - Spec: v1alpha1.ExportSpec{ - Host: fmt.Sprintf("%s.%s.svc.cluster.local", service.Name, service.Namespace), - Port: service.Port, - }, - }) -} - -func (c *ClusterLink) GetExport(name string) (*v1alpha1.Export, error) { - res, err := c.client.Exports.Get(name) - if err != nil { - return nil, err - } - - return res.(*v1alpha1.Export), nil -} - -func (c *ClusterLink) GetAllExports() (*[]v1alpha1.Export, error) { - res, err := c.client.Exports.List() - if err != nil { - return nil, err - } - - return res.(*[]v1alpha1.Export), nil + return c.cluster.Resources().Create(context.Background(), export) } func (c *ClusterLink) DeleteExport(name string) error { - if c.crdMode { - return c.cluster.Resources().Delete( - context.Background(), - &v1alpha1.Export{ - ObjectMeta: metav1.ObjectMeta{ - Name: name, - Namespace: c.namespace, - }, - }) - } - return c.client.Exports.Delete(name) + return c.cluster.Resources().Delete( + context.Background(), + &v1alpha1.Export{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: c.namespace, + }, + }) } func (c *ClusterLink) CreateImport(service *Service, peer *ClusterLink, exportName string) error { @@ -321,11 +204,7 @@ func (c *ClusterLink) CreateImport(service *Service, peer *ClusterLink, exportNa }, } - if c.crdMode { - return c.cluster.Resources().Create(context.Background(), imp) - } - - return c.client.Imports.Create(imp) + return c.cluster.Resources().Create(context.Background(), imp) } func (c *ClusterLink) UpdateImport(service *Service, peer *ClusterLink, exportName string) error { @@ -344,33 +223,7 @@ func (c *ClusterLink) UpdateImport(service *Service, peer *ClusterLink, exportNa }, } - if c.crdMode { - return c.cluster.Resources().Update(context.Background(), imp) - } - - return c.client.Imports.Update(imp) -} - -func (c *ClusterLink) GetImport(name string) (*v1alpha1.Import, error) { - res, err := c.client.Imports.Get(name) - if err != nil { - return nil, err - } - - return res.(*v1alpha1.Import), nil -} - -func (c *ClusterLink) GetAllImports() (*[]v1alpha1.Import, error) { - res, err := c.client.Imports.List() - if err != nil { - return nil, err - } - - return res.(*[]v1alpha1.Import), nil -} - -func (c *ClusterLink) DeleteImport(name string) error { - return c.client.Imports.Delete(name) + return c.cluster.Resources().Update(context.Background(), imp) } func (c *ClusterLink) CreatePolicy(policy *v1alpha1.AccessPolicy) error { @@ -380,62 +233,25 @@ func (c *ClusterLink) CreatePolicy(policy *v1alpha1.AccessPolicy) error { policy = &accessPolicyCopy } - if c.crdMode { - return c.cluster.Resources().Create(context.Background(), policy) - } - - return c.client.AccessPolicies.Create(policy) -} - -func (c *ClusterLink) UpdatePolicy(policy *v1alpha1.AccessPolicy) error { - return c.client.AccessPolicies.Update(&policy) -} - -func (c *ClusterLink) GetPolicy(name string) (*v1alpha1.AccessPolicy, error) { - res, err := c.client.AccessPolicies.Get(name) - if err != nil { - return nil, err - } - - return res.(*v1alpha1.AccessPolicy), nil -} - -func (c *ClusterLink) GetAllPolicies() (*[]v1alpha1.AccessPolicy, error) { - res, err := c.client.AccessPolicies.List() - if err != nil { - return nil, err - } - - return res.(*[]v1alpha1.AccessPolicy), nil + return c.cluster.Resources().Create(context.Background(), policy) } func (c *ClusterLink) DeletePolicy(name string) error { - if c.crdMode { - return c.cluster.Resources().Delete( - context.Background(), - &v1alpha1.AccessPolicy{ - ObjectMeta: metav1.ObjectMeta{ - Name: name, - Namespace: c.namespace, - }, - }) - } - return c.client.AccessPolicies.Delete(name) + return c.cluster.Resources().Delete( + context.Background(), + &v1alpha1.AccessPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: c.namespace, + }, + }) } func (c *ClusterLink) CreatePrivilegedPolicy(policy *v1alpha1.PrivilegedAccessPolicy) error { - if !c.crdMode { - return errors.New("privileged access policies are only supported in CRD mode") - } - return c.cluster.Resources().Create(context.Background(), policy) } func (c *ClusterLink) DeletePrivilegedPolicy(name string) error { - if !c.crdMode { - return errors.New("privileged access policies are only supported in CRD mode") - } - return c.cluster.Resources().Delete( context.Background(), &v1alpha1.PrivilegedAccessPolicy{ diff --git a/tests/e2e/k8s/util/fabric.go b/tests/e2e/k8s/util/fabric.go index de7c29783..a4a3321da 100644 --- a/tests/e2e/k8s/util/fabric.go +++ b/tests/e2e/k8s/util/fabric.go @@ -15,8 +15,6 @@ package util import ( "context" - "crypto/tls" - "crypto/x509" "fmt" "time" @@ -30,14 +28,11 @@ import ( "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" "github.com/clusterlink-net/clusterlink/pkg/bootstrap" "github.com/clusterlink-net/clusterlink/pkg/bootstrap/platform" - "github.com/clusterlink-net/clusterlink/pkg/client" "github.com/clusterlink-net/clusterlink/pkg/operator/controller" ) // PeerConfig is a peer configuration. type PeerConfig struct { - // CRUDMode indicates a CRUD-based controlplane (i.e. not CRD mode). - CRUDMode bool // DataplaneType is the dataplane type (envoy / go). DataplaneType string // Dataplanes is the number of dataplane instances. @@ -57,7 +52,6 @@ type peer struct { peerCert *bootstrap.Certificate controlplaneCert *bootstrap.Certificate dataplaneCert *bootstrap.Certificate - gwctlCert *bootstrap.Certificate } // CreateControlplaneCertificate creates the controlplane certificate. @@ -86,19 +80,6 @@ func (p *peer) CreateDataplaneCertificate() { }) } -// CreateGWCTLCertificate creates the gwctl certificate. -func (p *peer) CreateGWCTLCertificate() { - p.Run(func() error { - cert, err := bootstrap.CreateGWCTLCertificate(p.peerCert) - if err != nil { - return fmt.Errorf("cannot create controlplane certificate: %w", err) - } - - p.gwctlCert = cert - return nil - }) -} - // Fabric represents a collection of clusterlinks. type Fabric struct { AsyncRunner @@ -122,7 +103,6 @@ func (f *Fabric) CreatePeer(cluster *KindCluster) { p.peerCert = cert p.CreateControlplaneCertificate() p.CreateDataplaneCertificate() - p.CreateGWCTLCertificate() return p.Wait() }) @@ -282,30 +262,10 @@ func (f *Fabric) deployClusterLink(target *peer, cfg *PeerConfig) (*ClusterLink, port := uint16(service.Spec.Ports[0].NodePort) - cert := target.gwctlCert - certificate, err := tls.X509KeyPair(cert.RawCert(), cert.RawKey()) - if err != nil { - return nil, fmt.Errorf("cannot parse gwctl certificate: %w", err) - } - - caCertPool := x509.NewCertPool() - if !caCertPool.AppendCertsFromPEM(target.peerCert.RawCert()) { - return nil, fmt.Errorf("unable to parse peer certificate") - } - - c := client.New(target.cluster.IP(), port, &tls.Config{ - MinVersion: tls.VersionTLS12, - Certificates: []tls.Certificate{certificate}, - RootCAs: caCertPool, - ServerName: target.cluster.Name(), - }) - clink := &ClusterLink{ cluster: target.cluster, namespace: f.namespace, - client: c, port: port, - crdMode: !cfg.CRUDMode, } // wait for default service account to be created @@ -319,11 +279,6 @@ func (f *Fabric) deployClusterLink(target *peer, cfg *PeerConfig) (*ClusterLink, return nil, fmt.Errorf("error getting default service account: %w", err) } - if cfg.CRUDMode { - if err := clink.WaitForControlplaneAPI(); err != nil { - return nil, fmt.Errorf("error waiting for controlplane API server: %w", err) - } - } return clink, nil } diff --git a/tests/e2e/k8s/util/k8s_yaml.go b/tests/e2e/k8s/util/k8s_yaml.go index bc7b484aa..e90ab6ad4 100644 --- a/tests/e2e/k8s/util/k8s_yaml.go +++ b/tests/e2e/k8s/util/k8s_yaml.go @@ -55,13 +55,11 @@ func (f *Fabric) generateK8SYAML(p *peer, cfg *PeerConfig) (string, error) { } k8sYAMLBytes, err := platform.K8SConfig(&platform.Config{ - CRDMode: !cfg.CRUDMode, Peer: p.cluster.Name(), FabricCertificate: f.cert, PeerCertificate: p.peerCert, ControlplaneCertificate: p.controlplaneCert, DataplaneCertificate: p.dataplaneCert, - GWCTLCertificate: p.gwctlCert, Dataplanes: cfg.Dataplanes, DataplaneType: cfg.DataplaneType, LogLevel: logLevel, @@ -90,30 +88,6 @@ func (f *Fabric) generateK8SYAML(p *peer, cfg *PeerConfig) (string, error) { return "", fmt.Errorf("cannot switch ClusterRoleBinding name: %w", err) } - if cfg.CRUDMode { - k8sYAML, err = removeGWCTLPod(k8sYAML) - if err != nil { - return "", fmt.Errorf("cannot remove gwctl pod: %w", err) - } - - k8sYAML, err = removeGWCTLSecret(k8sYAML) - if err != nil { - return "", fmt.Errorf("cannot remove gwctl secret: %w", err) - } - - k8sYAML, err = removePeerSecret(k8sYAML) - if err != nil { - return "", fmt.Errorf("cannot remove peer secret: %w", err) - } - - if !cfg.ControlplanePersistency { - k8sYAML, err = removeControlplanePVC(k8sYAML, f.namespace) - if err != nil { - return "", fmt.Errorf("cannot remove controlplane PVC: %w", err) - } - } - } - if (os.Getenv("DEBUG")) == "1" { dpLogLevel := "trace" // More informative than the debug level. if cfg.ExpectLargeDataplaneTraffic && os.Getenv("CICD") == "1" { @@ -179,71 +153,6 @@ metadata: return replaceOnce(yaml, search, replace) } -func removeGWCTLPod(yaml string) (string, error) { - search := ` ---- -apiVersion: v1 -kind: Pod -metadata: - name: gwctl` - return remove(yaml, search, "\n---") -} - -func removeGWCTLSecret(yaml string) (string, error) { - search := ` ---- -apiVersion: v1 -kind: Secret -metadata: - name: gwctl` - return remove(yaml, search, "\n---") -} - -func removePeerSecret(yaml string) (string, error) { - search := ` ---- -apiVersion: v1 -kind: Secret -metadata: - name: cl-peer` - return remove(yaml, search, "\n---") -} - -func removeControlplanePVC(yaml, namespace string) (string, error) { - var err error - search := `--- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: cl-controlplane - namespace: ` + namespace + ` -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 100Mi -` - yaml, err = replaceOnce(yaml, search, "") - if err != nil { - return "", err - } - - search = ` - - name: cl-controlplane - persistentVolumeClaim: - claimName: cl-controlplane` - yaml, err = replaceOnce(yaml, search, "") - if err != nil { - return "", err - } - - search = ` - - name: cl-controlplane - mountPath: /var/lib/clink` - return replaceOnce(yaml, search, "") -} - func changeDataplaneDebugLevel(yaml, logLevel string) (string, error) { search := `args: ["--log-level", "debug", "--controlplane-host"` replace := `args: ["--log-level", "` + logLevel + `", "--controlplane-host"` @@ -258,7 +167,6 @@ func (f *Fabric) generateClusterlinkSecrets(p *peer) (string, error) { PeerCertificate: p.peerCert, ControlplaneCertificate: p.controlplaneCert, DataplaneCertificate: p.dataplaneCert, - GWCTLCertificate: p.gwctlCert, Namespace: f.namespace, }) if err != nil { diff --git a/tests/k8s.sh b/tests/k8s.sh deleted file mode 100755 index 194057471..000000000 --- a/tests/k8s.sh +++ /dev/null @@ -1,102 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) The ClusterLink Authors. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -ex - -SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) -TEST_DIR=$(mktemp -d) -CLI=$SCRIPT_DIR/../bin/clusterlink -DATAPLANE_TYPE="${1:-envoy}" - -function clean_up { - kind delete cluster --name peer1 - - cd - -} - -function clean_up_with_logs { - # export logs - kind export logs /tmp/clusterlink-k8s-e2e-$DATAPLANE_TYPE --name peer1 - - clean_up -} - -function test_k8s { - # create fabric with a single peer (peer1) - $CLI create fabric - $CLI create peer-cert --name peer1 --dataplane-type $DATAPLANE_TYPE - - # create kind cluster - kind create cluster --name peer1 - - # load images to cluster - kind load docker-image cl-controlplane --name peer1 - kind load docker-image cl-dataplane --name peer1 - kind load docker-image cl-go-dataplane --name peer1 - kind load docker-image gwctl --name peer1 - - # configure kubectl - kubectl config use-context kind-peer1 - - # wait for service account to be created - timeout 30 sh -c 'until kubectl -n default get serviceaccount default -o name; do sleep 0.1; done > /dev/null 2>&1' - - # create clusterlink objects - kubectl create -f ./peer1/k8s.yaml - - # start iperf3 server - kubectl run iperf-server --image=networkstatic/iperf3 -- iperf3 -s -p 1234 - - # wait for gwctl pod to run - kubectl wait --for=condition=ready pod/gwctl - - # install iperf3 and jq - kubectl exec -i gwctl -- timeout 30 sh -c 'until apk add iperf3 jq; do sleep 0.1; done > /dev/null 2>&1' - - # expose iperf3 server - kubectl expose pod iperf-server --name=foo --port=80 --target-port=1234 - - # wait for API server to come up - kubectl exec -i gwctl -- timeout 30 sh -c 'until gwctl get peer; do sleep 0.1; done > /dev/null 2>&1' - - # export iperf server - kubectl exec -i gwctl -- gwctl create export --name foo --host foo --port 80 - - # import - kubectl exec -i gwctl -- gwctl create peer --host cl-dataplane --port 443 --name peer1 - kubectl exec -i gwctl -- gwctl create import --name bla --port 9999 --peer peer1 - kubectl cp $SCRIPT_DIR/../examples/policies/allowAll.json gwctl:/tmp/allowAll.json - kubectl exec -i gwctl -- gwctl create policy --type access --policyFile /tmp/allowAll.json - - # get imported service port - PORT=$(kubectl exec -i gwctl -- /bin/bash -c "gwctl get import --name foo | jq '.Status.Listener.Port' | tr -d '\n'") - - # wait for imported service socket to come up - kubectl exec -i gwctl -- timeout 30 sh -c 'until nc -z $0 $1; do sleep 0.1; done' bla 9999 - # wait for iperf server to come up - kubectl wait --for=condition=ready pod/iperf-server - - # run iperf client - kubectl exec -i gwctl -- timeout 30 sh -c 'until iperf3 -c bla -p 9999 -t 1; do sleep 0.1; done > /dev/null 2>&1' -} - -cd $TEST_DIR -clean_up - -trap clean_up_with_logs INT TERM EXIT - -cd $TEST_DIR -test_k8s - -echo OK diff --git a/website/content/en/docs/main/tasks/operator.md b/website/content/en/docs/main/tasks/operator.md index 637ac7d80..204c575bd 100644 --- a/website/content/en/docs/main/tasks/operator.md +++ b/website/content/en/docs/main/tasks/operator.md @@ -150,7 +150,6 @@ To deploy the ClusterLink without using the CLI, follow the instructions below: kubectl create secret generic cl-peer -n clusterlink-system --from-file=ca=$CERTS /peer1/cert.pem kubectl create secret generic cl-controlplane -n clusterlink-system --from-file=cert=$CERTS /peer1/controlplane/cert.pem --from-file=key=$CERTS /peer1/controlplane/key.pem kubectl create secret generic cl-dataplane -n clusterlink-system --from-file=cert=$CERTS /peer1/dataplane/cert.pem --from-file=key=$CERTS /peer1/dataplane/key.pem - kubectl create secret generic gwctl -n clusterlink-system --from-file=cert=$CERTS /peer1/gwctl/cert.pem --from-file=key=$CERTS /peer1/gwctl/key.pem ``` 5. Deploy a ClusterLink K8s custom resource object: diff --git a/website/content/en/docs/v0.2/tasks/operator.md b/website/content/en/docs/v0.2/tasks/operator.md index 7db423ad7..f9f8e470b 100644 --- a/website/content/en/docs/v0.2/tasks/operator.md +++ b/website/content/en/docs/v0.2/tasks/operator.md @@ -150,7 +150,6 @@ To deploy the ClusterLink without using the CLI, follow the instructions below: kubectl create secret generic cl-peer -n clusterlink-system --from-file=ca=$CERTS /peer1/cert.pem kubectl create secret generic cl-controlplane -n clusterlink-system --from-file=cert=$CERTS /peer1/controlplane/cert.pem --from-file=key=$CERTS /peer1/controlplane/key.pem kubectl create secret generic cl-dataplane -n clusterlink-system --from-file=cert=$CERTS /peer1/dataplane/cert.pem --from-file=key=$CERTS /peer1/dataplane/key.pem - kubectl create secret generic gwctl -n clusterlink-system --from-file=cert=$CERTS /peer1/gwctl/cert.pem --from-file=key=$CERTS /peer1/gwctl/key.pem ``` 5. Deploy a ClusterLink K8s custom resource object: From 6dd6b28e2a38490bd56ba55e81196faff4f0d173 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 22 May 2024 12:18:16 +0300 Subject: [PATCH 13/53] --- (#601) updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pr-check.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pr-check.yml b/.github/workflows/pr-check.yml index 6f8887f30..43e2ae749 100644 --- a/.github/workflows/pr-check.yml +++ b/.github/workflows/pr-check.yml @@ -31,7 +31,7 @@ jobs: - name: Run vet check run: go vet ./... - name: Run linters - uses: golangci/golangci-lint-action@v5 + uses: golangci/golangci-lint-action@v6 with: version: v1.54.2 skip-pkg-cache: true From f9e2dbea723a04cf34c66b0136a998e9bdebfe09 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Thu, 23 May 2024 01:37:11 -0400 Subject: [PATCH 14/53] demos: Update clusterlink.py (#607) Update the documentation in the demo file: clusterlink.py. Signed-off-by: welisheva22 --- demos/utils/clusterlink.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/utils/clusterlink.py b/demos/utils/clusterlink.py index a928d3ad0..361b302d9 100644 --- a/demos/utils/clusterlink.py +++ b/demos/utils/clusterlink.py @@ -52,7 +52,7 @@ def delete_object(self, kind, name, namespace): except Exception as e: print(f"Error deleting {kind} '{name}': {e}") -# Imports Peers contains all the commands for managing peer CRD. +# Peers class contains all the commands for managing peer CRD. class Peers(CRDObject): def __init__(self, namespace): self.namespace=namespace From b01b971b9836f1de4f1b487f2be9996aa0488a27 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Thu, 23 May 2024 01:53:46 -0400 Subject: [PATCH 15/53] demos: Update bookinfo test.py (#606) Update the documentation in the Bookinfo demo file: test.py. Signed-off-by: welisheva22 Co-authored-by: Kfir Toledo --- demos/bookinfo/cloud/test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/bookinfo/cloud/test.py b/demos/bookinfo/cloud/test.py index 9569441c7..a73133005 100755 --- a/demos/bookinfo/cloud/test.py +++ b/demos/bookinfo/cloud/test.py @@ -15,7 +15,7 @@ ################################################################ #Name: Service node test -#Desc: create 1 proxy that send data to target ip +#Desc: create 1 proxy that sends data to target ip ############################################################### import os import sys From 9f5256b553c0729a5b03fffa25722f3536a14c2d Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Thu, 23 May 2024 02:11:29 -0400 Subject: [PATCH 16/53] demos/iPerf3: Update test.py (#605) Update the documentation in the iPerf3 demo file: test.py. Signed-off-by: welisheva22 Co-authored-by: Kfir Toledo --- demos/iperf3/test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/iperf3/test.py b/demos/iperf3/test.py index 0ce6b1034..08a7b83ba 100644 --- a/demos/iperf3/test.py +++ b/demos/iperf3/test.py @@ -55,7 +55,7 @@ def iperf3Test(cl1:Cluster, cl2:Cluster, testOutputFolder,logLevel="info" ,datap cl1.startCluster(testOutputFolder,logLevel, dataplane) cl2.startCluster(testOutputFolder,logLevel, dataplane) - # Create iPerf3 micto-services + # Create iPerf3 micro-services cl1.loadService(srcSvc, "mlabbe/iperf3",f"{folCl}/iperf3-client.yaml" ) cl2.loadService(destSvc, "mlabbe/iperf3",f"{folSv}/iperf3.yaml" ) From 13fe48a95cc9d6ec32ca0b6995afd5cc3d54f6d4 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Fri, 24 May 2024 05:05:26 -0400 Subject: [PATCH 17/53] Update install_clusterlink.sh (#614) documentation Changed spelling in comment (added an e in determine both times it appeared) code Added prefix CL_ to variable CL_ARCH Signed-off-by: welisheva22 --- hack/install_clusterlink.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hack/install_clusterlink.sh b/hack/install_clusterlink.sh index 060a002ed..047502162 100755 --- a/hack/install_clusterlink.sh +++ b/hack/install_clusterlink.sh @@ -1,7 +1,7 @@ #!/bin/sh set -e -# Detrmine the OS. +# Determine the OS. OS=$(uname) if [ "${OS}" = "Darwin" ] ; then CL_OS="darwin" @@ -10,14 +10,14 @@ else fi -# Detrmine the OS architecture. +# Determine the OS architecture. OS_ARCH=$(uname -m) case "${OS_ARCH}" in x86_64|amd64) CL_ARCH=amd64 ;; armv8*|aarch64*|arm64) - ARCH=arm64 + CL_ARCH=arm64 ;; *) echo "This ${OS_ARCH} architecture isn't supported" From e14c66fb1fe19f891f346c19b12b88656433a496 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Fri, 24 May 2024 06:20:09 -0400 Subject: [PATCH 18/53] Update gen-doc-version.sh (#612) documentation changed verbs to be consistent (removed an "s") changed "and" to "the" so it made sense Signed-off-by: welisheva22 --- hack/gen-doc-version.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hack/gen-doc-version.sh b/hack/gen-doc-version.sh index cda960574..996fdde51 100755 --- a/hack/gen-doc-version.sh +++ b/hack/gen-doc-version.sh @@ -21,9 +21,9 @@ # 2. Copy and git commit the contents of the last released docs directory # (`$PREVIOUS_DOCS_VERSION``) into the new directory, to establish a baseline # for documentation comparison. -# 3. Delete and replaces the contents of the new docs directory with the +# 3. Delete and replace the contents of the new docs directory with the # contents of the 'main' docs directory. -# 4. Update and version and/or revision specific value in the documentation. +# 4. Update the version and/or revision specific value in the documentation. # # The unstaged changes in the working directory can now easily be diff'ed # using 'git diff' to review all docs changes made since the previous From 0da9e7e72b0ce1866162f5b36075c601507139da Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Fri, 24 May 2024 06:35:41 -0400 Subject: [PATCH 19/53] Update install-devtools.sh (#611) documentation fixed small typo Signed-off-by: welisheva22 --- hack/install-devtools.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hack/install-devtools.sh b/hack/install-devtools.sh index bb538c744..77edd623c 100755 --- a/hack/install-devtools.sh +++ b/hack/install-devtools.sh @@ -3,7 +3,7 @@ set -e #-- docker/Python # Docker and python can be quite involved to get right (e.g., require sudo, -# setting up permissions, etc.), so we ask the user to install them manully. +# setting up permissions, etc.), so we ask the user to install them manually. # The devcontainer will install using apt-get, so that's probably ok for now. if [ -z "$(which docker)" ] || [ "$1" = "--force" ]; then echo "please install docker manually (https://docs.docker.com/engine/install/)" From 916babd772c8909533b1545e10401777dd77baf6 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Sun, 26 May 2024 04:35:57 -0400 Subject: [PATCH 20/53] Update instance_types.go (#613) * small documentation tweaks in instance_types.go * update CRD yaml --------- Signed-off-by: welisheva22 Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- config/crds/clusterlink.net_instances.yaml | 8 +++++--- pkg/apis/clusterlink.net/v1alpha1/instance_types.go | 8 ++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/config/crds/clusterlink.net_instances.yaml b/config/crds/clusterlink.net_instances.yaml index 1087f0cd1..98085efda 100644 --- a/config/crds/clusterlink.net_instances.yaml +++ b/config/crds/clusterlink.net_instances.yaml @@ -115,11 +115,12 @@ spec: type: string type: object status: - description: InstanceStatus defines the observed state of ClusterlLink + description: InstanceStatus defines the observed state of a ClusterlLink Instance. properties: controlplane: - description: ComponentStatus defines the status of component in ClusterLink. + description: ComponentStatus defines the status of a component in + ClusterLink. properties: conditions: additionalProperties: @@ -195,7 +196,8 @@ spec: type: object type: object dataplane: - description: ComponentStatus defines the status of component in ClusterLink. + description: ComponentStatus defines the status of a component in + ClusterLink. properties: conditions: additionalProperties: diff --git a/pkg/apis/clusterlink.net/v1alpha1/instance_types.go b/pkg/apis/clusterlink.net/v1alpha1/instance_types.go index 4889d035c..ad0aeee87 100644 --- a/pkg/apis/clusterlink.net/v1alpha1/instance_types.go +++ b/pkg/apis/clusterlink.net/v1alpha1/instance_types.go @@ -27,7 +27,7 @@ const ( ServiceReady StatusConditionType = "ServiceReady" ) -// IngressType represents the ingress type of the deployed ClusterLink. +// IngressType represents the ingress type of the deployed ClusterLink. type IngressType string const ( @@ -39,7 +39,7 @@ const ( IngressTypeLoadBalancer IngressType = "LoadBalancer" ) -// DataplaneType represents the dataplane type of the deployed ClusterLink. +// DataplaneType represents the dataplane type of the deployed ClusterLink. type DataplaneType string const ( @@ -54,7 +54,7 @@ const ( DefaultExternalPort = 443 ) -// ComponentStatus defines the status of component in ClusterLink. +// ComponentStatus defines the status of a component in ClusterLink. type ComponentStatus struct { // Conditions contain the status conditions. Conditions map[string]metav1.Condition `json:"conditions,omitempty"` @@ -70,7 +70,7 @@ type IngressStatus struct { Conditions map[string]metav1.Condition `json:"conditions,omitempty"` } -// InstanceStatus defines the observed state of ClusterlLink Instance. +// InstanceStatus defines the observed state of a ClusterlLink Instance. type InstanceStatus struct { Controlplane ComponentStatus `json:"controlplane,omitempty"` Dataplane ComponentStatus `json:"dataplane,omitempty"` From 3e9304138ff6a5f05b0eb4fca9bd8de116b27d79 Mon Sep 17 00:00:00 2001 From: Michal Malka Date: Sun, 26 May 2024 12:34:04 +0300 Subject: [PATCH 21/53] enable nilnil linter (#288) Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- .golangci.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.golangci.yaml b/.golangci.yaml index b2e0d4058..bb1bd4d75 100644 --- a/.golangci.yaml +++ b/.golangci.yaml @@ -46,7 +46,7 @@ linters: - nakedret - nestif - nilerr - # - nilnil + - nilnil # - noctx - nolintlint - nonamedreturns From 1aa5bca43c1a0a96b38ebea31587eb616de5e937 Mon Sep 17 00:00:00 2001 From: Ziv Nevo <79099626+zivnevo@users.noreply.github.com> Date: Sun, 26 May 2024 12:52:57 +0300 Subject: [PATCH 22/53] Send attributes of source workload to remote peer (#609) Signed-off-by: Ziv Nevo --- pkg/controlplane/api/authz.go | 7 ++++++- pkg/controlplane/authz/manager.go | 7 ++++--- pkg/controlplane/authz/server.go | 5 ++--- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/pkg/controlplane/api/authz.go b/pkg/controlplane/api/authz.go index ff5594d2b..13d008e03 100644 --- a/pkg/controlplane/api/authz.go +++ b/pkg/controlplane/api/authz.go @@ -13,7 +13,10 @@ package api -import "github.com/lestrrat-go/jwx/jwa" +import ( + "github.com/clusterlink-net/clusterlink/pkg/controlplane/authz/connectivitypdp" + "github.com/lestrrat-go/jwx/jwa" +) const ( // RemotePeerAuthorizationPath is the path remote peers use to send an authorization request. @@ -50,6 +53,8 @@ type AuthorizationRequest struct { ServiceName string // ServiceNamespace is the namespace of the requested exported service. ServiceNamespace string + // Attributes of the source workload, to be used by the PDP on the remote peer + SrcAttributes connectivitypdp.WorkloadAttrs } // AuthorizationResponse represents a response for a successful AuthorizationRequest. diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index fceb9f18d..b71144310 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -71,6 +71,8 @@ type egressAuthorizationResponse struct { type ingressAuthorizationRequest struct { // Service is the name of the requested exported service. ServiceName types.NamespacedName + // Attributes of the source workload, to be used by the PDP on the remote peer + SrcAttributes connectivitypdp.WorkloadAttrs } // ingressAuthorizationResponse (from remote peer controlplane) represents a response for an ingressAuthorizationRequest. @@ -263,6 +265,7 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR peerResp, err := cl.Authorize(&cpapi.AuthorizationRequest{ ServiceName: DstName, ServiceNamespace: DstNamespace, + SrcAttributes: srcAttributes, }) if err != nil { m.logger.Infof("Unable to get access token from peer: %v", err) @@ -322,7 +325,6 @@ func (m *Manager) parseAuthorizationHeader(token string) (string, error) { func (m *Manager) authorizeIngress( ctx context.Context, req *ingressAuthorizationRequest, - pr string, ) (*ingressAuthorizationResponse, error) { m.logger.Infof("Received ingress authorization request: %v.", req) @@ -344,13 +346,12 @@ func (m *Manager) authorizeIngress( resp.ServiceExists = true - srcAttributes := connectivitypdp.WorkloadAttrs{GatewayNameLabel: pr} dstAttributes := connectivitypdp.WorkloadAttrs{ ServiceNameLabel: req.ServiceName.Name, ServiceNamespaceLabel: req.ServiceName.Namespace, GatewayNameLabel: m.peerName, } - decision, err := m.connectivityPDP.Decide(srcAttributes, dstAttributes, req.ServiceName.Namespace) + decision, err := m.connectivityPDP.Decide(req.SrcAttributes, dstAttributes, req.ServiceName.Namespace) if err != nil { return nil, fmt.Errorf("error deciding on an ingress connection: %w", err) } diff --git a/pkg/controlplane/authz/server.go b/pkg/controlplane/authz/server.go index b91667f6b..7ad9c2f5b 100644 --- a/pkg/controlplane/authz/server.go +++ b/pkg/controlplane/authz/server.go @@ -141,7 +141,6 @@ func (s *server) PeerAuthorize(w http.ResponseWriter, r *http.Request) { return } - peerName := r.TLS.PeerCertificates[0].DNSNames[0] resp, err := s.manager.authorizeIngress( r.Context(), &ingressAuthorizationRequest{ @@ -149,8 +148,8 @@ func (s *server) PeerAuthorize(w http.ResponseWriter, r *http.Request) { Namespace: req.ServiceNamespace, Name: req.ServiceName, }, - }, - peerName) + SrcAttributes: req.SrcAttributes, + }) switch { case err != nil: http.Error(w, err.Error(), http.StatusInternalServerError) From 8bbbeeac8347ce624537b8266b736b047a70f23a Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Mon, 27 May 2024 13:55:47 +0300 Subject: [PATCH 23/53] [website] revise About page based on intro blog (#615) * revise About page based on intro blog Signed-off-by: Etai Lev Ran --- website/content/en/about/_index.md | 78 +++++++++++++++++++++++++----- 1 file changed, 66 insertions(+), 12 deletions(-) diff --git a/website/content/en/about/_index.md b/website/content/en/about/_index.md index f0e00c2f7..b78d0de4e 100644 --- a/website/content/en/about/_index.md +++ b/website/content/en/about/_index.md @@ -12,21 +12,75 @@ menu: {main: {weight: 30, pre: "" }} {{% pageinfo %}} ClusterLink is in **alpha** status and not ready for production use. {{% /pageinfo %}} -ClusterLink is a secure interconnectivity solution for working across the multi-cloud. It simplifies the connection between application services that are located in different domains, networks, and cloud infrastructures. -## The main usage scenarios +ClusterLink is an [open source][] project that offers a secure and performant + solution for interconnecting services across multiple clusters in + different domains, networks, and cloud infrastructures. -* Efficiently connecting services across multiple cloud clusters -* Providing seamless application-centric connectivity across existing private networks + Compared with other solutions in this space, ClusterLink control plane is simpler + to configure and operate at scale, and its data plane is more secure, scalable and + efficient. -## Core features +ClusterLink's management model is built on the following abstractions: -* Ensures global presence and availability -* Avoids vendor lock-in -* Allows for interoperability due to architectural openness -* Follows these principles through a Golang implementation: - * **Programmable**: exposes programmable interfaces between control, data, and management components. - * **Open and Extensible**: operates across administrative, technology, and vendor boundaries without interfering with the existing networks installed in the interconnected locations. - * **Connection Oriented**: supports predictable performance on business-meaningful connections such as service-to-service or client-to-service communication flows. +- **Fabric**: a set of collaborating clusters, all sharing the same root of trust. +- **Peer**: a specific cluster in a fabric. Each fabric peer makes independent + decisions on service sharing and access control. +- **Export**/**Import**: services must be explicitly shared by clusters before + they can be used. +- **Access policies**: ClusterLink supports fine-grained segmentation with a + "default deny" policy, adhering to "zero trust" principles. + +## Architecture + +ClusterLink consists of several main components that work together to securely + connect workloads across multiple Kubernetes clusters, both on-premises and on + public clouds. These run as regular Kubernetes deployments and can take advantage + of existing mechanisms such as horizontal scaling, rolling deployments, etc. + +ClusterLink uses the Kubernetes API servers for its configuration store. The + **control plane** is responsible for watching for changes in relevant built-in + and custom resources and configuring the data plane Pods. + The control plane is also responsible for managing local Kubernetes + services and endpoints corresponding to imported remote services. By using + standard Kubernetes services, ClusterLink integrates seamlessly with the Kubernetes + network model, and can work with any Kubernetes distribution, CNI and IP address + management scheme. + +The local service endpoints refer to **data plane** Pods, responsible for + workload-to-service secure tunnels to other clusters. The data plane uses + [HTTP CONNECT][] with [mutual TLS][] for security. + The use of HTTPS over tcp/443 removes the need for VPNs and special firewall + configurations. Certificate-based mTLS guarantees in-transit data + encryption and limits allowed connections to other fabric peers only. In addition, + all data plane connections between clusters are explicitly approved by the + control plane and must pass independent egress and ingress access policies + before any workload data is carried across. + +### Use cases + +In addition to the typical multicluster networking use cases, such as + HA/DR, cloud bursting, and connecting microservices deployed across + geographically distributed clusters, ClusterLink can also provide + significant benefits which are not well served by other solutions. + Specifically, ClusterLink can address requirements of use cases where: + +- **clusters are aligned with organizational units** - sharing *internal* + microservices with other clusters and namespaces should be limited to those + belonging to the same unit, while also communicating with *exposed* services + belonging to other units. +- services are owned by **different administrative domains** (e.g., different + development teams) - thus judicious sharing and more stringent access + controls across clusters are needed. +- it is desirable to **increase scalability and limit information sharing** by + minimizing information exchanged between clusters - with ClusterLink, each + cluster manages its own naming and load balancing, requiring considerable + less cross-cluster metadata for its communication. +- there is a need for **separation of concerns** - that is, between network administrators + and application owners. {{% /blocks/section %}} + +[open source]: https://github.com/clusterlink-net/clusterlink +[HTTP CONNECT]: https://en.wikipedia.org/wiki/HTTP_tunnel +[mutual TLS]: https://en.wikipedia.org/wiki/Mutual_authentication#mTLS From 39207487c1c382c19850b65038d46ddbdee5df47 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 27 May 2024 14:17:17 +0300 Subject: [PATCH 24/53] build(deps): bump sigs.k8s.io/controller-runtime in the k8s group (#616) Bumps the k8s group with 1 update: [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime). Updates `sigs.k8s.io/controller-runtime` from 0.18.2 to 0.18.3 - [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases) - [Changelog](https://github.com/kubernetes-sigs/controller-runtime/blob/main/RELEASE.md) - [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.18.2...v0.18.3) --- updated-dependencies: - dependency-name: sigs.k8s.io/controller-runtime dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 4 ++-- go.sum | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 289e0d2ae..f8c5572ae 100644 --- a/go.mod +++ b/go.mod @@ -22,7 +22,7 @@ require ( k8s.io/apimachinery v0.30.1 k8s.io/client-go v0.30.1 k8s.io/utils v0.0.0-20230726121419-3b25d923346b - sigs.k8s.io/controller-runtime v0.18.2 + sigs.k8s.io/controller-runtime v0.18.3 sigs.k8s.io/e2e-framework v0.3.0 ) @@ -90,7 +90,7 @@ require ( gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect - k8s.io/apiextensions-apiserver v0.30.0 // indirect + k8s.io/apiextensions-apiserver v0.30.1 // indirect k8s.io/klog/v2 v2.120.1 // indirect k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect diff --git a/go.sum b/go.sum index abd8cbea9..04f1bd070 100644 --- a/go.sum +++ b/go.sum @@ -271,8 +271,8 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= k8s.io/api v0.30.1 h1:kCm/6mADMdbAxmIh0LBjS54nQBE+U4KmbCfIkF5CpJY= k8s.io/api v0.30.1/go.mod h1:ddbN2C0+0DIiPntan/bye3SW3PdwLa11/0yqwvuRrJM= -k8s.io/apiextensions-apiserver v0.30.0 h1:jcZFKMqnICJfRxTgnC4E+Hpcq8UEhT8B2lhBcQ+6uAs= -k8s.io/apiextensions-apiserver v0.30.0/go.mod h1:N9ogQFGcrbWqAY9p2mUAL5mGxsLqwgtUce127VtRX5Y= +k8s.io/apiextensions-apiserver v0.30.1 h1:4fAJZ9985BmpJG6PkoxVRpXv9vmPUOVzl614xarePws= +k8s.io/apiextensions-apiserver v0.30.1/go.mod h1:R4GuSrlhgq43oRY9sF2IToFh7PVlF1JjfWdoG3pixk4= k8s.io/apimachinery v0.30.1 h1:ZQStsEfo4n65yAdlGTfP/uSHMQSoYzU/oeEbkmF7P2U= k8s.io/apimachinery v0.30.1/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= k8s.io/client-go v0.30.1 h1:uC/Ir6A3R46wdkgCV3vbLyNOYyCJ8oZnjtJGKfytl/Q= @@ -283,8 +283,8 @@ k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7F k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSnlTLKgpAAttJvpI= k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= -sigs.k8s.io/controller-runtime v0.18.2 h1:RqVW6Kpeaji67CY5nPEfRz6ZfFMk0lWQlNrLqlNpx+Q= -sigs.k8s.io/controller-runtime v0.18.2/go.mod h1:tuAt1+wbVsXIT8lPtk5RURxqAnq7xkpv2Mhttslg7Hw= +sigs.k8s.io/controller-runtime v0.18.3 h1:B5Wmmo8WMWK7izei+2LlXLVDGzMwAHBNLX68lwtlSR4= +sigs.k8s.io/controller-runtime v0.18.3/go.mod h1:TVoGrfdpbA9VRFaRnKgk9P5/atA0pMwq+f+msb9M8Sg= sigs.k8s.io/e2e-framework v0.3.0 h1:eqQALBtPCth8+ulTs6lcPK7ytV5rZSSHJzQHZph4O7U= sigs.k8s.io/e2e-framework v0.3.0/go.mod h1:C+ef37/D90Dc7Xq1jQnNbJYscrUGpxrWog9bx2KIa+c= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= From c00e7f7a6f5c7bfcf745efcc6a5e7a23d2a4074a Mon Sep 17 00:00:00 2001 From: Ziv Nevo <79099626+zivnevo@users.noreply.github.com> Date: Mon, 27 May 2024 17:38:27 +0300 Subject: [PATCH 25/53] Add import and export labels as destination attributes (#618) Also, Setting service name to be the export name --------- Signed-off-by: Ziv Nevo --- pkg/controlplane/authz/manager.go | 23 +++++++++++++------- tests/e2e/k8s/suite.go | 1 + tests/e2e/k8s/test_policy.go | 35 +++++++++++++++++++++++++------ tests/e2e/k8s/util/clusterlink.go | 2 ++ tests/e2e/k8s/util/kind.go | 2 ++ 5 files changed, 50 insertions(+), 13 deletions(-) diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index b71144310..0752376ff 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -43,6 +43,7 @@ const ( ServiceNameLabel = "clusterlink/metadata.serviceName" ServiceNamespaceLabel = "clusterlink/metadata.serviceNamespace" + ServiceLabelsPrefix = "service/metadata." GatewayNameLabel = "clusterlink/metadata.gatewayName" ) @@ -206,6 +207,11 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR return nil, fmt.Errorf("cannot get import %v: %w", req.ImportName, err) } + dstAttributes := connectivitypdp.WorkloadAttrs{} + for k, v := range imp.Labels { // add import labels to destination attributes + dstAttributes[ServiceLabelsPrefix+k] = v + } + lbResult := NewLoadBalancingResult(&imp) for { if err := m.loadBalancer.Select(lbResult); err != nil { @@ -230,11 +236,10 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR } } - dstAttributes := connectivitypdp.WorkloadAttrs{ - ServiceNameLabel: imp.Name, - ServiceNamespaceLabel: imp.Namespace, - GatewayNameLabel: importSource.Peer, - } + dstAttributes[ServiceNameLabel] = importSource.ExportName + dstAttributes[ServiceNamespaceLabel] = importSource.ExportNamespace + dstAttributes[GatewayNameLabel] = importSource.Peer + decision, err := m.connectivityPDP.Decide(srcAttributes, dstAttributes, req.ImportName.Namespace) if err != nil { return nil, fmt.Errorf("error deciding on an egress connection: %w", err) @@ -347,10 +352,14 @@ func (m *Manager) authorizeIngress( resp.ServiceExists = true dstAttributes := connectivitypdp.WorkloadAttrs{ - ServiceNameLabel: req.ServiceName.Name, - ServiceNamespaceLabel: req.ServiceName.Namespace, + ServiceNameLabel: export.Name, + ServiceNamespaceLabel: export.Namespace, GatewayNameLabel: m.peerName, } + for k, v := range export.Labels { // add export labels to destination attributes + dstAttributes[ServiceLabelsPrefix+k] = v + } + decision, err := m.connectivityPDP.Decide(req.SrcAttributes, dstAttributes, req.ServiceName.Namespace) if err != nil { return nil, fmt.Errorf("error deciding on an ingress connection: %w", err) diff --git a/tests/e2e/k8s/suite.go b/tests/e2e/k8s/suite.go index d9064d26c..7b62f8c2d 100644 --- a/tests/e2e/k8s/suite.go +++ b/tests/e2e/k8s/suite.go @@ -47,6 +47,7 @@ var httpEchoService = util.Service{ Name: "http-echo", Namespace: v1.NamespaceDefault, Port: 8080, + Labels: map[string]string{"env": "test"}, } // TestSuite is a suite for e2e testing on k8s clusters. diff --git a/tests/e2e/k8s/test_policy.go b/tests/e2e/k8s/test_policy.go index bd97bbc69..87f67a972 100644 --- a/tests/e2e/k8s/test_policy.go +++ b/tests/e2e/k8s/test_policy.go @@ -32,8 +32,9 @@ func (s *TestSuite) TestPolicyLabels() { require.Nil(s.T(), cl[1].CreatePeer(cl[0])) importedService := &util.Service{ - Name: httpEchoService.Name, - Port: 80, + Name: httpEchoService.Name, + Port: 80, + Labels: httpEchoService.Labels, } require.Nil(s.T(), cl[1].CreateImport(importedService, cl[0], httpEchoService.Name)) @@ -44,8 +45,9 @@ func (s *TestSuite) TestPolicyLabels() { authz.GatewayNameLabel: cl[1].Name(), } dstLabels := map[string]string{ // allow traffic only to echo in cl1 - authz.ServiceNameLabel: httpEchoService.Name, - authz.GatewayNameLabel: cl[0].Name(), + authz.ServiceNameLabel: httpEchoService.Name, + authz.GatewayNameLabel: cl[0].Name(), + authz.ServiceLabelsPrefix + "env": "test", } allowEchoPolicy := util.NewPolicy(allowEchoPolicyName, v1alpha1.AccessPolicyActionAllow, srcLabels, dstLabels) require.Nil(s.T(), cl[1].CreatePolicy(allowEchoPolicy)) @@ -109,15 +111,36 @@ func (s *TestSuite) TestPolicyLabels() { // 8. Replace the policy in cl[1] with a policy having a wrong service name - connection should be denied require.Nil(s.T(), cl[1].DeletePolicy(allowEchoPolicyName)) - badSvcLabels := map[string]string{ + attrsWithBadSvcName := map[string]string{ authz.ServiceNameLabel: "bad-svc", authz.GatewayNameLabel: cl[0].Name(), } - badSvcPolicy := util.NewPolicy("bad-svc", v1alpha1.AccessPolicyActionAllow, nil, badSvcLabels) + badSvcPolicy := util.NewPolicy("bad-svc", v1alpha1.AccessPolicyActionAllow, nil, attrsWithBadSvcName) require.Nil(s.T(), cl[1].CreatePolicy(badSvcPolicy)) _, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, true, &services.ConnectionResetError{}) require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) + + // 9. Add an allow policy in cl[1], but with a wrong service label - connection should still be denied + attrsWithBadSvcLabels := map[string]string{ + authz.ServiceLabelsPrefix + "env": "prod", + } + badLabelPolicy := util.NewPolicy("bad-label", v1alpha1.AccessPolicyActionAllow, nil, attrsWithBadSvcLabels) + require.Nil(s.T(), cl[1].CreatePolicy(badLabelPolicy)) + + _, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, true, &services.ConnectionResetError{}) + require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) + + // 10. Add an allow policy in cl[1], now with the right service label - connection should be allowed + attrsWithGoodSvcLabels := map[string]string{ + authz.ServiceLabelsPrefix + "env": "test", + } + GoodLabelPolicy := util.NewPolicy("good-label", v1alpha1.AccessPolicyActionAllow, nil, attrsWithGoodSvcLabels) + require.Nil(s.T(), cl[1].CreatePolicy(GoodLabelPolicy)) + + data, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, true, nil) + require.Nil(s.T(), err) + require.Equal(s.T(), cl[0].Name(), data) } func (s *TestSuite) TestPrivilegedPolicies() { diff --git a/tests/e2e/k8s/util/clusterlink.go b/tests/e2e/k8s/util/clusterlink.go index 236eb7c71..6eadf4afa 100644 --- a/tests/e2e/k8s/util/clusterlink.go +++ b/tests/e2e/k8s/util/clusterlink.go @@ -168,6 +168,7 @@ func (c *ClusterLink) CreateExport(service *Service) error { ObjectMeta: metav1.ObjectMeta{ Name: service.Name, Namespace: c.namespace, + Labels: service.Labels, }, Spec: v1alpha1.ExportSpec{ Port: service.Port, @@ -193,6 +194,7 @@ func (c *ClusterLink) CreateImport(service *Service, peer *ClusterLink, exportNa ObjectMeta: metav1.ObjectMeta{ Name: service.Name, Namespace: c.namespace, + Labels: service.Labels, }, Spec: v1alpha1.ImportSpec{ Port: service.Port, diff --git a/tests/e2e/k8s/util/kind.go b/tests/e2e/k8s/util/kind.go index 43fea25b2..a2d596a7a 100644 --- a/tests/e2e/k8s/util/kind.go +++ b/tests/e2e/k8s/util/kind.go @@ -61,6 +61,8 @@ type Service struct { Namespace string // Port is the service external listening port. Port uint16 + // A key-value map to organize and categorize (scope and select) services. + Labels map[string]string } // PodAndService represents a kubernetes service and a backing pod. From ef2756401a18bc26aedf590b549db74fd2d7ade5 Mon Sep 17 00:00:00 2001 From: welisheva22 Date: Mon, 27 May 2024 13:41:48 -0400 Subject: [PATCH 26/53] Create SECURITY.md (#604) * Create initial version of SECURITY.md * enable private vulnerability reporting --------- Signed-off-by: welisheva22 Signed-off-by: Etai Lev Ran Co-authored-by: Etai Lev Ran --- SECURITY.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 SECURITY.md diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..8fe36f0f4 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,51 @@ +# Security policy + +Thank you for your interest in the security of the ClusterLink project. + We created this project with security in mind - enabling simple, performant + **and** secure communication across boundaries in the hybrid cloud. + +We are in the **alpha** phase of this project and do not yet recommend using + ClusterLink in production. However, we would welcome your contributions to the + code base and/or documentation including pointing out potential security issues + so that collectively we can create a solid, robust solution that continuously + improves. + +## Security bulletins + +For information regarding the security of this project please join the + [users mailing list][]. + +## Reporting a vulnerability + +We're extremely grateful for security researchers and users that report + vulnerabilities to the ClusterLink Open Source Community. All reports + are thoroughly investigated by a set of community volunteers. + +We use [GitHub private vulnerability reporting][] for ClusterLink. Private vulnerability + reporting provides an easy way for vulnerability reporters to privately disclose + security risks to repository maintainers, within GitHub, and in a way that immediately + notifies the repository maintainers of the issue. + +You will receive a reply from one of the maintainers within a week, acknowledging receipt + of the vulnerability report. You may be contacted to discuss the reported item further. + Please bear with us as we seek to understand the breadth and scope of the reported + problem, recreate it, and confirm if there is a vulnerability present. + +### When Should I Report a Vulnerability? + +- You think you discovered a potential security vulnerability in ClusterLink + components or features +- You are unsure how a vulnerability affects ClusterLink +- You think you discovered a vulnerability in another project that ClusterLink + depends on. For projects with their own vulnerability reporting and disclosure + process, please report the vulnerability directly there. + +### When Should I NOT Report a Vulnerability? + +- You need help tuning ClusterLink components for security (e.g., advice and help + on setting access control policies for specific use cases) +- You need help applying security related updates +- Your issue is not security related + +[users mailing list]: https://groups.google.com/g/clusterlink-users +[GitHub private vulnerability reporting]: https://github.com/clusterlink-net/clusterlink/security/advisories/new From c3da278dde804677aab7f79f24c1092ac83b8731 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Tue, 28 May 2024 08:07:55 +0300 Subject: [PATCH 27/53] website: Create a nginx toturial (#608) * Move the shared content of the tutorials to a shared folder. * Add a Nginx toturial. --------- Signed-off-by: Kfir Toledo --- .../testdata/clusterlink/allow-policy.yaml | 11 + .../testdata/clusterlink/export-nginx.yaml | 7 + .../testdata/clusterlink/import-nginx.yaml | 11 + .../testdata/clusterlink/peer-client.yaml | 9 + .../testdata/clusterlink/peer-server.yaml | 9 + demos/nginx/testdata/nginx-job.yaml | 12 + demos/nginx/testdata/nginx-server.yaml | 31 ++ .../en/docs/main/tutorials/bookinfo/index.md | 26 +- .../en/docs/main/tutorials/iperf/index.md | 239 +--------------- .../en/docs/main/tutorials/nginx/index.md | 264 ++++++++++++++++++ website/layouts/shortcodes/readfile.html | 41 --- .../files/tutorials/allow-all-policy.md | 66 +++++ .../files/tutorials/cli-installation.md | 12 + .../files/tutorials/deploy-clusterlink.md | 62 ++++ website/static/files/tutorials/envsubst.md | 14 + website/static/files/tutorials/peer.md | 64 +++++ 16 files changed, 588 insertions(+), 290 deletions(-) create mode 100644 demos/nginx/testdata/clusterlink/allow-policy.yaml create mode 100644 demos/nginx/testdata/clusterlink/export-nginx.yaml create mode 100644 demos/nginx/testdata/clusterlink/import-nginx.yaml create mode 100644 demos/nginx/testdata/clusterlink/peer-client.yaml create mode 100644 demos/nginx/testdata/clusterlink/peer-server.yaml create mode 100644 demos/nginx/testdata/nginx-job.yaml create mode 100644 demos/nginx/testdata/nginx-server.yaml create mode 100644 website/content/en/docs/main/tutorials/nginx/index.md delete mode 100644 website/layouts/shortcodes/readfile.html create mode 100644 website/static/files/tutorials/allow-all-policy.md create mode 100644 website/static/files/tutorials/cli-installation.md create mode 100644 website/static/files/tutorials/deploy-clusterlink.md create mode 100644 website/static/files/tutorials/envsubst.md create mode 100644 website/static/files/tutorials/peer.md diff --git a/demos/nginx/testdata/clusterlink/allow-policy.yaml b/demos/nginx/testdata/clusterlink/allow-policy.yaml new file mode 100644 index 000000000..0d3ef23ae --- /dev/null +++ b/demos/nginx/testdata/clusterlink/allow-policy.yaml @@ -0,0 +1,11 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: AccessPolicy +metadata: + name: allow-policy + namespace: default +spec: + action: allow + from: + - workloadSelector: {} + to: + - workloadSelector: {} diff --git a/demos/nginx/testdata/clusterlink/export-nginx.yaml b/demos/nginx/testdata/clusterlink/export-nginx.yaml new file mode 100644 index 000000000..236adb2bf --- /dev/null +++ b/demos/nginx/testdata/clusterlink/export-nginx.yaml @@ -0,0 +1,7 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Export +metadata: + name: nginx + namespace: default +spec: + port: 80 diff --git a/demos/nginx/testdata/clusterlink/import-nginx.yaml b/demos/nginx/testdata/clusterlink/import-nginx.yaml new file mode 100644 index 000000000..893b23ef7 --- /dev/null +++ b/demos/nginx/testdata/clusterlink/import-nginx.yaml @@ -0,0 +1,11 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: nginx + namespace: default +spec: + port: 80 + sources: + - exportName: nginx + exportNamespace: default + peer: server diff --git a/demos/nginx/testdata/clusterlink/peer-client.yaml b/demos/nginx/testdata/clusterlink/peer-client.yaml new file mode 100644 index 000000000..673b5dd33 --- /dev/null +++ b/demos/nginx/testdata/clusterlink/peer-client.yaml @@ -0,0 +1,9 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Peer +metadata: + name: client + namespace: clusterlink-system +spec: + gateways: + - host: "${CLIENT_IP}" + port: 30443 diff --git a/demos/nginx/testdata/clusterlink/peer-server.yaml b/demos/nginx/testdata/clusterlink/peer-server.yaml new file mode 100644 index 000000000..6abafd3ac --- /dev/null +++ b/demos/nginx/testdata/clusterlink/peer-server.yaml @@ -0,0 +1,9 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Peer +metadata: + name: server + namespace: clusterlink-system +spec: + gateways: + - host: "${SERVER_IP}" + port: 30443 diff --git a/demos/nginx/testdata/nginx-job.yaml b/demos/nginx/testdata/nginx-job.yaml new file mode 100644 index 000000000..6dd486a26 --- /dev/null +++ b/demos/nginx/testdata/nginx-job.yaml @@ -0,0 +1,12 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: curl-nginx-homepage +spec: + template: + spec: + containers: + - name: curl + image: curlimages/curl:latest + command: ["curl", "http://nginx.default.svc.cluster.local"] + restartPolicy: Never diff --git a/demos/nginx/testdata/nginx-server.yaml b/demos/nginx/testdata/nginx-server.yaml new file mode 100644 index 000000000..caf9750f6 --- /dev/null +++ b/demos/nginx/testdata/nginx-server.yaml @@ -0,0 +1,31 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + replicas: 1 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:latest + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx +spec: + selector: + app: nginx + ports: + - protocol: TCP + port: 80 + targetPort: 80 diff --git a/website/content/en/docs/main/tutorials/bookinfo/index.md b/website/content/en/docs/main/tutorials/bookinfo/index.md index 2f00fa50a..3398df09c 100644 --- a/website/content/en/docs/main/tutorials/bookinfo/index.md +++ b/website/content/en/docs/main/tutorials/bookinfo/index.md @@ -20,17 +20,7 @@ System illustration: ## Install ClusterLink CLI -1. Install ClusterLink CLI on Linux or Mac using the installation script: - - ```sh - curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - - ``` - -1. Verify the installation: - - ```sh - clusterlink --version - ``` +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} ## Initialize clusters @@ -130,19 +120,7 @@ In this step, we enable connectivity access for the BookInfo application and reviews-v3 (server2). We establish connections between the peers, export the reviews service on the server side, import the reviews service on the client side, and create a policy to allow the connection. -Note that the provided YAML configuration files refer to environment variables - (defined below) that should be set when running the tutorial. The values are - replaced in the YAMLs using `envsubst` utility. - -{{% expand summary="Installing `envsubst` on macOS" %}} -In case `envsubst` does not exist, you can install it with: - -```sh -brew install gettext -brew link --force gettext -``` - -{{% /expand %}} +{{% readfile file="/static/files/tutorials/envsubst.md" %}} ```sh kubectl config use-context kind-client diff --git a/website/content/en/docs/main/tutorials/iperf/index.md b/website/content/en/docs/main/tutorials/iperf/index.md index 99aae9890..9296b65a5 100644 --- a/website/content/en/docs/main/tutorials/iperf/index.md +++ b/website/content/en/docs/main/tutorials/iperf/index.md @@ -11,17 +11,7 @@ The tutorial uses two kind clusters: ## Install ClusterLink CLI -1. Install ClusterLink CLI on Linux or Mac using the installation script: - - ```sh - curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - - ``` - -1. Verify the installation: - - ```sh - clusterlink --version - ``` +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} ## Initialize clusters @@ -88,82 +78,20 @@ Install iPerf3 (client and server) on the clusters: *Client cluster*: ```sh -export IPERF3_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests -kubectl apply -f $IPERF3_FILES/iperf3-client/iperf3-client.yaml +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests +kubectl apply -f $TEST_FILES/iperf3-client/iperf3-client.yaml ``` *Server cluster*: ```sh -export IPERF3_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests -kubectl apply -f $IPERF3_FILES/iperf3-server/iperf3.yaml +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests +kubectl apply -f $TEST_FILES/iperf3-server/iperf3.yaml ``` ## Deploy ClusterLink -1. Create the fabric and peer certificates for the clusters: - - *Client cluster*: - - ```sh - clusterlink create fabric - clusterlink create peer-cert --name client - ``` - - *Server cluster*: - - ```sh - clusterlink create peer-cert --name server - ``` - - For more details regarding fabric and peer see [core concepts][]. - -1. Deploy ClusterLink on each cluster: - - *Client cluster*: - - ```sh - clusterlink deploy peer --name client --ingress=NodePort --ingress-port=30443 - ``` - - *Server cluster*: - - ```sh - clusterlink deploy peer --name server --ingress=NodePort --ingress-port=30443 - ``` - - {{< notice note >}} - This tutorial uses NodePort to create an external access point for the kind clusters. - By default `deploy peer` creates an ingress of type LoadBalancer, - which is more suitable for Kubernetes clusters running in the cloud. - {{< /notice >}} - -1. Verify that ClusterLink control and data plane components are running: - - It may take a few seconds for the deployments to be successfully created. - - *Client cluster*: - - ```sh - kubectl rollout status deployment cl-controlplane -n clusterlink-system - kubectl rollout status deployment cl-dataplane -n clusterlink-system - ``` - - *Server cluster*: - - ```sh - kubectl rollout status deployment cl-controlplane -n clusterlink-system - kubectl rollout status deployment cl-dataplane -n clusterlink-system - ``` - - {{% expand summary="Sample output" %}} - - ```sh - deployment "cl-controlplane" successfully rolled out - deployment "cl-dataplane" successfully rolled out - ``` - - {{% /expand %}} +{{% readfile file="/static/files/tutorials/deploy-clusterlink.md" %}} ## Enable cross-cluster access @@ -171,85 +99,11 @@ In this step, we enable connectivity access between the iPerf3 client and server For each step, you have an example demonstrating how to apply the command from a file or providing the complete custom resource (CR) associated with the command. -Note that the provided YAML configuration files refer to environment variables - (defined below) that should be set when running the tutorial. The values are - replaced in the YAMLs using `envsubst` utility. - -{{% expand summary="Installing `envsubst` on macOS" %}} -In case `envsubst` does not exist, you can install it with: - -```sh -brew install gettext -brew link --force gettext -``` - -{{% /expand %}} +{{% readfile file="/static/files/tutorials/envsubst.md" %}} ### Set-up peers -Add the remote peer to each cluster: - -*Client cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` -curl -s $IPERF3_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: Peer -metadata: - name: server - namespace: clusterlink-system -spec: - gateways: - - host: "${SERVER_IP}" - port: 30443 -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} - -*Server cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` -curl -s $IPERF3_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: Peer -metadata: - name: client - namespace: clusterlink-system -spec: - gateways: - - host: "${CLIENT_IP}" - port: 30443 -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} +{{% readfile file="/static/files/tutorials/peer.md" %}} {{< notice note >}} The `CLIENT_IP` and `SERVER_IP` refers to the node IP of the peer kind cluster, which assigns the peer YAML file. @@ -265,7 +119,7 @@ In the server cluster, export the iperf3-server service: {{% tab header="File" %}} ```sh -kubectl apply -f $IPERF3_FILES/clusterlink/export-iperf3.yaml +kubectl apply -f $TEST_FILES/clusterlink/export-iperf3.yaml ``` {{% /tab %}} @@ -296,7 +150,7 @@ In the client cluster, import the iperf3-server service from the server cluster: {{% tab header="File" %}} ```sh -kubectl apply -f $IPERF3_FILES/clusterlink/import-iperf3.yaml +kubectl apply -f $TEST_FILES/clusterlink/import-iperf3.yaml ``` {{% /tab %}} @@ -323,71 +177,7 @@ spec: ### Set-up access policies -Create access policies on both clusters to allow connectivity: - -*Client cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -kubectl apply -f $IPERF3_FILES/clusterlink/allow-policy.yaml -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: AccessPolicy -metadata: - name: allow-policy - namespace: default -spec: - action: allow - from: - - workloadSelector: {} - to: - - workloadSelector: {} -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} - -*Server cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -kubectl apply -f $IPERF3_FILES/clusterlink/allow-policy.yaml -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: AccessPolicy -metadata: - name: allow-policy - namespace: default -spec: - action: allow - from: - - workloadSelector: {} - to: - - workloadSelector: {} -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} - -For more details regarding policy configuration, see [policies][] documentation. +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} ## Test service connectivity @@ -448,19 +238,18 @@ iperf Done. ``` 1. Unset the environment variables: + *Client cluster*: ```sh - unset KUBECONFIG IPERF3_FILES IPERF3CLIENT + unset KUBECONFIG TEST_FILES IPERF3CLIENT ``` *Server cluster*: ```sh - unset KUBECONFIG IPERF3_FILES + unset KUBECONFIG TEST_FILES ``` [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start -[core concepts]: {{< relref "../../concepts/_index.md" >}} -[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/content/en/docs/main/tutorials/nginx/index.md b/website/content/en/docs/main/tutorials/nginx/index.md new file mode 100644 index 000000000..e814794fc --- /dev/null +++ b/website/content/en/docs/main/tutorials/nginx/index.md @@ -0,0 +1,264 @@ +--- +title: nginx +description: Running basic connectivity between nginx server and client across two clusters using ClusterLink. +--- + +In this tutorial, we'll establish connectivity across clusters using ClusterLink to access a remote nginx server. +The tutorial uses two kind clusters: + +1) Client cluster - runs ClusterLink along with a client. +2) Server cluster - runs ClusterLink along with a nginx server. + +## Install ClusterLink CLI + +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} + +## Initialize clusters + +This tutorial uses [kind][] as a local Kubernetes environment. + You can skip this step if you already have access to existing clusters, just be sure to + set KUBECONFIG accordingly. + +To setup two kind clusters: + +1. Install kind using [kind installation guide][]. +1. Create a directory for all the tutorial files: + + ```sh + mkdir nginx-tutorial + ``` + +1. Open two terminals in the tutorial directory and create a kind cluster in each terminal: + + *Client cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=client + ``` + + *Server cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=server + ``` + + {{< notice note >}} + kind uses the prefix `kind`, so the name of created clusters will be **kind-client** and **kind-server**. + {{< /notice >}} + +1. Setup `KUBECONFIG` on each terminal to access the cluster: + + *Client cluster*: + + ```sh + kubectl config use-context kind-client + cp ~/.kube/config $PWD/config-client + export KUBECONFIG=$PWD/config-client + ``` + + *Server cluster*: + + ```sh + kubectl config use-context kind-server + cp ~/.kube/config $PWD/config-server + export KUBECONFIG=$PWD/config-server + ``` + +{{< notice tip >}} +You can run the tutorial in a single terminal and switch access between the clusters +using `kubectl config use-context kind-client` and `kubectl config use-context kind-server`. +{{< /notice >}} + +## Deploy nginx client and server + +Setup the ```TEST_FILES``` variable, and install nginx on the server cluster. + +*Client cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata +``` + +*Server cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata +kubectl apply -f $TEST_FILES/nginx-server.yaml +``` + +## Deploy ClusterLink + +{{% readfile file="/static/files/tutorials/deploy-clusterlink.md" %}} + +## Enable cross-cluster access + +In this step, we enable access between the client and server. + For each step, you have an example demonstrating how to apply the command from a + file or providing the complete custom resource (CR) associated with the command. + +{{% readfile file="/static/files/tutorials/envsubst.md" %}} + +### Set-up peers + +{{% readfile file="/static/files/tutorials/peer.md" %}} + +{{< notice note >}} +The `CLIENT_IP` and `SERVER_IP` refers to the node IP of the peer kind cluster, which assigns the peer YAML file. +{{< /notice >}} + +### Export the nginx server endpoint + +In the server cluster, export the nginx server service: + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/export-nginx.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Export +metadata: + name: nginx + namespace: default +spec: + port: 80 +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up import + +In the client cluster, import the nginx service from the server cluster: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/import-nginx.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: nginx + namespace: default +spec: + port: 80 + sources: + - exportName: nginx + exportNamespace: default + peer: server +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up access policies + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +## Test service connectivity + +Test the connectivity between the clusters with a batch job of the ```curl``` command: + +*Client cluster*: + +```sh +kubectl apply -f $TEST_FILES/nginx-job.yaml +``` + +Verify the job succeeded: + +```sh +kubectl logs jobs/curl-nginx-homepage +``` + +{{% expand summary="Sample output" %}} + +```sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +``` + +{{% /expand %}} + +## Cleanup + +1. Delete the kind clusters: + *Client cluster*: + + ```sh + kind delete cluster --name=client + ``` + + *Server cluster*: + + ```sh + kind delete cluster --name=server + ``` + +1. Remove the tutorial directory: + + ```sh + cd .. && rm -rf nginx-tutorial + ``` + +1. Unset the environment variables: + *Client cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Server cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + +[kind]: https://kind.sigs.k8s.io/ +[kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start diff --git a/website/layouts/shortcodes/readfile.html b/website/layouts/shortcodes/readfile.html deleted file mode 100644 index 5f97b66fb..000000000 --- a/website/layouts/shortcodes/readfile.html +++ /dev/null @@ -1,41 +0,0 @@ -{{/* Store ordinal, to be retrieved by parent element */}} -{{ if ge hugo.Version "0.93.0" }} - {{ .Page.Store.Add "Ordinal" 1 }} -{{ end }} - -{{/* Handle the "file" named parameter or a single unnamed parameter as the file -path */}} -{{ if .IsNamedParams }} - {{ $.Scratch.Set "fparameter" ( .Get "file" ) }} -{{ else }} - {{ $.Scratch.Set "fparameter" ( .Get 0 ) }} -{{ end }} - - -{{/* If the first character is "/", the path is absolute from the site's -`baseURL`. Otherwise, construct an absolute path using the current directory */}} - -{{ if eq (.Scratch.Get "fparameter" | printf "%.1s") "/" }} - {{ $.Scratch.Set "filepath" ($.Scratch.Get "fparameter") }} -{{ else }} - {{ $.Scratch.Set "filepath" "/" }} - {{ $.Scratch.Add "filepath" $.Page.File.Dir }} - {{ $.Scratch.Add "filepath" ($.Scratch.Get "fparameter") }} -{{ end }} - - -{{/* If the file exists, read it and highlight it if it's code. -Throw a compile error or print an error on the page if the file is not found */}} - -{{ if fileExists ($.Scratch.Get "filepath") }} - {{ if eq (.Get "code") "true" }} - {{- highlight ($.Scratch.Get "filepath" | readFile | htmlUnescape | - safeHTML ) (.Get "lang") "" -}} - {{ else }} - {{- $.Scratch.Get "filepath" | os.ReadFile -}} - {{ end }} -{{ else if eq (.Get "draft") "true" }} - -

The file {{ $.Scratch.Get "filepath" }} was not found.

- -{{ else }}{{- errorf "Shortcode %q: file %q not found at %s" .Name ($.Scratch.Get "filepath") .Position -}}{{ end }} diff --git a/website/static/files/tutorials/allow-all-policy.md b/website/static/files/tutorials/allow-all-policy.md new file mode 100644 index 000000000..e11d37d48 --- /dev/null +++ b/website/static/files/tutorials/allow-all-policy.md @@ -0,0 +1,66 @@ + +Create access policies on both clusters to allow connectivity: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: AccessPolicy +metadata: + name: allow-policy + namespace: default +spec: + action: allow + from: + - workloadSelector: {} + to: + - workloadSelector: {} +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: AccessPolicy +metadata: + name: allow-policy + namespace: default +spec: + action: allow + from: + - workloadSelector: {} + to: + - workloadSelector: {} +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies](../../concepts/policies) documentation. diff --git a/website/static/files/tutorials/cli-installation.md b/website/static/files/tutorials/cli-installation.md new file mode 100644 index 000000000..7e8f5e000 --- /dev/null +++ b/website/static/files/tutorials/cli-installation.md @@ -0,0 +1,12 @@ + +1. Install ClusterLink CLI on Linux or Mac using the installation script: + + ```sh + curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - + ``` + +2. Verify the installation: + + ```sh + clusterlink --version + ``` diff --git a/website/static/files/tutorials/deploy-clusterlink.md b/website/static/files/tutorials/deploy-clusterlink.md new file mode 100644 index 000000000..224cf22cf --- /dev/null +++ b/website/static/files/tutorials/deploy-clusterlink.md @@ -0,0 +1,62 @@ + +1. Create the fabric and peer certificates for the clusters: + + *Client cluster*: + + ```sh + clusterlink create fabric + clusterlink create peer-cert --name client + ``` + + *Server cluster*: + + ```sh + clusterlink create peer-cert --name server + ``` + + For more details regarding fabric and peer see [core concepts][]. + +2. Deploy ClusterLink on each cluster: + + *Client cluster*: + + ```sh + clusterlink deploy peer --name client --ingress=NodePort --ingress-port=30443 + ``` + + *Server cluster*: + + ```sh + clusterlink deploy peer --name server --ingress=NodePort --ingress-port=30443 + ``` + + This tutorial uses **NodePort** to create an external access point for the kind clusters. + By default `deploy peer` creates an ingress of type **LoadBalancer**, + which is more suitable for Kubernetes clusters running in the cloud. + +3. Verify that ClusterLink control and data plane components are running: + + It may take a few seconds for the deployments to be successfully created. + + *Client cluster*: + + ```sh + kubectl rollout status deployment cl-controlplane -n clusterlink-system + kubectl rollout status deployment cl-dataplane -n clusterlink-system + ``` + + *Server cluster*: + + ```sh + kubectl rollout status deployment cl-controlplane -n clusterlink-system + kubectl rollout status deployment cl-dataplane -n clusterlink-system + ``` + + {{% expand summary="Sample output" %}} + + deployment "cl-controlplane" successfully rolled out + deployment "cl-dataplane" successfully rolled out + + {{% /expand %}} + +[core concepts]: {{< relref "../../concepts/_index.md" >}} diff --git a/website/static/files/tutorials/envsubst.md b/website/static/files/tutorials/envsubst.md new file mode 100644 index 000000000..a17eea837 --- /dev/null +++ b/website/static/files/tutorials/envsubst.md @@ -0,0 +1,14 @@ + +Note that the provided YAML configuration files refer to environment variables + (defined below) that should be set when running the tutorial. The values are + replaced in the YAMLs using `envsubst` utility. + +{{% expand summary="Installing `envsubst` on macOS" %}} +In case `envsubst` does not exist, you can install it with: + +```sh +brew install gettext +brew link --force gettext +``` + +{{% /expand %}} diff --git a/website/static/files/tutorials/peer.md b/website/static/files/tutorials/peer.md new file mode 100644 index 000000000..5867fa33d --- /dev/null +++ b/website/static/files/tutorials/peer.md @@ -0,0 +1,64 @@ + +Add the remote peer to each cluster: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` +curl -s $TEST_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Peer +metadata: + name: server + namespace: clusterlink-system +spec: + gateways: + - host: "${SERVER_IP}" + port: 30443 +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` +curl -s $TEST_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Peer +metadata: + name: client + namespace: clusterlink-system +spec: + gateways: + - host: "${CLIENT_IP}" + port: 30443 +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} From dd010ab91b5754e26f94a0dae82c3ff0cbb37677 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Wed, 29 May 2024 08:44:12 +0300 Subject: [PATCH 28/53] website: Add multi-hop (relay) task (#617) Add a task on how to use ClusterLink as a relay. Signed-off-by: Kfir Toledo --- .../clusterlink/import-nginx-relay.yaml | 11 ++ .../testdata/clusterlink/peer-relay.yaml | 9 + demos/nginx/testdata/nginx-relay-job.yaml | 12 ++ .../content/en/docs/main/tasks/relay/index.md | 165 ++++++++++++++++++ .../en/docs/main/tasks/relay/nginx-relay.png | Bin 0 -> 112485 bytes .../en/docs/main/tutorials/nginx/index.md | 32 +--- .../files/tutorials/nginx/nginx-output.md | 32 ++++ 7 files changed, 230 insertions(+), 31 deletions(-) create mode 100644 demos/nginx/testdata/clusterlink/import-nginx-relay.yaml create mode 100644 demos/nginx/testdata/clusterlink/peer-relay.yaml create mode 100644 demos/nginx/testdata/nginx-relay-job.yaml create mode 100644 website/content/en/docs/main/tasks/relay/index.md create mode 100644 website/content/en/docs/main/tasks/relay/nginx-relay.png create mode 100644 website/static/files/tutorials/nginx/nginx-output.md diff --git a/demos/nginx/testdata/clusterlink/import-nginx-relay.yaml b/demos/nginx/testdata/clusterlink/import-nginx-relay.yaml new file mode 100644 index 000000000..c867ea92f --- /dev/null +++ b/demos/nginx/testdata/clusterlink/import-nginx-relay.yaml @@ -0,0 +1,11 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: nginx-relay + namespace: default +spec: + port: 80 + sources: + - exportName: nginx + exportNamespace: default + peer: server diff --git a/demos/nginx/testdata/clusterlink/peer-relay.yaml b/demos/nginx/testdata/clusterlink/peer-relay.yaml new file mode 100644 index 000000000..de56cd65e --- /dev/null +++ b/demos/nginx/testdata/clusterlink/peer-relay.yaml @@ -0,0 +1,9 @@ +apiVersion: clusterlink.net/v1alpha1 +kind: Peer +metadata: + name: relay + namespace: clusterlink-system +spec: + gateways: + - host: "${RELAY_IP}" + port: 30443 diff --git a/demos/nginx/testdata/nginx-relay-job.yaml b/demos/nginx/testdata/nginx-relay-job.yaml new file mode 100644 index 000000000..c157a7cbb --- /dev/null +++ b/demos/nginx/testdata/nginx-relay-job.yaml @@ -0,0 +1,12 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: curl-nginx-relay-homepage +spec: + template: + spec: + containers: + - name: curl + image: curlimages/curl:latest + command: ["curl", "http://nginx-relay.default.svc.cluster.local"] + restartPolicy: Never diff --git a/website/content/en/docs/main/tasks/relay/index.md b/website/content/en/docs/main/tasks/relay/index.md new file mode 100644 index 000000000..e7905bab4 --- /dev/null +++ b/website/content/en/docs/main/tasks/relay/index.md @@ -0,0 +1,165 @@ +--- +title: Relay Cluster +description: Running basic connectivity between nginx server and client through a relay cluster using ClusterLink. +--- + +This task involves creating multi-hop connectivity between a client and a server using relay clusters. +Multi-hop connectivity using a relay may be necessary for several reasons, such as: + +1. When the client needs to use an indirect connection due to network path limitations. +2. Using multiple relays allows for explicit selection between multiple network paths without impacting or changing the underlying routing information. +3. Using multiple relays provides failover for the network paths. + +In this task, we'll establish multi-hop connectivity across clusters using ClusterLink to access a remote nginx server. +In this case, the client will not access the service directly in the server cluster but will pass through a relay cluster. +The example uses three clusters: + +1) Client cluster - runs ClusterLink along with a client. +2) Relay cluster - runs ClusterLink and connects the services between the client and the server. +3) Server cluster - runs ClusterLink along with an nginx server. + +System illustration: + +drawing + +## Run basic nginx Tutorial + +This is an extension of the basic [nginx toturial][]. Please run it first and set up the nginx server and client cluster. + +## Create relay Cluster with ClusterLink + +1. Open third terminal for the relay cluster: + + *Relay cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=relay + ``` + +1. Setup `KUBECONFIG` the relay cluster: + + *Relay cluster*: + + ```sh + kubectl config use-context kind-relay + cp ~/.kube/config $PWD/config-relay + export KUBECONFIG=$PWD/config-relay + ``` + +1. Create peer certificates for the relay: + + *Relay cluster*: + + ```sh + clusterlink create peer-cert --name relay + ``` + + {{< notice note >}} + The relay cluster certificates should use the same Fabric CA files as the server and the client. + {{< /notice >}} + +1. Deploy ClusterLink on the relay cluster: + + *Relay cluster*: + + ```sh + clusterlink deploy peer --name relay --ingress=NodePort --ingress-port=30443 + ``` + +## Enable cross-cluster access using the relay + +1. Establish connectivity between the relay and the server by adding the server peer, importing the nginx service from the server, + and adding the allow policy. + + *Relay cluster*: + + ```sh + export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata + export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` + curl -s $TEST_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - + kubectl apply -f $TEST_FILES/clusterlink/import-nginx.yaml + kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml + ``` + +1. Establish connectivity between the relay and the client by adding the relay peer to the client cluster, + exporting the nginx service in the relay, and importing it into the client. + + *Relay cluster*: + + ```sh + kubectl apply -f $TEST_FILES/clusterlink/export-nginx.yaml + ``` + + *Client cluster*: + + ```sh + export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata + export RELAY_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' relay-control-plane` + curl -s $TEST_FILES/clusterlink/peer-relay.yaml | envsubst | kubectl apply -f - + kubectl apply -f $TEST_FILES/clusterlink/import-nginx-relay.yaml + ``` + +## Test service connectivity + +Test the connectivity between the clusters (through the relay) with a batch job of the ```curl``` command: + +*Client cluster*: + +```sh +kubectl apply -f $TEST_FILES/nginx-relay-job.yaml +``` + +Verify the job succeeded: + +```sh +kubectl logs jobs/curl-nginx-relay-homepage +``` + +{{% readfile file="/static/files/tutorials/nginx/nginx-output.md" %}} + +## Cleanup + +1. Delete the kind clusters: + *Client cluster*: + + ```sh + kind delete cluster --name=client + ``` + + *Server cluster*: + + ```sh + kind delete cluster --name=server + ``` + + ```sh + kind delete cluster --name=relay + ``` + +1. Remove the tutorial directory: + + ```sh + cd .. && rm -rf nginx-tutorial + ``` + +1. Unset the environment variables: + *Client cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Server cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Rekay cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + +[nginx toturial]: {{< relref "../../tutorials/nginx/_index.md" >}} diff --git a/website/content/en/docs/main/tasks/relay/nginx-relay.png b/website/content/en/docs/main/tasks/relay/nginx-relay.png new file mode 100644 index 0000000000000000000000000000000000000000..110b308c68360848869d4aa36b7b9a4739c2253f GIT binary patch literal 112485 zcmeFYd03NY+CCh+=uBOhX>D;~sa30piin6TN!l_lh#D0Y7f4hD27(|4h!CE{zD{9W z0A-amMOj2N5ilTxBwB=6wjhgPOJYR`A)64$L-J(#-NAOIGw;mt{q_C#9)}((gy+8R z^SZ9{I-m%}L3k|q@DL+u~tgRlJI*auGFk9nlSyT3y* zI&Ea&A0LH&yZ>7hsw!=v;mpV2-+#QY=L89b+WOS;d*1qUCx=idbE(f?zCCgYuP{<> zxauGYIsCJA-~3O+qi=4UKEI=KXI#yREuJFpmzn1r8jf=h)_!%dA$e)k9fypsHu*LF z{m%NspH}>`^UrIe%P2`rt9D*p>T@f7{k4*}YYyD}((Tg7X=|TH>-+B-t2zN*cdI@}*Dun*aOte+c{^ z0{@4={~_>y2>kzzKp5AKqEI8Dc zbseno2JQ3YIj00ox#@drJk78YJaaONGCMIgY85DHyqkNTRyFy<0(vuEVxYH$)}-*@ z(6jluaFnYVmc*+P7okv3*A$6(rdJyCqVqLOSIW>8qjCVPQd+rEL`_Z z)3ZX)P$wv?0=)FI3!NB6vm|SOH(qb!x2CKXny%4pDL`c5G3}FlTB#z|O3MxrI3jqq z9t1Qj7OjN?1JE-ZYoS|RirK$t)jU*-Ur6eLn%yo}GEbD)?ghYwV(R6PTNdY$B3XI=9!XWa+m7~fv@b*5 z*w~=!82iCQ)GY?Pd1~xZK*45qLwxAKf}y^H04lh~+}ymnF6zmX2A$}7<&fDykc>k0 zIM%#q?)!{K1_`mU^~RLq-dAATDMJhKll|D)uDdwX)warfD?GrkbSi4eUX25Iy5r1g z+(zV0$AUJ~aqWw^(%!@D!~tc~D8W3`c!9uq!qutYEL;e7vB8UNy&X9-@L4`o6==KG zCq1SGZI~*@ceN+ja~d0rG_n+nx;%jPyqoyAeWP;Lvi9QphT_p=W|Zli3xPWRW@1bd z9@1gfx!0@%D=T^wx2%O7m;psrFfZ|$BUj_RqvV9iF96UGI^& z@g{=Ek{La27*gkBW63kMI03 zhTrdkxt<+=M$JNX% z&L$suZ1=S|*vAmJjGL_X|*{9SehLj+B4_X)E4e%dVD|8kxsxy3B>uQQOV%YUL*zK*{A&o;gxwG*Nq~ zAVP2+$+Uv-+E)bVUCvRl+1c!!jd~i>`ruGV>Q>6M;mSHe=JIk|GC1|)H~87^a`QWo z8@NEc$UyjMxWA3usLExlk?R1-2{KKq+KqX)(#nLINj6wLn}(oI#sqL+1)>c*~O~2V)Wa4jTMp3IL-QIi#J+(J}Uv6i3HZJ7A z8nSv;=Y*bJ*&|Bty^#`bepH@m>IXSIS*<%%wv8^V;(WSFVh-)mfrC82b zy_pfomA=tyuZJCbaLBa#e+c?)LRd(blh>@z=4#qVftxIM^5)wcw^#3WNuID;ISGB<0 zrnV$-v-J`H{IzPgY-5$;Rv2%%V!*GBJIG*w9q0WBI8x09yoVd{dZpLe8)mPVKkwNl zpNHD$lY637ksgwlvY-a%@*!*FCfKLygd`?blbjc?zIt6gWOnYj!cK&(&?GpIOjD^u z_Qs0^t{`tt|Ghdi8>&$hhS`fopW%lTb-{x3{&-hht8d=AQ34Kke5!+)H~~Tn=~cUF zorKE(*B^P7Q&lT+!qhkNI4lKIxxo=2b1Pwd;%eP&s2Zl39I3J4Gx%-x0@!z9*xl`S;Am8!&=Cp5$nAkpK9 zgbsMOVe+)SC5rlB$tFq^`vWGkEk{HpYyM&wo&IPoZOjSmZa4O=(^CNWmZIuV(}2A$ z>q+w0R@(j$`g#9TU_INH%`+Lin<0!8HLZ;~%rOTMu9Iv~Pu5D?sS`uDe$4|Z$fy((|N8LPB!QHJxnakI8U>-`s`5>rk3QvHZ zt>%HYrf#|cd(kG6X*$*ZG|0L02H4k@@@i%_$5mM)=utciwvY55L54t#GVzfSpUXNTQ=#@Lxfpb1h&or_W zRl)TA6jg1AlZl5-G(HD`7)5%Tgl@E{HxpWkuT{d3y;(zlg50t)>`MF!B;xDLR;r#) z8~-etCOW1SCFgPK0>1~{{BikIR6ow;XtYs_lerbEkcua@GRR53!RkI?@6xLUtSfJf+!_rc^;Vs%z)}0NgJeXr-Rg4nH=7 zM@}^Z)w75F9i0H;DIozepqDJDn-p+WGPWC6y>Rj>*!ShN@goTD-RZRbh*I)vM1w#Q zumS@lROe1yiW^qDGTQV3>WQFrQD(^9JM9OQ%K&60A5Y(*sV1?#NXj`Rjfli&#-kk(UuCiFzn97YhYr^K?)Ix)eh5ybyTBX)cjDCJsH#$_}jEd1!#t#?9Bhelvj9jSUU;ov_Gk*P1RKeXPW-QDFb+N2vZw zBS@Ta<=H5cG46pS5&uYeS!r*+0z^2W&1JC0E`F$7P1RN~TXYsyfAyoHIjzy?5*;O8 zkHikt3}sQ4yIN~sjP@zb!<_yUtg(sSXwhnKt^gPEG6~c`HCzVGejf^t?jHR7B-nr} zh&x8pIB;)riflSrPzfITpnXqR+yEk2kDc*BKJS=T88=f@2P8T+$7$*W_T7l-AQ00im@q)=!8ab#O5?Hs z-VLx)uXs%{8P<(#{229A?OM~luE-7hxe9RBn!>VVMv?wih*r<+-gm6=Z3n#pk)1N1 zT?U>rz5%&7kl%v|%!Jqqk@6D^E=P#tf&)zMXI`38b)*@0C0+oZyI{}JRXJh$lY5E+ zKqqK?7+3)kSasftGyRN*9-)nQerJVEpuj4E{$jyh6l#{UpyqIHZUgDa2<>>g`D_Eg z9jydP#H6rr(z1}M1^5|;pk=uQ^4W%JY&oZ4Jq7ko8#d*}eU3e!1OBqF;ociu<1ZOx zb-aQ6=0i)NArBNy!A=K|i}KLHJ^-~0|Ek^7yerfc|9Y_+`L2L&d6?UF7r(sH73N)@ zk(MlbYsaNdxt=j4yl%EMNGEC+Z<)nE4$ne2D0;#Ko5A6=R~{hSZT z`idZN6Xs9gbH}e#Tjv;eC_5g)EXP`K4$cIS6w)RI34U=3Yb*U4g(_mU9|$YkP;<;P z&az^OT?6&3l<+s>oZ4PTmDP>3TUJzPKjTf0E}+B?)?KeotE<#K2ED7%J@A`JW0WrI z(j@rEqg}zc1Bm9sG%>d#kY#~*r)2ttpP9W4PUOpr>XFwjyp__mU`e%ihvG=cy=|vB z?-2T|q;4b!ly$1&m&$eKBYE}L)TUzG@LYpvv@C6-?WrTgKO&TK*y}0r>^ry~0CnR= zXyna=(&R4!MEYZ;qAe_Kvt2_26vn;?D5MYq&4zx!Mc_#_`9>f30Cl>KocPvPkY8!{ z*APaW!BR?p4tzfaN15XC>P@{?<|$DSj<%^Y3@rH#$12f4B-8Z!=?P+0U?!(gAywYT zL|O%+Cm#YfrLBWT>}Mwl*R3rHZ4h7KxGQ^t1>8&1+Easlr$BHc!2JHRb$PtB4?8)q z!+sv>!tvbF?i)u#m~o4MhtPf5t|4Ipu(-&uJM5L7%bRtK-VO=~mE?%T4K}q=@!ILQ zVEc052TZTb|I8dJTZx&fC8SvY0emJgaDd)}57a8F$#ssPWpK?@QNI`0D~Md`Aj- zkl$8>*UrSnV!0NZB$N9RHhp{@3%=XCff6&wY!izZufan?{axA;DS6R&vmZUp<fmj>ThSe(^r7pRpz4$U#L1DioAb>0UwB8UymS6>qGN4!2leM1Ybo#;FT zVnIL8cLtezG!AsI#_gSbjc+zRFVDo0mr9nZPSj>3WlrVm)9G`i> zz{hK*MU*J+r%o0``yVQLf|&=9xPhjjtN;+cjry|6d~D`eo;}rUJ+krrdV5{`bDIaN zEEq0GcG^U!QCZij^`zBZTXMazS-0 zi2(zAPJ`yiWD*Rm>Lh??fWZz^q@Awj)m0|)z??$G6uw&^*5zKZ#PSg=|*|uoi0q%RY8kx0qXKEtvC7#*(=ODJ+m!kkF!EgJ!dyPmPhnB zQsM`NZFUmIbZj73p{3;?m=n(816H~$%d%9MsF0(W45ml< zBTcRgplV9`8$Je1St~7=8ne)zN{OYaBm854{Ajj>J6C0+vCz*B7$DlXl^83vG+e>s z5es79fe+iqn-;}V;JfzvhdW4?GnM;kHIM6ZAa0-%hP&G6m(~})KC0G77?Qv@pI!)- z9H3kp5DXm+1en|ctDT1sM32r>mM6%v z&}gMPm>K175Cwwb-UUCk=1o;_w+;$&sw04xA4ynPIY+> z#9=q(aJYe!q#PlrK=i#!A_~@?2PzXlQ%x?18Wl2St7V~kAXl&hg@Ii2Cf3Eo^WZI7d>Y<}%9XK2HKILg0 zM=P6MjC%EUtv{28l;hw1#2)h>)nu9*FZHl7pQio5+JOBmpPm+o^&Aa#v|WfI9?_~* z2LQQ^@!UXJe<1`EK^rU#H2oArxItP{@EQF0cDi82W#`P?O3&o1fd{%kFwD)JI;w$W z8Dx7!3os_uj?j6ieI22HLOnV)AOw{y8f~LW4Dh+}&j}{P37Bp?(TC!9E`+~JP^UMw zzIkcL_PV$6OWABmLCr=-navMa#ZS;9;tBN>b;0G(EF+tpk%Q(aB83btdOd$cU^hAL5}stfvR+4yY$G#h&;OG1JsU> z>EEKBQiG$^s;OXJlnI&C1!DBit>SpIJDHazR87A;M-kIa-4mN-vy4IPyR)}`87_rm z-_V*rK@}YZ^^{B|YpR)U=0XP$soNkWE*&L@=!)DDzv|p^Au_M-(&zBwaiRi16zf>y zPquS(IjQmd3t*)ER)M0%Mo%ZSVLW5bVQ62Bx{Y%wJ2JWaBNXwdR$I*@ll9l->+H{) zJg4d{$vLXB#&e84lh08kM{lZ*z#sT4H3f~+3EG5Nydw$igpIa{)xDCDI3Ipf{T{XK zdG`_&fox0_c!fGSwQ|Ja z>w?_oYnI@8gWw*MEF|c!)1G*ONx)byxmNS})7!1+AA?K}i0)$g${!F%A2Tm0>=m6GngzpPrrW-+TmZENMVKx|G$bMG(F?9pPENay zg)T+~YxlaG$nDFMtOmiaR88F)a*LIgL?O@0M*=?HlI@@p>NR&Sm`mt0&q0|y2pkw; zv!MYvA6W0T7Lp|MUbC>eflFI2gT!`v7S}htahb;I-fY{W2R;e7b=h&KOAL!sQdC37 z%EvSvv?x#+9b&UgKWPpkAJYXfCzJmu?P|1H=dT(lBi7@(M-B}4++jLniKz9hi(VDB zwLW-du)75Ys@NRRrDdHyOe!pWv+v^z52ixB$_+IP;*)g?iRskN&p^#QiVIFttF*zA zC=+FB0GNlS1bCs?BrqTITK=?GZ~Eb7kga}UxoN8+!M5ShQSnOrT~RkF`}kn_D75GT zwG|^4y#zd95QB*#kCV)#&(?@ejN-VH8PSY0Cp+4l^N~cSRcP^ZOJ-_n<|mPOnMg_d zX;J-y_V5mhW%lX=HDO8?xy#O*yf$w6{-RZ}j^*MGfAx!L4|e1#9SN()Eqas)QbB5V zAFNip@ygrYn6J_r5v%J~UPcx`(m}a$V|gYcIH(=_V>djO{A~hW=IYa3b$l@AE40+d zEnP1$BFHN70WiG zCEy3M$AY7b)1;8NZf~$^K3VMdJx1ZP+W&dH@Nx5@}PU=(0Ib*~qN**`ASomcUnGi9cdt z-})##l0k<$_Z`bT@i5-XGSAmqTt#{39*It$#*hEx^|B6TL52gZc6;E57pUEWOK)8M zE_;GobOFQF%m8lTfhL3SetDz6O5-9{42em0>C?*o0 zgN_S~hgJDouf$5Lw^q8U&j7yYiv!n)zo>4+j0&1QSnWAOC(q=Zc^`klmt`XufXR9! zk<53ZnB4;Kl0>jSyZFjc`yGWC8tU8zBs=_OhH z-i^whNzD8m1YlO{d=m#G`H}%pxiuE)u8xytl1)G5{I*Zkkp1($Y0lBO>)f*02j5eN zokqD1F2~869cxZ%?7-u6e^LhT!8* zu)l}4n|_xA9FqVYA=;eNV1B(DJDoqd1{EZCIrud@^3#aUK+_3OUxDBV-fk;uOGn&s z*@Yve(hXIQaeQRi>qMZ?*p)d(j;?SMuwaNlc$?Ey1ygfFbJ*=;nu|v&bXN|Kn)cC= z9|L6t(hX>?rPiiSG#UQG_UF7n@~(vISgVCM?u`lVMmoZ)yW`UX;k5s?^;V2TH0i{Q zG&Rri-k5^ybxM#ARWlstM=#0sPFLhy>|d*+!rm=7^6KYmQsZ>M+H9@tlKqDT%<3F6 zIo2Fv1&ZZwdG&9xs!j(r^v3ah?j9;tb9ynbx0|ZRSU5dvX#{Pdi!~NnD4U~&Nqe_{ zR%}INUUo&h!gdIX=>*=Y;bCmS1LL;)}DU735 z3`2?vdr2^2di}{34u;aed2ICF*0+)sUDBfOoZnJ59!*HJw1Ux^BwanB74i#tqzX6| z!upjOfWk8+0C+7^kksKaS3i7ehqfq8htJ2#Pny)zu1|F`6QjR5^iMs}bP(K3h|3-! zfc0j;wJO&b#WueVrR&4J{`mBEH(@XO+=EiC=L^-T6^40PvDFG2C$#_7QpiKhM@cIc zqoP2LLfxT`0QGZ%4Q_AMY*H)GXzX8c4j#8}n4p-q7mnDJ^0S|m?<8P|x9Ldfuv_qa zd1r9&F`~H1(!x3bxMjUkQ!0}DO*)mZhMK&O51qZNsnQ5uZa4GQWb?CR>@C~}Te2~3 zA8&7cm)q&u8*&>xo%~GIV}H;gt=_9BkaqK-LAuiSREahaSt?AFEKqZ z$34TDP)DkY$CJ@;^BA#Jp4oXQ=&E`o4;JmxHN>R)4F!-II!gmqUoeA#pIzxj<>_pJ z4%kKv{#w?)9^-f02V}VeLi8}*RBcY&L){Lc8qO4JTf*} zPHJBjs2N+4Y)C`_zWX+k$}>F6k{YkXd~n%0b^+ac%x1V1J|N628OhjYIDZ3uFLM6d zX-;^ie0k89qS(cwhMgW|<25l63Qxk;x0LCZa6(y3dyo?0?vw5b7o1{nymDkVJ9VM_ zMt?V{WCqh_3#h_4M|pv4#pXm+&jdDoterO25~+3qN?|RK6x_lQ1`~B>Aak~tL|?X9PoGdr}G0e z9R8&x*Dn@9l1eoT7d9?`38Ve(NFKQKRGhZ3CV+4&wUVVq*SMRVnL%`EiM$KmY~E<= zmhC_vS}{5sBgV9n9?Mt6p?OH^z$ous_0U zt_2E$>uc`_oQGaL(o=7bH__$W0O$Ekr5xqPsICWP0UbydtW9mu&H?=+^NCT93}D~yxW?W(VJu%|ri zAP!3@3eS*J@Q%x?x{IQ83qo$qFA7XuzLeh;+O>~Q?kiIFkNz0f^?*hE@!_FVU+M5G zs4*yjrkqLDc`3I}*RS;LAResLa&lAmw_qe)(grQGz>2WI=wTqvVb@) zKlT2H&l8!PM3FWc736~?Xehfa5}^A=ddTuJVWZ8_)PYkyx;XccP(rKm%2$OhGD_NC z+xh)RH7kU}D+hLr>4xrsC8SYwMk=m*GLR*$jHj)E#4fT~NMC^53`~o9 zASBaQsU;Zol@$=p`g~=4YAuf|kZ*rmf9CeKd{^f8fDFTZHHY%_y9O@_VHTJ>UQzMgX!%Ou<|UA$ZU4$&+3{V(m0~BE!wrt=3(_58AwcrK?&H6L1(A zE(yvgZu3)WdZny_iBM;B%XDnesh@(Jk{Vk^2Yc#k&tth5tQ+6Jo!`GsDeq9*S}4@l zWp7^aZ9JeF_mHUl(xKoNT<&9J!A=kV!xZ|tR(?NA?F`qaV&KbhKGOZ&Ret#suI|%C zmGxNGtyJ}Ww4_Ti4=O5;F>HoZ>^>kPAW^^G^kCm)|Mg;Kw1>ZNjEIxYg;LTq?@A7jIXmW>43fl*GMeFA&^yV6kwX@Cdx+`YQPR zftkF*?jwT9VyT;LMlaXDyXxmIB$mYtmeB8F)VEH`IGNnY2*?iUrq*7MVK6#`T8^n$ z_|c3o8BQ?*@myEE(?P)7oiTkR)jG3`LKrCwN*t(_pN7{Hok8_Zm zJ)RQW3O5z=GgE!S>G?4cTVvwyLyT}H`5HX6t04&>S_vtnCdxhaOb$~`#a?zUfjBzO zY71|N?eeKv)I-FlZs02($`aWnBk+kgmTX;D!2yI&*v;h^aH%;li4XITEzLIYk+Q~6 z{}OfmA*69Bn9nQdQjFlJ2PSt)%0)Qc!W$2@@Y7y3`|JyOL1)iVZsdy@H4^yzNq5An z#`%^Owy9%xSn3DM?g)&9XHyZvP<8_D&RG2LFj+>yh9L>vSV=Ic{ZEkC8a$kv7XNoH z9BBcvwHLcmMu{w8;#*Gbw!WwPDbqC3aNS(zEiD0EcSFuAoJ#hZIhE-TM zZwVPSG)DplHZS&Mw^zN5>Dw5+(&8zciF*;GCOza&xcvAkz+acQ>dM6wZ$YpJct%_2 z?J|5pO9Gzw47P4-%OQk?=yBnT8OPXZ;4%5)X9`}ZD9z%XxvT8abq-M+&yjO7Yd>Cdg zY3b8Hd8?eVAc%o>4-_obi!<)kbB8%sixoE8>fiyvB<4USoOk4Fs3B zdW1_$b$u&+!h{6p_4*Rff@T>YX{OP5JUxcX7?j&i*~#uAMoTous&K|9bu+*1fjEvV zIelub>V0MczY0$tahO^DjiuzTe=GTzsU~?Q#n}3Ekd7mbivU?^bGIW_$xX-Ua+xMCu2i!YKvYUxHpGx}$2Bfl|EO-UlWg{U`Rs{t zs$kkRs8Z*itH($C)9Ip^L}8a)0@pnKUpaYSchu!Y1$lI>1i;92oz~Z>f0(IX*C*Ap z!XD7BYRSkD2U-sEz=qSfp!a1&P##_8g=m?5jF1k zA6RmELtPn3S*P;FpOFz;jxj0_sBR`+lEvIFbrkHROPT#JW;i?5)>r9z zq4D1v(geGlTzEDVuSZ`EqBlXUBLgP9j2Jh7VWh<{*4VNS6|rO5H;=mS6OeTRgliI! z)|4l90G(U!-omB-Fg@cF-w9I8(x9X@m0I`8dND?QcZ%gL9uYBd1eaPU4U$TLt4QVE`FPg(pzd=hk})rG51=q*W{pQ$cPl^i4S%DjZ50c-}-AoW%eOGuyld4&#z zOFUf#lH=j{WX5z}rrF$8uSJJKMF`l7QkNuVPiPuTN9sz0(jkGQcB-GPW~qdr1FZ@# z5`uP0VoY5Mr*Xpxni$3LaDs1Y&cD&c!5|T74@_@K=MuIG(N{6>&-w|xH(l-ec4nE} zPSh`Sm+JKQA&E_;SLd`5c1U9*hF^D44sv~pGZ%+*5?7@kqIsW)+LLXt0#AU6OE|~) z9xz?`m91xXvD)+U`7lA9;lp)DJ(pUSpS-X{=CY$~E)0WwN$5ef%aKiV%4|NYgY>?@Ae9t6FMiT8*QoiCuS#h)?rleFG>%;#V5=)1|Y1^#=qm=R8RY)eE z~5>>@s-77fDTKsBsDIFddoP)L${Xu(Z*Xw3Z{ zI_?4g$XzWYglBxDd$b&Dx9;d5A4yu+zX{?CH!yJg5(nx1fe~V16Z)z%oySl?cQEjk zaJBRKN{9;=vIVZ%cxEp~;Hv(?j~n7w0X@7n2rdPfBCgN@f_u*q)Ggq#vS-%t#`$3(BsdQf@@z^wki=k(DUmwv5RitsRm7ws{BZK!K z)sn!dj(B~ebz1a&coH(9AHU>iFLFCriy^^0n{@5W7@TSu%1f88?by1H8Sj4Irorlv zcJS=Ds-QEspvjW_Ihp*7RP~0m@3O(=43Ja_O{+D)SFR> zhOY;?kjqNhV6Z%syw9VFLdOb4;;Ti>X_zWthqxmU_?Py zAcEahD_H{mO_)Cx?+$oL*~=X0Z-vM4J)6C$m<9n!pO>O!4wZZD{x!W_Jo)8U7V5wJYRUJq{x!Cg z7ssz$qDwE_)lR}E_FjugeR~8b5Pg8g*|%j3oj4w=rV0YXX|H;!elC^|mTE&aT{Eu= zq_?HKG(S#Sk1jn%v5I7}+i+m!wNpNw$M42sKq%@;$G<}?JFcc`U#_qUuJ zBnmQ8ZHFAhe0T;v_!1`n`DwpJ+fjjN{in9$KhO_fanG@g8k^n6PNp5Aq>XgV06NCN zyFpj=h}$4T)|Ef8I)V&-Zc&<=#ZW|$=#*EpfcCJ;u4k2>WHkd)_EZgZJ10-;M1nb{ zc=)@~ISB)=(66YzLT{U6=k2NWG@&$hf|Pb&s}qwu2Z?f0V4y59Fu9X9A&iKS>C^oJ zNP4c1l2`;rH?eT#2!tc47^egoU<4{0|Q!ytUslys0ZrSH7r5%NsS>vdxf_% z`MIg~eiq1Bmf2HXJaw{66AQ{1!*fiF=LyG2n`E;qq*In8)8^`D=8f;AaDjbYv2CgH z;;_#q2Hg(31LHo6}>i1&2n+2GjkF5vA(FUOWyYSHoXZ zSdZa&p0(fRB!F2t2`JW+Ac8@lY`fi22R2#s+WLQ)Y72(DkiD`1ACKB&0E63bKr+Uf zf1e{EjaO;wqmF2f^Bd~6mx1;HsBk5c5H9f7aB(e-b+=Oog2!iG?PxZ34+=+b(|;IL zAF_+`psu#9X>%5~J+=&CA-@~KPAR!!eKt zlw%=N-WA(9nU_*3V3m*Xv$;`+8J@P1tknJCVxdobREZ|GFCfYzGKa*&6?A5m1QAQY zSG>|>PXfawIjErDuv@0A&ip=1vGZP>^D+`@KmXNzjyOiq(l0oZ-kH_AIi~!0TnjJ{ zk~gy;3v#9h*RCfDqzx3R_K?EHnCT#AW1f^%GQ|W-2=dS!UsKhKG3C)~oJ+s~8?CNeMy}Z9 zat^+?h&Z&A=}VU^0EzcL>`F~;swjJI-4D95el6S`gQamd=_l!HH(OdrqTPxW$fA+* zn~dlng>;LLbZ@wIgQa9ozL_ag7Y_4F(YF4T1+?*?@jN~BQfjb#as*7SsFrL?tG4{- zk$<-819yg9+{f_E?v^&;Jl7`rFmzYi`eK1G9a4WksOb&px!uwzaZ9RFfyB${nNeX`|mTt1mxG`gJNwJIp`Tz8#%8sbMy#b}Y?xIo;z-wBu zIR0AMY;m`a?8A@ph)G%LYxpmD)!T(EmByHV+{0hP zf*&t&5a8VX52 zQa5tqc+kKy#CdIyaZ%bYT@+8-i~@aV4%Jedk^ijC9Y1t%cw@qz^{u*Y*r8JfG>deN zE%pdq%~sblIQ*D#{4qM^=v?#}!iYt(6&c>@p(;yiSy*QPE)th($!+~YwxC=68NEjk z_dbh%pG~9vXJR)7IYqOsKP%=-qz!Z6|Dz)PoIHa(n@Pj|da;l4e=<||XIDQK7BO48 z_}gZXe)X!$VWq%CLW4I{erFQJh!MttbvxM^yaM;tJb4k5>Z z8@E?@(?)Yr`%)UZYEEek!ruw|;V+N^^-bXRuPG#MP8JWn5A7n0;Smd7x%tiTaLA&3 z3aJ(=tD|<~2-qBN_*14s$=ueqg#+%Ts^yrVfvn_V_Be(1<~=?o0`#2}Ka4dS4>K@uS{!&!hzN{^(0wwG z^i|B!KS6l&`>78D`wgE`kUCdD1h#{npTCDNUU8sAcB$S5EXC9 zh}!=Th>Y{HNGr1W)Fl})jFPxw&u05;%X=J(M!4PZfpF`Rn273Q#_q3vOQ3V9biza_kPM>VNrv)^X=lgj4vzdXQW>+bt(fx1 zAjvv;&<%AVx3+Su=qR1&ZlM7ZH-rz=f=z-Xb?t8~#E{q~_nH_Jh(RPpo=c~JLN^7a4Li!` zWN3xEzq1x9KIzBVB?=;{D`L|;e*hkwk{JIWs7P9(O`OArTX!50R~$l?l~eca1PRnV za~_Z`=%MfL4}Qnf*{5G{ngS^a`oa1V`OtPPRE^PlNk@rnz7S98FlJ%|O}$G3)dkFf zs&3b>T>)2H0?k}!n_mKxQU;MH-gF*viT*?&XzLBf=fnj3zQ88fjZSASOt@rPtbhfu zx3}(a(rBLWum5OZ6*+}fg>T$djsWv9^bt_BrJDgbLfOq26N9x;>B{t>6Zi8!Y+v=i zJO7ai<2#)C;^z=zjwq!EaRzOiU;GL%L=GYkw zHx=qlUyM?E#X0D63i{7LIL9#j90g%;s4FOo|A-PWu&g7-XN<0 zTyfv-ec#g@vb8A}Q}~e_J9e$_lVfp~>9FCS)8T2KmM2d6-0dfRDw0Ss2R%&3 z`HALf*FYAyZKc}jca#&ApyGt{ij_YS&SY!di{z%EKaqGL{5`_ki#Tb6 za}63 z3})0{3|%9&f57?!f?{_S4sGq0Cg4~%e^U|_PzrnOVrL6jvy<5w^Q{i zN`xzG4GL}i;_KdNGfUHeB~jiW7~b8$IQK_kYvjKv?~Oim$`@=EtpLdGDs{%HFxHl+#xQCy%36hj(LtOznDml#hn`9G&Z@QBRc?ep1v6MLq zm+vi0Oqr+knX9;ZjJLR@YgOVRi!Sok6`yI`!(U6n(3%~-_yzPgZT*hA;F3MU?B2$@ zmYZDi*)SToFIm4NuniaoAoEvHt>)dBo9O;^C^;gdINcL02`nMSl#Zm=tHD^(knRl! z&&?uqP-fn$dBAuatrBu^OVBvK2@M|8OkoOUER!Y}PN?r3RGVInlWcc-@LL@0dU}~{ zw?T!GOg5zB7Po8vKiw|hZfQ|7gYtkQ5mYr6Y%oIVHK{v* zCV>2_-x4v3F6JyrYAK@h_vi$eg>~nb-qnQwqIkK z(e8uW2mV6}9VXGM91BBp@(RrX79rV?4hc_Ea3L>ql1fKg(P`x2Q3-8~6jG<0xp7A$D`Hxck?4Wxn;gFOOS(eq50 z6+Xkk9ienU>bnJP9L?Is5bv3Kt)G`bhJ4KuTCT*sYJpJ{eND;`5OY%{ zFEU~hx5f-CKcz&9cZ{LIgznBSZZ1R677-|-@t-{x5I)Z%cm}tdy#K}7l4MDp-K|VdD?WmDn6~8R_m-a zR$!!+^=VID!tsvX@hxDePoJAMleqSJFB-p5+H#ZOAk&zBnf|w_WZ7Ag)n65X@g(sI zHWzjmWsH*uj<(qBX=D0D4vTq6k3o8i>JkGzOh>Y(@)iSue|N{2G)R+BMyq{kq_TTW zMLv8Lu+U2d>e4&fgtRO!`5#Mnj)t=L2(t<%T<5>HTyF1_jGn!_2nyQxMzo3~xx8Kb zsoU7)A?e_l&3C`*O#!mbmfo%#j+AS>ti7q(wGv*k`0kK|NFS6kwV~zrpa`h59vDF7Q@5ri@_2P4c)U2xm`KWCt`k92UftCP0bLiPK9=E@&%%;>kQ3TSgO zR|VlKy~n7VuEzwj7sYorFlV?(I#e`dY0=1o_e*XnBu0Srk@cSkI{h{r1>VPz&HDKW zGdBJbnk)J@SjQbjZUbbed&laddeCZ&%x0xb4#r}7Yd`#R(gexYwX9Kcsy#j}#DO_* zQ076-1=Fuq2(DlfV4dw@#IsVZE`4)EEdV+`MgjCXIOL2j*S(Ka=i5+rP9^{7mE&CD zZinV`7*)w_P@Avi-8ypL=0CC)9tDG$b0l|>^RMYybOku^vRU^M;!wFZw1WL0s)Q-p zL!kG9dm{Ky^f=3|`jPcqa`I0CtL#Z&_oQogeO$fZzc>r`d&Qv&@J@Nj7zMm0pDPE- zp0RF#Ob8Kj9~@R)EC-g*iVxrG{`L)veWA=}_ZMAeCq12_=P=#36{t@skd2>M+Uw3U64&!` z_9Yva8<9ow4id1w-*aWPoRbb}-9fMu9v)J2FcNqfB(d7*{x!#`mLLyo;79lp=%5Wx z$J?LJ4CVsS0RV8o6#7O?3rt&<$@M zO%%b;w3IJ7yRQw(MSCRWX%{0pVq;^OK{>_DmF2hob+Z445R8qacGw)@G8AKMtww3f zWqNxi=!!s!jWw)0gamvcn{_T#yC3nOrEt~B-s&q*EAer>sI?xvf=3XtN`w)ApSs<{ zvO8=?P9Wrfy~=CVu6qe|g(trj!|d3V5a8WeqwJcA#9x_e?8DSD28C5)P$59EzzI#r zhrdZkTZ)KR|J)@lT`xQsf#>H*Pv523vTaDvuY&ycQRQh57UV4n*#ZDi-?xnr^!jrW4crO^Wf%h2!HI|Q(ilTUDRF2Vqw>b~1VU7DZvnYo8X-|<6-&y!k|3$b z1l7qYnPxBBAzk|{pZ@tb=r*Ke^8A%#zTWRJU*gQO;a z<`H_RlWlA$B`rnpaLQUvh<7Qmu$3!Z3qjRto1r8wtp%KxtuFRfOBk56nz2SUHEnT~ zJoK;QL{h8e$~b>Io`Y)1v^bqx*wB^!3dO0vAOQq0pT%7kllKP(Eg6OS7PbAVGCYT^ zgSc+E`i}~xj5oPv0BebWl#z;Mc?Xjm7fPdr^Lh)k8_WkwY959Q6E7%m`ONV6B&IB=kD z{GwXc*GQicr}fu4B)jlcfSjN?9K%#$f@Zw(`)8;}hAAjVJx%gIO6Ldq#b`jlv$3F5 z?FjHY6|V->9}Xx%ZwHh{DIa}RyimT{eL13kr%3NHYS{?nihPD+?DttxbW(`MMQku% zu%)pLb%h{t_QA@g|-OP`jhl+RO=#&%x3l*YcZ8tPx)v_uMC-w%6pt_MAd`X+mXRw2yj>W8#$0PM}S7Mh7))YX(KJsz?X_L zyP2+*e4R=jZA9-eEF97~P5zLK4J>G~WiYumktu`R$qex?y@sXV382yp=$G$1W4KV7 z(d{V9O8K63rq5r+5$c3@52`O!j~EVi@B9LEEgr5CtNRcQ9EtzP8v}R}LwMpx7}5FX zPmSbofy7IEI!qZ6(a&Vna`7iG`hssJ(j@T)f^@`nFSj566spUZmNmnT=%~t#6w(x> z)d9svx{(kWT6LRF=O{- z-ak`m2dT{jMtv2Ze_<1tZ87VJGk7?!wY}st3e*K4X*H%_iiP@ui;H}BbfzMLt)jOf z$3PpE<^upk8QHl}0I)sf4UTbncnk(br+`W>co1X)BwWd9&s;1oCqf(a0r}Ui1&(wv zJL0+ny(*kq`7MFi^w)&euG7ZQ$$oCz=7n8i4Jz2O{=hKRNHDg!v*TqM^d`%+<6owX ze|^=_xNTw5b=T8o9h{FP;}a(6I|;N}M#>5V$Di@f9ZwlAM^$wSZg#M2ReTOWL&V?= z*=s|@&Dfd;{xqB%eV`Td9fWG}zMg5@5G~3zZfFW1etD50iJwuDz|xFMw?|;@$+9iAYW=nh@FYV_n|Cd;L=BrU)CTF1jo7u|@QfeO&9z z+I2Su#ePsn-U}{@`8L*}g`boY0i;!n<$r+~|D+VHjhp0wD5^;mzq}1Ua&l7iL;Z z1gYNT;(q+#IPyu5X`&&PJ>%IouL^W`jl3d}lx^0-zG2}{6*^28Xi2R@tVCiT=y}#^ zz(t!Cv!Um`oL^syQ+?pG6773aeOF4Rk7y-f6wA;xR3G(B>dsD6e{#Q3?SXiDE%1tn z(S8&SjI^a&U=F1@a$!&V`BqsztQF9>KBAMYPXY5~qerC$MgwA-i(pEQA?Xa?ZMoHA zb}k5ve~09U0b}7tC^BLdf^s3^ z1!m!YwqR3yQ#6&%R&$4^_I);b872Ou6AtGnx;{Y_V4X(HEM`4d8a|cSL~7c0H~)j8 z@XX*A(b%wAc}HBB5Opw##IF`)vd=@ahn}OCBS}X`HGD{{&gLGm#na0{>)El-HqNx> zjRCH7UYK}wA5is7T6SxOW}2uPEvAbin)s5PrnF&>M4*8l=AdqlTMSu~M}vB%51f34 zs%chG$FbeD#xk$$Nig)3Kgb!3MzA;=9b@w&i0NkOVG)X1qM0d+yqyuR&)4gMG$w$!5Q&}!JdQ+&h|4p8ZG4SBE_=E0;iLX zPpm`8ekM4kD2NkfHF2NzoD<8k#_==`)bkDX9@hqwl$_>CFCVcbFg|gtS`j;G(3(V* z^omFN6O`9I`44Fy-5Oe$!vQZT_;9~WY6hJ~@q|vJEFDrCRdX_iOI@I>LglM`;N{7e zK*#XqK@o6jUKa_87W}8?YfyBBQ^1=Za;l(7>SZ^zSIG@7fY$KRud3eecRAX!9is-+ z684hIM?f^o>iV;-3LcbY0Oi8qKWl91 z)sM>e{S$8BM_t)ahzkCr``sXeNt*_n@!JUvEHCLs3N|D4G}UfFMU#22g%gMiGuq&f zYZ4X;MBpU>AAo2|u7Ly;k6$<^@~BlPY5Gm!WN_Nsdbvt3l*SUEgSIb6*T64k76KS> zlNA{vp(r8&EN5UpbpGLt;E3YKAn?%Q`SyXxRVpP=co|dwXU84rWAp%2)dW9YyM1ze zk~;@AB{QL>qzbo65{}DCTmHxxT5jlnyWE6KOmC2Y?NvOl{(Y~7iaFHfCu9Fhm6{_* z4Q`skr(8ObdG=hD7mr5&w~w}lB()akCC>rUD(%13YAH;4q<=x7FMjyozy2YI(mI$s zBK-CADiw!q5Ef4Sw^(qJtf)3 z_krR2!0>%w_&zZFdjmry_=M;`LbDpG)ll2?A9$v^rn*1e``K}0eQ)i&bKYmX6TST$ zK3ntcKX07*`I9YA{*m$djqoKa&1DuwZTD{Js@87_-So?+J`X)j?wm%{-TX4V%VcBr z&Ffzeqq{va1rAtbjEidw*+nQ4XVb(!mw98=r7hCON@qHDoywl=Rg7`j+!x^A*)8<# zWm!v9j)@P49LWxWF2!GQ)-JXLUn5poPj2XLngd^2J@e`QaiJ7O`uXRde?PX*WPiCY z_$Uz3IhXF63tjTsO1CRZ17Gn{=`g4g=qbRbWRCsR6GtSa8-V-mayy)QF>r&5O0)sp zFKS>bxKx+DPk5-j5d7fG^nvk&WN0R9{j~-!9G{&GZm62&Tv&7zSlsY|Mt21M7jS99 z_b@7BY&-b7WAi>8BfX z%l7PR1fMq~WCzx_1g{2{2D*x*+v$hEXWTkmQR6oB2(Q?ns0Zs1m?MZ~w>J0|By_v7 zqT@ym3UrqD)YiX+K^!9#$kakBy5pxB{prV58g>f&uh{j@xA1$W>)z%zR-~sUG0t4n zV-tV+F_lW(y;5H+CqcykPdnaauW1rQh=kvdcP`!$Mt3OK*n* zDW!aO6vq%x@Ba4NZ*RWr~vYn03ux;n4S7 z!E{;Yaky-KO|`e7ua@}fAijA;hvmd2FOgb>a{TFv8eUPd12&Pk?#>=PmFSB9`CfGfu~bcex>LIy zak}gH*|jQmh4M&; z39mID`R#3}q~1oXJrsRJ&t)FD4f#oKor?KUUzmd+@(Q`P6|GS;eRLjt&F$7di-mD0 zZ6qL%?`w)SZlUi26zf$^9jmj4E*V@JD|EI4-+sgk3gRc-8y`ZJcHDNYO51k9%o9%p zhi{D2UY_iur~4KW#pxS3XE}mvsS0Tz0^x@hy1w(X|2ry(!rq?|g8~%@^&JSC18Rda&nVQORuGw_92=>6x^C>_)nxFS}!_u1ewpf6;R4Uo6f; z!{7TYjP=$AoPr`QaIsyZoxrX4Sq*k7ePo4Q!=2oPEsaHY_L#p-BxcS|CW%7)>EERT zPNyu>Rq1#fc8|+zZzkO11{7fH+6~vfjZNiEe~4=pR&E@7Y=8<0CVg;Bqtlvb9+NGx zrV5C~M48y6XKWTo(0SZwH@OZy-9simB#pp{>Pyr*_9PGx=4 zE=PmzVPBY|U_L`Ht+wWEu$$s+TigLQw}r}r^k=;1ZMdR)uZeSD%Lm6id*<^w5}gA+ z4R8DClJ}2(FYy1rWbQr8-W%6{RatlhvDyEtlybR9hP&YxgE&f^YAwncj*msJJm{Q5 zhP+ik%3C?*UjuJEjQsG@DP<;oBE+%E&#b_e>X0A*D6!eYpxjR!aDMdkUO`L_w^TRv z!P`WGMu%5s=2Rd1mzMP*>QqZnPQEocXE@T!gz&h&d_dSZqcl)XJoOC7Kt1Z5*ZS(y z0EGi^XvKza6(sH{zjnBHO$zguz-XE;eWGOBk@`zqtrI!N)(x90V@h+6)8Hra^q3vu zpTZBk!yX!@$gMfU(TBj=hXSL=T8C6}^K!@}#S@j{#AtE=2|bS{vG?_<_r+~(j474 z7QRi6bB0j?CRS0mzgZnD8LuvT5T#kXK{Ojr9iDCo6y2IP>hW*; zx#%5s@iG7O9uvaoZ-LP>LrcqQqgEH!;^&L|bM*mE;WqrGDc`;WKF_xB&j&^`cQ2=w zgF}NQd-_5wO_X^v?qRWqO$g!*vDjnB-T^_XPj0WKPoyPR4BVYz9OZ`oGqr+n7*>(s z7jT{^*P%}+b_!s>efs2J0&usCtwUbezc z_kRz{e%vfb>nb16bF@4GvoSfgdg6M|1rHGyvWF%hdMk}<#+ZfhfBjp9|C{OGmGM4~ zo>J^A9|&=To%m4?^NTR?{sqfrrON#i&GZSqbuRTXeVWe?n=iZrIJ>P+0&s?hn^?u7 zO$g1Q#`g9%d=77n>LL~7#g99wbzJCJX7w)U7v@I(4U^#SAdQ$_Ef-Trb}!69mBY|l4F0RcbRrTs?C$h)C2shg@34=?;h0zx!sPX{Tl;6$c{{Z|gKhUQpje+Q}eYUL4RJ;0m)c$WN+cy@Dh^Ce+s;**}w;@J}5 z3Q9GNHA?v^f$t#sdn4iQz#r~gzOts&`1aJjyOLVHP*P3Kuo0ZYBb;ge--UDDCR(|M zK7j{c15}5PKQhS9A|K1iF&`*U3S5uVs{ER~oT*a;YmC`Mf7Hf<*@D| zvb)KJtNk(l6Z!A5{&4;%a8Zuser8!6UCj?K-#zKjO}_Mmv!Y_aCVP~Bf%xnH?zF=1 z*_`wRwBV$cb1W=}u*vxNF0E0k9hh2iUvl|?-1R9x!^hx%m(G*RanHXsAqXylU~2yC zvMzO7_7=7?sUnxl8Yq**i1Eo>)@Y{0iqpUUe~)X;6Q^S1ObC)+=o1f{mt%a5O+0s@ zM+sIgImZn^JW6iJu~|6Hno=?y}srwE=?ZE+H0^;K3 z_BEyITWD1K3=M6wdt`Rcb5D}MO267qt(*44>Cmry-+?Ap3)B3a>pk=<{k8$qg_t6r z6Hn&-7$_2z(r6#q*HZTV?{RbcogvI_;27`HC!WSG$E-9p*-~qv#?PH1zLaqjn{&C`JE;+2sLpTNA}y0Hr@RA#evO9= z0b=Uj50_&;oZVwzqkCv$(SgF*yMl^={A`UpD_-95*>{Le_DqaG3Mc@6%wwas)6y?> zEiK^4^SK3X)DgvOLa5=pIDyTPP+l!6)iA!DMV`H~QR|w$Ap55<)F~4JeYaQ{0X zXYu8F3y0pJKCr$Yn@>1cxVHd6M>SQH$z zlYf#Sol(aE=3AwP6ta$}h2n}4wM3McCU995BhvyFMrz(Fmh-F9?|m}@S7OMr0v3+M z&&IUIiO8ozgtyJQ$^04D=w8{%GUGc1d6+OUfR4+ucX029XPS_4_kVILTGqbFQ_w%= z6H07|>vbf_BUf`h7Td8dz|Q(yiTH`y1@X>xKKq_c`tb#Oxe;ac$&-+ z_+BwnqZQto&fuJV7}>5e0=w?lAEfO8&_b?}o@2=~TYMv|ByzZ;L+ENH zZ+W>|%j0>T*&Q!{p>fQ-p74fgmz?~(dW@E*n8KbEaaF%Dnt1lg{XV3-rPn8u&x0vM#X-l3} z-z)Tm*#|itllJgoCJ4hbwZ?5;g3Dv&nD~(4>&*mDnUmNBSlYIOO`Al66c=m)m+MNF z*pZ#mPt-#OSD`PTa?OJ&rUhv^e*F*rAc?a(Ra7E5V3|T8>UR^O>uc?NjWAaflStoU z0E_KwR{1){S4&S2kmG-?A~JN5G*yCW0>ux?9}LXXdiXZb;tav!VjP4R)=a|~T-j|v z*UKSLUErq>!}Asw$%!p#71{`*h!R?ESC`)^=PG=MHLQPRU)vBJ`O5dtJ^}-ZRn|{G zu4#_aK4s58^xAM2g&{!7`^FB@tYy|J$56g7Jwaej*tO(Huh*89o#{-W(@)zK&%by?9p5G&U5Y`;2fzfk) zQ?*BCF6_J|suWxZ8YI0o2borSxC^`y?+o?C&&9Y%*rEm9tzgaC(24x{U&ibdasy|$ zOhEql>#BRnzGz*+_c?HNTtb^oc>c!+ouv7tB|DdWmFU-Bq3t4X)3$MG^3$?M3)0c0 zalfIEr=$i|9jJ|kLE1?9s0a3td{BGd&*uDOk@0e@>=vkenxs>)@fpT#X%4T#O)XC_ z1FECe{K3SY>?F_L=+|b2PR@G@L&Vkd+FdPUOfsQG7SBS+?-MX2DGw{locnyuEt1X) zn>tNzLb>L_lXsD`Voj;!n*Em8MZ+F#V4NbxnLmFU`-{*@Ma_lNwJWSdGFyf`W$Rh` zL2m|{TgE%TB;~0=8;Q4#C(jwyv7H6%m+=7}clCEwQMw_+4ZMK@fw zyp)_Y*KD`%O(lE*HdGmwtJYjHIktG-_3>Qu)AnsN=~OZ;zTKmOpTKZzm##mPv;*R; z-h`u^0b}KFJ9wVue~w3Oq@n1;wGPXVHSWTc#0pp7TJ)-2e&FpRPeY`z=YguMe=J8S z9%(TVoDdkGd6aTBc+>+Pxn-j`*mFCZbG+@gy~-Tb zs#R1ziT!=ev4~vjr#awTH;ZOr6WODD(wL~8B!05OE7rgA>=L&sh!gY#AvsCjS0c3* z!DzRnE3wRU`PE!s%qCG31NI4{;#NZuD<-hzxJvwps>dk-4ec*n0tH8F0+b#9Hqu@C zM^Fb@dtSS3wF;BY-NcxczD7#sFfYvTS^DNSu_6|C+M^P+75(eW73i!ip((h^lNu|pb)87`Re9wR z<4%64awh3HFyYVyh8QOX1Re4^^J#DMiWa!DBrYrwqL%aFF&kePBG0F*HJ9VFcP$Nj z{ngAQMi!x%b{9T?%-St|X-&Z{tbTW`W87_FIPv3yT~ERYd+l9Szc!js-VK|7z<1`3 zD6a=b3w>tLtL)#@jP*}Ro=c+r^OHQ+AzdFX9@K1`KBVP$biSLf7>S|8t$Vp+;U_xd zZq}Ide@wah^=nJS;-zP4vn> z6G!d+Yt>kdTLaa!ys4T9(xb6@l0Z97O#5>kdVjEUO)sFVvug3?zQhvtI+LY&q2!W0mq;m?Z& zD2;ClZ}_<_`=fURLbo$j(jG=DUfOU#F~(EoDs0%m!_BO(l=kqQt;`Ur%rQt@)Q}y*|=@jnOA9?&CL%~ zf~h;nqb*}b2J!tozy29a7sdhSGxRq~ovE{sI%vlVoGJzy(`i2DAODBkp9mSGLQB3t zJHF*p{2@hzgv%4AxR$H+w%e;n_X&a-lACXkAHbRATj0!!+oznfp<{_?bY->GwsarH zf5@S|7#TTWyU|-6ys4C79dvNQfh z^*7FJowqPBLgV?3FJV zz-r}sX^VbyOjWkFl7h2G(x>+dy<2X6^ff%7VA4|iwIcv+)4!4g&fEna>P=AHXNh^` zh(YS^{1YqU}F7+-!N$DtSIi&+K@|c(obXl>AUsE3&*#S z9$nHtd@NI$=jDobtSd?&SSXHQWotaCR{MVq0c=U|+(#lyp zNHn7a;{5XJ*ykdN#?04w9%mjCXU9pKOz~^Aqp7=Vf8U|BCX%-Evtdcz#eau{OSwO6 zxMdicDCd%A3+WSs6jz!o8&@$RQ;7SNk!F&7@jP1i6f@H^Y?=|cD7Xui-_A5+MZ^fR zv~L=k5|%`UkE;!RN0eyBbXY7~-|Z51vo`MGs9L&xm9{Wd@PXN9ZmBc;-sn+zr>q2! z_}{{hJojVEPNw8~E87i}Cxkzk<-svZiZjHPR5$-USol&e< zk6^#0{dZQfB7*WF&J+qNai`=Oa? zb#~rTjns8$D&tNE-H|ZsQ=w)feCX!fsKd;h?3HEF4#;jl*#CEMi33f!>kKL|QLh?( zFzWVF55;DM!YV$x3yu5A*ohvPJaE?$h#&Wu|+os}HD{hEviJPN?3y z%h|04aK)8A2D`dh?KbBY2q2gFGv7-^aij$Flk9h0J;JjFlcQPn6 z*@$IcuJXI9a`vlnV@3q=^h-)!7oPEb`J&9@X5(_#!>iSAnI;JA)$YVD@6TtTDG2)N z0?F_FHBv)dtfy{Cp9>&pjAUp=5c~ph4Q-WKW#z zB@@0z6GKs4XKiyO8DfAkB#|)xTseoSNN*m!*h{VSpQKnWHYby|4+={+tWE3VCP*TD z&W>%f;fFzer91yFZ<5F|#r9ZN(dJ2Q!rzhxXksO!&nAP6g2N}-73nu6I&`tJlQ6Cq zolQVbA&+P$*bZbz%#b3lEmHH14QtoiP-{*`7AWbspHs`Qr$#}4Vyf@J@lPAeW=Ny* zUT&6rpX_ev-{ikg8*ahO+SY=C%p4O|Dm5U>4+xJCy5Mo*kmVT7Xu6_uar*meb;e#I zX;|9Ivi2kl%pQl=%MP0lmb7u{;Hi`^Y{ACNNd)i_v_|FS0#iFV(YL0$^p{fqtXNesJ_wx znd?`WNjA+(y%SJH6GgTm~QQ+IjG7&i@mS;WzWes0HRjoW^kvgWQqu2px0OCpqU z!XIJ~Ym-Y5JFy-x_@wIq%Xk%j=id`nupqg?9yX@qU{6Z6xw6DN;a=pfsZ%r>1}VH< zD(2_48YKS};MfO{g2DciN9QHu*nh1kDLf3V?2yPRHED9+-{as!yHuz({<(bGHlmR4 ztekFAddM9JUFzs3X=ll+n7-PJ(@T>!hkNVLqoZ^dOzhOTva&t{Vj2A6BUEd0d$3rPazM97Wy2&W}Hz64}fKTeeNa8N7l zJm7-39-7+7+)gx^3A2zAe1|T+mK1VpmaR&t-G6p91|Yk zL7H=MZBOiRM0HSHv`=;Q9{<63t*^2ZT_QJN?`LcFgwG=-`~NIFNu&p%Gs5>5jty&= zuVR$^b?s^P?qk3Hj-s45_GB@n%Dg;CyauWgv>C{JC#z8j!gUTd&2d}Xm#fVv} zq3Ol7n8!@)`sBh|9oZETc{}?6TkE&%)+g%QFx{_32VAA*=0B}rRYuTKL7NCuy{3qf zf_Nk8vOqLIn4CQjm>6>Kx{fJ$tI8jh&5277!Hzm#>}VR=(eOR-w09Hs26_x5|F?XK z(A6ZrgMh+@aPsSX6kU_NwSPAv$KIr^4rkm}_e;AZxO3%s6lyXz6xS`8a7?!^!b~@& zr=NS^mMOPqZ)fjhA7sM=leACek6FAHj1?$jpfi2HUtM?!jCsxJ{HrJPRnD}%s(KsY z%pf32{alQKeIDA^F0KrFV=KYFQsg7wNkdBgPj3Jnpee{?=47OsQl;V%hK{Ytg+>~1 zUQ!&(T-kb#iM%gf2#I#4{v2+nfqFWZ4(74jQ&EYbO=Vv~Hs; z>q(b)(o%Lm=+BfpuyqzT!`sB*Y<@*S2C27KsUNabw<(NV1`5=|9TS)IsWqv!BkKFf zle6>s!1odgpj<Rg?tCTaQjDT6fne{L-Qv6Urrvd zHJVV2Nr$x=iEh-I;4kDyR_$ETRc3v2QBFu4OC;jOfpI_l%e7 zQ|n*UwgfCl*?U=)y{gRhE>6MFOCm_Ltg7bo?3StH8#|kuPp4MRb8C@EW)sVKp$;gz zYYFXpcwyfY3;QMoX*m&R?5fM)zA=(fmP|NiG@>|H3g^#qU+40biGe@q9Wludt>1v( z%hbDs=RS+H*=ExA+dxm)7M&GcI>tKH>{z65rvEFc8C>Q~Urr1mnScD@1Lq9DA$tL8 z9VqM@k+Huwt`|tMAbyO%PBNFq)&yytqK-eGZIOfnSNh7AE^qPLw!I^+>T{^CFzm1O zg>Opd+3U$%KMU>BlMY%V+}(=nA6tVjPIJg>z^{HK@gOPeejx?e8tPyuUH2|`3_&1^P$XGO-j(aX7?5#o3Z zP(}Gn4>pmLeB9931oCjZH()IF?Yu#MFxjvmYystBcd5QQW%>u}l55SM#?2)JhoVau z>31Kayuj3g(?YqImJX55A0|Bouo3(c8)YZ@9J;g)2lX8qWyK)UCzwma4%%93tNXwW zdSPZFiRBVdD!QUPvS9O^eG&fj$3Yt#URZ}Vp-yx~tu*^o{uf05wkTDpTa=z25ser{ z^~lTyyT><8GQ)NNq!B-seSZm1_f#5xOgyq^z5ldYmE&#F)`OH*MIKt4 zc1{u0F3N+)0j%;(-1&VQzpw>{Nt{Z?=m(7F5@bLU)AiqBd1ch7unNRbP z3lM>%h>s1-w>OhIxv`TUih!)wS!Nk?bm(Bw=oRe808Tq zCkWAfyDh{$@n-nx4TQQs*#)Ftrim?P8E7W&3IZfTcrr8_F+b_*4434>T`^ClW+s)7 zt9z;CYGihVN!y=io*t@t?6k+0U42$=yi-5uTU|avfyhqmaG|D*vWi14HrVtQZvZY& zUd-yW?ELN2LHqL|B!zT&`8Jr9+0Tm13avHCSIc$+m#s&+rjZ_*7aed3!du=8yiuwr z|1H_&OTZ_M^+cz#=f2ir)=n(@-82*(*{qX;S5IyrgR78@FeInZkc)p6wftZY1ZRuf zxlmxIly}9IRrroyLd8EyrB+$`pSRb39+?jckT2V-M#?Hk=(AMQTyJ%|!OwD`W<>fS zmcPc4eOR{TvhhJ-RLb?qc~*!2L?=mJ0)E$5QR9doP-U8r#oO|l@(UQgpbcvl0cJxf zBeMB1hVpsSnB%ZLGi8PXzLC-&dV0Ai?p$;F~u<=OpY(iJ>Xc z71R*EH3G1d9w2AJfY4Nd&<`&b31(_QgIKn6QX)leKIhrUya*Kpq8*teR9zH-A9ovX|+Rj!bl5!O1RA+ zy&C89SqS-ukc5uvi9d~T9l&GZ3uw7R(29<5g_lbwFe0^bcmiLWfo*+}WJ^1><06ER z4+CujFA!H8{L?16eF3{{DxrXI{CaTiMh%m;izgE4ss}Kx#oy-J44@Xh}SDGp)A3h1FgXB-CLp;K``R)Tj1O zD@2&KnMR|+wu3C7z6Htx*4|?&bL?DkbC3Bq;IT0hA~fKL;$tv!EFQ>J_tLA$MZf-L z7-miMr9J&^oiYtHl3MfSE4~W0b$46!py=kk6=Y7(@y*Xl5mGQNTo4zii%!ek)xB#Z zQloQvA0=d6Gyqa~#tTnRHRR2lK_YpAp7D*c`Op1pCkW3kR3v%2?vpXi)428vLj(dJ zNH#C?vQ-mfmfwUszhrvD=}&m4PycKTw)x{1#4w);3T!gsO#!7{-+-^ZRMHQ4QpA_)v~_*Xhp7 z*(PzO%IeloCMChCLWEi;%Y$vx!RF$zAozfSBz&2!ZhGB5!`OB+VzV^m09@bn2lO3v zPV!IR&1ty$w(`s~*iLCqmS`rv3y@c#IcuS+oVGiA>n-~t+Cf~924(q7-(${q(Vv+m z^~g*+&=U&|56nchyl$1zsQjz_!p0w=BsT`g1PNpk_j6>E=eFOEzT}+8DrVAhN!# z4`Pl(#ya|OlQBasHjl5Qr>wlHEy$s?U+3P0g04c1(o?NH;X@^br+;tkAALz}x;@kT zIzQw@E+GvaI~wI3H!a*9hk!h2(<-y1Iv1{6ZK@A@6N9(+FML5^i^F#lueffR))$)E zjE}%i4%X5rC3`7FwKT>@pYP++=9?fu_n6rNZ;0=C=>%?X!<5PyCV};si0e6gedCy`pi)X3kp8wHz|E1$` z>4kNZe*M+_rqpE*qeR4Hdr{eedfpS$+$jtrqSUy&WU?vXnc$|{0ti<{0}#dx6y1FG zgp1Trhr<=4*iTm!pbm9l(+*g8@ytVe55b-uop{A>%MhWf)*!+@$zGyKF3y{HMJp-gLO%-}lm9 zPXJ_XV|=uW@ew58MW|vQ2+##O?_H-W(06Nd7YgoVvThA76oQia@cOzv(ae)W)jk>1 z8FB~4=biAKO49hzneE^TN{8P#5r3kiu8QVSGLYU*v!?_OkW8bAIA(+SR7d%AlY3M` zXM9OWi7vIi_|xY%irA`o55y20DgF@_aXH>GY*7DlP2jt_VxE^`ocYw7f_pKfqei-i zUisJfVO2utB_DuC<3_*!T195>EG#*PQ5~Pw5Wobk96Reb^%QL^Ql~naOd$t zZmFkanvNw1FS<=kmn>SVe6G zE$_Swt1Bb)OfWON#iC&a5NaA53fsYG(Ca*y*b3x2`!YIdz(=|Mif1OWMYlXQI1ca| zIt-MRAXN%#EWGLPov65#QOCcwp$iWSyW$653pdlO@Rber3rV59Pj0GLbg=xUiL1s1 zXrFA(z--Mnc4oJWiVEJZ3V0r5sb#_m0Rj<_J^q_ZP)}Xj0j6lptWw83SoRD*`&ZKs1Yv_WsvrpVlqKGocjpmlx26GE1Tx^=ReGHr#=wR+rwcD7S$ElLnt2kq5v>R!Ub_vgM$>5{TK z&~D83=fF_~9i053m6Ld&rN&F)UfW&k8T|vPzb~d|BH~wADGe{~cD#gdh}B!;(>%s%Q@&Oo(bdj9g>rXq5UiqpwC9PHL-MdYt<8;ZTP1wRKK zI;x(2I{{-+#Bco3IlII%=0w1~q{&mmje%6xzI_3?9d#iLcVuH#Z;$ES@^Iby)1IbPPvN^!PWT_#tfm^~v?cAsczZ-9{V|g*-r_ zma}_B!$JuxF$*|U{w?!JqS!0 zVn`xFjLXW}+CB@j3)oMsAq0Mj@XP0K3n8Ci$grY{`>MGCiKFHN4g045?1WiCov`cH zpIOMNHL@dHJHvG;^%{(s{33qPSen4LI%M5oa>FnlXtV*Lip29rKZs_m;EhJ{P@VCS z#813N3Ma2iuQ#M4NXhyOpzath3BEL9;2!O{DxEN|T&I+KCsS@fz!pKk&LlPM4p89L&h%x z;f^1z4Ve$fZR-^I!h7)u4X>tR49kw$4@R#mD(k%tGS^XWGSd*!O1v1#Ojbcz1LxERGJ9rELKRxc8q{iFc_ftisbRI7}#Uz!6 zTtrm56D;QyJeIgN5G$*`c4=0T+!MWf0O)ns!>ANI3YdD&cyhIm*A~>sbr*_1J*>Qo zl*uRxNy=Ez=Iok2um(g93u>;q^NB@N1~d*(<5|6%PV(_b@=q_*nqrP-*QX==4>7;n0tsgZ?u zr#zNJ*5Km9uUk8@gQ&dTfH)*7>>?FK90Q?bBzV#QJ?cmMbd0ohcK~6JWWK?^Ht^nV zCO>>TM0?YT=$zx-8~-I@q{%on zG#-G`E`&~s)(k@;auX@v8tGEp-QA*5&!3-P2RAZ}giV|i#BTn~%-Y$)fj5`QK1w<} zE7vfifLgFIsjbJ9nE|V>9s8m6i;&)Dac7TR%2i?B?8nT>d&qnrdVf>w+3iH7cUPKY z#HDNVtXaO=0q)n#aK3aHR~N_{9D#>8Wz+`8&G&Nk_A^a-rh^+$+CF59q77?R&KRB` z{9#u6*_OfTLDeotsw9JbTV8;|qX{q)qOUn*72h9{BEUq@!`+C!)BwV9x)|*yO~Ho{ z&|+X7p-kvOk(z=Xxf+7w15rQfb7#tfRBb^8We}sH@_F1|kg(#MG+{-O(CUWq#;)>4 z1T&{91aTR^Pg?EXJI;ujllPMOn^FEuQsM)*sD$$kRLdTV`yyh~q|>?(%FdA9y*O6M z?3JV{OIR1$DTHA-h)#y?zj5hTp~>RJCt+6BNefNwR8YyanJM79fbJIAi$8oCR7G5R z=2557@KrtQP1>UGH{LzPC;_$ZWU+1!P}4@6-e^Ro+-v>-e%yFMD|*gmFcPI~M_K8E z2wM#lO6X^kXtC|t?R&Z%6+>h%)FVW8`;)|^9UwNrMQXs=^;+x5+3dhEwPrlxdjk-4 z0gXL~_SvzZu&aFgGaHayr1v=z$~5|F-hMI#Y2 zSOVBlzh$UsAJZgDUec1F-G$dYYSaVk(H%t9ELC!ntWIySVacv=HL-gvvb(TSO} z9XzPzNU(=hDda%LEnBhRQNNPv@!=wR40XuGa`|8%6G4`_M2F`e^XT!{%EAZC+L-l3 z`vBK!ybK&pEbawe5RyaG+;=UNNU(v1K>NXH&B73yT%eyycP7j##5GO)BsvM%6bSDN zpEFVqk$J%4!>f)eBABp7lwCk>R1v%1G^>HyGKf|O@QIV}N~W}SS==`zxlp#Dkhy4z z?E#>zo`YVMM$-*j{ExZ@CGQ z8Fl`JYnohN6mlLGM4xd_W@@4l%(euCrsCcnrXq{x<+&nDIfQr2O|V0}65f$i!lb;{ zv`be@pwGRv6^kXYXIt9VHf_iSo+%uhBb_{q${%}vEz-TMXc8X)5nk%h3&QKG%&m+_ z_G7scqkCy58c?(lN^^+JMRw5(n}ITK_0s1z&(T&X3A0+MQEAj&AwCX}r&?EK$*CZT zuZJX5q|u&mL*qhIz5zWFw=)4cw6F4oKu?br^faDBaW;_M4yzEK)BuwqJHV@G!{@w| z;TJk?Hc@MMJF%*F?hhr@&Vnm7v2q6MxCLlwYLt1-; zRwfk%B|xf7%8Wpm6BQvpX>D}j3`||_&@4fb3 z>srIUuHmQ7786NwEK}UmJ1Pz;Yw|czfQ8B%(S%22tClVgG-1G+uxzt{AvB*rbn%(J z7G?SYY++HTSSyD<@oMH7=*80&CqU;XV1(V^~Gb-Lqa9R@Fkp;xD zyrng1h&1e=PfrO5!ZpbVI9Y?)R*9CDwKLM~SG?apRv-n~Lh|w!#+UYeZ#`JlgDLNi zRHeb%OXhlseWyy{Ub%HIOT63)&|CkwmjloHi_x2S`$uY#`Dke^g8j0wF(9S=(~3_% z1*n1zBtZrXC1$;&2mM{uw)!`tPuHoTd38jY9RL)405145`4y5Wr_02U2rPlba)_0A zVAC9#Vp4atOA&vrd}k}E2h&nYk`l!)B*=0}%6*LLncal>hXi-a#dCUAQjn*$D)O}S zkvcHr1Uf~%YsvF#=khjz{z};$lPE~CEYBgU` z|9_(k#5PLXZ1w3y;f4;&2_0Q_OF2pHo{4i;*tfRB_I9}>D<+CeuMm^lm<_*==wO5M zr^g}`8^w|iz3*IyV4jy^mQ8YMSa&xKmModFWpZGBtsnJ>NIVU^H_TK75Oq_|7XW%O z!T)xQdbNUmYGQSs$Zm8r0OuK`Vas@&tH^GF&%^frVEGhSf@5}|Iv};zTeH8tgzD_O zFXx@T>Mg}?I4u~`mDgMB|+ z4+i5U73s-JOO*DdboP~@3CNLfy0jfD+H1o!?i--gV!vo59jUY|qv{8kixjIE>|0Po zB5Xb{n%QxU=wCKSTtJkY)n7nRhw}j%H`F<^cy#lWVjjXy{2q9489kE;3ci{h3fhFp z@+?Oy5zoq%KYdo=@cftDr^Go;hG0Zb6V&EH&P+Fj;59lc2-W32{Um8q7kluDYU%b& z+1w{MZksmkL*~mY4`a4o8&O)79`oR!veUcUAE(A#y(*AfN=OW@%DZxHsNyz} z?6TB1@q*|t!pjCnqH?`z9_vuxXu5M`+17H9g@}(Z4YS$9k(;81srr5sr@E?CRF2&n z<*iLU$=OucX$DpJ7LiBB+zCdVCo^^r+5v>NJAAymo^#!)DKpYeG|Smnn=VQfN_Mu2 zs^FAc)BTme$tW+z4kU#M>!?M5eYA&@kH{5@CDfB|bt^RQ=%N5oh)=JSR^hH+Eg z2DH~Prr{~HYxixUcqWdtN#V%c?nY|{pF=+Yb`UA;&<&=E;eu(Dvy8@<;pVi|K*3m% z9JL9gAnt%W@^iJ}wn+5)eKb)rC}Ox~yV2vI0aB20)1pw#S{uL(@=pYFZ^>Ow37wc; zU-!rfy=z*cODqe$-3RU40>&HmY1x9AxhvuPQAqrP%kr(MWhJ56JLB-e##Hd!Q~?pN z{J71Zf!#uDG^2+!zNU#%F~RQ_OsjL}(8iw+1~2WSi%18x74Bqr@FuXnN1Xax0hXL~ zV>Q%FJg#s@oYcWWMHo-R zaH>HfDI7e+(APZ;x)oTKxQZR0`(nTt+CG$RfuI!P+NZWLhUfDd@zY< z)TA#=}i>Vf#(vtSS%Y<1cP4XhOqlJ>%i@+ zUc=7FKBKNEDUlVfDhV@}U)o8MZt2xtezHC{Q@vlDPGws>u#Sev40rS3@fPAb#k%^A%!90V4~0tP(`UCciiqA}z)%!;3)l z*J$s$A(R|qE}ZDd9SMKD;-|cx8s3AwjvRrPl#xSp1Ko958RmoUq#dtB74qOl#L(Q| zI-^=Pa%+@(ZUn-wetBF@20J$QH$*JCS^-|wRaDNN4myXvtSk)_qBo+HAS||g+N@(` zzneA41!&EoYcx!XH%}lO?6_9xRg0}(@k7^?eh6%Ky=Mb`4ZTBMW!S6*hJ|W!-qR)*@WUpHLf9FKKEdrig$omz3ZWGPX^@ITponkYidoY_n z)JkH@(TKW-r}D5ad&zTG(iDbcv7nOn2 zglCa8^b3HT8ql>SeO*;nFmNUt73?EFkSf8@qT16qT!iVcsTX?b zqqtxMH~gASYc?vN=5qWr{t(vq(TQ5kHt!}4rXznC6OT6DPV^q8jyxX)jU#0 zP?HPzgWS1Pq5zY@_ujpdynS8=HfFU{ZjrmzB$cdZk3y_P_2SNi`oL;u}(!L&Zk_bUq=aDE>1Kz(kd8+rG0^D zLaTDG`^u8$TJtWAl=gQd)B%s|TnDqO3`toqJH(8-uBb&BH>EYAQvl>DLE~9ZOTtV2 zq|>Q3AP8UkF`prBe7{#3Pgbthp`O9>1=d1I%jLgZ50(b|OHZ3zkL4Kxyis4lu<^ zNq9joCqc;LB>!~c`BZb58*TP$O0@&`;nTs?NHRfEl z$Ess#NaeksN)(^gX?3c8c)G5KI5L#z7{~%{pP_Wv3b(KVExm%(y#|XexwCc?SmPGc z(qYxlfxkU=x8SOemh(98%&)W$KSZLx0E1q*EX~6GdLk{m!^Lm%eC(RJs5~mD^)hMRfA#5QvNQm59uk?$B=yDKwltzom;G=tqT_Gjtcxx7)a|;Z5 zSY$+=q!|TmI~@ZvkX58VDx!V`QOxvEyjAl360s-eu38BvnvBBZ8fS`Uv=9HxF`JM| z{iARP=4{Dg-rVj;0&9uQDur{?01qPr4(HSDX{zRCs}p!LXoD%U)WUXNNL)-jA-}d8 zUOWxaWcc*g;*tu{|Lm#)Ttu1dGwR7*9i8s-uIk2``Ir2LZ?>9AOB5qf7s52~Ua>iN z4{6RyRG>w!Y3iR_H>M~IQ$cm0Y2-7%<|BC^iSF&n@jf-pu^-YHO!Q{aOQ^xY-wS(S zMHLjEtBS;}sAo?6^bK^J&e3a{;cQ!hKWX1L9=ceuw$^l|bX8XlvP3Ed7;wd(xf&Fr z2d{{OTsfM=Ae6C7DQr;m1-b^oHBiSgZZ3OE({9E`j{*~yhC>{(eeoYyTu&6T2#3E? zw2FURZ521$Zw|^(j(r?K^9nuHdH20fVH23A0~C>j+2DS$EZp)7G&xl^*rz|a7wh8% z%>0N{C=OQU*QxnnkS2iHM!rqd-EtbCaxi6w)-+nMmdwPHhMRfWzzcZMMUU|URbcB30zqYG=!hMKsz<3yK7rd%5jKA>(=oBBs@gB(^~H44n2B3ME`8gYipT*QuW=ggceyf> ze@)VLqr!MkU+=qytbHiXbZ7a&$gFDuQw++!8Poyr-5 z@F<0q{h1xGzA!Z7r~(KB%!`d=1%8B7Tv^!^kuTd#s!^slJ($P%9A+sIxrnGkcJ_~l z;9OU%ohG^7gS5!X0V9izZc6ehUfv1p%ZIy8SqXt5gQ-In%DS@bDiaDUYCf&#gX_gI zjGg#cxpeZAX{erz5u!nT*VbTI_Fya_KSCqhgYu;8x@d2Ys=1bOo}gUgJW>}1!FFd^ z8N*@|Xo96fMu1>cH(JyYzc_AE049K3&GMLGPa&|aX!g+Zlv#nH2sA|inrAGdw;EwkRNxO2JvBbMVBHhasX06GEDGgl%{~@t@Z`eFvVqsH}jh6D#1A;5r##% z3LiYxhB-$J&+ngGsNT5xSo=Cbs+fbphJ(Q%SK#6=D~2J}%BM-b)v0wJca)efAFbKv zy|E7_LZ>#PQFIuyAr1V9db7k`UFi&dFYUhWsqQr?z#0Kyf}{*7d=3C7$>7q4ZpD&| z^DnPVADyJ8%)DxRE@Y-rC&GjsPgGsq0C5^%m*Gu{9(M`m!&w z4kgVuKBag%$g@XV$!Bm&lu0>hVQhf|NOst_{1Y`_Ge3)oVDj7g{@2K*$rsSn#_f?u zyR!YsEB+QtB0YjmhgDy9TU<=`YU5DIiw2(eYV;n>(?j=A_G(hlW+fZclOMI!`Qs!? zLJQLZV)B!AsNSYx&{xFQQFUol~F+$m;JE4;~>5yl7sJc0Cy4bxcv)1PR%4`U1k; zRx4buNivk2f1?mEs3~Z6d=q9AWB9@@d&9Pwf{%zciZOh6EtI$5NyqpOCiDWhxv`hX zLi4(aMs7VKEo9wv;Gj?MQh5z2NZl9IKWQ8zm?a@|pd+e}BMo>52+?EcBA50t7~oV? z&X(Qb`5sCiQ&{&nbz;AgMNs_j$+aOSl}&O8%dy^HL?z2p7#oSULbn4m$=%De5QK>$CB)1$tAQzxFOo!qB!el4NgQBH`Y2A$Am)q23Vj{GSly9wL)0+3sL|8lu{aR`F zqZ|qpUGov~lClzX1P!^T?#~2u?p|<*!E^EqsFxjct~vmfHT?KR4FZ%H%LH)VX|ayA z6WZ7S*P}I=e;`AlnR)gZ7}$%#V82#88PGh< z2wVqlY;v}dox%vW(5W#eJV8s)cJKN{tjHh4t$FOPf~>oTnwm;M9+K=z^IaAWrEYH482o#KzvZifC!V0|uh-HKs4 z5ddFg-=u$YTV}bsv1UjOmcgzL$N@E5v^E`P!}QIfx^S|;jX?|Y(5#*@Url1SuT~!{ zuY&dP>txQ%%~f9wyd!|aSy&XBa~g1q7Jw>+;_(X7s7-5pIWkj0WcCNI`$2-o2qxc* z6^ASqHzC7ii?8^y*tu(QsIwBit`6$8>{k%a%9tR`Y?Q2ZvRKa0>&qXqDEve;AYU|M zn;-wfG1!OW8g1EBs=C1NpbUY2;rb^?8|hV$r7sW9og(_*(&=w=pGAaJz}d9)WH4|z zn<(%a4R^abTf_z_jH089b~Uq$^nid9vYf*NHh<$LY&CI%CT5|t&;)pCPvqyVXfn9R z#AJDDEn2%&gc;~cab)V6=RE{w*^CYTKB6a*{lmbl+hGE~6jV!BN*$6#&>JF#T14$x z9(9_3dr%!9%;C8eAFevscR3Z)rppst=PFONY`R!D*pj@g&8QKC#&Tu@X2`bxmiqoUaQ9y0=3v9xIFSjmf zPF9ij?VE$qvV1UZZWkTNK7&^X7ms>9@!m(@9oTZG`!rGKG;zHxj|`wT(J1T^(Y3aL zP}+AZnFcQw`=uQ=(<$w6)>Y>;F^wooFRcO#rXg3wwoGI8QE5#OOFg;LN>{TKI+$jw zolLM7?Q5;5o|m+}6AwB`mBd*P4=RzMY@>MRG7xDn^)dCUVldQuC~Hm1?d`;diE2CtThGhByRy4xq28hz+om z8y|_l^JzeNq4VSo<;9|gv3Jh$$Ztj(6Ivkfc1@dXS&sI#P)K5Om`6 zri*+?9rhG+oe%;2xA7@7xqH?*p4Ilq1XS`VA@)iW_AR(~2I9qTktxY=<2_A&>eM^z z#xzF?Nzqh~%C_wA-ssq;-`S_p z*=OR9tqZVxO3Vy$wcV1;ukJPhodzvpLt2o+cxY*3P!2Dcp&@OPmt7*FPLAlISt)y` zsdum;x}>G5h_WU>b0}6fp<_%O*MG-r2-LLS94M~=PUVh}9+9OQIMpjJyu#g;qwnBd zL|nnDpQ+v*n6nVfKXzJPe!3FlVm|GO0DsIVkl7TX*$L0*_tT`)lP1d6bjP&{FI}w~ zBJ2Jr1A4SFM57%8u2+SG)TF_v9e%Qc{D=UYKVSAHbl^<_fj1EjnDwiu)Z5s6ci<%O zEThGFZmkoC?M}(rYI3LH`;8V#;^zw4ZD|QPmN`zT`GgocY+6ha&xTmq%VgT5CP@!U zetlAHNtiyFUuDK8NZSGX6`fLR*qd3zp5K13c^v2jaOkfZAAR=vAW41tF*}Z_GlNZZ zQwtC^>=Lp?oExaaF7leJ2aA6=2Bx{9se3>XR)-eiz1b`9oaK}YYW8OThmv1>v7U2l z#E&_b{`g3N+cNOfo)D$1{!z~4;x)&2hBf}43ysLa)laoiJct-Fp?qhPS+@|?+8;bprg709m4_xys+fTkg`F!BeTwb z*`_7@paFeFM%78;Su4x@;AVEhalO&W2$J$Xf;TmArJ(alygCDBo+>t%o$C9T+kmz` zjY2nN9#d8nZlaNOY2LhUV%^w&o)!mDtC%xTrVGdv@7ZhRk_7rJKr8fZJjHW>r7B{m znxc=OWV{jxkjrNVAnhZoY2h(qBOG<5?|>pKftoja&~_omo9#}FWcGyyD9Qz072<^| zga`)`!@=&-wo2p`cb3To-nLTWurNN5neZV1vpmQCU3geq+UPR=nB4zk91yMf>!bD1Aps7CP3YiSp;VKaHfobam$R@o^V#Ndse zVW`%r>`nDOPaFF$HkGUif814NJs5etR%UM_%{W%Ac-Ux{tHd_DuqlneU-`DY{4VH2 z*}oyU0No$clJ}DgXTmIKp@W4HOG2ZkG0NAs;kuEbf@f92>s(AK6hfmb#$dwLjSDu} z66}SoAKi)0_sS3l)J*!^(=pz2YM|YDYx+P5>svDtC|2w^92&m6l~g?`p!3uM)x{^2 z+=_YWw!ns~udDlg49a-#4^O{3ec-e&K(81V+h8x8)AT{fV^3GgZG}IBb@qif&j%Y1 z$SVRTtV>CiB*kxd)lJRAKv!BHwd+Ngn>4|Y3K`tb~4;~9OU4D`Cyl#f)W|-_iqGKD~e(10bg3^uS|H# z^?JF+IL8Tl#Pm?{rnH`jop;-~_edfG`r+q7L zeu{b_^#&mDiqY4+@a|i?J-LT@J0vU7Ybz3}8nP4X6N}961alojQ_dKDOY8(J-9eGN zptpuZ7!t8wx12FG9vS#gOKXQsxL)9Ho1N-R3;6BXZ=LzoOvhmB-LB|Ke;M}6o-P4;p;Vt9Y1=H}8B-1_2KWn12k0n{%t ztGu&M7F~9j*_Fz}+A!f%;x+r;2RM=DJA3jXJ<2n+RL9S?yKGsZPj3wyO>u=Zc#82lM zw({b|?PDZkO37j~1|9|2HoyaY)?p_HMf7?7WOcpr$*TPypRC$)9GtvN`OB}Dx?mw% zhMCAK$1Bkht)TTZT@v3>qP~VJ7!1~k=pFwSOW=F|dPns;ESBkx8< z;AIj|1X8#FX*)FlvK`d9i~oB z@idr4QhftS1a8Xk(gz9yj;rD2@uS#)3{1#CLhs=1#bzORN`pZPj0XD>QiUVqR#7p- z9!RSY`vGVx>uX;82jIy`XIh*SniZHIe@K!8r#u6Z$I^yyiNx};0G$eSN=&bqE`oOw z_sPpTqfQl6S4y=GD*_yNj9d_CFm;V!dmOyme+*sQ%mKnx6U$noM^clk_8E68giXyc z2Y^t`u;8I?u!uf-at5?Lr^RsRJbFlz?Wv5GFD0#InZl>q-{r-&3x zr@p=Tvb~+6(moyVXjlv$ckr@HoJDY((;(|*_Wg_wZ1f%+F?QjY14MJ06aCe)DF|Ln zpA4+sRu(Qp5T@?sJ}0Yfr4|v$4}H}X9fW0c5x$HrngBcfoz-H;7VnKUL8&JJla9MX zgbRD-XbquLq=&Bgb$_2XVIY1625)vj6;b_1$kfmfk!34?R6HYHsqAJOfU+S)Bn8a0 zOxeXC21%ukUV&gk=d-bP{J8m6DamT;O?^U;r%XcuqlYGD<&v(FTvyp;0uChZGKQPk zzf+6?{#BvSH9Mpf)P4FP&&MoRd!OY?WUFb0?d*#%KFOgMfd0V4Jr^{gBn{I)cWAo40pM2_23d z__Q|rqnv@8)hnR9_)P8xrpcf&>e2i2Z_TksYD}SAgXD>ZAjv>!2l1hA#IPKlvWB>C zIn!U*_m;Nloc0C zo4GOy=p5pYkbfF$FutcHy27zE1)LvKP6+Ks2z6OeRs(1Z8@%9UvrRup#0bc zR`R+~S)vHyD!V}hRML|hR2qPD@ttJitie>lY8_#PIQ%@8X4rOoVcCHE{oku~56gth znp^`lhOnZd?R zn1fTHA&L8lm}B&~bjAC$>gXlD!{#7%AS)sc)9s6k4f;%$T#1d>P3b6^3fXaZJ4(4S zR|`&2^zubnCFJGETNMjijYtK`g_(fYbSn-d_MkF;92M}s#IF=cUN7wXnRN)+^1@#3 z4^wk@>6&Sn|E;>(&?NH^<*CX?#*eRtKksN{629-!x<+_MZ4C60zV$z5`6#j5F}14w zDd+^}2Yo%Gv{niL`LUIlLb0X+c=~zMlrOFZ$et|T7?Of0Xh+S9c6E&T1u3bH6f^)y zXQN5`Af~jyj&~WU!zl<<{4T!L8+XNGhPcN`Sfo(nlU+b!mE>rL0`})*-i{` z@MGV~zQD}>6_ihZDC?@i=cVHl2nU}4&n9XGE7D8ob*#Q5bRf)z_9hLzs&)P>)p~%e zV5#eDphYKrKQVY4b`SOrL8m~T2iEkObLElIx!Wkh(Lc+fT)GA}E(1c(MOM=0%*9s=t9*h&&A2a-qAPWt@75N|4WtIi#?vs-O9sKM^ zN~dMCNNkW`-bkWBA`e0tpDKzUQZTk5CDvK(9W=7Yo3J&x;Ko(iQK@Ukr4Gu(yY|_EdMiF? z|7nJQp@rUN(On>Srw|Cu;Ak5j8u>U|$28%%@xQ3UOV0x$b3RtppehzZJJ6DSmK{d+Q3-PU;ArAG1B;-m&OkRf?e3v}zbA<%yBs19@WHlYhDF z*aI{?*->pnT>^epd{&g3TbEYDejG=`{0%%(vdvC)hr7>uSC0xati+X0+R+T}YQMp{ zSB>KWdV+#c>`e3D^NE614Tr+eSLJr)Keoex3wXO~_e(pS zc$I2HKq)B*|2DEzPN&wMjhRsv<=MFw;Bz8I4y^SFc>Mgu&^J-x3($!Xi9(b7>O!+u z!Owqecg4=cL|0M}-+>0gPM5gU-`xqzm9dHQp|NU5M_SA9HiE+JqnTgL`32sX1 zWv8jvvbn1a$K3+5WJ~qua}E`Y^OMx!SR37&FvpSl7Mx^A!J0Wxu54KSZ)H}5EqZZN zn$%k$JSybYUAvAi?5#drq`{}XA17GsXN9{g1@{O3;jU_wQx{%wIy#=(#g4FU`Kaan za{gq9K)jcahgVJ=9<+*7PM`g+Y2x96&V;zLGc(0bH4b(bf~=zsYj@$BJraUCQDZ=` z)To(vdD@Cht$u??mg#1Dn&%fFl=Z#oxCLLp3tjmt0T9V>ie+Pdf77K%7_I%Fvdkw6 zUY6To+Vn_TAl6e-<3bMqO|e3}K`oR8w5^FiR<=Ep z{3$-$oI9hCaQ;N3kh>Olr=!i@sH1n3TQ-DsiU}7|gH-U*ooYiLb2QF~-)f_lD%+?B zdL0Mgtsx67uNwVvCT8#OA^ugx7w!Pcr1R%+e)J(Agns{LvQKOn)F}3jo2d_I0vCY_ zO1+h(wy*Qf6}dCzmkvjbY*kXgv#sK672Nu>_bV5&`)eisSqVXNH;VIiB$vRvo0sX| z{jE$+|2<#2fO4L^x4Cz!!nSnap+c8+D)k08BMB4%io(h!Mj~hL=^vwPJ++Dt6x{kP zqr|vUX2MHJ|Ks&fDAQK)$hnUCkNmBq&n^Ur`ha}jMvkUoY-S?$>4#1HN2qiAC@)02oNA<%k178&d;QU+ zY8CO)+$c&S>__;BP}Dr8(1%cV8_q0u8_F{GRNC6>L{G{M3_!t07V$gn~bA>5LTT);rd>@P5Eh(s74_ zBG;OfJ>KHoy*vEy^@Fsb_f&bWtG9`L3c|q~TK_0Jv>^6xew8qerE~Oi{+=M9`;G9` zGawGlpl{mY8Ird*^~P-vp-Iz?;B&2^GiV4kz5VLKwgYyno34oI7Hf%T*u!pEj-n zB83*}nSE4gNfxD}kGwFTRtkTCw!cBfDQfJo-hPKpZg%eNI71I=eI`?jxV*u~%5WmRzZYq@D_MG#>mEA(%< zG72G{Al9!6{dcDtJoJ`N@#I9$Q)jcUjJqi(mUc|tUbr^XRFG|HqV1(&9%Q$A0u^eV zu`!PXc^c|SE74vnK}4wIQiw$C4Ma#%-r*1bt2va&vu(@NcmCU2+5Sq&@zjFC8-_;N zX#*R>y;+^xR)ASdS=HKi&eiyva!xe1e}k3LlHir(4l>rN&KZ*O2O9om6`UTbZHobkQeyd+D!c=x%VHAr!-2Rp<1V|N54YdN)6%s=` z@H6-JC-STP8Q$`v>={LQ&yMQ#(--fk+@;3PmR4LytyDU%6uZAA!wqOU_r>h9q2a7k zH4YFBLuK{gqS?P+K>|BbKkvic6IB#YV_arA=@6vKoZ7;mVn|uND()_bn?WLkH>YM@0}W1AQfQ${nKh1 zp>#QWUsog4{QtdA-nb|b#&-NY9uOSV-$p_Q4jc^dzp(Fn_9X1;`hMDnag*nOpDv6c zoH_IMwdv1Zc||FzqNi4k%QJxz8KvQG+mxZ7!wdE2o-g<=c?JgjF#KH%72j<>pHhzm z>AcA$@w!8wz3uQ^=M=9Am5PXTvhU*Gb!AdI);a+U2(MG3ZuaEH8@DmYam(XmMq}?( zp8~!>sf3MRtP=F(EA=6z=bT!;TnaDx{B7p`zpPvZZ1Q3#&e{m{9X`Ny%{`52d{~#9 zA(&29^zVU;yw!$7yWif4mmmKv5aTxYABRc}t40(y1ls3C~%8BDsByv)y$&2&X?h4~uaFnv8emnb7a)TT+++ zDRAf|1F4{9*8pmEDD`pHRmPA>6vd%*V69W4_~%($K~y%USU0stp$2U69dKCdj{pCN zQ67d1x_q%5D_J=CpMl5~RaxSS9Pr|vx1<8&SPPr0wLVzU$<>Qr!n~z_`^`toO`=EF z$&#l2593M(YQ-SLCuy0qjQ;YEATN`!k zQ7dpI^%mUUX6p>w^OkI@1af(oU6~I*TC!-gt9xX<^S+i8SZCg;H!pN(ivy| zdn+2i-XjXBwPcQ+{Tr#+ElP20BU6>(D=~&$m3opUbERalPzuBqzp(_JOUo?B+cTaBgR8`9nD{Cwf3(Fd*9=)&5#1_!Fx}-qDZ{ru|>&c#_)Cy z(8)DD`DXD(dD4<*FZ8Cn0p3f;1d|YVjH_Ged-RM1Hn>7RhOX9sWd9%(~z3K;)a*o$cBg?>aFIrjzbOZ`$%kwz0=!Tu+@O_yDKV z_d>p<)#ANd1O5L4+pT5h3%d8#q!%BEw8$rrSDapzHUWt zIX%HNRBPMMwRlTV{IV07+gz%+7QN|?#k3>>U@R|vP5LpGDEwW?HI-%l=zfeaQ1 z-da@bxcbcPJjqwaIdjxCFDf4Ad&3cd4re4ZKHLbo<{RkS(f7wfrPMf5D-6NxeOFZ5 zF<`(^&UR6WaGG#P12-oYF@>R6vU$5-(apLbdatU5bX$Vv!kdhd+eJJ)fq zLeo92FXne6H0oW^MW!PwG6eH5*)9*rDnCJ9KinCgJ>l}=>s;42KhrZR;Q|r&6Rj_J zGq)S0K41c zxG4iuL4d}dffcM5odv;Fl%V&B0(8XUQ@!toW`{f`0RQvJHwu?c? zonE(6jXKhO*>+Qsn0eMW8WkFYpD}Q{+_` zffR|}p>rz#{O5y1p1-dtxck$_FRH$>y|nX%=U3IlSBoVH7|Hcwt6^ksF+s2>v8XhD zdH-|YlG5f9sRc(`UYuu&KkK1;HYH;5IAtMO*YZ%Om5PeGKnL8J+&M%Amjt!B{3XII z+n%P|79T%0tF1R8nGJ!u$FMRw$@}B=>}l|JIbsZ^#fP` z^wJuY{tL#9?`iNYSkJR(#%>?dn(UPWZ$4!I|cpb_8}g6#xi6)Bo{%@zIOGUO;Iu2J=X28p5SQ4LOu#^= z;&)<@q4#<_*Q(r|rv;s&z-%rTxMWV`8!C>;?yeKun|qI6f}Ymw6fB)w+VP`{4TT?( zL;juSRl5kdHvaW9jg$cXhW{8X<+9Iqz#*08Q)5O1|I&5GP`l9dA1Xf{^iMnrL57|S z3k_)U!|48RzT+a)+0v%*+T4u%*$dg?2(|nQE(;#q)n4_lU7BQi=+aKx^qyCM2T^6? zQK{rBu#jJE4UT4etEg;4Wp_>X>)2d2?4-qCnb@tOvZ-+U*T{%=eWJ|OqqDt&n{b&8 zy@1izeeze=ml9Du8A@#Ym>2>?MkTJ^KMnlFzx2tKWLGKVcHZ+?IgIhDu@ARl+C{Hq zUGUd`(R$}bS^cs9#Nqdz?f#1g@qMP7YTPGF;A^5EvEa>1>V8Rm@`|B~s}u)P+Nk)3 z98;9uc)d(p?$4^}%S#s;Rgw?k1hXA~_26LDuU7x85iI(HtPPklQmRtGIDMRvWHG|Lf%L~)(V67J;9L7nsb*1>h z=Yd>S0-p0cmQ=C*uNprn?tp(&`koxlx~o{<)?XqrIMq1)DHsYT4;-A3)-7GDG2>I~ zy3Q9c&3NaFxunFt`e6|-N`LCmUtX`fYFuBkHMR2^69=f;pru_{+a`=d90WCNDlZpXz{okwwbW2MeAm0)H)B zpEvk2(5Bvoh*dqvg=f$cwsya|Mh##RIwQ0X;zxpiZ?97Si@O?{67e}pCs^2?U_2HA znPxr@u|P7PpQF|!$l8)m*EbTDDCqq75W4IuE}KDYBnU2((%1#*+Gad4a4vuSS9vnL zdB_8695_tMd=)rH0_4&HktH9+ zf@8cG)A$KlFZjK`c=6M!r>&DIrZb)$f{Sp}ormWyjt342#~vZ{%D{hmYL1`|HX-kH zoxg$Qk}hYESWJsu_{E_dv1|+1i`wZ-h6L~bi@$vQY#t;$Pi$Sy!H!OzZc2aNqW5&@ zR3m?s!xWnCNz{(ZeicoZeaU4fqi4MEgsTVw2Z7s4qvT}?7fg2kwPV%3hqeyO$alLK ztp#Z>l4=c#zU}v+PBaO(me-1*hzyBed8dVh)wr!2e{|LpNT!TDFo2|d%PPH!N%+;p^ z1M|(ff9M({Fg0#{89|p?aF<}M>Ac2UhQq8UI@C}}Lq@~LqM^dKVB5CdP)OTWru!7* zReRMqp^$lBXYtsYiv`$Lslm4Zuh9MrUa2lbLy1EjMUZoOl=Ih%O)EDT*JCSa*c|I7 z{ttAH1vV&*Cb$$#C`cbZ*+jk}AZ-)HCUl#eddF0`5&A)w;1ee0ZmOn z#R^WOjyW*hrvKOOpwC7&8)*)8zw7tZ?Db#W{!DQ9W2pzs6JsTWBdB%%I8f2yW$ zW*UoYg1b~m@mu)d%*nQ7k0eNKh1^Bb|2YJZVuxQA%>}PlAmRUZR&#z1t7gL>XYwWr z$Y1rgSK)04UzkpZ7!L~<;M}+gPiNwlcYcZM*Kpw`u#2agu$SSH(-sLj-42o+iv29c%1vwDIr>K-ZJD?cxo^7; zgs9lgYZP!#`! z+zTdi!<%qppz>gPWhK8ssw)p-N_Fd}S%>AULTU|(zhq_JDR7)=#%Xh5GHUBI$Arr9 zL#IgJKVuF~Mp`>CQbwlF6~j&ym2XIf_}*#+xI&qQ3%k4QRa7*N8k}r;ap9Az$$RP- z)Q?IZQHrF;>T@5(xWAtbUy3cHW@v~_U@}R%)V&-nuo(J~8#raFoslDwaUQ?zVp9U5 zKRF|RuW89j?-5$6ZEo0dakiETX-gmI3nh3(H%H*>mZtl}PNa&68>1&0yIw9y62+l| z)5)WhpjQ@oK6p+Ho$3^v1&5pRuodzJ8&6N2#t>C=JW#37(( z5ZYM{+aeABaM`LVaQZ<_FDm5K2blnCwE}D-MSip3f_+)2`a&Z{34gb#^}arm9XMSNXqkByajyED~X#WzDV zH`BlOPn`0G1H{(PM6r5LG#nNNk4u>+l5guJw&qdETybE=hHvW^%sOhh?WJ6=3zA4?j0)-^ zX^hKoo#BdYBQqe@a()&`b^xsejC+s2?umOPD|_eO2l!O zM=6@(9JP?F;F`?5{5gjE3~rz0NCeKhiTY|*#>|mNe$n_~8Qs=>!4%wod#ytCb-mYP6>wUSiIpW@s4kT=A z+lH5E%~h2zSQSU=K{BIkl%cy-bHZ(V@qI6oDML_J`sqb(!>Q z8agpCeY${z`rE65_PhlI@7_v;tS#E`DiP9) zj%{W`N#>o_v27bv>1c_h-f~*h#te?_dG?Y`LrW5=7`bQHDBG6GcL{nW1Luwe!{%LB zqA0&J{8MIpS6e!wo~u5xR6sRqK{p|K$Xz;+=Xn!{UVDb}Kjj(B(vv-mTN?xo>EuhN z8@3UT5cFJ7LycSpl0;~uVseDD4`E`{OmM!SiJFLakika1TJsstyQA&d+@0iC)iS0= zRL@9JJx@I$mkYznn0b*>bTBh2bAvNoU=kmMx6sD#fHSyK;_VuXk-AYYiy5z^dZvgs zJkh8DH@0pX9@MEHo)Cmg{@<=)aANEU8*12c1lRcjmTnAeFdz(}BT@g#Swik#wAj^n z1eYHoY5FImEluYgdk`C)D{#%8(KlEx$L_PC?+*zhv2B$G!$p-iCOw$kg_d@QT^f5? zy^G7PI4se?$Cn>$qu_(Xa}c=h#Cqm)U$0y8S?eZ5HJ4G_M@EA~6Q9=oYEEtV7mntS zH8#@NsFtgL0vf!}+W?_TJ;?~YUjVT7?7uC~>^TJm^klq#;li-U*NeF!toRD4;&}@W zclKQypu?+&UJ|Y4tgQ5Dd2H%DzcML;t;xhAk^(YJ>DImG-E;%36}c@M02vMdtxey&r*c_|MBtghYr!T!ngu zy4rC=Wc9&`oA(fU!BadI3EXPt1QE-GM`5|=`d?9LXlmQbxQrlaD_M(ztr!h&dMJ+d zi`M4GXu`EQa|U%?)VU&wII6Xq&0F&f)CZf{f95K9J}XGx+DSI=JYUUp|1fTlwW+D= z{8fh~L_RYNoiL9UL1=P^&2rG(d`i3g%*I3J6lRXY+yo1`9?oUixu+>XZOt&*8 z;9g=I5lwy(85<;t+8Mz$L~x>r*qJ_>7s|HY%5E2SW~^XC7=4;LRzTH{KW-=4Ze}Bg zUl=7@b&4x+QqK%#ki;PL@+da1*Jd?BUi&iZBnkh`KFwoR`ZaeYQ-7R*>YdJAJ&s~CId&Y>6Q=pm`MQbnPZP=a6hEs}AY#7yBGB3!411}n@|MJ8_Ljc+VO2=*;Pmr?eBedI#P*IZp=LMC&@-Dw)O5eucYKDsB`qmQ8F68ggv~@Q?bh-}~ zSoC^PXi+I2hFKM3W$EBvxv6gWMo$4l zd;bg!{Rs_yrqNae(aoE?|44AwUN6y-x9-6~F3A|~QX^CN9u)N1;0`O@88cWU9 zd07B#FBr>~+|vb%8f}~iEJ}turD`=}f{%Aw$O{f@W)~1{sL!^nJq!EO`}~`DiK*^G z-*7CcXdZt^Z&Kl@hM4`rbF_?On0s}}9(wOIaq@S#`_+SDT3hnQq_k$gN_nUkQaB(t zy8uuoMEdff-*;l=#U)cGA$!F{SQM@>Ee6SL$E0QPPh3C`f$ z8Y+s%a(UYmy6_{rKp_FR*_=PKx9@XijeiMd$Lf9E(gS(m{yf>EQrvh{9JAHI?PK15 z>@KnKHv=wi>PaYM$0jSzHpl}JpS*Gw@bDM5?`9M8Q()Td8%3|2qhnq8?i9+4xO3 zeZ!}A;a})IYyDkOtPwUqo2wQP7@TkoD~e&1Z}TBn{nCeGTZ1M>Hg^+)uDq*#FnU7VtYsT9{DQNay;#gCP) zQk2HCj`L7eevG#d4ME^~;U-&dF|wzZoGUNifbw-NC%5_PRzvffI@ z!UXH#`7d`^4mz*ACKCh5%aAbGEwQeJx$Mz{KVm}D$$ z$5fc%M0Vu5kubwIAtR4SXl&mfT!QJ1NXtn(lEWE_i0EBdvDsFD1NIb^S~)ZV119Va z;8FlaK4L5_PYuD{J`bZh^Z;wiIy$7M3H#T9AV20nRu@87I)LDvNZ4bj?;1Tcs0eoC z^DM5SW^dF$?iOkt`#vapv^jDr0{b^-_k`tbp25)|%4ti44AYN)oG5QOAK$|7={#@+ z#Nd=rMKx?9su7UfTBjt@?$BAGTcA7bKql$KH^ zEt!Y!KTw-I#Kfd3LBK*6du>+OP7*a@e1+&QiS;o7ilrB;{@o|4d3HBZW~pz6c1gJV~VfZRF@l+Q_bhU%|!gaU(+9#;>eDX!(buq|KR!leIekRB8vbf za_Hb*Kcny;0{D=MjsIo~{Bh|W8??%@G)Z$9p@=5E)U!ghYIeoJD14W<5nf+VsgPP|?wrkm0>BV!kSx3(oL z20LX*pVg(jTum-5`77t;AK81n#hw~|>Z)QzF43DTTY7QN-I2NJ_js2jn#jt|+g=T-m`P1cwD zt08LMo2#@p#{XMbPGqfmz17b%D>&r!$cqb~R}h9XNeOiRAL}xAbp7aMb%^ysV~1ZG ziSOj%apceSE}l_0cd&6E?je#!CS-)S`ldlA<=_*vA0-@Xky1^5a_dno8*iFmcyiwM zqVQM~lX8^w7gFPh@_==0*2iOhEQS+A!<|anrnx~HSPgwO-**kSCM$`_waG~m8X{r} z|0j>?pTt;_qJTu4dTlNaVq=KpRbY#EM?}*e2;eo>5Uf@?^}PHmZ|NUJ_)(Jf4Kum* zSR+l$wSJ{n2B|4+nXxg>XMij-X6T}c8C}xaM={Q8Wb9a;^XM}ehT}YZz!TE%!Noc_ z984RDvdiI^Urnz5XmY3#-4>xk9kA?TU(q}Ud#I$}qqNJVH z4q>OBY}33q=6`bN{!#o3*=_{hYnKzMcZgh1)vIz+ zg5lYD+mpikE$Ey`A4@tf(ukJ+h<^T~N$V6&7&H0W=Y;A?HNgQ{xbV46OPIBBG9%2+ zm=?NcqSX6i5pOdS+wZMz9(-8oyQ{1Hjp7DvKIvray4deXoU zO69#>KYL>4I0hd$L#N8FV&@q`nE6I1M%W%>A|tHUD7Pz28F)8m!AAqmjP3}cMx3Hf zDu0(?H}x)-Z>4ufk`ZQfZX}uN`FQO4M^o%5Tl6+4xzz(t6i}1RHAPURx8J;CXWOLI zku!Gf#VI3UcCMV(&RO4&JqcfXZs@nMGl*o9^h$-MRVhHJ)}rdIkRKL)=bn_wm8B;Y z1aoEf)wF*_SvZ=0H0@PM2fIxz3EVc6-Of;bHBO2@HM1Fp6Ao{t$}_M-hR^%eudIb0 zst-XNssr-L$Z3?VAax|v&az`~jH~%yxt;$+mstbs`W(BR5g;X0CD{zxg0PbhxkFlPsKQOu;zRV&$2%3T*R0G88$N z;hfZZ4q}}bPBx=c!c}JU7k_3IyU>p${n%0Pv8x8wE~6~IMnB{}{!B;p$C3~D*6(2)^%Khfo;LLf z<^MGq{}Yw}w`849g5|&EA$$@n|1BT$lVJHISUy=S|4}&f$zu6$rDmUu#Q#>v_sK5* zZxw-`T%rG3k@!ild=f059R9x-Ab+yUKiTDzxN6I6ukWuy#16^@hPbSG!*=&O#V;m z^OO4g|Eu~`m8rBt)24*UgjkSF4%%I|Af5WV?TH?Lmg=M*T4Z%CVkrc&Vp?$v)U&*8 zaoBY)nH;qcOg8+<$)HRdpB5!qt{Iv}#m)_xHrazTYe*AZA{r`2N~C|5&sYit+VY_+ z(F~nn6to769}g@U@^KHW{G%!6PGLL%cQLb}RG7!t&^c>0i3a$g#WSt|)Ww+;N^DI5 zMuYsbiYE;t)oL!dMo*s{1Il1crsu-Jn2sQ_=}PdlWybHY!6z4cS@yEzc61$0}`T_g&PlDrfp=~|DMsXJ*6zo03$A={Ry+O z95cPR!Io=bstn9J0~&_9rN6iu;hEn3XV=iEC@E}r)~zVYAGpZ+3m0_(Qa#ctO=22# zDB;ib{!Ud4&7Tz_J?-ol;0$YLH4K*pz9Ja4_Q$1U+eKZbfl6vloiPl#=y+M^FCmm!aNHl z0ijMaY#`vVarQ7-B-siK`8r?-)4?6A4!DGdgkyDuh)R?YKg2O@-$vx3lbdS)=U7ub zi~b;UzDS@s-e5aXX)5!|dIo@8l-zYB)byp{X~mi}%Yizam_4LV5Qvi-|5*NP1c|7! zRYo6`Na|kds_=UC*dnf$b-{hBiH+=J)_Kqxmlz=V(dvKBx6cw~i=DmTs$kWt=>6ao zgAcdhR2QnDx;;PLmjcIwvJNVa%F_{#n>FrgjAYUm;W9R2YIHaKSO85-yo=jRDaxF;w_-zqFo{RQ@#p@+PT_mQT?{lVnpg1YCQ< z9x!eMHjjdF{935vXB%R@Fji!X13MAd)q$Pte#;$*D`-gPo)F5$j)5NP;lu2B4&Q0)d4^0i$O(<>k&H~7P{#($SUvj+D>!#KN0tgrz23)b{7&& zyEd#zx8f&oiDbwUB~jK*nw}4m4l)`HWGs8r8<^!_Fe|r=Xn+xwgO^D zD7ZQg!R{l@OfR^`jRLty+O4=xc@B&Ru2il+dKBj{-9U}6J{Yb{X!n1u$%0t*;j z(|$4lG-S>I6{Kg=P|+87w`!lwW=I{4$8jDF!?9Y_q5p2w1UMxqEu; z)d-LDw#j?oh#n#BaZ1#hcGv1C!r0*AFt!eBA>^|BG(5hhA&o*UmeDZ`zQMpw)$^A` zeLXM=2)eq<1Sy%V`%On^ETBGMYudL<2_o;X(Rq+TMT%#z4KXFXFewP^c+DJ{(xGJy zseYcK@S1Hxb*r^Tvbwr7NQt6uIJY0rDfEdwRw~3>6ZU~YUtBXZ$hc`7s8wt4gUR_e5x>8{mDSM&XiQ$$ zo6F|HO;q(0uDY6r;mtY~T?Lm+RS?rCgJa}=|j@I??PwQDbK^*AE| z@_O)Vv;%0`eQf*y#z{rT0P1MIkfZ?QgNU7aYf5M7kg*EIVlW6!{VawFO82_Z9`1nJ z%|Q4RE5_wq%LOa}dGEs_b+HkWnYs)ezEIXQm32e(Xr+*7*AD9OvJmN|S0mtKv}gY5 zU^}kksb&WFZ(5Hf!pUm0Fp2Iwbf!ZEDb0~uJl9uQUazh0G)|=v_TY6}eLm5G%$D%W z;+ZKvw>M1ed)EdvUERJ5rtWnfUiKN}IV)v#unmUq6Fr`XHV3=u+zMz^YY;#C^URaQ zxe2p|0LJ#5vgCY6%9IeD!l8N>JE$&2=IdmvrGTT|tI{3H#gPpo$kXm7(;bu1gLt>c}7pS@1$+GNLGGV4niWVy9RKF zcnY*@u1WhIm{dU@b^yDRw4B?WRSnb%)t$Zr$j(aY-79?KvhMVZM^C(*C8Zx#V4O~)jHT%kl*k(P$1 z6bqP~S&5T22oojM^5znNp<*_ZsMDIuQ`ENn?f`&G4B~|jm#6G;ivbJSLSxz*65SjG z`j-JmixcRC3fTnHNtCs^;5~TVQR4uO7UB2xt)GW#P!zmV;3p%dUO(WQj*qv<-Ld6_r zI18NJ3|oxj*zSZ|psnD>G_bWuV}8Hn%9u_q-PskPZpoyPW$*J~*CK#*49*}KCduRG zWJ`o)X=!S5@#ILPyeiH*-wmm1qG%Fz_F;*;o0H*K(w54b)e5Nmczk zQ}s{`-nHyGf+Uw3M?Le9E$au8B&|Riq_X{ zb3kaM@~uVXQ|hV?+nflgaAgZ(`zu^c|uh;gjLYR9N+gZ6;e z4Pu4berYNW-CLblmdxx{HNC@kF20I-)eEIgBqkaVG`N{xz`W;eg;NayP*GbL{E zA($Cm9qczd#rIR%})c&iE+wua{2T0iP z z04S51JrF-!8<$3zNw7IQ4>G-Ypm45n{p|4}wC+7?yDaJr)GpRIHLT0UovgAzAyw@! z42I2hPF#e$gj;5vkFVRJlz?ek7sQMFBITuGAGV{6*#Hga!wI~^M5UAh<^t^IPt^wq zk0_@*&!VWu;HLp}Rz-QWR(6Z^G(w3}Iq#YWx!Q95ZeTtz3g|rS@2`JjQEjSvtaH0I zw*@vP3b5FjDz{j(_DBG6yK{h>r82xX4=&NVvZ{HL0|y5T#lzVHYOS~ClEKOjntF0r zpro0YZU?h{_6KxWC_{SV61#KA+Tl7oApF{jLslx>y0RiW;(+0}6ylrIa_(;6lbwgV zOF`$vL)ImRWQiWE7A!K)A+N|MdhB>&vXiR8h68x>TEW2UP2@F~FoFiTuGA{}fnkoCKu2z)$T z?WAzYs)RsBAGPj3?&Vab`qz3e5U0vBkt&!m1dPq)=*#UxO``3q|a1WoPTp zWNx13s)vrr%?0SEc}ypski9IR1fuIc&vm-1!(sm_{I}70=k;1JEGIaSa1eA zL&Bn#X;U3*ieG{Kt*I!-O%LpH{u+ZH=Y_3Y0I6A>!rI)n|3G^BqznkmM@_x8{7G+} zWpXHEM1d6NkStBeiUu2<_I!Ten53_4?1{RXG{vxoOE_Ne@x*~5oRX%4N^~`-5E`LX9hxo<>z|>c)o&F_dQp<$wPCf;_5U_gZlI%I_(^Hh$W}7t77RIHe^-kb} z6t!tjDcJp?8&%~>IZS8NHam0}|1Jm$ILhuuXA}9+EIqKT`vC#D7P#}Q&@Q&ps9x!o zimxqhw{{zd!wtTU(??30pq7H@>2%qC-HPwEIC)7rD@Z0`+`(V!M1ZW%G+64K{jjP@ z9Xor)VzrCITmc?lw_HCSn7wNO&>hE&b>B9LoL7eyKKNb8(3)7qWAbVgqF*A&UkSu1 zW+Y4J4O?tg2DAJbPMMN^Y6w(A>ksdHfhs5vwXwK;5^dZa4MT{B#CYU2MEf&vRe2n3e+@PuvMi&Bdxk69p9!r%ku7W!tJkKoQAI{ zZkinznHeB^9mi>_b8ikZ*$QQ1HBr&1*|a}-)PGW19B zt5R6E9Rz$X@>B3-Qz-#77hKRgeo+9isGtyRz5SVYlxdxHJE|K+5H}<;iGYNQ7eG#X zrk`vvzY~~Mxl^yZ6tv(uWW5)P}dzaY*Zk?eyxL4<=;wy`bb&T}@M80TLAS+Rg(Sbt66z=wQ~BrkY?&NDT4l4EB^8aLZ9P^QYa!z0O2 zCdwJnk*|VH`LQ2vnOZt2rVZ**^;-3G@=60IRa|2noLHl(RB1&Wa4oeSA@olSCz=Wc z_5DP-53fVoEC?PaI!^EwUR;@iCr#l4v}QP8*5a-lY+V^qKrETcL+{q%N?*`JiAPog z`DQRnjvDXjTfP#`p7}7GvzYd;p#8ptmf%e8?sh+QT~M@fq_}Q8IgNBDRXwxbfP{3$ zmyij)TSSddgNp@J!6hs?mnBj(T~3gAOf0^d4}&s7V;g-5KmK`Ne-H>1VucM7`@ygud` zHNR5p#Pky*Sz_Et(io+T*Kr&vl}nHcTI4ACw2CS|8-hut}D>1`39{aB6DY&x*Z}cKQ>>;pn*L z7`X>^_2X)CVOpw)w*XEo!43|_Cvt3MFFNC;i9)DANT6sR3@bs5`9YUj@j}4VS(*bR zj(6Qow)q7;MhuVh%)Vk4s(Kb)CZjaPxc+-bmz&K#J@pDhhaqd%|)?nSkes1?{2 zz4F)Phq(s#tz5dlZU1)WGrRr_71D#FGkZdtIyt2$!eI z$_QpBb0vmQGW$z6iA%B~@)zeI$pUo&3Z>NK8o0D5Vi4B&fyJCqmWgMiNELNTf;nvIWF66j2P+l}SE|y;ZB^hH5&8;e>+YL^XFf z2Wp|SwTj8Wnn#hx5*ndVH0sn5H}l_hAz|o-n>l4pzi|Y-CtSKBF_)$+ZS2sFjxe;} zV&vj6r1t$VDM}Z@#L%4TusX?wj19_Hii3o)k>Uzt-$l*>*n^RXT;r zcax}^u>quJk3><^;_m}vw*x`m&DC0V&_%V+I*vvI9@)*6dplyda`O&;%ld%`M2=GA zI3B63k40(DO7Y{wWP-+;Cne3^7T`Rn`2|A$B#x}1(#}0o);@);uC`25=)TW!m^0+~0b%c^>&jk$-JJ51ln#2l4Fo>iXP6`vn!ekNdv5LKw z7Aj$jPf2m7#kk*8nz|yT`UHWL!igXeI*u1`;2$bOtK!(B26(p z^oQcGXGZxgGiRgwCp8$`q?8es;(e7v5B{NvvQCR2SkN27eSA#T!z%5NsUxGu9FwTC zT^uC2K*?eFpTo4`rTK6=TV!pUIx>wb=njiPw-U5t*95DO$z$C2$fOx=4NX(2vb{i* zM?6ram#C~BYNC&7`?2@SRS}P+L5H;+Swnh$ji4pza;S;)(kjr1M~afGQAvln9-c`9 z%<>9D`})P&GQe3{;J5q)l-H4dH_Bv$mJ#BFBg3k*ACcKVMTyGlQwJ>DZnrW;yC=oL zT?}KQc+ovE8-+|pQZ3(XHFUN_#TS$IrZK;ECpnT(L=VcM(pbbp^1+$f2Bt*|Zxt7P zgB#3XFwl>{q7Nf8vMNWWe zdq)+u+HTO(G$@t1lP8^+-rd?Q4O2T>a5$ojfH0XC_jqDoqG0R+`JgBA&J@kklaSOB zeWg42mKhiQS}gyyfoWQw+ZL{QnAP!=Wa}gqHwCQ>MdqxCj*=^eMU{W3!|gY%R|AYc__1DPNeixwT~5g6c5#YG{!x>>?FG`; zJz%4r7E0Y@^H%9&XsqnX?`zBh-nN{%}$<;d~ffN%cal=Tlt0hzP1%l-h)BF^|5WyV#17A7%vX)fVar6ijPm?%sw*durWDM2AyF{nFVA5q>cmaLbPHjcMGU=Mg9~JY zh01bR;^2O+x{Nn6s=(G!1^2V8yij*as$x5)DtU(Pbe@>UDPTT_)5TodXCf*~k;Cei z|JDH0Th6gos$>*Q;4o zDIqWm+*uZ!H~HJ_mS!8i*zFCpKs}#(-UK zxsTF(vlV6@kd7@;#tr5T%%d5-VHk8?xaDa&mFzQKtrF82$&{+sky@O5^cvT68<#0C z$OK<4P_(TqfZb0`bdd&Y4N8NF&%03uY4^HKFBfbaot}ROlLq^4nMmJ=GW0NO1I)+IZs4af>KkFP629QD_K0yX z!y5B$Z~ihT_1@Fyy>DZDW9rK79uUF1&;GL2aI-P8@SVl>0aCnP>*Vkg@^d5n_dg(g z-9vx0DSK{?m)GjKczP9EY%gehOnkr@`p(<%9(V1=x|7ww+=G7WO?%+yxQmpQCeHlP zoxxD-qm_{huTFE*!$W!RGBGfXoh(^Vc6dw4*(JWom zRv^AoA|R3iQ@Z=hvQWj!#uAb_(sS$?_+Aoow5XdDKkE^F^xW=zF$#-BI8bx=Ik*|w zKWjLU}Oss&VwX!uE!> zIU!~GTaQ-_HAjnY?vv! z@c>Q2wl|Ipz|1xu>#2WTX??B7@I?zry<#WN`bMf_klJ*Qt>WAFXXPf;&)qc$R(#9y zpPexB-WDdp1jX9h(*6@QlpFL=MNY{b*_r!PY>5IiNVb>Sd%@vJmt06!iM zJ_csp7P)z={qfqLNK;D!ETOXz(R^}iZ9s@c#6+d`A}Sq0qT~=|85#vzC3f*a*27{4 zsO@h9ii^v|BD8`VJjX#%HgSLm%X(!QYW`bsEMA5W@; zVv@Uqrx*i|rnVi9-z-ie~rc5 zQyTtItc^o++tzb;PjM{~=2BkwTDMe^C(}_iJ-@kNCg?NY;|+TE;iTCjZNp72w!&`H zW9FYS4Dg}yv2ZFRMjX=v=GMb_KsDC2e$L&)b!r*gg8E*r zV9Xu|W~|x;wt4f`a+mZolUSf6H#p{Jo1tVVbi~MF5Yf^0Tu3FG1i-f1JF|>ioPG*a z;ehiZBF{yzSM;^LEs(w{X3u7@SK#^^@tGQ~A%od7$vbQ2?b8qMz~uHd;<@4)0aJWn z8$;R{rL89}*Tz~3yeag-_le57fwKjP%1K@fgDDAl1*YOnyb=A?F$RYSq37^7E~we> z1w>P-bre2~GlcUeQD9X9MuE1*OMUpOI${+;zf_A%?^gQpz#+O#(U_yXIR8X_0G?W% zuhqO4#f(%kji5b^c*ypuI~?nJJ0eD+*9*STv4%hH?(RksMh^cS6L1=_o7cIvGBzsZgtCT8k@HUik z(y@Z$B?#r>;DnVe9iH-NOPX`1u%Y}n-1y=aq!;|>8v~Os-^U|Qu%E|9hiXli(YAs( z$c58d(?ZLO1GIsKB_L2^SfC5pf+(+w9^DRtM{eNqv~6d@yRd^`(5})OL_Wa`;hZT+dl|9k#Lvg0IDQo8%flK8lMm*Pnaqb zyrmLw17h`i$eqRuY$M`NX+Xlq#gRzdhxU+aw{M%s{rG;^q(%*!avG%xL$n5UeQL~rRvXG7akXhgqy}G$sKv``6f@e2epD?-K-4CR zqFctya;sxC0)Gv0K&xR6z-CeRJ{!C=ApyZ=zJQiL6^Gt^7sOB6SlUx#EiWW+TL`JyejT? z3A%%vaBb*S@z8$>z^;_Z9Dr)|6kID8-87MKtI#_%x^9&KWe@! za7pK2LZz(^*s?9F?&5w6X`_s>BkJ>&k~v!xG2hu{i-MSkgS|K;-iL_G?<6>*IKzAbtb>(St1-~Ys{xTY-KP~EdC_t7e~tJ5{~i7cD2;a3SOHoIoouu8t| z41co9^v71aOZTK9!|xs&%kztChOgTU2$UNbRs@bIhJN7_NmVdr>qcizC+&cf9eCoxp*$Le8`sne|PQq11 z*Sp+(+6@s-?KSk0=(`dlh3ON=OvU%(CXY0bI^}AjDL?((h{-w4KDA{eMr6BS!1#LY zjp%%TjM<**pF$ow-O#|SLbmyb+{ZhLw_dMLvMqiwdbSt-R?O5tru{-rcF#qoD5$Sr&2q3Enos=iur?>SK{CEef>I}7w1ALi5!;{U*88gX9TJ63IKNJ(^5}WqT&ARMRdG;h41i}ct z;|E|TIm=JM2(3{Dr_HazTCgdbsw#=aUvXy%H-on{v0Sv@G-M5k-9yevg16kQ52P$n zmgK_Dom)~DDQ)6spI=Qg?33;Fe^3-r?KG3KnpWmqyitbqr7{2Ay_9l1b^C#8h*KBUzL5yMp?|*-lD$9i4$_KuEAzh zxBBVvGYa=?yULLC{)cl$DQuIp?qR>=UkD!b#+Ldu=`z*sEXy^Th%3XyblkE zf6{r+7`-ggp)TAfLvfc@27fs}-pgPb#V^nksrKMk^#0i>wH>xjm11p7cEJ`K_n0 z$8ry?+p8y|DG;*9Ik+get-NUOd!wM&HCF~dL#(a!i+ou8HNxr~@2gRp6t&F}=2v^Z zN8=~Fx^~C6vFbFfm9KXAXt1r4O4G=5`-iVeRWmo7`?r6@l?o=1bX>ls*G?%&7#;$+y%53=Qp%D)awla83Y zocW5jB>%{cUIDTqx5V@0rI(AZG%vOK%EZj*=)H#*Jch2nG`iv#HF1LV!220m`NXNH z@5#O3=D{LUqSKPoMMd7*7Jcp7>u|bd&&!Kfl9#Ui`e1+e+RBJ)2}=#WE;cv+*(0Lr z@pl#eUa*l99BW-Dm%FQpwl{d;xC7x)?e+#?2`@+?p#XMh+c7yiaye5D3 zDwmz-0*bERcluJzy>5J?&13Z3_o36yqlIr^zi7tE>)d_z z#LLD8l43pJK4Ag=Df*fH&a@YsN=sE9FrU2x)BbOvJMKBBn69-5jJ*WEW@;9@-^g>u z|BCmu;DFM@qdAAwo=uJoSVpqNWw-5uFvVqmr%RSYua|Z#3wrBnQ#OQuCk*I!>0Q}Z z7=Ef@^>vzY{Tk=0F9up`GRLirJQbm@p0wP0Ot7d!e|YU4_9n@;?B)K%;nz(|ocC0` zKfR9odQrnH;has`ct=Ci@T05i*Jj<&m`8%qHP&7^=(N4-{{)Q<#Zp+e+s#sP+eL`sA$X2p_%-pKts1r&_$pT^*2-A(aQ)nX$ptAj<-1-H4Ddc=z;6RhHpI0`wdWjJ}sH}6iM(nI@#l8O~9N* zKIKx~P}?!09>Os+%26BHJQIir3+UQ*RhcAQ3IEEV1@=b0wf2E6%lxza1Flz%<}1HI zH6z1sqF3kzmHrfYCpxL@^MPXnkh$+WhUL%AzvbNCGM86uipi7m7OW2WZ0V`k%yDNm zYHo7^;lW1uz3&cwmN_MuExds@3H?&+t=ZDk3ix^ z%olrIPTY0zR$Ni=cgw5wkzjX6oVQ}1ZB*<|*(7{=*;@Tm&tmA0*H_HsrH;cDs$X>` zG!P-po7ST*-aBMcebECis)6j7_sfaz&OP7o{q4eees`jtxtX{vxijDB_;)T+(XQzJ zo11yJ=?i_s4MM-Tzm@hKBnUjCidTFHlN-u@II-PDv$*f+FDGT2qiH@18$QqqmtJk| zxJ8>y@by55YBu^V>i!Tm@0F0k2Fs2>_s<^oEQ$g?O<|c3Z?(~Y~GUV1@lhIidzv2Uw-K8)Lj4W{pQWdvs=HJ zI@yHDauAR*972%OrOTT~hWviq9c`3-#6$`|Vq*P4*nNjCetA2pd1Um;rr;wcW3%gf zmwdjG;CqhWe+|9z=x4(dd-X~`54$|i+*Vs?R^*A$Te$)Am#Yyr{Nc`l9ryf9DJA;a zGX30R`;AWy%U)e|m~#?O>ZxRRJL--R%r%q5r_3)mtoZh}%6$}@C|?%h#@RV1X^fTi zT(7>o{??794@?e1ZxR!;@$+)U1G(R6SM5o@-*xz*$wP-D0OH=-wqcuH?KR+eXxZV?^Np1fE}9#)$DCMJl=g0G@yd{F zzn-nEgJHG!b=k^zCqO~52MQ!F-kY_rIr-&%Il>z2 zUH{qmXj<9z%cOqen%;fLN`^7&9tU9hSVYN{CtlnI~2I)ixv?r_I8RvLgx&I5d z5*y_*!mNv6mbtkW>VXT-t?A*l+z>MgHcGo|N^>{WlG-*(ZU21ZT@`Ufa|*Qld={^P{RW@dv$O_7d3zc*}>(W9}e5=kDFIfKp#4R9)4!G_w-sgMi{{UD1u%-Y2 literal 0 HcmV?d00001 diff --git a/website/content/en/docs/main/tutorials/nginx/index.md b/website/content/en/docs/main/tutorials/nginx/index.md index e814794fc..e489420ba 100644 --- a/website/content/en/docs/main/tutorials/nginx/index.md +++ b/website/content/en/docs/main/tutorials/nginx/index.md @@ -194,37 +194,7 @@ Verify the job succeeded: kubectl logs jobs/curl-nginx-homepage ``` -{{% expand summary="Sample output" %}} - -```sh - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - - - -Welcome to nginx! - - - -

Welcome to nginx!

-

If you see this page, the nginx web server is successfully installed and -working. Further configuration is required.

- -

For online documentation and support please refer to -nginx.org.
-Commercial support is available at -nginx.com.

- -

Thank you for using nginx.

- - -``` - -{{% /expand %}} +{{% readfile file="/static/files/tutorials/nginx/nginx-output.md" %}} ## Cleanup diff --git a/website/static/files/tutorials/nginx/nginx-output.md b/website/static/files/tutorials/nginx/nginx-output.md new file mode 100644 index 000000000..c050df241 --- /dev/null +++ b/website/static/files/tutorials/nginx/nginx-output.md @@ -0,0 +1,32 @@ + +{{% expand summary="Sample output" %}} + +```sh + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +``` + +{{% /expand %}} From 0d5bf003fb52116d0c6ff2d5c3b9be9c20d16438 Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Wed, 22 May 2024 09:49:40 +0300 Subject: [PATCH 29/53] TLS Refactoring This commit makes the following changes: - Move peer server TLS termination from the controlplane to the dataplane. - Change tunnel establishment method from POST to CONNECT. - Peer authorization token returned using a header instead of a json-encoded http body. - Change controlplane ext_authz server from HTTP go gRPC. - Remove SNI proxies in both the controlplane and the dataplane. - Support dynamic peer certificates in the dataplane via SDS from controlplane. - Support dynamic peer certificates in the controlplane (not yet tested). - Remove peer CA certificate, and instead use a new "site CA" for signing controlplane and dataplane certificates (used for controlplane<->dataplane traffic). Signed-off-by: Or Ozeri --- cmd/cl-controlplane/app/server.go | 77 +++--- cmd/cl-dataplane/app/envoy.go | 23 +- cmd/cl-dataplane/app/envoyconf.go | 220 +++++++-------- cmd/cl-dataplane/app/server.go | 20 +- cmd/cl-go-dataplane/app/server.go | 53 ++-- cmd/clusterlink/cmd/create/create_peer.go | 58 +++- cmd/clusterlink/cmd/deploy/deploy_peer.go | 13 +- cmd/clusterlink/config/config.go | 7 + go.mod | 3 +- go.sum | 3 - pkg/bootstrap/cert.go | 44 +-- pkg/bootstrap/crypt.go | 1 - pkg/bootstrap/platform/config.go | 11 +- pkg/bootstrap/platform/k8s.go | 56 ++-- pkg/controlplane/api/authz.go | 13 +- pkg/controlplane/api/heartbeat.go | 2 +- pkg/controlplane/api/servername.go | 10 - pkg/controlplane/api/xds.go | 8 +- pkg/controlplane/authz/manager.go | 87 +++--- pkg/controlplane/authz/server.go | 260 ++++++++++++------ pkg/controlplane/control/manager.go | 5 +- pkg/controlplane/control/peer.go | 39 ++- pkg/controlplane/peer/client.go | 34 +-- pkg/controlplane/xds/manager.go | 70 ++++- pkg/controlplane/xds/server.go | 1 + pkg/dataplane/api/servername.go | 28 -- pkg/dataplane/client/fetcher.go | 31 ++- pkg/dataplane/client/xds.go | 29 +- pkg/dataplane/server/dataplane.go | 174 ++++++++++-- pkg/dataplane/server/listener.go | 76 +++-- pkg/dataplane/server/server.go | 141 +++++++--- .../controller/instance_controller.go | 17 +- pkg/util/jsonapi/client.go | 10 +- pkg/util/sniproxy/server.go | 77 ------ pkg/util/tls/util.go | 45 ++- tests/e2e/k8s/suite.go | 1 + tests/e2e/k8s/test_loadbalancing.go | 10 + tests/e2e/k8s/util/async.go | 7 + tests/e2e/k8s/util/fabric.go | 39 ++- tests/e2e/k8s/util/k8s_yaml.go | 2 + .../content/en/docs/main/concepts/peers.md | 2 +- .../content/en/docs/main/tasks/operator.md | 4 +- 42 files changed, 1084 insertions(+), 727 deletions(-) delete mode 100644 pkg/util/sniproxy/server.go diff --git a/cmd/cl-controlplane/app/server.go b/cmd/cl-controlplane/app/server.go index 459a02268..7a433914e 100644 --- a/cmd/cl-controlplane/app/server.go +++ b/cmd/cl-controlplane/app/server.go @@ -17,6 +17,7 @@ import ( "context" "fmt" "os" + "path" "github.com/bombsimon/logrusr/v4" "github.com/sirupsen/logrus" @@ -37,10 +38,8 @@ import ( "github.com/clusterlink-net/clusterlink/pkg/controlplane/xds" "github.com/clusterlink-net/clusterlink/pkg/util/controller" "github.com/clusterlink-net/clusterlink/pkg/util/grpc" - "github.com/clusterlink-net/clusterlink/pkg/util/http" "github.com/clusterlink-net/clusterlink/pkg/util/log" "github.com/clusterlink-net/clusterlink/pkg/util/runnable" - "github.com/clusterlink-net/clusterlink/pkg/util/sniproxy" "github.com/clusterlink-net/clusterlink/pkg/util/tls" "github.com/clusterlink-net/clusterlink/pkg/versioninfo" ) @@ -49,9 +48,6 @@ const ( // logLevel is the default log level. logLevel = "warn" - // StoreFile is the path to the file holding the persisted state. - StoreFile = "/var/lib/clink/controlplane.db" - // CAFile is the path to the certificate authority file. CAFile = "/etc/ssl/certs/clink_ca.pem" // CertificateFile is the path to the certificate file. @@ -59,10 +55,14 @@ const ( // KeyFile is the path to the private-key file. KeyFile = "/etc/ssl/private/clink-controlplane.pem" - // httpServerAddress is the address of the localhost HTTP server. - httpServerAddress = "127.0.0.1:1100" - // grpcServerAddress is the address of the localhost gRPC server. - grpcServerAddress = "127.0.0.1:1101" + // PeerTLSDirectory is the path to the directory holding the peer TLS certificates. + PeerTLSDirectory = "/etc/ssl/certs/clink" + // PeerCertificateFile is the name to the peer certificate file. + PeerCertificateFile = "cert.pem" + // PeerKeyFile is the name of the peer private-key file. + PeerKeyFile = "key.pem" + // FabricCertificateFile is the name of the fabric CA file. + FabricCertificateFile = "ca.pem" // NamespaceEnvVariable is the environment variable // which should hold the clusterlink system namespace name. @@ -71,6 +71,21 @@ const ( SystemNamespace = "clusterlink-system" ) +// PeerCertificateFilePath returns the path to the peer certificate file. +func PeerCertificateFilePath() string { + return path.Join(PeerTLSDirectory, PeerCertificateFile) +} + +// PeerKeyFilePath returns the path to the peer private key file. +func PeerKeyFilePath() string { + return path.Join(PeerTLSDirectory, PeerKeyFile) +} + +// FabricCertificateFilePath returns the path to the fabric CA file. +func FabricCertificateFilePath() string { + return path.Join(PeerTLSDirectory, FabricCertificateFile) +} + // Options contains everything necessary to create and run a controlplane. type Options struct { // LogFile is the path to file where logs will be written. @@ -111,23 +126,15 @@ func (o *Options) Run() error { } logrus.Infof("ClusterLink namespace: %s", namespace) - parsedCertData, err := tls.ParseFiles(CAFile, CertificateFile, KeyFile) + controlplaneCertData, _, err := tls.ParseFiles(CAFile, CertificateFile, KeyFile) if err != nil { return err } - dnsNames := parsedCertData.DNSNames() - if len(dnsNames) != 2 { - return fmt.Errorf("expected peer certificate to contain 2 DNS names, but got %d", len(dnsNames)) - } - - serverName := dnsNames[0] - grpcServerName := dnsNames[1] - - expectedGRPCServerName := api.GRPCServerName(serverName) - if grpcServerName != expectedGRPCServerName { - return fmt.Errorf("expected second DNS name to be '%s', but got: '%s'", - expectedGRPCServerName, grpcServerName) + peerCertData, rawPeerCertData, err := tls.ParseFiles( + FabricCertificateFilePath(), PeerCertificateFilePath(), PeerKeyFilePath()) + if err != nil { + return err } config, err := rest.InClusterConfig() @@ -171,27 +178,26 @@ func (o *Options) Run() error { } controlplaneServerListenAddress := fmt.Sprintf("0.0.0.0:%d", api.ListenPort) - sniProxy := sniproxy.NewServer(map[string]string{ - serverName: httpServerAddress, - grpcServerName: grpcServerAddress, - }) - - httpServer := http.NewServer("controlplane-http", parsedCertData.ServerConfig()) - grpcServer := grpc.NewServer("controlplane-grpc", parsedCertData.ServerConfig()) + grpcServer := grpc.NewServer("controlplane-grpc", controlplaneCertData.ServerConfig()) - authzManager, err := authz.NewManager(parsedCertData, mgr.GetClient(), namespace) + authzManager, err := authz.NewManager(mgr.GetClient(), namespace) if err != nil { return fmt.Errorf("cannot create authorization manager: %w", err) } + if err := authzManager.SetPeerCertificates(peerCertData); err != nil { + return fmt.Errorf("authorization manager cannot set peer certificates: %w", err) + } + err = authz.CreateControllers(authzManager, mgr) if err != nil { return fmt.Errorf("cannot create authz controllers: %w", err) } - authz.RegisterHandlers(authzManager, httpServer) + authz.RegisterService(authzManager, grpcServer.GetGRPCServer()) - controlManager := control.NewManager(mgr.GetClient(), parsedCertData, namespace) + controlManager := control.NewManager(mgr.GetClient(), namespace) + controlManager.SetPeerCertificates(peerCertData) err = control.CreateControllers(controlManager, mgr) if err != nil { @@ -201,6 +207,9 @@ func (o *Options) Run() error { xdsManager := xds.NewManager() xds.RegisterService( context.Background(), xdsManager, grpcServer.GetGRPCServer()) + if err := xdsManager.SetPeerCertificates(rawPeerCertData); err != nil { + return err + } if err := xds.CreateControllers(xdsManager, mgr); err != nil { return fmt.Errorf("cannot create xDS controllers: %w", err) @@ -209,9 +218,7 @@ func (o *Options) Run() error { runnableManager := runnable.NewManager() runnableManager.Add(controller.NewManager(mgr)) runnableManager.Add(controlManager) - runnableManager.AddServer(httpServerAddress, httpServer) - runnableManager.AddServer(grpcServerAddress, grpcServer) - runnableManager.AddServer(controlplaneServerListenAddress, sniProxy) + runnableManager.AddServer(controlplaneServerListenAddress, grpcServer) return runnableManager.Run() } diff --git a/cmd/cl-dataplane/app/envoy.go b/cmd/cl-dataplane/app/envoy.go index 6944c6aff..b897c7a8e 100644 --- a/cmd/cl-dataplane/app/envoy.go +++ b/cmd/cl-dataplane/app/envoy.go @@ -18,7 +18,6 @@ import ( "fmt" "os" "os/exec" - "strings" "text/template" cpapi "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" @@ -29,9 +28,8 @@ const ( envoyPath = "/usr/local/bin/envoy" ) -func (o *Options) runEnvoy(peerName, dataplaneID string) error { +func (o *Options) runEnvoy(dataplaneID string) error { envoyConfArgs := map[string]interface{}{ - "peerName": peerName, "dataplaneID": dataplaneID, "controlplaneHost": o.ControlplaneHost, @@ -43,10 +41,8 @@ func (o *Options) runEnvoy(peerName, dataplaneID string) error { "keyFile": KeyFile, "caFile": CAFile, - "controlplaneInternalHTTPCluster": cpapi.ControlplaneInternalHTTPCluster, - "controlplaneExternalHTTPCluster": cpapi.ControlplaneExternalHTTPCluster, - "controlplaneGRPCCluster": cpapi.ControlplaneGRPCCluster, - "egressRouterCluster": cpapi.EgressRouterCluster, + "controlplaneCluster": cpapi.ControlplaneCluster, + "egressRouterCluster": cpapi.EgressRouterCluster, "egressRouterListener": cpapi.EgressRouterListener, "ingressRouterListener": cpapi.IngressRouterListener, @@ -54,17 +50,8 @@ func (o *Options) runEnvoy(peerName, dataplaneID string) error { "certificateSecret": cpapi.CertificateSecret, "validationSecret": cpapi.ValidationSecret, - "controlplaneGRPCSNI": cpapi.GRPCServerName(peerName), - "dataplaneSNI": api.DataplaneSNI(peerName), - - "dataplaneEgressAuthorizationPrefix": strings.TrimSuffix(cpapi.DataplaneEgressAuthorizationPath, "/"), - "dataplaneIngressAuthorizationPrefix": strings.TrimSuffix(cpapi.DataplaneIngressAuthorizationPath, "/"), - - "importNameHeader": cpapi.ImportNameHeader, - "importNamespaceHeader": cpapi.ImportNamespaceHeader, - "clientIPHeader": cpapi.ClientIPHeader, - "authorizationHeader": cpapi.AuthorizationHeader, - "targetClusterHeader": cpapi.TargetClusterHeader, + "authorizationHeader": cpapi.AuthorizationHeader, + "targetClusterHeader": cpapi.TargetClusterHeader, } var envoyConf bytes.Buffer diff --git a/cmd/cl-dataplane/app/envoyconf.go b/cmd/cl-dataplane/app/envoyconf.go index 417c0d561..51d2f590d 100644 --- a/cmd/cl-dataplane/app/envoyconf.go +++ b/cmd/cl-dataplane/app/envoyconf.go @@ -17,7 +17,7 @@ const ( envoyConfigurationTemplate = ` node: id: {{.dataplaneID}} - cluster: {{.peerName}} + cluster: cl-dataplane admin: address: socket_address: @@ -39,7 +39,7 @@ dynamic_resources: transport_api_version: V3 grpc_services: - envoy_grpc: - cluster_name: {{.controlplaneGRPCCluster}} + cluster_name: {{.controlplaneCluster}} retry_policy: retry_back_off: base_interval: 0.5s @@ -53,19 +53,8 @@ dynamic_resources: initial_fetch_timeout: 1s ads: {} static_resources: - secrets: - - name: {{.certificateSecret}} - tls_certificate: - certificate_chain: - filename: {{.certificateFile}} - private_key: - filename: {{.keyFile}} - - name: {{.validationSecret}} - validation_context: - trusted_ca: - filename: {{.caFile}} clusters: - - name: {{.controlplaneGRPCCluster}} + - name: {{.controlplaneCluster}} type: LOGICAL_DNS dns_refresh_rate: 1s connect_timeout: 1s @@ -79,7 +68,7 @@ static_resources: explicit_http_config: http2_protocol_options: {} load_assignment: - cluster_name: {{.controlplaneGRPCCluster}} + cluster_name: {{.controlplaneCluster}} endpoints: - lb_endpoints: - endpoint: @@ -91,60 +80,16 @@ static_resources: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext - sni: {{.controlplaneGRPCSNI}} max_session_keys: 0 # TODO: remove once controlplane no longer uses inet.af/tcpproxy common_tls_context: - tls_certificate_sds_secret_configs: - - name: {{.certificateSecret}} - validation_context_sds_secret_config: - name: {{.validationSecret}} - - name: {{.controlplaneInternalHTTPCluster}} - type: LOGICAL_DNS - dns_refresh_rate: 1s - connect_timeout: 1s - typed_dns_resolver_config: - name: envoy.network.dns_resolver.getaddrinfo - typed_config: - "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.getaddrinfo.v3.GetAddrInfoDnsResolverConfig - load_assignment: - cluster_name: {{.controlplaneInternalHTTPCluster}} - endpoints: - - lb_endpoints: - - endpoint: - hostname: {{.peerName}} - address: - socket_address: - address: {{.controlplaneHost}} - port_value: {{.controlplanePort}} - transport_socket: - name: envoy.transport_sockets.tls - typed_config: - "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext - sni: {{.peerName}} - max_session_keys: 0 # TODO: remove once controlplane no longer uses inet.af/tcpproxy - common_tls_context: - tls_certificate_sds_secret_configs: - - name: {{.certificateSecret}} - validation_context_sds_secret_config: - name: {{.validationSecret}} - - name: {{.controlplaneExternalHTTPCluster}} - type: LOGICAL_DNS - dns_refresh_rate: 1s - connect_timeout: 1s - typed_dns_resolver_config: - name: envoy.network.dns_resolver.getaddrinfo - typed_config: - "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.getaddrinfo.v3.GetAddrInfoDnsResolverConfig - load_assignment: - cluster_name: {{.controlplaneInternalHTTPCluster}} - endpoints: - - lb_endpoints: - - endpoint: - hostname: {{.peerName}} - address: - socket_address: - address: {{.controlplaneHost}} - port_value: {{.controlplanePort}} + tls_certificates: + - certificate_chain: + filename: {{.certificateFile}} + private_key: + filename: {{.keyFile}} + validation_context: + trusted_ca: + filename: {{.caFile}} - name: {{.egressRouterCluster}} connect_timeout: 1s typed_extension_protocol_options: @@ -176,35 +121,25 @@ static_resources: domains: ["*"] routes: - match: - path: / + connect_matcher: {} route: cluster_header: {{.targetClusterHeader}} auto_host_rewrite: true - prefix_rewrite: / upgrade_configs: - upgrade_type: CONNECT http_filters: - name: envoy.filters.http.ext_authz typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz - http_service: - server_uri: - uri: {{.peerName}} - cluster: {{.controlplaneInternalHTTPCluster}} - timeout: 0.250s - path_prefix: {{.dataplaneEgressAuthorizationPrefix}} - authorization_response: - allowed_upstream_headers: - patterns: - - exact: {{.targetClusterHeader}} - - exact: {{.authorizationHeader}} + grpc_service: + envoy_grpc: + cluster_name: {{.controlplaneCluster}} + retry_policy: + retry_back_off: + base_interval: 0.5s + max_interval: 1s clear_route_cache: true transport_api_version: V3 - allowed_headers: - patterns: - - exact: {{.importNameHeader}} - - exact: {{.importNamespaceHeader}} - - exact: {{.clientIPHeader}} - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router @@ -213,22 +148,8 @@ static_resources: socket_address: address: 0.0.0.0 port_value: {{.dataplaneListenPort}} - listener_filters: - - name: envoy.filters.listener.tls_inspector - typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector filter_chains: - - filter_chain_match: - server_names: ["{{.peerName}}"] - filters: - - name: envoy.filters.network.tcp_proxy - typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy - stat_prefix: tcp-proxy-controlplane - cluster: {{.controlplaneExternalHTTPCluster}} - - filter_chain_match: - server_names: ["{{.dataplaneSNI}}"] - transport_socket: + - transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext @@ -236,8 +157,16 @@ static_resources: common_tls_context: tls_certificate_sds_secret_configs: - name: {{.certificateSecret}} + sds_config: + resource_api_version: V3 + initial_fetch_timeout: 1s + ads: {} validation_context_sds_secret_config: name: {{.validationSecret}} + sds_config: + resource_api_version: V3 + initial_fetch_timeout: 1s + ads: {} filters: - name: envoy.filters.network.http_connection_manager typed_config: @@ -249,34 +178,85 @@ static_resources: domains: ["*"] routes: - match: - path: / + connect_matcher: {} route: cluster_header: {{.targetClusterHeader}} upgrade_configs: - upgrade_type: CONNECT - connect_config: - allow_post: true + connect_config: {} + - match: + prefix: / + direct_response: + status: 200 upgrade_configs: - upgrade_type: CONNECT http_filters: - - name: envoy.filters.http.ext_authz + - name: composite typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz - http_service: - server_uri: - uri: {{.peerName}} - cluster: {{.controlplaneInternalHTTPCluster}} - timeout: 0.250s - path_prefix: {{.dataplaneIngressAuthorizationPrefix}} - authorization_response: - allowed_upstream_headers: - patterns: - - exact: {{.targetClusterHeader}} - clear_route_cache: true - transport_api_version: V3 - allowed_headers: - patterns: - - exact: {{.authorizationHeader}} + "@type": type.googleapis.com/envoy.extensions.common.matching.v3.ExtensionWithMatcher + extension_config: + name: composite + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.composite.v3.Composite + matcher: + on_no_match: + action: + name: action-no-match + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.composite.v3.ExecuteFilterAction + typed_config: + name: envoy.filters.http.ext_authz + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz + grpc_service: + envoy_grpc: + cluster_name: {{.controlplaneCluster}} + retry_policy: + retry_back_off: + base_interval: 0.5s + max_interval: 1s + clear_route_cache: true + include_peer_certificate: true + with_request_body: + max_request_bytes: 65536 + transport_api_version: V3 + allowed_headers: + patterns: + - exact: {{.authorizationHeader}} + matcher_list: + matchers: + - predicate: + single_predicate: + input: + name: method-matcher + typed_config: + "@type": type.googleapis.com/envoy.type.matcher.v3.HttpRequestHeaderMatchInput + header_name: :method + value_match: + exact: CONNECT + ignore_case: true + on_match: + action: + name: connect-action + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.composite.v3.ExecuteFilterAction + typed_config: + name: envoy.filters.http.ext_authz + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz + grpc_service: + envoy_grpc: + cluster_name: {{.controlplaneCluster}} + retry_policy: + retry_back_off: + base_interval: 0.5s + max_interval: 1s + clear_route_cache: true + include_peer_certificate: true + transport_api_version: V3 + allowed_headers: + patterns: + - exact: {{.authorizationHeader}} - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router diff --git a/cmd/cl-dataplane/app/server.go b/cmd/cl-dataplane/app/server.go index 1477ab761..6cd2876fb 100644 --- a/cmd/cl-dataplane/app/server.go +++ b/cmd/cl-dataplane/app/server.go @@ -22,9 +22,7 @@ import ( "github.com/spf13/cobra" "github.com/spf13/pflag" - "github.com/clusterlink-net/clusterlink/pkg/dataplane/api" "github.com/clusterlink-net/clusterlink/pkg/util/log" - "github.com/clusterlink-net/clusterlink/pkg/util/tls" ) const ( @@ -81,27 +79,11 @@ func (o *Options) Run() error { }() } - // parse TLS files - parsedCertData, err := tls.ParseFiles(CAFile, CertificateFile, KeyFile) - if err != nil { - return err - } - - dnsNames := parsedCertData.DNSNames() - if len(dnsNames) != 1 { - return fmt.Errorf("expected peer certificate to contain a single DNS name, but got %d", len(dnsNames)) - } - - peerName, err := api.StripServerPrefix(dnsNames[0]) - if err != nil { - return err - } - // generate random dataplane ID dataplaneID := uuid.New().String() logrus.Infof("Dataplane ID: %s.", dataplaneID) - return o.runEnvoy(peerName, dataplaneID) + return o.runEnvoy(dataplaneID) } // NewCLDataplaneCommand creates a *cobra.Command object with default parameters. diff --git a/cmd/cl-go-dataplane/app/server.go b/cmd/cl-go-dataplane/app/server.go index 763361013..1b256e6fb 100644 --- a/cmd/cl-go-dataplane/app/server.go +++ b/cmd/cl-go-dataplane/app/server.go @@ -18,11 +18,15 @@ import ( "net" "os" "strconv" + "time" "github.com/google/uuid" "github.com/sirupsen/logrus" "github.com/spf13/cobra" "github.com/spf13/pflag" + "google.golang.org/grpc" + "google.golang.org/grpc/backoff" + "google.golang.org/grpc/credentials" cpapi "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" "github.com/clusterlink-net/clusterlink/pkg/dataplane/api" @@ -42,9 +46,6 @@ const ( CertificateFile = "/etc/ssl/certs/clink-dataplane.pem" // KeyFile is the path to the private-key file. KeyFile = "/etc/ssl/private/clink-dataplane.pem" - - // dataplaneServerAddress is the address of the dataplane HTTP server for accepting ingress dataplane connections. - dataplaneServerAddress = "127.0.0.1:8443" ) // Options contains everything necessary to create and run a dataplane. @@ -73,26 +74,36 @@ func (o *Options) RequiredFlags() []string { } // Run the go dataplane. -func (o *Options) runGoDataplane(peerName, dataplaneID string, parsedCertData *tls.ParsedCertData) error { +func (o *Options) runGoDataplane(dataplaneID string, parsedCertData *tls.ParsedCertData) error { controlplaneTarget := net.JoinHostPort(o.ControlplaneHost, strconv.Itoa(cpapi.ListenPort)) - logrus.Infof("Starting go dataplane, Name: %s, ID: %s", peerName, dataplaneID) + logrus.Infof("Starting go dataplane, ID: %s", dataplaneID) + + controlplaneClient, err := grpc.NewClient( + controlplaneTarget, + grpc.WithTransportCredentials(credentials.NewTLS(parsedCertData.ClientConfig("cl-controlplane"))), + grpc.WithConnectParams(grpc.ConnectParams{ + Backoff: backoff.Config{ + BaseDelay: 100 * time.Millisecond, + Multiplier: 1.6, + Jitter: 0.2, + MaxDelay: time.Second, + }, + })) + if err != nil { + return fmt.Errorf("error initializing controlplane client: %w", err) + } - dataplane := dpserver.NewDataplane(dataplaneID, controlplaneTarget, peerName, parsedCertData) + dataplaneServerAddress := fmt.Sprintf(":%d", api.ListenPort) + dataplane := dpserver.NewDataplane(dataplaneID, controlplaneClient, parsedCertData) go func() { err := dataplane.StartDataplaneServer(dataplaneServerAddress) logrus.Errorf("Failed to start dataplane server: %v.", err) }() - go func() { - err := dataplane.StartSNIServer(dataplaneServerAddress) - logrus.Error("Failed to start dataplane server", err) - }() - // Start xDS client, if it fails to start we keep retrying to connect to the controlplane host - tlsConfig := parsedCertData.ClientConfig(cpapi.GRPCServerName(peerName)) - xdsClient := dpclient.NewXDSClient(dataplane, controlplaneTarget, tlsConfig) - err := xdsClient.Run() + xdsClient := dpclient.NewXDSClient(dataplane, controlplaneClient) + err = xdsClient.Run() return fmt.Errorf("xDS Client stopped: %w", err) } @@ -111,17 +122,7 @@ func (o *Options) Run() error { } // parse TLS files - parsedCertData, err := tls.ParseFiles(CAFile, CertificateFile, KeyFile) - if err != nil { - return err - } - - dnsNames := parsedCertData.DNSNames() - if len(dnsNames) != 1 { - return fmt.Errorf("expected peer certificate to contain a single DNS name, but got %d", len(dnsNames)) - } - - peerName, err := api.StripServerPrefix(dnsNames[0]) + parsedCertData, _, err := tls.ParseFiles(CAFile, CertificateFile, KeyFile) if err != nil { return err } @@ -130,7 +131,7 @@ func (o *Options) Run() error { dataplaneID := uuid.New().String() logrus.Infof("Dataplane ID: %s.", dataplaneID) - return o.runGoDataplane(peerName, dataplaneID, parsedCertData) + return o.runGoDataplane(dataplaneID, parsedCertData) } // NewCLGoDataplaneCommand creates a *cobra.Command object with default parameters. diff --git a/cmd/clusterlink/cmd/create/create_peer.go b/cmd/clusterlink/cmd/create/create_peer.go index ba7b15369..418c7c0b5 100644 --- a/cmd/clusterlink/cmd/create/create_peer.go +++ b/cmd/clusterlink/cmd/create/create_peer.go @@ -59,8 +59,44 @@ func (o *PeerOptions) saveCertificate(cert *bootstrap.Certificate, outDirectory return os.WriteFile(filepath.Join(outDirectory, config.PrivateKeyFileName), cert.RawKey(), 0o600) } -func (o *PeerOptions) createControlplane(peerCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateControlplaneCertificate(o.Name, peerCert) +func (o *PeerOptions) createCA() (*bootstrap.Certificate, error) { + cert, err := bootstrap.CreateCACertificate() + if err != nil { + return nil, err + } + + outDirectory := config.CADirectory(o.Name, o.Fabric, o.Path) + if err := os.Mkdir(outDirectory, 0o755); err != nil { + return nil, err + } + + if err := o.saveCertificate(cert, outDirectory); err != nil { + return nil, err + } + + return cert, nil +} + +func (o *PeerOptions) createPeerCert(fabricCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { + cert, err := bootstrap.CreatePeerCertificate(o.Name, fabricCert) + if err != nil { + return nil, err + } + + outDirectory := config.PeerDirectory(o.Name, o.Fabric, o.Path) + if err := os.Mkdir(outDirectory, 0o755); err != nil { + return nil, err + } + + if err := o.saveCertificate(cert, outDirectory); err != nil { + return nil, err + } + + return cert, nil +} + +func (o *PeerOptions) createControlplane(caCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { + cert, err := bootstrap.CreateControlplaneCertificate(caCert) if err != nil { return nil, err } @@ -77,8 +113,8 @@ func (o *PeerOptions) createControlplane(peerCert *bootstrap.Certificate) (*boot return cert, nil } -func (o *PeerOptions) createDataplane(peerCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateDataplaneCertificate(o.Name, peerCert) +func (o *PeerOptions) createDataplane(caCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { + cert, err := bootstrap.CreateDataplaneCertificate(caCert) if err != nil { return nil, err } @@ -110,26 +146,20 @@ func (o *PeerOptions) Run() error { return err } - peerDirectory := config.PeerDirectory(o.Name, o.Fabric, o.Path) - if err := os.Mkdir(peerDirectory, 0o755); err != nil { - return err - } - - peerCertificate, err := bootstrap.CreatePeerCertificate(o.Name, fabricCert) - if err != nil { + if _, err := o.createPeerCert(fabricCert); err != nil { return err } - err = o.saveCertificate(peerCertificate, peerDirectory) + caCert, err := o.createCA() if err != nil { return err } - if _, err := o.createControlplane(peerCertificate); err != nil { + if _, err := o.createControlplane(caCert); err != nil { return err } - if _, err := o.createDataplane(peerCertificate); err != nil { + if _, err := o.createDataplane(caCert); err != nil { return err } diff --git a/cmd/clusterlink/cmd/deploy/deploy_peer.go b/cmd/clusterlink/cmd/deploy/deploy_peer.go index 9bb5fc3a8..35ec7c3f1 100644 --- a/cmd/clusterlink/cmd/deploy/deploy_peer.go +++ b/cmd/clusterlink/cmd/deploy/deploy_peer.go @@ -159,8 +159,14 @@ func (o *PeerOptions) Run() error { return err } - peerCertificate, err := bootstrap.ReadCertificates( - config.PeerDirectory(o.Name, o.Fabric, o.Path), false) + peerCert, err := bootstrap.ReadCertificates( + config.PeerDirectory(o.Name, o.Fabric, o.Path), true) + if err != nil { + return err + } + + caCert, err := bootstrap.ReadCertificates( + config.CADirectory(o.Name, o.Fabric, o.Path), false) if err != nil { return err } @@ -181,7 +187,8 @@ func (o *PeerOptions) Run() error { platformCfg := &platform.Config{ Peer: o.Name, FabricCertificate: fabricCert, - PeerCertificate: peerCertificate, + PeerCertificate: peerCert, + CACertificate: caCert, ControlplaneCertificate: controlplaneCert, DataplaneCertificate: dataplaneCert, Dataplanes: o.DataplaneReplicas, diff --git a/cmd/clusterlink/config/config.go b/cmd/clusterlink/config/config.go index 759b5d3b4..66fe0ce42 100644 --- a/cmd/clusterlink/config/config.go +++ b/cmd/clusterlink/config/config.go @@ -33,6 +33,8 @@ const ( ControlplaneDirectoryName = "controlplane" // DataplaneDirectoryName is the directory name containing dataplane server configuration. DataplaneDirectoryName = "dataplane" + // CADirectoryName is the directory name containing site CA configuration. + CADirectoryName = "ca" // GHCR is the path to the GitHub container registry. GHCR = "ghcr.io/clusterlink-net" @@ -60,6 +62,11 @@ func DataplaneDirectory(peer, fabric, path string) string { return filepath.Join(PeerDirectory(peer, fabric, path), DataplaneDirectoryName) } +// CADirectory returns the path for a site CA. +func CADirectory(peer, fabric, path string) string { + return filepath.Join(PeerDirectory(peer, fabric, path), CADirectoryName) +} + // FabricCertificate returns the fabric certificate name. func FabricCertificate(name, path string) string { return filepath.Join(FabricDirectory(name, path), CertificateFileName) diff --git a/go.mod b/go.mod index f8c5572ae..542ff6c5d 100644 --- a/go.mod +++ b/go.mod @@ -9,13 +9,13 @@ require ( github.com/envoyproxy/go-control-plane v0.12.0 github.com/go-chi/chi v4.1.2+incompatible github.com/google/uuid v1.6.0 - github.com/inetaf/tcpproxy v0.0.0-20240214030015-3ce58045626c github.com/lestrrat-go/jwx v1.2.29 github.com/sirupsen/logrus v1.9.3 github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 golang.org/x/net v0.25.0 + google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.1 k8s.io/api v0.30.1 @@ -86,7 +86,6 @@ require ( gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect google.golang.org/appengine v1.6.8 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect diff --git a/go.sum b/go.sum index 04f1bd070..0c3701105 100644 --- a/go.sum +++ b/go.sum @@ -1,4 +1,3 @@ -github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a/go.mod h1:QmP9hvJ91BbJmGVGSbutW19IC0Q9phDCLGaomwTJbgU= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= @@ -75,8 +74,6 @@ github.com/imdario/mergo v0.3.15 h1:M8XP7IuFNsqUx6VPK2P9OSmsYsI/YFaGil0uD21V3dM= github.com/imdario/mergo v0.3.15/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= -github.com/inetaf/tcpproxy v0.0.0-20240214030015-3ce58045626c h1:gYfYE403/nlrGNYj6BEOs9ucLCAGB9gstlSk92DttTg= -github.com/inetaf/tcpproxy v0.0.0-20240214030015-3ce58045626c/go.mod h1:Di7LXRyUcnvAcLicFhtM9/MlZl/TNgRSDHORM2c6CMI= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= diff --git a/pkg/bootstrap/cert.go b/pkg/bootstrap/cert.go index 86bd87ee7..236a0f256 100644 --- a/pkg/bootstrap/cert.go +++ b/pkg/bootstrap/cert.go @@ -18,8 +18,6 @@ import ( "path/filepath" "github.com/clusterlink-net/clusterlink/cmd/clusterlink/config" - "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" - dpapi "github.com/clusterlink-net/clusterlink/pkg/dataplane/api" ) // Certificate represents a clusterlink certificate. @@ -54,13 +52,11 @@ func CreateFabricCertificate(name string) (*Certificate, error) { return &Certificate{cert: cert}, nil } -// CreatePeerCertificate creates a peer certificate. -func CreatePeerCertificate(name string, fabricCert *Certificate) (*Certificate, error) { +// CreateCACertificate creates a site CA certificate for controlplane <-> dataplane trust. +func CreateCACertificate() (*Certificate, error) { cert, err := createCertificate(&certificateConfig{ - Parent: fabricCert.cert, - Name: name, - IsCA: true, - DNSNames: []string{name}, + Name: "cl-ca", + IsCA: true, }) if err != nil { return nil, err @@ -69,14 +65,13 @@ func CreatePeerCertificate(name string, fabricCert *Certificate) (*Certificate, return &Certificate{cert: cert}, nil } -// CreatePeerCertificate creates a controlplane certificate. -func CreateControlplaneCertificate(peer string, peerCert *Certificate) (*Certificate, error) { +// CreateControlplaneCertificate creates a controlplane certificate. +func CreateControlplaneCertificate(caCert *Certificate) (*Certificate, error) { cert, err := createCertificate(&certificateConfig{ - Parent: peerCert.cert, + Parent: caCert.cert, Name: "cl-controlplane", IsServer: true, - IsClient: true, - DNSNames: []string{peer, api.GRPCServerName(peer)}, + DNSNames: []string{"cl-controlplane"}, }) if err != nil { return nil, err @@ -85,14 +80,29 @@ func CreateControlplaneCertificate(peer string, peerCert *Certificate) (*Certifi return &Certificate{cert: cert}, nil } -// CreatePeerCertificate creates a dataplane certificate. -func CreateDataplaneCertificate(peer string, peerCert *Certificate) (*Certificate, error) { +// CreateDataplaneCertificate creates a dataplane certificate. +func CreateDataplaneCertificate(caCert *Certificate) (*Certificate, error) { cert, err := createCertificate(&certificateConfig{ - Parent: peerCert.cert, + Parent: caCert.cert, Name: "cl-dataplane", + IsClient: true, + DNSNames: []string{"cl-dataplane"}, + }) + if err != nil { + return nil, err + } + + return &Certificate{cert: cert}, nil +} + +// CreatePeerCertificate creates a peer certificate. +func CreatePeerCertificate(peer string, fabricCert *Certificate) (*Certificate, error) { + cert, err := createCertificate(&certificateConfig{ + Parent: fabricCert.cert, + Name: peer, IsServer: true, IsClient: true, - DNSNames: []string{dpapi.DataplaneServerName(peer)}, + DNSNames: []string{peer}, }) if err != nil { return nil, err diff --git a/pkg/bootstrap/crypt.go b/pkg/bootstrap/crypt.go index 202a8a54f..789cb895b 100644 --- a/pkg/bootstrap/crypt.go +++ b/pkg/bootstrap/crypt.go @@ -79,7 +79,6 @@ func createCertificate(config *certificateConfig) (*certificate, error) { if config.IsCA { cert.BasicConstraintsValid = true - cert.PermittedDNSDomains = config.DNSNames cert.KeyUsage = x509.KeyUsageCertSign | x509.KeyUsageCRLSign } else { cert.DNSNames = config.DNSNames diff --git a/pkg/bootstrap/platform/config.go b/pkg/bootstrap/platform/config.go index f02cc292a..a926e4cd7 100644 --- a/pkg/bootstrap/platform/config.go +++ b/pkg/bootstrap/platform/config.go @@ -24,15 +24,18 @@ type Config struct { // Namespace is the namespace the components deployed. Namespace string - // FabricCertificate is the fabric certificate. - FabricCertificate *bootstrap.Certificate - // PeerCertificate is the peer certificate. - PeerCertificate *bootstrap.Certificate + // CACertificate is the CA certificate. + CACertificate *bootstrap.Certificate // ControlplaneCertificate is the controlplane certificate. ControlplaneCertificate *bootstrap.Certificate // DataplaneCertificate is the dataplane certificate. DataplaneCertificate *bootstrap.Certificate + // FabricCertificate is the fabric CA certificate. + FabricCertificate *bootstrap.Certificate + // PeerCertificate is the peer certificate. + PeerCertificate *bootstrap.Certificate + // Dataplanes is the number of dataplane servers to run. Dataplanes uint16 // DataplaneType is the type of dataplane to create (envoy or go-based) diff --git a/pkg/bootstrap/platform/k8s.go b/pkg/bootstrap/platform/k8s.go index 811245f5a..42d75a8ea 100644 --- a/pkg/bootstrap/platform/k8s.go +++ b/pkg/bootstrap/platform/k8s.go @@ -17,7 +17,6 @@ import ( "bytes" "encoding/base64" "fmt" - "path/filepath" "text/template" cpapp "github.com/clusterlink-net/clusterlink/cmd/cl-controlplane/app" @@ -32,10 +31,10 @@ const ( apiVersion: v1 kind: Secret metadata: - name: cl-fabric + name: cl-ca namespace: {{.namespace}} data: - ca: {{.fabricCA}} + ca: {{.ca}} --- apiVersion: v1 kind: Secret @@ -54,6 +53,16 @@ metadata: data: cert: {{.dataplaneCert}} key: {{.dataplaneKey}} +--- +apiVersion: v1 +kind: Secret +metadata: + name: cl-peer + namespace: {{.namespace}} +data: + {{.peerCertificateFile}}: {{.peerCert}} + {{.peerKeyFile}}: {{.peerKey}} + {{.fabricCertFile}}: {{.fabricCert}} ` k8sTemplate = `--- @@ -77,10 +86,13 @@ spec: volumes: - name: ca secret: - secretName: cl-fabric + secretName: cl-ca - name: tls secret: secretName: cl-controlplane + - name: peer-tls + secret: + secretName: cl-peer containers: - name: cl-controlplane image: {{.containerRegistry}}cl-controlplane:{{.tag}} @@ -101,6 +113,9 @@ spec: mountPath: {{.controlplaneKeyMountPath}} subPath: "key" readOnly: true + - name: peer-tls + mountPath: {{.peerTLSMountPath}} + readOnly: true env: - name: {{ .namespaceEnvVariable }} valueFrom: @@ -127,7 +142,7 @@ spec: volumes: - name: ca secret: - secretName: cl-fabric + secretName: cl-ca - name: tls secret: secretName: cl-dataplane @@ -261,12 +276,12 @@ func K8SConfig(config *Config) ([]byte, error) { "namespaceEnvVariable": cpapp.NamespaceEnvVariable, "dataplaneAppName": dpapp.Name, - "persistencyDirectoryMountPath": filepath.Dir(cpapp.StoreFile), - "controlplaneCAMountPath": cpapp.CAFile, "controlplaneCertMountPath": cpapp.CertificateFile, "controlplaneKeyMountPath": cpapp.KeyFile, + "peerTLSMountPath": cpapp.PeerTLSDirectory, + "dataplaneCAMountPath": dpapp.CAFile, "dataplaneCertMountPath": dpapp.CertificateFile, "dataplaneKeyMountPath": dpapp.KeyFile, @@ -294,13 +309,18 @@ func K8SConfig(config *Config) ([]byte, error) { // K8SCertificateConfig returns a kubernetes secrets that contains all the certificates. func K8SCertificateConfig(config *Config) ([]byte, error) { args := map[string]interface{}{ - "fabricCA": base64.StdEncoding.EncodeToString(config.FabricCertificate.RawCert()), - "peerCA": base64.StdEncoding.EncodeToString(config.PeerCertificate.RawCert()), - "controlplaneCert": base64.StdEncoding.EncodeToString(config.ControlplaneCertificate.RawCert()), - "controlplaneKey": base64.StdEncoding.EncodeToString(config.ControlplaneCertificate.RawKey()), - "dataplaneCert": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawCert()), - "dataplaneKey": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawKey()), - "namespace": config.Namespace, + "ca": base64.StdEncoding.EncodeToString(config.CACertificate.RawCert()), + "controlplaneCert": base64.StdEncoding.EncodeToString(config.ControlplaneCertificate.RawCert()), + "controlplaneKey": base64.StdEncoding.EncodeToString(config.ControlplaneCertificate.RawKey()), + "dataplaneCert": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawCert()), + "dataplaneKey": base64.StdEncoding.EncodeToString(config.DataplaneCertificate.RawKey()), + "peerCertificateFile": cpapp.PeerCertificateFile, + "peerKeyFile": cpapp.PeerKeyFile, + "fabricCertFile": cpapp.FabricCertificateFile, + "peerCert": base64.StdEncoding.EncodeToString(config.PeerCertificate.RawCert()), + "peerKey": base64.StdEncoding.EncodeToString(config.PeerCertificate.RawKey()), + "fabricCert": base64.StdEncoding.EncodeToString(config.FabricCertificate.RawCert()), + "namespace": config.Namespace, } var certConfig bytes.Buffer @@ -356,13 +376,7 @@ func K8SClusterLinkInstanceConfig(config *Config, name string) ([]byte, error) { // used for deleting the secrets. func K8SEmptyCertificateConfig(config *Config) ([]byte, error) { args := map[string]interface{}{ - "fabricCA": "", - "peerCA": "", - "controlplaneCert": "", - "controlplaneKey": "", - "dataplaneCert": "", - "dataplaneKey": "", - "namespace": config.Namespace, + "namespace": config.Namespace, } var certConfig bytes.Buffer diff --git a/pkg/controlplane/api/authz.go b/pkg/controlplane/api/authz.go index 13d008e03..fb793fafc 100644 --- a/pkg/controlplane/api/authz.go +++ b/pkg/controlplane/api/authz.go @@ -21,10 +21,6 @@ import ( const ( // RemotePeerAuthorizationPath is the path remote peers use to send an authorization request. RemotePeerAuthorizationPath = "/authz" - // DataplaneEgressAuthorizationPath is the path the dataplane uses to authorize an egress connection. - DataplaneEgressAuthorizationPath = "/authz/egress/" - // DataplaneIngressAuthorizationPath is the path the dataplane uses to authorize an ingress connection. - DataplaneIngressAuthorizationPath = "/authz/ingress/" // ImportNameHeader holds the name of the imported service. ImportNameHeader = "x-import-name" @@ -39,6 +35,9 @@ const ( // TargetClusterHeader holds the name of the target cluster. TargetClusterHeader = "host" + // AccessTokenHeader holds the access token for an exported service, sent back by the server. + AccessTokenHeader = "x-access-token" + // JWTSignatureAlgorithm defines the signing algorithm for JWT tokens. JWTSignatureAlgorithm = jwa.RS256 // ExportNameJWTClaim holds the name of the requested exported service. @@ -56,9 +55,3 @@ type AuthorizationRequest struct { // Attributes of the source workload, to be used by the PDP on the remote peer SrcAttributes connectivitypdp.WorkloadAttrs } - -// AuthorizationResponse represents a response for a successful AuthorizationRequest. -type AuthorizationResponse struct { - // AccessToken holds an access token which can be used to access the requested exported service. - AccessToken string -} diff --git a/pkg/controlplane/api/heartbeat.go b/pkg/controlplane/api/heartbeat.go index 976c7013e..5d913053e 100644 --- a/pkg/controlplane/api/heartbeat.go +++ b/pkg/controlplane/api/heartbeat.go @@ -15,5 +15,5 @@ package api const ( // HeartbeatPath is the path for Heartbeat requests from remote peers. - HeartbeatPath = "/healthz " + HeartbeatPath = "/healthz" ) diff --git a/pkg/controlplane/api/servername.go b/pkg/controlplane/api/servername.go index 1c454cae0..ebf1c2214 100644 --- a/pkg/controlplane/api/servername.go +++ b/pkg/controlplane/api/servername.go @@ -13,17 +13,7 @@ package api -import "fmt" - const ( // ListenPort is the port used by the dataplane to access the controlplane. ListenPort = 444 - - // gRPCServerNamePrefix is the prefix such that . is the gRPC server name. - gRPCServerNamePrefix = "grpc" ) - -// GRPCServerName returns the gRPC server name of a specific peer. -func GRPCServerName(peer string) string { - return fmt.Sprintf("%s.%s", gRPCServerNamePrefix, peer) -} diff --git a/pkg/controlplane/api/xds.go b/pkg/controlplane/api/xds.go index 5825ac375..8b58a9dcd 100644 --- a/pkg/controlplane/api/xds.go +++ b/pkg/controlplane/api/xds.go @@ -16,12 +16,8 @@ package api const ( // cluster names. - // ControlplaneInternalHTTPCluster is the cluster name of the controlplane HTTP server for local dataplanes. - ControlplaneInternalHTTPCluster = "controlplane-internal-http" - // ControlplaneExternalHTTPCluster is the cluster name of the controlplane HTTP server for remote clients. - ControlplaneExternalHTTPCluster = "controlplane-external-http" - // ControlplaneGRPCCluster is the cluster name of the controlplane gRPC server. - ControlplaneGRPCCluster = "controlplane-grpc" + // ControlplaneCluster is the cluster name of the controlplane gRPC server. + ControlplaneCluster = "controlplane" // EgressRouterCluster is the cluster name of the internal egress router. EgressRouterCluster = "egress-router" // ExportClusterPrefix is the prefix of clusters representing exported services. diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index 0752376ff..608ac28dd 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -58,8 +58,6 @@ type egressAuthorizationRequest struct { // egressAuthorizationResponse (to local dataplane) represents a response for an egressAuthorizationRequest. type egressAuthorizationResponse struct { - // ServiceExists is true if the requested service exists. - ServiceExists bool // Allowed is true if the request is allowed. Allowed bool // RemotePeerCluster is the cluster name of the remote peer where the connection should be routed to. @@ -100,10 +98,12 @@ type Manager struct { loadBalancer *LoadBalancer connectivityPDP *connectivitypdp.PDP - peerName string - peerTLS *tls.ParsedCertData - peerLock sync.RWMutex - peerClient map[string]*peer.Client + selfPeerLock sync.RWMutex + peerTLS *tls.ParsedCertData + peerName string + + peerClientLock sync.RWMutex + peerClient map[string]*peer.Client podLock sync.RWMutex ipToPod map[string]types.NamespacedName @@ -120,20 +120,22 @@ func (m *Manager) AddPeer(pr *v1alpha1.Peer) { m.logger.Infof("Adding peer '%s'.", pr.Name) // initialize peer client + m.selfPeerLock.RLock() cl := peer.NewClient(pr, m.peerTLS.ClientConfig(pr.Name)) + m.selfPeerLock.RUnlock() - m.peerLock.Lock() + m.peerClientLock.Lock() m.peerClient[pr.Name] = cl - m.peerLock.Unlock() + m.peerClientLock.Unlock() } // DeletePeer removes the possibility for egress dataplane connections to be routed to a given peer. func (m *Manager) DeletePeer(name string) { m.logger.Infof("Deleting peer '%s'.", name) - m.peerLock.Lock() + m.peerClientLock.Lock() delete(m.peerClient, name) - m.peerLock.Unlock() + m.peerClientLock.Unlock() } // AddAccessPolicy adds an access policy to allow/deny specific connections. @@ -191,7 +193,7 @@ func (m *Manager) getPodInfoByIP(ip string) *podInfo { func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationRequest) (*egressAuthorizationResponse, error) { m.logger.Infof("Received egress authorization request: %v.", req) - srcAttributes := connectivitypdp.WorkloadAttrs{GatewayNameLabel: m.peerName} + srcAttributes := connectivitypdp.WorkloadAttrs{GatewayNameLabel: m.getPeerName()} podInfo := m.getPodInfoByIP(req.IP) if podInfo != nil { srcAttributes[ServiceNamespaceLabel] = podInfo.namespace @@ -249,9 +251,9 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR continue } - m.peerLock.RLock() + m.peerClientLock.RLock() cl, ok := m.peerClient[importSource.Peer] - m.peerLock.RUnlock() + m.peerClientLock.RUnlock() if !ok { return nil, fmt.Errorf("missing client for peer: %s", importSource.Peer) @@ -267,7 +269,7 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR DstNamespace = req.ImportName.Namespace } - peerResp, err := cl.Authorize(&cpapi.AuthorizationRequest{ + accessToken, err := cl.Authorize(&cpapi.AuthorizationRequest{ ServiceName: DstName, ServiceNamespace: DstNamespace, SrcAttributes: srcAttributes, @@ -277,25 +279,10 @@ func (m *Manager) authorizeEgress(ctx context.Context, req *egressAuthorizationR continue } - if !peerResp.ServiceExists { - m.logger.Infof( - "Peer %s does not have an import source for %v", - importSource.Peer, req.ImportName) - continue - } - - if !peerResp.Allowed { - m.logger.Infof( - "Peer %s did not allow connection to import %v: %v", - importSource.Peer, req.ImportName, err) - continue - } - return &egressAuthorizationResponse{ - ServiceExists: true, Allowed: true, RemotePeerCluster: cpapi.RemotePeerClusterName(importSource.Peer), - AccessToken: peerResp.AccessToken, + AccessToken: accessToken, }, nil } } @@ -354,7 +341,7 @@ func (m *Manager) authorizeIngress( dstAttributes := connectivitypdp.WorkloadAttrs{ ServiceNameLabel: export.Name, ServiceNamespaceLabel: export.Namespace, - GatewayNameLabel: m.peerName, + GatewayNameLabel: m.getPeerName(), } for k, v := range export.Labels { // add export labels to destination attributes dstAttributes[ServiceLabelsPrefix+k] = v @@ -392,8 +379,37 @@ func (m *Manager) authorizeIngress( return resp, nil } +func (m *Manager) getPeerName() string { + m.selfPeerLock.RLock() + defer m.selfPeerLock.RUnlock() + return m.peerName +} + +func (m *Manager) SetPeerCertificates(peerTLS *tls.ParsedCertData) error { + dnsNames := peerTLS.DNSNames() + if len(dnsNames) == 0 { + return fmt.Errorf("expected peer certificate to contain at least one DNS name") + } + + m.selfPeerLock.Lock() + defer m.selfPeerLock.Unlock() + + m.peerName = dnsNames[0] + m.peerTLS = peerTLS + + m.peerClientLock.Lock() + defer m.peerClientLock.Unlock() + + // re-initialize peer clients + for pr, cl := range m.peerClient { + m.peerClient[pr] = peer.NewClient(cl.Peer(), m.peerTLS.ClientConfig(pr)) + } + + return nil +} + // NewManager returns a new authorization manager. -func NewManager(peerTLS *tls.ParsedCertData, cl client.Client, namespace string) (*Manager, error) { +func NewManager(cl client.Client, namespace string) (*Manager, error) { // generate RSA key-pair for JWT signing // TODO: instead of generating, read from k8s secret rsaKey, err := rsa.GenerateKey(rand.Reader, 2048) @@ -411,18 +427,11 @@ func NewManager(peerTLS *tls.ParsedCertData, cl client.Client, namespace string) return nil, fmt.Errorf("unable to create JWK verifing key: %w", err) } - dnsNames := peerTLS.DNSNames() - if len(dnsNames) == 0 { - return nil, fmt.Errorf("expected peer certificate to contain at least one DNS name") - } - return &Manager{ client: cl, namespace: namespace, connectivityPDP: connectivitypdp.NewPDP(), loadBalancer: NewLoadBalancer(), - peerName: dnsNames[0], - peerTLS: peerTLS, peerClient: make(map[string]*peer.Client), jwkSignKey: jwkSignKey, jwkVerifyKey: jwkVerifyKey, diff --git a/pkg/controlplane/authz/server.go b/pkg/controlplane/authz/server.go index 7ad9c2f5b..4c16db41b 100644 --- a/pkg/controlplane/authz/server.go +++ b/pkg/controlplane/authz/server.go @@ -14,16 +14,22 @@ package authz import ( + "context" "encoding/json" "fmt" "net/http" "strings" + corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3" + authv3 "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3" + typev3 "github.com/envoyproxy/go-control-plane/envoy/type/v3" "github.com/sirupsen/logrus" + "google.golang.org/genproto/googleapis/rpc/code" + "google.golang.org/genproto/googleapis/rpc/status" + "google.golang.org/grpc" "k8s.io/apimachinery/pkg/types" "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" - utilhttp "github.com/clusterlink-net/clusterlink/pkg/util/http" ) const ( @@ -35,141 +41,213 @@ type server struct { logger *logrus.Entry } -// RegisterHandlers registers the HTTP handlers for dataplane authz requests. -func RegisterHandlers(manager *Manager, srv *utilhttp.Server) { - router := srv.Router() - server := &server{ - manager: manager, - logger: logrus.WithField("component", "controlplane.authz.server"), +// Check a dataplane connection. +func (s *server) Check(ctx context.Context, req *authv3.CheckRequest) (*authv3.CheckResponse, error) { + if req == nil || + req.Attributes == nil || + req.Attributes.Source == nil || + req.Attributes.Request == nil || + req.Attributes.Request.Http == nil { + s.logger.Errorf("Invalid check request: %+v", req) + return nil, fmt.Errorf("invalid check request: %v", req) + } + + var resp *authv3.CheckResponse + if req.Attributes.Source.Address != nil && + req.Attributes.Source.Address.GetEnvoyInternalAddress() != nil { + resp = s.checkEgress(ctx, req) + } else { + resp = s.checkIngress(ctx, req) } - router.Post(api.DataplaneEgressAuthorizationPath, server.DataplaneEgressAuthorize) - router.Post(api.DataplaneIngressAuthorizationPath, server.DataplaneIngressAuthorize) + s.logger.WithFields(logrus.Fields{ + "request": req, + "response": resp, + }).Debugf("Check.") - router.Get(api.HeartbeatPath, server.Heartbeat) - router.Post(api.RemotePeerAuthorizationPath, server.PeerAuthorize) + return resp, nil } -// DataplaneEgressAuthorize authorizes access to an imported service. -func (s *server) DataplaneEgressAuthorize(w http.ResponseWriter, r *http.Request) { - // TODO: verify that request originates from local dataplane - - ip := r.Header.Get(api.ClientIPHeader) - if ip == "" { - http.Error(w, fmt.Sprintf("missing '%s' header", api.ClientIPHeader), http.StatusBadRequest) - return +// check an egress dataplane connection. +func (s *server) checkEgress(ctx context.Context, req *authv3.CheckRequest) *authv3.CheckResponse { + httpReq := req.Attributes.Request.Http + headers := httpReq.Headers + + expectedHeaders := []string{api.ClientIPHeader, api.ImportNameHeader, api.ImportNamespaceHeader} + for _, header := range expectedHeaders { + if _, ok := headers[header]; !ok { + errorString := fmt.Sprintf("Missing '%s' header.", header) + return buildDeniedResponse(code.Code_INVALID_ARGUMENT, typev3.StatusCode_BadRequest, errorString) + } } - importName := r.Header.Get(api.ImportNameHeader) - if importName == "" { - http.Error(w, fmt.Sprintf("missing '%s' header", api.ImportNameHeader), http.StatusBadRequest) - return + resp, err := s.manager.authorizeEgress(ctx, &egressAuthorizationRequest{ + ImportName: types.NamespacedName{ + Namespace: headers[api.ImportNamespaceHeader], + Name: headers[api.ImportNameHeader], + }, + IP: headers[api.ClientIPHeader], + }) + if err != nil { + return buildDeniedResponse(code.Code_INTERNAL, typev3.StatusCode_InternalServerError, err.Error()) } - importNamespace := r.Header.Get(api.ImportNamespaceHeader) - if importNamespace == "" { - http.Error(w, fmt.Sprintf("missing '%s' header", api.ImportNamespaceHeader), http.StatusBadRequest) - return + if !resp.Allowed { + errorString := fmt.Sprintf( + "Access denied for '%s/%s'.", headers[api.ImportNamespaceHeader], headers[api.ImportNameHeader]) + return buildDeniedResponse(code.Code_PERMISSION_DENIED, typev3.StatusCode_Forbidden, errorString) } - resp, err := s.manager.authorizeEgress(r.Context(), &egressAuthorizationRequest{ - ImportName: types.NamespacedName{ - Namespace: importNamespace, - Name: importName, + return buildAllowedResponse(&authv3.OkHttpResponse{ + Headers: []*corev3.HeaderValueOption{ + { + Header: &corev3.HeaderValue{ + Key: api.TargetClusterHeader, + Value: resp.RemotePeerCluster, + }, + }, + { + Header: &corev3.HeaderValue{ + Key: api.AuthorizationHeader, + Value: bearerSchemaPrefix + resp.AccessToken, + }, + }, }, - IP: ip, }) +} +// check an ingress dataplane connection. +func (s *server) checkIngress(ctx context.Context, req *authv3.CheckRequest) *authv3.CheckResponse { + httpReq := req.Attributes.Request.Http switch { - case err != nil: - http.Error(w, err.Error(), http.StatusInternalServerError) - return - case !resp.ServiceExists: - w.WriteHeader(http.StatusNotFound) - return - case !resp.Allowed: - w.WriteHeader(http.StatusUnauthorized) - return + case httpReq.Method == http.MethodGet && httpReq.Path == api.HeartbeatPath: + // heartbeat request always simply allowed + return buildAllowedResponse(&authv3.OkHttpResponse{}) + case httpReq.Method == http.MethodPost && httpReq.Path == api.RemotePeerAuthorizationPath: + return s.checkAuthorizationRequest(ctx, httpReq) + case httpReq.Method == http.MethodConnect: + return s.checkServiceAccessRequest(httpReq) } - w.Header().Set(api.TargetClusterHeader, resp.RemotePeerCluster) - w.Header().Set(api.AuthorizationHeader, bearerSchemaPrefix+resp.AccessToken) + errorString := fmt.Sprintf("No handler defined for %s %s.", httpReq.Method, httpReq.Path) + return buildDeniedResponse(code.Code_INVALID_ARGUMENT, typev3.StatusCode_BadRequest, errorString) } -// DataplaneIngressAuthorize authorizes a remote peer dataplane access to an exported service. -func (s *server) DataplaneIngressAuthorize(w http.ResponseWriter, r *http.Request) { - authorization := r.Header.Get(api.AuthorizationHeader) - if authorization == "" { - http.Error(w, fmt.Sprintf("missing '%s' header", api.AuthorizationHeader), http.StatusBadRequest) - return +// check an ingress connection for accessing an exported service. +func (s *server) checkServiceAccessRequest(req *authv3.AttributeContext_HttpRequest) *authv3.CheckResponse { + authorization, ok := req.Headers[api.AuthorizationHeader] + if !ok { + errorString := fmt.Sprintf("Missing '%s' header.", api.AuthorizationHeader) + return buildDeniedResponse(code.Code_INVALID_ARGUMENT, typev3.StatusCode_BadRequest, errorString) } if !strings.HasPrefix(authorization, bearerSchemaPrefix) { - http.Error(w, fmt.Sprintf("authorization header is not using the bearer scheme: %s", authorization), - http.StatusBadRequest) - return + errorString := "Authorization header is not using the bearer scheme." + return buildDeniedResponse(code.Code_INVALID_ARGUMENT, typev3.StatusCode_BadRequest, errorString) } token := strings.TrimPrefix(authorization, bearerSchemaPrefix) targetCluster, err := s.manager.parseAuthorizationHeader(token) if err != nil { - fmt.Printf("Error: %v\n", err) - http.Error(w, err.Error(), http.StatusUnauthorized) - return + return buildDeniedResponse(code.Code_PERMISSION_DENIED, typev3.StatusCode_Forbidden, err.Error()) } - w.Header().Set(api.TargetClusterHeader, targetCluster) -} - -// Heartbeat returns a response for heartbeat checks from remote peers. -func (s *server) Heartbeat(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) + return buildAllowedResponse(&authv3.OkHttpResponse{ + Headers: []*corev3.HeaderValueOption{ + { + Header: &corev3.HeaderValue{ + Key: api.TargetClusterHeader, + Value: targetCluster, + }, + }, + }, + }) } -// PeerAuthorize authorizes a remote peer controlplane request for accessing an exported service, -// yielding an access token. -func (s *server) PeerAuthorize(w http.ResponseWriter, r *http.Request) { - var req api.AuthorizationRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - http.Error(w, err.Error(), http.StatusBadRequest) - return - } - - if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 || len(r.TLS.PeerCertificates[0].DNSNames) != 2 || - r.TLS.PeerCertificates[0].DNSNames[0] == "" { - http.Error(w, "certificate does not contain a valid DNS name for the peer gateway", http.StatusBadRequest) - return +// check an ingress connection for authorizing access to an exported service. +func (s *server) checkAuthorizationRequest( + ctx context.Context, + req *authv3.AttributeContext_HttpRequest, +) *authv3.CheckResponse { + var authzReq api.AuthorizationRequest + if err := json.NewDecoder(strings.NewReader(req.Body)).Decode(&authzReq); err != nil { + s.logger.Errorf("Cannot decode authorization request: %v.", err) + return buildDeniedResponse(code.Code_INVALID_ARGUMENT, typev3.StatusCode_BadRequest, err.Error()) } resp, err := s.manager.authorizeIngress( - r.Context(), + ctx, &ingressAuthorizationRequest{ ServiceName: types.NamespacedName{ - Namespace: req.ServiceNamespace, - Name: req.ServiceName, + Namespace: authzReq.ServiceNamespace, + Name: authzReq.ServiceName, }, - SrcAttributes: req.SrcAttributes, + SrcAttributes: authzReq.SrcAttributes, }) switch { case err != nil: - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return buildDeniedResponse(code.Code_INTERNAL, typev3.StatusCode_InternalServerError, err.Error()) case !resp.ServiceExists: - w.WriteHeader(http.StatusNotFound) - return + errorString := fmt.Sprintf( + "Exported service '%s/%s' not found.", + authzReq.ServiceNamespace, authzReq.ServiceName) + return buildDeniedResponse(code.Code_NOT_FOUND, typev3.StatusCode_NotFound, errorString) case !resp.Allowed: - w.WriteHeader(http.StatusUnauthorized) - return + errorString := fmt.Sprintf( + "Permission denied for '%s/%s'.", + authzReq.ServiceNamespace, authzReq.ServiceName) + return buildDeniedResponse(code.Code_PERMISSION_DENIED, typev3.StatusCode_Forbidden, errorString) } - responseBody, err := json.Marshal(api.AuthorizationResponse{AccessToken: resp.AccessToken}) - if err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return buildAllowedResponse(&authv3.OkHttpResponse{ + ResponseHeadersToAdd: []*corev3.HeaderValueOption{ + { + Header: &corev3.HeaderValue{ + Key: api.AccessTokenHeader, + Value: resp.AccessToken, + }, + }, + }, + }) +} + +// RegisterService registers an ext_authz service backed by Manager to the given gRPC server. +func RegisterService(manager *Manager, grpcServer *grpc.Server) { + srv := newServer(manager) + authv3.RegisterAuthorizationServer(grpcServer, srv) +} + +func buildAllowedResponse(resp *authv3.OkHttpResponse) *authv3.CheckResponse { + return &authv3.CheckResponse{ + Status: &status.Status{ + Code: int32(code.Code_OK), + }, + HttpResponse: &authv3.CheckResponse_OkResponse{ + OkResponse: resp, + }, + } +} + +func buildDeniedResponse(rpcCode code.Code, httpCode typev3.StatusCode, message string) *authv3.CheckResponse { + return &authv3.CheckResponse{ + Status: &status.Status{ + Code: int32(rpcCode), + Message: message, + }, + HttpResponse: &authv3.CheckResponse_DeniedResponse{ + DeniedResponse: &authv3.DeniedHttpResponse{ + Status: &typev3.HttpStatus{ + Code: httpCode, + }, + Body: message, + }, + }, } +} - w.Header().Set("Content-Type", "application/json; charset=utf-8") - if _, err := w.Write(responseBody); err != nil { - s.logger.Errorf("Cannot write http response: %v.", err) +func newServer(manager *Manager) *server { + return &server{ + manager: manager, + logger: logrus.WithField("component", "controlplane.authz.server"), } } diff --git a/pkg/controlplane/control/manager.go b/pkg/controlplane/control/manager.go index 4f88fb5fb..256ed9f85 100644 --- a/pkg/controlplane/control/manager.go +++ b/pkg/controlplane/control/manager.go @@ -39,7 +39,6 @@ import ( dpapp "github.com/clusterlink-net/clusterlink/cmd/cl-dataplane/app" "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" - "github.com/clusterlink-net/clusterlink/pkg/util/tls" ) const ( @@ -889,11 +888,11 @@ func endpointSliceChanged(endpointSlice1, endpointSlice2 *discv1.EndpointSlice) } // NewManager returns a new control manager. -func NewManager(cl client.Client, peerTLS *tls.ParsedCertData, namespace string) *Manager { +func NewManager(cl client.Client, namespace string) *Manager { logger := logrus.WithField("component", "controlplane.control.manager") return &Manager{ - peerManager: newPeerManager(cl, peerTLS), + peerManager: newPeerManager(cl), client: cl, namespace: namespace, ports: newPortManager(), diff --git a/pkg/controlplane/control/peer.go b/pkg/controlplane/control/peer.go index 708edcae3..78abc8b4c 100644 --- a/pkg/controlplane/control/peer.go +++ b/pkg/controlplane/control/peer.go @@ -54,8 +54,10 @@ type peerMonitor struct { // peerManager manages peers status. type peerManager struct { - client client.Client - peerTLS *tls.ParsedCertData + client client.Client + + peerTLSLock sync.RWMutex + peerTLS *tls.ParsedCertData lock sync.Mutex monitors map[string]*peerMonitor @@ -83,6 +85,20 @@ func (m *peerMonitor) SetPeer(pr *v1alpha1.Peer) { m.pr = pr } +func (m *peerMonitor) SetClientCertificates(peerTLS *tls.ParsedCertData) { + m.lock.Lock() + defer m.lock.Unlock() + + m.client = peer.NewClient(m.pr, peerTLS.ClientConfig(m.pr.Name)) +} + +func (m *peerMonitor) getClient() *peer.Client { + m.lock.Lock() + defer m.lock.Unlock() + + return m.client +} + func (m *peerMonitor) Start() { defer m.wg.Done() @@ -105,7 +121,7 @@ func (m *peerMonitor) Start() { break } - heartbeatOK := m.client.GetHeartbeat() == nil + heartbeatOK := m.getClient().GetHeartbeat() == nil if healthy == heartbeatOK { if !healthy { ticker.Reset(unhealthyInterval) @@ -296,13 +312,26 @@ func newPeerMonitor(pr *v1alpha1.Peer, manager *peerManager) *peerMonitor { return monitor } +func (m *peerManager) SetPeerCertificates(peerTLS *tls.ParsedCertData) { + m.peerTLSLock.Lock() + defer m.peerTLSLock.Unlock() + + m.peerTLS = peerTLS + + m.lock.Lock() + defer m.lock.Unlock() + + for _, mon := range m.monitors { + mon.SetClientCertificates(peerTLS) + } +} + // newPeerManager returns a new empty peerManager. -func newPeerManager(cl client.Client, peerTLS *tls.ParsedCertData) peerManager { +func newPeerManager(cl client.Client) peerManager { logger := logrus.WithField("component", "controlplane.control.peerManager") return peerManager{ client: cl, - peerTLS: peerTLS, monitors: make(map[string]*peerMonitor), stopCh: make(chan struct{}), statusUpdatesCh: make(chan *v1alpha1.Peer), diff --git a/pkg/controlplane/peer/client.go b/pkg/controlplane/peer/client.go index 070bef829..bc1dcf51c 100644 --- a/pkg/controlplane/peer/client.go +++ b/pkg/controlplane/peer/client.go @@ -30,6 +30,7 @@ import ( // Client for accessing a remote peer. type Client struct { + pr *v1alpha1.Peer // jsonapi clients for connecting to the remote peer (one per each gateway) clients []*jsonapi.Client logger *logrus.Entry @@ -93,42 +94,25 @@ func (c *Client) getResponse( } // Authorize a request for accessing a peer exported service, yielding an access token. -func (c *Client) Authorize(req *api.AuthorizationRequest) (*RemoteServerAuthorizationResponse, error) { +func (c *Client) Authorize(req *api.AuthorizationRequest) (string, error) { body, err := json.Marshal(req) if err != nil { - return nil, fmt.Errorf("unable to serialize authorization request: %w", err) + return "", fmt.Errorf("unable to serialize authorization request: %w", err) } serverResp, err := c.getResponse(func(client *jsonapi.Client) (*jsonapi.Response, error) { return client.Post(api.RemotePeerAuthorizationPath, body) }) if err != nil { - return nil, err - } - - resp := &RemoteServerAuthorizationResponse{} - if serverResp.Status == http.StatusNotFound { - return resp, nil - } - - resp.ServiceExists = true - if serverResp.Status == http.StatusUnauthorized { - return resp, nil + return "", err } if serverResp.Status != http.StatusOK { - return nil, fmt.Errorf("unable to authorize connection (%d), server returned: %s", + return "", fmt.Errorf("unable to authorize connection (%d), server returned: %s", serverResp.Status, serverResp.Body) } - var authResp api.AuthorizationResponse - if err := json.Unmarshal(serverResp.Body, &authResp); err != nil { - return nil, fmt.Errorf("unable to parse server response: %w", err) - } - - resp.Allowed = true - resp.AccessToken = authResp.AccessToken - return resp, nil + return serverResp.Headers.Get(api.AccessTokenHeader), nil } // GetHeartbeat get a heartbeat from other peers. @@ -148,6 +132,11 @@ func (c *Client) GetHeartbeat() error { return nil } +// Peer object this client corresponds to. +func (c *Client) Peer() *v1alpha1.Peer { + return c.pr +} + // NewClient returns a new Peer API client. func NewClient(peer *v1alpha1.Peer, tlsConfig *tls.Config) *Client { clients := make([]*jsonapi.Client, len(peer.Spec.Gateways)) @@ -156,6 +145,7 @@ func NewClient(peer *v1alpha1.Peer, tlsConfig *tls.Config) *Client { } return &Client{ + pr: peer, clients: clients, logger: logrus.WithFields(logrus.Fields{ "component": "controlplane.peer.client", diff --git a/pkg/controlplane/xds/manager.go b/pkg/controlplane/xds/manager.go index 0c316bed5..cb2f13610 100644 --- a/pkg/controlplane/xds/manager.go +++ b/pkg/controlplane/xds/manager.go @@ -35,7 +35,7 @@ import ( "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" cpapi "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" - dpapi "github.com/clusterlink-net/clusterlink/pkg/dataplane/api" + utiltls "github.com/clusterlink-net/clusterlink/pkg/util/tls" ) // Manager manages the core routing components of the dataplane. @@ -47,6 +47,7 @@ import ( type Manager struct { clusters *cache.LinearCache listeners *cache.LinearCache + secrets *cache.LinearCache logger *logrus.Entry } @@ -56,21 +57,30 @@ func (m *Manager) AddPeer(peer *v1alpha1.Peer) error { m.logger.Infof("Adding peer '%s'.", peer.Name) clusterName := cpapi.RemotePeerClusterName(peer.Name) - dataplaneSNI := dpapi.DataplaneSNI(peer.Name) - epc, err := makeEndpointsCluster(clusterName, peer.Spec.Gateways, dataplaneSNI) + epc, err := makeEndpointsCluster(clusterName, peer.Spec.Gateways, peer.Name+":443") if err != nil { return err } + sdsConfig := &core.ConfigSource{ + ConfigSourceSpecifier: &core.ConfigSource_Ads{ + Ads: &core.AggregatedConfigSource{}, + }, + InitialFetchTimeout: durationpb.New(time.Second), + ResourceApiVersion: core.ApiVersion_V3, + } + tlsConfig := &tls.UpstreamTlsContext{ - Sni: dataplaneSNI, + Sni: peer.Name, CommonTlsContext: &tls.CommonTlsContext{ TlsCertificateSdsSecretConfigs: []*tls.SdsSecretConfig{{ - Name: cpapi.CertificateSecret, + Name: cpapi.CertificateSecret, + SdsConfig: sdsConfig, }}, ValidationContextType: &tls.CommonTlsContext_ValidationContextSdsSecretConfig{ ValidationContextSdsSecretConfig: &tls.SdsSecretConfig{ - Name: cpapi.ValidationSecret, + Name: cpapi.ValidationSecret, + SdsConfig: sdsConfig, }, }, }, @@ -141,7 +151,6 @@ func (m *Manager) AddImport(imp *v1alpha1.Import) error { tunnelingConfig := &tcpproxy.TcpProxy_TunnelingConfig{ Hostname: egressRouterHostname, - UsePost: true, HeadersToAdd: []*core.HeaderValueOption{ { Header: &core.HeaderValue{ @@ -202,6 +211,52 @@ func (m *Manager) DeleteImport(name types.NamespacedName) error { return m.listeners.DeleteResource(listenerName) } +// SetPeerCertificates sets the TLS certificates used for peer-to-peer communication. +func (m *Manager) SetPeerCertificates(rawCertData *utiltls.RawCertData) error { + m.logger.Info("Setting peer certificates.") + + certificateSecret := &tls.Secret{ + Name: cpapi.CertificateSecret, + Type: &tls.Secret_TlsCertificate{ + TlsCertificate: &tls.TlsCertificate{ + CertificateChain: &core.DataSource{ + Specifier: &core.DataSource_InlineBytes{ + InlineBytes: rawCertData.Certificate(), + }, + }, + PrivateKey: &core.DataSource{ + Specifier: &core.DataSource_InlineBytes{ + InlineBytes: rawCertData.Key(), + }, + }, + }, + }, + } + + if err := m.secrets.UpdateResource(certificateSecret.Name, certificateSecret); err != nil { + return fmt.Errorf("error setting certificate secret: %w", err) + } + + validationSecret := &tls.Secret{ + Name: cpapi.ValidationSecret, + Type: &tls.Secret_ValidationContext{ + ValidationContext: &tls.CertificateValidationContext{ + TrustedCa: &core.DataSource{ + Specifier: &core.DataSource_InlineBytes{ + InlineBytes: rawCertData.CA(), + }, + }, + }, + }, + } + + if err := m.secrets.UpdateResource(validationSecret.Name, validationSecret); err != nil { + return fmt.Errorf("error setting validation secret: %w", err) + } + + return nil +} + func makeAddressCluster(name, addr string, port uint16, hostname string) (*cluster.Cluster, error) { return makeEndpointsCluster(name, []v1alpha1.Endpoint{{Host: addr, Port: port}}, hostname) } @@ -287,6 +342,7 @@ func NewManager() *Manager { return &Manager{ clusters: cache.NewLinearCache(resource.ClusterType, cache.WithLogger(logger)), listeners: cache.NewLinearCache(resource.ListenerType, cache.WithLogger(logger)), + secrets: cache.NewLinearCache(resource.SecretType, cache.WithLogger(logger)), logger: logger, } } diff --git a/pkg/controlplane/xds/server.go b/pkg/controlplane/xds/server.go index 5317da488..da8b06c99 100644 --- a/pkg/controlplane/xds/server.go +++ b/pkg/controlplane/xds/server.go @@ -36,6 +36,7 @@ func RegisterService(ctx context.Context, manager *Manager, grpcServer *grpc.Ser Caches: map[string]cache.Cache{ resource.ClusterType: manager.clusters, resource.ListenerType: manager.listeners, + resource.SecretType: manager.secrets, }, } diff --git a/pkg/dataplane/api/servername.go b/pkg/dataplane/api/servername.go index 943541575..1a41c885f 100644 --- a/pkg/dataplane/api/servername.go +++ b/pkg/dataplane/api/servername.go @@ -13,35 +13,7 @@ package api -import ( - "fmt" - "strings" -) - const ( - // serverPrefix is the prefix such that . is the dataplane server name. - serverPrefix = "dataplane" // ListenPort is the dataplane external listening port. ListenPort = 443 ) - -// DataplaneServerName returns the dataplane server name for a specific peer. -func DataplaneServerName(peer string) string { - return fmt.Sprintf("%s.%s", serverPrefix, peer) -} - -// DataplaneSNI returns the dataplane SNI for a specific peer. -func DataplaneSNI(peer string) string { - return fmt.Sprintf("%s:%d", DataplaneServerName(peer), ListenPort) -} - -// StripServerPrefix strips the dataplane server prefix from the dataplane server name, yielding the peer name. -func StripServerPrefix(serverName string) (string, error) { - toStrip := serverPrefix + "." - if !strings.HasPrefix(serverName, toStrip) { - return "", fmt.Errorf("expected dataplane server name to start with '%s', but got: '%s'", - toStrip, serverName) - } - - return strings.TrimPrefix(serverName, toStrip), nil -} diff --git a/pkg/dataplane/client/fetcher.go b/pkg/dataplane/client/fetcher.go index 6a721aed8..3caee38a6 100644 --- a/pkg/dataplane/client/fetcher.go +++ b/pkg/dataplane/client/fetcher.go @@ -22,6 +22,7 @@ import ( cluster "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3" core "github.com/envoyproxy/go-control-plane/envoy/config/core/v3" listener "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3" + tlsv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3" client "github.com/envoyproxy/go-control-plane/pkg/client/sotw/v3" "github.com/envoyproxy/go-control-plane/pkg/resource/v3" "github.com/sirupsen/logrus" @@ -98,6 +99,22 @@ func (f *fetcher) handleListeners(resources []*anypb.Any) error { return nil } +func (f *fetcher) handleSecrets(resources []*anypb.Any) error { + for _, res := range resources { + secret := &tlsv3.Secret{} + err := anypb.UnmarshalTo(res, secret, proto.UnmarshalOptions{}) + if err != nil { + return err + } + f.logger.Debugf("Secret: %s.", secret.Name) + if err := f.dataplane.AddSecret(secret); err != nil { + return fmt.Errorf("error adding secret %s: %w", secret.Name, err) + } + } + + return nil +} + func (f *fetcher) Run() error { for { resp, err := f.client.Fetch() @@ -118,6 +135,11 @@ func (f *fetcher) Run() error { if err != nil { f.logger.Errorf("Failed to handle listeners: %v.", err) } + case resource.SecretType: + err := f.handleSecrets(resp.Resources) + if err != nil { + f.logger.Errorf("Failed to handle secrets: %v.", err) + } default: return fmt.Errorf("unknown resource type") } @@ -129,9 +151,14 @@ func (f *fetcher) Run() error { } } -func newFetcher(ctx context.Context, conn *grpc.ClientConn, resourceType string, dp *server.Dataplane) (*fetcher, error) { +func newFetcher( + ctx context.Context, + controlplaneClient grpc.ClientConnInterface, + resourceType string, + dp *server.Dataplane, +) (*fetcher, error) { cl := client.NewADSClient(ctx, &core.Node{Id: dp.ID}, resourceType) - err := cl.InitConnect(conn) + err := cl.InitConnect(controlplaneClient) if err != nil { return nil, err } diff --git a/pkg/dataplane/client/xds.go b/pkg/dataplane/client/xds.go index 84c70e67f..66a73737d 100644 --- a/pkg/dataplane/client/xds.go +++ b/pkg/dataplane/client/xds.go @@ -15,29 +15,24 @@ package client import ( "context" - "crypto/tls" "errors" "fmt" "sync" - "time" "github.com/envoyproxy/go-control-plane/pkg/resource/v3" "github.com/sirupsen/logrus" "google.golang.org/grpc" - "google.golang.org/grpc/backoff" - "google.golang.org/grpc/credentials" "github.com/clusterlink-net/clusterlink/pkg/dataplane/server" ) // resources indicate the xDS resources that would be fetched. -var resources = [...]string{resource.ClusterType, resource.ListenerType} +var resources = [...]string{resource.ClusterType, resource.ListenerType, resource.SecretType} // XDSClient implements the client which fetches clusters and listeners. type XDSClient struct { dataplane *server.Dataplane - controlplaneTarget string - tlsConfig *tls.Config + controlplaneClient grpc.ClientConnInterface lock sync.Mutex errors map[string]error logger *logrus.Entry @@ -46,20 +41,7 @@ type XDSClient struct { func (x *XDSClient) runFetcher(resourceType string) error { for { - conn, err := grpc.Dial( - x.controlplaneTarget, - grpc.WithTransportCredentials(credentials.NewTLS(x.tlsConfig)), - grpc.WithConnectParams(grpc.ConnectParams{ - Backoff: backoff.DefaultConfig, - MinConnectTimeout: time.Second, - })) - if err != nil { - x.logger.Errorf("Failed to dial controlplane xDS server: %v.", err) - time.Sleep(1 * time.Second) - continue - } - - fetcher, err := newFetcher(context.Background(), conn, resourceType, x.dataplane) + fetcher, err := newFetcher(context.Background(), x.controlplaneClient, resourceType, x.dataplane) if err != nil { x.logger.Errorf("Failed to initialize %s fetcher: %v.", resourceType, err) continue @@ -110,11 +92,10 @@ func (x *XDSClient) Run() error { } // NewXDSClient returns am xDS client which can fetch clusters and listeners from the controlplane. -func NewXDSClient(dataplane *server.Dataplane, controlplaneTarget string, tlsConfig *tls.Config) *XDSClient { +func NewXDSClient(dataplane *server.Dataplane, controlplaneClient grpc.ClientConnInterface) *XDSClient { return &XDSClient{ dataplane: dataplane, - controlplaneTarget: controlplaneTarget, - tlsConfig: tlsConfig, + controlplaneClient: controlplaneClient, errors: make(map[string]error), logger: logrus.WithField("component", "xds.client"), clustersReady: make(chan bool, 1), diff --git a/pkg/dataplane/server/dataplane.go b/pkg/dataplane/server/dataplane.go index 6d380fae7..48a91afef 100644 --- a/pkg/dataplane/server/dataplane.go +++ b/pkg/dataplane/server/dataplane.go @@ -14,16 +14,22 @@ package server import ( + "crypto/tls" + "crypto/x509" "fmt" - "net/http" + "net" "strconv" "strings" - "time" + "sync" cluster "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3" listener "github.com/envoyproxy/go-control-plane/envoy/config/listener/v3" + tlsv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3" + authv3 "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3" "github.com/go-chi/chi" + "github.com/go-chi/chi/middleware" "github.com/sirupsen/logrus" + "google.golang.org/grpc" "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" utiltls "github.com/clusterlink-net/clusterlink/pkg/util/tls" @@ -32,16 +38,18 @@ import ( // Dataplane implements the server and api client which sends authorization to the control plane. // Assumption: The caller implements lock mechanism which operating with clusters and listeners. type Dataplane struct { - ID string - peerName string - router *chi.Mux - apiClient *http.Client - parsedCertData *utiltls.ParsedCertData - controlplaneTarget string - clusters map[string]*cluster.Cluster - listeners map[string]*listener.Listener - listenerEnd map[string]chan bool - logger *logrus.Entry + ID string + router *chi.Mux + authzClient authv3.AuthorizationClient + parsedCertData *utiltls.ParsedCertData + clusters map[string]*cluster.Cluster + listeners map[string]*listener.Listener + listenerEnd map[string]chan bool + + tlsConfigLock sync.RWMutex + tlsConfig *tls.Config + + logger *logrus.Entry } // GetClusterTarget returns the cluster address:port from the cluster map. @@ -55,11 +63,24 @@ func (d *Dataplane) GetClusterTarget(name string) (string, error) { // GetClusterHost returns the cluster hostname after trimming ":". func (d *Dataplane) GetClusterHost(name string) (string, error) { + hostname, err := d.GetClusterHostname(name) + if err != nil { + return "", err + } + + host, _, err := net.SplitHostPort(hostname) + if err != nil { + return "", fmt.Errorf("cluster hostname '%s' cannot be parsed: %w", hostname, err) + } + return host, nil +} + +// GetClusterHostname returns the cluster hostname. +func (d *Dataplane) GetClusterHostname(name string) (string, error) { if _, ok := d.clusters[name]; !ok { return "", fmt.Errorf("unable to find %s in cluster map", name) } - return strings.Split( - d.clusters[name].LoadAssignment.GetEndpoints()[0].LbEndpoints[0].GetEndpoint().Hostname, ":")[0], nil + return d.clusters[name].LoadAssignment.GetEndpoints()[0].LbEndpoints[0].GetEndpoint().Hostname, nil } // AddCluster adds/updates a cluster to the map. @@ -107,24 +128,119 @@ func (d *Dataplane) GetListeners() map[string]*listener.Listener { return d.listeners } +// AddSecret adds a secret (dataplane cert or CA). +func (d *Dataplane) AddSecret(secret *tlsv3.Secret) error { + switch secret.Name { + case api.CertificateSecret: + return d.addCertificateSecret(secret) + case api.ValidationSecret: + return d.addValidationSecret(secret) + } + + return fmt.Errorf("unknown secret: %s", secret.Name) +} + +func (d *Dataplane) addCertificateSecret(secret *tlsv3.Secret) error { + tlsSecret := secret.GetTlsCertificate() + if tlsSecret == nil { + return fmt.Errorf("not a TLS certificate secret") + } + + certChain := tlsSecret.CertificateChain + if certChain == nil { + return fmt.Errorf("no certificate chain") + } + + certBytes := certChain.GetInlineBytes() + if certBytes == nil { + return fmt.Errorf("no certificate chain bytes embedded") + } + + privateKey := tlsSecret.PrivateKey + if privateKey == nil { + return fmt.Errorf("no private key") + } + + keyBytes := privateKey.GetInlineBytes() + if keyBytes == nil { + return fmt.Errorf("no private key bytes embedded") + } + + certificate, err := tls.X509KeyPair(certBytes, keyBytes) + if err != nil { + return fmt.Errorf("error parsing certificate: %w", err) + } + + d.tlsConfigLock.Lock() + defer d.tlsConfigLock.Unlock() + newTLSConfig := d.tlsConfig.Clone() + newTLSConfig.Certificates = []tls.Certificate{certificate} + d.tlsConfig = newTLSConfig + + return nil +} + +func (d *Dataplane) addValidationSecret(secret *tlsv3.Secret) error { + validationContext := secret.GetValidationContext() + if validationContext == nil { + return fmt.Errorf("not a validation context secret") + } + + trustedCa := validationContext.TrustedCa + if trustedCa == nil { + return fmt.Errorf("no trusted CA") + } + + caBytes := trustedCa.GetInlineBytes() + if caBytes == nil { + return fmt.Errorf("no CA bytes embedded") + } + + caCertPool := x509.NewCertPool() + if !caCertPool.AppendCertsFromPEM(caBytes) { + return fmt.Errorf("error parsing CA") + } + + d.tlsConfigLock.Lock() + defer d.tlsConfigLock.Unlock() + newTLSConfig := d.tlsConfig.Clone() + newTLSConfig.ClientCAs = caCertPool + newTLSConfig.RootCAs = caCertPool + d.tlsConfig = newTLSConfig + + return nil +} + // NewDataplane returns a new dataplane HTTP server. -func NewDataplane(dataplaneID, controlplaneTarget, peerName string, parsedCertData *utiltls.ParsedCertData) *Dataplane { +func NewDataplane( + dataplaneID string, + controlplaneClient grpc.ClientConnInterface, + parsedCertData *utiltls.ParsedCertData, +) *Dataplane { + logger := logrus.WithField("component", "dataplane.server.http") + + router := chi.NewRouter() + router.Use(middleware.Recoverer) + if logrus.GetLevel() >= logrus.DebugLevel { + router.Use(middleware.RequestLogger(&middleware.DefaultLogFormatter{ + Logger: logger, + NoColor: true, + })) + } + dp := &Dataplane{ - ID: dataplaneID, - peerName: peerName, - router: chi.NewRouter(), - apiClient: &http.Client{ - Timeout: 10 * time.Second, - Transport: &http.Transport{ - TLSClientConfig: parsedCertData.ClientConfig(peerName), - }, + ID: dataplaneID, + router: router, + authzClient: authv3.NewAuthorizationClient(controlplaneClient), + parsedCertData: parsedCertData, + clusters: make(map[string]*cluster.Cluster), + listeners: make(map[string]*listener.Listener), + listenerEnd: make(map[string]chan bool), + tlsConfig: &tls.Config{ + MinVersion: tls.VersionTLS12, + ClientAuth: tls.RequireAndVerifyClientCert, }, - parsedCertData: parsedCertData, - controlplaneTarget: controlplaneTarget, - clusters: make(map[string]*cluster.Cluster), - listeners: make(map[string]*listener.Listener), - listenerEnd: make(map[string]chan bool), - logger: logrus.WithField("component", "dataplane.server.http"), + logger: logger, } dp.addAuthzHandlers() diff --git a/pkg/dataplane/server/listener.go b/pkg/dataplane/server/listener.go index 6865dde05..129fd2f55 100644 --- a/pkg/dataplane/server/listener.go +++ b/pkg/dataplane/server/listener.go @@ -14,12 +14,15 @@ package server import ( + "context" "fmt" "net" - "net/http" "strconv" "strings" + corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3" + authv3 "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3" + "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" ) @@ -70,7 +73,12 @@ func (d *Dataplane) serveEgressConnections(name string, listener net.Listener) e conn.Close() continue } - tlsConfig := d.parsedCertData.ClientConfig(targetHost) + + d.tlsConfigLock.RLock() + tlsConfig := d.tlsConfig.Clone() + d.tlsConfigLock.RUnlock() + + tlsConfig.ServerName = targetHost go func() { err := d.initiateEgressConnection(targetPeer, accessToken, conn, tlsConfig) @@ -84,27 +92,59 @@ func (d *Dataplane) serveEgressConnections(name string, listener net.Listener) e // getEgressAuth returns the target cluster and authorization token for the outgoing connection. func (d *Dataplane) getEgressAuth(name, sourceIP string) (string, string, error) { //nolint:gocritic // unnamedResult - url := "https://" + d.controlplaneTarget + api.DataplaneEgressAuthorizationPath - egressAuthReq, err := http.NewRequest(http.MethodPost, url, http.NoBody) + components := strings.SplitN(name, "/", 2) + authzReq := &authv3.CheckRequest{ + Attributes: &authv3.AttributeContext{ + Source: &authv3.AttributeContext_Peer{ + Address: &corev3.Address{ + Address: &corev3.Address_EnvoyInternalAddress{ + EnvoyInternalAddress: &corev3.EnvoyInternalAddress{}, + }, + }, + }, + Request: &authv3.AttributeContext_Request{ + Http: &authv3.AttributeContext_HttpRequest{ + Headers: map[string]string{ + api.ImportNamespaceHeader: components[0], + api.ImportNameHeader: components[1], + api.ClientIPHeader: sourceIP, + }, + }, + }, + }, + } + + resp, err := d.authzClient.Check(context.Background(), authzReq) if err != nil { + d.logger.Errorf("Error authorizing egress request: %v.", err) return "", "", err } - egressAuthReq.Close = true - components := strings.SplitN(name, "/", 2) + okResp, ok := resp.HttpResponse.(*authv3.CheckResponse_OkResponse) + if !ok { + if deniedResp, denied := resp.HttpResponse.(*authv3.CheckResponse_DeniedResponse); denied { + return "", "", fmt.Errorf("egress connection denied: %s", deniedResp.DeniedResponse.Body) + } + return "", "", fmt.Errorf("unknown authorization response: %+v", resp) + } - egressAuthReq.Header.Add(api.ClientIPHeader, sourceIP) - egressAuthReq.Header.Add(api.ImportNamespaceHeader, components[0]) - egressAuthReq.Header.Add(api.ImportNameHeader, components[1]) - egressAuthResp, err := d.apiClient.Do(egressAuthReq) - if err != nil { - d.logger.Errorf("Unable to send auth/egress request: %v.", err) - return "", "", err + // get target and access token from response headers + var accessToken, targetCluster string + for _, header := range okResp.OkResponse.Headers { + if header.Header.Key == api.TargetClusterHeader { + targetCluster = header.Header.Value + } else if header.Header.Key == api.AuthorizationHeader { + accessToken = header.Header.Value + } } - defer egressAuthResp.Body.Close() - if egressAuthResp.StatusCode != http.StatusOK { - d.logger.Infof("Failed to obtain egress authorization: %s", egressAuthResp.Status) - return "", "", fmt.Errorf("failed egress authorization:%s", egressAuthResp.Status) + + if targetCluster == "" { + return "", "", fmt.Errorf("missing target cluster") + } + + if accessToken == "" { + return "", "", fmt.Errorf("missing access token") } - return egressAuthResp.Header.Get(api.TargetClusterHeader), egressAuthResp.Header.Get(api.AuthorizationHeader), nil + + return targetCluster, accessToken, nil } diff --git a/pkg/dataplane/server/server.go b/pkg/dataplane/server/server.go index 4b259b620..c8eae42b7 100644 --- a/pkg/dataplane/server/server.go +++ b/pkg/dataplane/server/server.go @@ -16,18 +16,14 @@ package server import ( "crypto/tls" "fmt" + "io" "net" "net/http" - "strconv" "time" - cpapi "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" - "github.com/clusterlink-net/clusterlink/pkg/dataplane/api" - "github.com/clusterlink-net/clusterlink/pkg/util/sniproxy" -) + authv3 "github.com/envoyproxy/go-control-plane/envoy/service/auth/v3" -const ( - httpSchemaPrefix = "https://" + cpapi "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" ) // StartDataplaneServer starts the Dataplane server. @@ -41,66 +37,114 @@ func (d *Dataplane) StartDataplaneServer(dataplaneServerAddress string) error { WriteTimeout: 2 * time.Second, IdleTimeout: 120 * time.Second, MaxHeaderBytes: 10 * 1024, - TLSConfig: d.parsedCertData.ServerConfig(), + TLSConfig: &tls.Config{ + MinVersion: tls.VersionTLS12, + ClientAuth: tls.RequireAndVerifyClientCert, + GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) { + // this function is defined for the sake of skipping certificate file reading + // in net/http.Server.ServeTLS + return nil, fmt.Errorf("invalid") + }, + GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) { + // return certificate set by the controlplane (using the SDS protocol) + d.tlsConfigLock.RLock() + defer d.tlsConfigLock.RUnlock() + return d.tlsConfig, nil + }, + }, } return server.ListenAndServeTLS("", "") } -// StartSNIServer starts the SNI Proxy in the dataplane. -func (d *Dataplane) StartSNIServer(dataplaneServerAddress string) error { - dataplaneListenAddress := ":" + strconv.Itoa(api.ListenPort) - sniProxy := sniproxy.NewServer(map[string]string{ - d.peerName: d.controlplaneTarget, - api.DataplaneServerName(d.peerName): dataplaneServerAddress, - }) - - d.logger.Infof("SNI proxy starting at %s.", dataplaneListenAddress) - err := sniProxy.Listen(dataplaneListenAddress) - if err != nil { - return fmt.Errorf("unable to create listener for server on %s: %w", - dataplaneListenAddress, err) - } - return sniProxy.Start() -} - func (d *Dataplane) addAuthzHandlers() { - d.router.Post("/", d.dataplaneIngressAuthorize) + d.router.NotFound(d.dataplaneIngressAuthorize) } func (d *Dataplane) dataplaneIngressAuthorize(w http.ResponseWriter, r *http.Request) { - forwardingURL := httpSchemaPrefix + d.controlplaneTarget + cpapi.DataplaneIngressAuthorizationPath + if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 || len(r.TLS.PeerCertificates[0].DNSNames) == 0 { + http.Error(w, "certificate does not contain a valid DNS name for the peer gateway", http.StatusBadRequest) + return + } - forwardingReq, err := http.NewRequest(r.Method, forwardingURL, r.Body) + headers := make(map[string]string) + allowedHeaders := []string{cpapi.AuthorizationHeader} + for _, header := range allowedHeaders { + if value := r.Header.Get(header); value != "" { + headers[header] = value + } + } + + defer func() { + if err := r.Body.Close(); err != nil { + d.logger.Warnf("Cannot close response body: %v.", err) + } + }() + + body, err := io.ReadAll(r.Body) if err != nil { - d.logger.Error("Forwarding error in NewRequest", err) - http.Error(w, err.Error(), http.StatusInternalServerError) + errorString := fmt.Sprintf("Unable to read response body: %v.", err) + d.logger.Errorf(errorString) + http.Error(w, errorString, http.StatusInternalServerError) return } - forwardingReq.ContentLength = r.ContentLength - for key, values := range r.Header { - for _, value := range values { - forwardingReq.Header.Add(key, value) - } + + authzReq := &authv3.CheckRequest{ + Attributes: &authv3.AttributeContext{ + Source: &authv3.AttributeContext_Peer{ + Principal: r.TLS.PeerCertificates[0].DNSNames[0], + }, + Request: &authv3.AttributeContext_Request{ + Http: &authv3.AttributeContext_HttpRequest{ + Method: r.Method, + Path: r.URL.Path, + Headers: headers, + Body: string(body), + }, + }, + }, } - resp, err := d.apiClient.Do(forwardingReq) + resp, err := d.authzClient.Check(r.Context(), authzReq) if err != nil { - d.logger.Error("Forwarding error in sending operation", err) + d.logger.Errorf("Error authorizing ingress request: %v.", err) http.Error(w, err.Error(), http.StatusInternalServerError) return } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - d.logger.Infof("Failed to obtain ingress authorization: %s.", resp.Status) - http.Error(w, err.Error(), http.StatusInternalServerError) + if okResp, ok := resp.HttpResponse.(*authv3.CheckResponse_OkResponse); ok { + d.routeIngress(w, r, okResp.OkResponse) + return + } + if deniedResp, ok := resp.HttpResponse.(*authv3.CheckResponse_DeniedResponse); ok { + d.logger.Infof("Ingress connection denied: %s", deniedResp.DeniedResponse.Body) + http.Error(w, deniedResp.DeniedResponse.Body, http.StatusForbidden) + return + } + + d.logger.Errorf("Unknown authorization response: %+v", resp) + http.Error(w, "Unknown authorization response.", http.StatusInternalServerError) +} + +func (d *Dataplane) routeIngress(w http.ResponseWriter, r *http.Request, authzResp *authv3.OkHttpResponse) { + if r.Method != http.MethodConnect { + for _, header := range authzResp.ResponseHeadersToAdd { + w.Header().Set(header.Header.Key, header.Header.Value) + } + w.WriteHeader(http.StatusOK) return } - d.logger.Infof("Got authorization to use service: %s.", resp.Header.Get(cpapi.TargetClusterHeader)) + // get target cluster (for export tunnel) + var targetCluster string + for _, header := range authzResp.Headers { + if header.Header.Key == cpapi.TargetClusterHeader { + targetCluster = header.Header.Value + break + } + } - serviceTarget, err := d.GetClusterTarget(resp.Header.Get(cpapi.TargetClusterHeader)) + serviceTarget, err := d.GetClusterTarget(targetCluster) if err != nil { d.logger.Errorf("Unable to get cluster target: %v.", err) http.Error(w, err.Error(), http.StatusInternalServerError) @@ -161,7 +205,14 @@ func (d *Dataplane) initiateEgressConnection(targetCluster, authToken string, ap d.logger.Error(err) return err } - url := httpSchemaPrefix + target + + targetHostname, err := d.GetClusterHostname(targetCluster) + if err != nil { + d.logger.Errorf("Unable to get cluster hostname: %v.", err) + return err + } + + url := "https://" + targetHostname d.logger.Debugf("Starting to initiate egress connection to: %s.", url) peerConn, err := tls.Dial("tcp", target, tlsConfig) @@ -177,7 +228,7 @@ func (d *Dataplane) initiateEgressConnection(targetCluster, authToken string, ap }, } - egressReq, err := http.NewRequest(http.MethodPost, url, http.NoBody) + egressReq, err := http.NewRequest(http.MethodConnect, url, http.NoBody) if err != nil { return err } diff --git a/pkg/operator/controller/instance_controller.go b/pkg/operator/controller/instance_controller.go index b14827dcb..fa3482746 100644 --- a/pkg/operator/controller/instance_controller.go +++ b/pkg/operator/controller/instance_controller.go @@ -223,7 +223,7 @@ func (r *InstanceReconciler) applyControlplane(ctx context.Context, instance *cl Name: "ca", VolumeSource: corev1.VolumeSource{ Secret: &corev1.SecretVolumeSource{ - SecretName: "cl-fabric", + SecretName: "cl-ca", }, }, }, @@ -235,6 +235,14 @@ func (r *InstanceReconciler) applyControlplane(ctx context.Context, instance *cl }, }, }, + { + Name: "peer-tls", + VolumeSource: corev1.VolumeSource{ + Secret: &corev1.SecretVolumeSource{ + SecretName: "cl-peer", + }, + }, + }, }, Containers: []corev1.Container{ { @@ -266,6 +274,11 @@ func (r *InstanceReconciler) applyControlplane(ctx context.Context, instance *cl SubPath: "key", ReadOnly: true, }, + { + Name: "peer-tls", + MountPath: cpapp.PeerTLSDirectory, + ReadOnly: true, + }, }, Env: []corev1.EnvVar{ { @@ -297,7 +310,7 @@ func (r *InstanceReconciler) applyDataplane(ctx context.Context, instance *clust Name: "ca", VolumeSource: corev1.VolumeSource{ Secret: &corev1.SecretVolumeSource{ - SecretName: "cl-fabric", + SecretName: "cl-ca", }, }, }, diff --git a/pkg/util/jsonapi/client.go b/pkg/util/jsonapi/client.go index 48612a395..22d86eeed 100644 --- a/pkg/util/jsonapi/client.go +++ b/pkg/util/jsonapi/client.go @@ -38,8 +38,9 @@ type Client struct { // Response for a request. type Response struct { - Status int - Body []byte + Status int + Headers *http.Header + Body []byte } // Get sends an HTTP GET request. @@ -114,8 +115,9 @@ func (c *Client) do(method, path string, body []byte) (*Response, error) { requestLogger.Debugf("Response body: %v.", body) return &Response{ - Status: resp.StatusCode, - Body: body, + Status: resp.StatusCode, + Headers: &resp.Header, + Body: body, }, nil } diff --git a/pkg/util/sniproxy/server.go b/pkg/util/sniproxy/server.go deleted file mode 100644 index 01ed6ad47..000000000 --- a/pkg/util/sniproxy/server.go +++ /dev/null @@ -1,77 +0,0 @@ -// Copyright (c) The ClusterLink Authors. -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package sniproxy - -import ( - "net" - "time" - - "github.com/inetaf/tcpproxy" - "github.com/sirupsen/logrus" - - "github.com/clusterlink-net/clusterlink/pkg/util/tcp" -) - -// Server for proxying connections by checking client SNI. -type Server struct { - tcp.Listener - - routes map[string]string - server *tcpproxy.Proxy - - logger *logrus.Entry -} - -// Start the server. -func (s *Server) Start() error { - listenAddress := s.GetAddress() - for sni, targetAddress := range s.routes { - target := tcpproxy.To(targetAddress) - target.DialTimeout = 100 * time.Millisecond - s.server.AddSNIRoute(listenAddress, sni, target) - } - - return s.server.Run() -} - -// Stop the server. -func (s *Server) Stop() error { - return s.server.Close() -} - -// GracefulStop does a graceful stop of the server. -func (s *Server) GracefulStop() error { - return s.server.Close() -} - -// NewServer returns a new server. -// routes map (server name) -> (target host:port). -func NewServer(routes map[string]string) *Server { - logger := logrus.WithFields(logrus.Fields{ - "component": "sni-proxy", - }) - - sniproxy := &Server{ - Listener: tcp.NewListener("sni-proxy"), - routes: routes, - server: &tcpproxy.Proxy{}, - logger: logger, - } - - sniproxy.server.ListenFunc = func(_, _ string) (net.Listener, error) { - return sniproxy.GetListener(), nil - } - - return sniproxy -} diff --git a/pkg/util/tls/util.go b/pkg/util/tls/util.go index c39d8cc4b..b550d8728 100644 --- a/pkg/util/tls/util.go +++ b/pkg/util/tls/util.go @@ -21,42 +21,65 @@ import ( ) // ParseFiles parses the given TLS-related files. -func ParseFiles(ca, cert, key string) (*ParsedCertData, error) { +func ParseFiles(ca, cert, key string) (*ParsedCertData, *RawCertData, error) { rawCA, err := os.ReadFile(ca) if err != nil { - return nil, fmt.Errorf("unable to read CA file '%s': %w", ca, err) + return nil, nil, fmt.Errorf("unable to read CA file '%s': %w", ca, err) } rawCertificate, err := os.ReadFile(cert) if err != nil { - return nil, fmt.Errorf("unable to read certificate file: %w", err) + return nil, nil, fmt.Errorf("unable to read certificate file: %w", err) } rawPrivateKey, err := os.ReadFile(key) if err != nil { - return nil, fmt.Errorf("unable to read private key file: %w", err) + return nil, nil, fmt.Errorf("unable to read private key file: %w", err) } certificate, err := tls.X509KeyPair(rawCertificate, rawPrivateKey) if err != nil { - return nil, fmt.Errorf("unable to parse certificate keypair: %w", err) + return nil, nil, fmt.Errorf("unable to parse certificate keypair: %w", err) } caCertPool := x509.NewCertPool() if !caCertPool.AppendCertsFromPEM(rawCA) { - return nil, fmt.Errorf("unable to parse CA file") + return nil, nil, fmt.Errorf("unable to parse CA file") } x509cert, err := x509.ParseCertificate(certificate.Certificate[0]) if err != nil { - return nil, fmt.Errorf("unable to parse x509 certificate: %w", err) + return nil, nil, fmt.Errorf("unable to parse x509 certificate: %w", err) } return &ParsedCertData{ - certificate: certificate, - ca: caCertPool, - x509cert: x509cert, - }, nil + certificate: certificate, + ca: caCertPool, + x509cert: x509cert, + }, &RawCertData{ + certificate: rawCertificate, + key: rawPrivateKey, + ca: rawCA, + }, nil +} + +// RawCertData contains a raw TLS certificate, private key, and CA. +type RawCertData struct { + certificate []byte + key []byte + ca []byte +} + +func (c *RawCertData) Certificate() []byte { + return c.certificate +} + +func (c *RawCertData) Key() []byte { + return c.key +} + +func (c *RawCertData) CA() []byte { + return c.ca } // ParsedCertData contains a parsed CA and TLS certificate. diff --git a/tests/e2e/k8s/suite.go b/tests/e2e/k8s/suite.go index 7b62f8c2d..14ca7f7c5 100644 --- a/tests/e2e/k8s/suite.go +++ b/tests/e2e/k8s/suite.go @@ -149,6 +149,7 @@ func convertCaseCamelToKebab(s string) string { // BeforeTest creates the test namespace before each test, and removes the previous test namespace. func (s *TestSuite) BeforeTest(_, testName string) { + s.fabric.ClearErrors() testName = convertCaseCamelToKebab(testName) if err := s.fabric.SwitchToNewNamespace(testName, false); err != nil { s.T().Fatal(err) diff --git a/tests/e2e/k8s/test_loadbalancing.go b/tests/e2e/k8s/test_loadbalancing.go index 96d2d2964..de88a3fa4 100644 --- a/tests/e2e/k8s/test_loadbalancing.go +++ b/tests/e2e/k8s/test_loadbalancing.go @@ -119,6 +119,16 @@ func (s *TestSuite) TestLoadBalancingStatic() { require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), imp)) + // wait for all peers to be reachable + for i := 0; i < 3; i++ { + require.Nil(s.T(), cl[0].WaitForPeerCondition(&v1alpha1.Peer{ + ObjectMeta: metav1.ObjectMeta{ + Name: cl[i].Name(), + Namespace: cl[i].Namespace(), + }, + }, v1alpha1.PeerReachable, true)) + } + // test static lb scheme for i := 0; i < 30; i++ { data, err := cl[0].AccessService(httpecho.GetEchoValue, importedService, true, nil) diff --git a/tests/e2e/k8s/util/async.go b/tests/e2e/k8s/util/async.go index 45f48ef2a..01a66eae8 100644 --- a/tests/e2e/k8s/util/async.go +++ b/tests/e2e/k8s/util/async.go @@ -63,3 +63,10 @@ func (r *AsyncRunner) Wait() error { r.wg.Wait() return r.Error() } + +// ClearErrors clears all errors. +func (r *AsyncRunner) ClearErrors() { + r.lock.Lock() + defer r.lock.Unlock() + r.err = nil +} diff --git a/tests/e2e/k8s/util/fabric.go b/tests/e2e/k8s/util/fabric.go index a4a3321da..863f5e490 100644 --- a/tests/e2e/k8s/util/fabric.go +++ b/tests/e2e/k8s/util/fabric.go @@ -49,15 +49,41 @@ type peer struct { AsyncRunner cluster *KindCluster + fabricCert *bootstrap.Certificate peerCert *bootstrap.Certificate + caCert *bootstrap.Certificate controlplaneCert *bootstrap.Certificate dataplaneCert *bootstrap.Certificate } +// CreatePeerCertificate creates the peer certificate. +func (p *peer) CreatePeerCertificate() { + p.Run(func() error { + cert, err := bootstrap.CreatePeerCertificate(p.cluster.Name(), p.fabricCert) + if err != nil { + return fmt.Errorf("cannot create peer certificate: %w", err) + } + + p.peerCert = cert + return nil + }) +} + +// CreateCACertificate creates the site CA certificate. +func (p *peer) CreateCACertificate() error { + cert, err := bootstrap.CreateCACertificate() + if err != nil { + return fmt.Errorf("cannot create site CA certificate: %w", err) + } + + p.caCert = cert + return nil +} + // CreateControlplaneCertificate creates the controlplane certificate. func (p *peer) CreateControlplaneCertificate() { p.Run(func() error { - cert, err := bootstrap.CreateControlplaneCertificate(p.cluster.Name(), p.peerCert) + cert, err := bootstrap.CreateControlplaneCertificate(p.caCert) if err != nil { return fmt.Errorf("cannot create controlplane certificate: %w", err) } @@ -70,7 +96,7 @@ func (p *peer) CreateControlplaneCertificate() { // CreateDataplaneCertificate creates the dataplane certificate. func (p *peer) CreateDataplaneCertificate() { p.Run(func() error { - cert, err := bootstrap.CreateDataplaneCertificate(p.cluster.Name(), p.peerCert) + cert, err := bootstrap.CreateDataplaneCertificate(p.caCert) if err != nil { return fmt.Errorf("cannot create dataplane certificate: %w", err) } @@ -95,12 +121,11 @@ func (f *Fabric) CreatePeer(cluster *KindCluster) { p := &peer{cluster: cluster} f.peers = append(f.peers, p) f.Run(func() error { - cert, err := bootstrap.CreatePeerCertificate(p.cluster.Name(), f.cert) - if err != nil { - return fmt.Errorf("cannot create peer certificate: %w", err) + p.fabricCert = f.cert + p.CreatePeerCertificate() + if err := p.CreateCACertificate(); err != nil { + return err } - - p.peerCert = cert p.CreateControlplaneCertificate() p.CreateDataplaneCertificate() diff --git a/tests/e2e/k8s/util/k8s_yaml.go b/tests/e2e/k8s/util/k8s_yaml.go index e90ab6ad4..b1f3b39a6 100644 --- a/tests/e2e/k8s/util/k8s_yaml.go +++ b/tests/e2e/k8s/util/k8s_yaml.go @@ -58,6 +58,7 @@ func (f *Fabric) generateK8SYAML(p *peer, cfg *PeerConfig) (string, error) { Peer: p.cluster.Name(), FabricCertificate: f.cert, PeerCertificate: p.peerCert, + CACertificate: p.caCert, ControlplaneCertificate: p.controlplaneCert, DataplaneCertificate: p.dataplaneCert, Dataplanes: cfg.Dataplanes, @@ -165,6 +166,7 @@ func (f *Fabric) generateClusterlinkSecrets(p *peer) (string, error) { Peer: p.cluster.Name(), FabricCertificate: f.cert, PeerCertificate: p.peerCert, + CACertificate: p.caCert, ControlplaneCertificate: p.controlplaneCert, DataplaneCertificate: p.dataplaneCert, Namespace: f.namespace, diff --git a/website/content/en/docs/main/concepts/peers.md b/website/content/en/docs/main/concepts/peers.md index 81595f047..517daaf02 100644 --- a/website/content/en/docs/main/concepts/peers.md +++ b/website/content/en/docs/main/concepts/peers.md @@ -118,7 +118,7 @@ $ kubectl get secret --namespace clusterlink-system NAME TYPE DATA AGE cl-controlplane Opaque 2 19h cl-dataplane Opaque 2 19h -cl-fabric Opaque 1 19h +cl-ca Opaque 1 19h cl-peer Opaque 1 19h ``` diff --git a/website/content/en/docs/main/tasks/operator.md b/website/content/en/docs/main/tasks/operator.md index 204c575bd..9379481f7 100644 --- a/website/content/en/docs/main/tasks/operator.md +++ b/website/content/en/docs/main/tasks/operator.md @@ -146,8 +146,8 @@ To deploy the ClusterLink without using the CLI, follow the instructions below: ```sh export CERTS = - kubectl create secret generic cl-fabric -n clusterlink-system --from-file=ca=$CERTS /cert.pem - kubectl create secret generic cl-peer -n clusterlink-system --from-file=ca=$CERTS /peer1/cert.pem + kubectl create secret generic cl-ca -n clusterlink-system --from-file=ca=$CERTS /cert.pem + kubectl create secret generic cl-peer -n clusterlink-system --from-file=ca.pem=$CERTS /cert.pem --from-file=cert.pem=$CERTS /peer1/cert.pem --from-file=key.pem=$CERTS /peer1/key.pem kubectl create secret generic cl-controlplane -n clusterlink-system --from-file=cert=$CERTS /peer1/controlplane/cert.pem --from-file=key=$CERTS /peer1/controlplane/key.pem kubectl create secret generic cl-dataplane -n clusterlink-system --from-file=cert=$CERTS /peer1/dataplane/cert.pem --from-file=key=$CERTS /peer1/dataplane/key.pem ``` From 993683a3e6c41d2cb28f81650e42e9aecef09fe0 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Wed, 29 May 2024 14:36:57 +0300 Subject: [PATCH 30/53] website: Add clerification to iperf3 tutorials (#620) 1. Add an explanation about the certificate creation needs to use the same fabric CA. 2. Add a check for peer status to verify peer connectivity. Signed-off-by: Kfir Toledo --- .../en/docs/main/tutorials/iperf/index.md | 4 - .../en/docs/main/tutorials/nginx/index.md | 3 - .../files/tutorials/deploy-clusterlink.md | 5 + website/static/files/tutorials/peer.md | 160 +++++++++++------- 4 files changed, 104 insertions(+), 68 deletions(-) diff --git a/website/content/en/docs/main/tutorials/iperf/index.md b/website/content/en/docs/main/tutorials/iperf/index.md index 9296b65a5..5a6afbfb6 100644 --- a/website/content/en/docs/main/tutorials/iperf/index.md +++ b/website/content/en/docs/main/tutorials/iperf/index.md @@ -105,10 +105,6 @@ In this step, we enable connectivity access between the iPerf3 client and server {{% readfile file="/static/files/tutorials/peer.md" %}} -{{< notice note >}} -The `CLIENT_IP` and `SERVER_IP` refers to the node IP of the peer kind cluster, which assigns the peer YAML file. -{{< /notice >}} - ### Export the iPerf server endpoint In the server cluster, export the iperf3-server service: diff --git a/website/content/en/docs/main/tutorials/nginx/index.md b/website/content/en/docs/main/tutorials/nginx/index.md index e489420ba..bda0d5e80 100644 --- a/website/content/en/docs/main/tutorials/nginx/index.md +++ b/website/content/en/docs/main/tutorials/nginx/index.md @@ -104,9 +104,6 @@ In this step, we enable access between the client and server. {{% readfile file="/static/files/tutorials/peer.md" %}} -{{< notice note >}} -The `CLIENT_IP` and `SERVER_IP` refers to the node IP of the peer kind cluster, which assigns the peer YAML file. -{{< /notice >}} ### Export the nginx server endpoint diff --git a/website/static/files/tutorials/deploy-clusterlink.md b/website/static/files/tutorials/deploy-clusterlink.md index 224cf22cf..5a75c5ed3 100644 --- a/website/static/files/tutorials/deploy-clusterlink.md +++ b/website/static/files/tutorials/deploy-clusterlink.md @@ -14,6 +14,11 @@ clusterlink create peer-cert --name server ``` + All peer certificates (i.e., for the `client` and `server` clusters, in this tutorial) should be created from the same fabric CA files. + In this tutorial, we assume the server has access to the Fabric certificate created in the `default_fabric` folder. + In this tutorial, we assume the `server` cluster creation has access to the fabric certificate stored in the + `default_fabric` folder. If it doesn't, the fabric certificate should be copied from the `client` to the `server`. + For more details regarding fabric and peer see [core concepts][]. 2. Deploy ClusterLink on each cluster: diff --git a/website/static/files/tutorials/peer.md b/website/static/files/tutorials/peer.md index 5867fa33d..dce15284e 100644 --- a/website/static/files/tutorials/peer.md +++ b/website/static/files/tutorials/peer.md @@ -1,64 +1,102 @@ Add the remote peer to each cluster: -*Client cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` -curl -s $TEST_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: Peer -metadata: - name: server - namespace: clusterlink-system -spec: - gateways: - - host: "${SERVER_IP}" - port: 30443 -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} - -*Server cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` -curl -s $TEST_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: Peer -metadata: - name: client - namespace: clusterlink-system -spec: - gateways: - - host: "${CLIENT_IP}" - port: 30443 -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} + *Client cluster*: + + {{< tabpane text=true >}} + {{% tab header="File" %}} + + ```sh + export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` + curl -s $TEST_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - + ``` + + {{% /tab %}} + {{% tab header="Full CR" %}} + + ```sh + export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` + echo " + apiVersion: clusterlink.net/v1alpha1 + kind: Peer + metadata: + name: server + namespace: clusterlink-system + spec: + gateways: + - host: "${SERVER_IP}" + port: 30443 + " | kubectl apply -f - + ``` + + {{% /tab %}} + {{< /tabpane >}} + + *Server cluster*: + + {{< tabpane text=true >}} + {{% tab header="File" %}} + + ```sh + export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` + curl -s $TEST_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - + ``` + + {{% /tab %}} + {{% tab header="Full CR" %}} + + ```sh + export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` + echo " + apiVersion: clusterlink.net/v1alpha1 + kind: Peer + metadata: + name: client + namespace: clusterlink-system + spec: + gateways: + - host: "${CLIENT_IP}" + port: 30443 + " | kubectl apply -f - + ``` + + {{% /tab %}} + {{< /tabpane >}} + + The `CLIENT_IP` and `SERVER_IP` refers to the node IP of the peer kind cluster, which assigns the peer YAML file. + +To verify that the connectivity between the peers is established correctly, +please check if the condition `PeerReachable` has been added to the peer CR status in each cluster. + + ```sh + kubectl describe peers.clusterlink.net -A + ``` + + {{% expand summary="Sample output" %}} + + ``` + Name: client + Namespace: clusterlink-system + Labels: + Annotations: + API Version: clusterlink.net/v1alpha1 + Kind: Peer + Metadata: + Creation Timestamp: 2024-05-28T12:47:33Z + Generation: 1 + Resource Version: 807 + UID: 1fdeafff-707a-43e2-bb3a-826f003a42ed + Spec: + Gateways: + Host: 172.18.0.4 + Port: 30443 + Status: + Conditions: + Last Transition Time: 2024-05-28T12:47:33Z + Message: + Reason: Heartbeat + Status: True + Type: PeerReachable + ``` + + {{% /expand %}} From e5d1f979171eef896073c12c50c1b4f3f53c5cbc Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Sun, 2 Jun 2024 08:41:39 +0300 Subject: [PATCH 31/53] CLI: Enhancement for creation of ClusterLink components' certificates (#623) Create the ClusterLink components' certificates on the deploy peer instead of in the create peer-cert command. Signed-off-by: Kfir Toledo --- cmd/clusterlink/cmd/create/create_peer.go | 67 -------------- cmd/clusterlink/cmd/deploy/deploy_peer.go | 87 +++++++++---------- .../content/en/docs/main/tasks/operator.md | 5 +- 3 files changed, 44 insertions(+), 115 deletions(-) diff --git a/cmd/clusterlink/cmd/create/create_peer.go b/cmd/clusterlink/cmd/create/create_peer.go index 418c7c0b5..656bf47eb 100644 --- a/cmd/clusterlink/cmd/create/create_peer.go +++ b/cmd/clusterlink/cmd/create/create_peer.go @@ -59,24 +59,6 @@ func (o *PeerOptions) saveCertificate(cert *bootstrap.Certificate, outDirectory return os.WriteFile(filepath.Join(outDirectory, config.PrivateKeyFileName), cert.RawKey(), 0o600) } -func (o *PeerOptions) createCA() (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateCACertificate() - if err != nil { - return nil, err - } - - outDirectory := config.CADirectory(o.Name, o.Fabric, o.Path) - if err := os.Mkdir(outDirectory, 0o755); err != nil { - return nil, err - } - - if err := o.saveCertificate(cert, outDirectory); err != nil { - return nil, err - } - - return cert, nil -} - func (o *PeerOptions) createPeerCert(fabricCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { cert, err := bootstrap.CreatePeerCertificate(o.Name, fabricCert) if err != nil { @@ -95,42 +77,6 @@ func (o *PeerOptions) createPeerCert(fabricCert *bootstrap.Certificate) (*bootst return cert, nil } -func (o *PeerOptions) createControlplane(caCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateControlplaneCertificate(caCert) - if err != nil { - return nil, err - } - - outDirectory := config.ControlplaneDirectory(o.Name, o.Fabric, o.Path) - if err := os.Mkdir(outDirectory, 0o755); err != nil { - return nil, err - } - - if err := o.saveCertificate(cert, outDirectory); err != nil { - return nil, err - } - - return cert, nil -} - -func (o *PeerOptions) createDataplane(caCert *bootstrap.Certificate) (*bootstrap.Certificate, error) { - cert, err := bootstrap.CreateDataplaneCertificate(caCert) - if err != nil { - return nil, err - } - - outDirectory := config.DataplaneDirectory(o.Name, o.Fabric, o.Path) - if err := os.Mkdir(outDirectory, 0o755); err != nil { - return nil, err - } - - if err := o.saveCertificate(cert, outDirectory); err != nil { - return nil, err - } - - return cert, nil -} - // Run the 'create peer-cert' subcommand. func (o *PeerOptions) Run() error { if _, err := idna.Lookup.ToASCII(o.Name); err != nil { @@ -150,19 +96,6 @@ func (o *PeerOptions) Run() error { return err } - caCert, err := o.createCA() - if err != nil { - return err - } - - if _, err := o.createControlplane(caCert); err != nil { - return err - } - - if _, err := o.createDataplane(caCert); err != nil { - return err - } - return nil } diff --git a/cmd/clusterlink/cmd/deploy/deploy_peer.go b/cmd/clusterlink/cmd/deploy/deploy_peer.go index 35ec7c3f1..aff94c0c7 100644 --- a/cmd/clusterlink/cmd/deploy/deploy_peer.go +++ b/cmd/clusterlink/cmd/deploy/deploy_peer.go @@ -41,11 +41,12 @@ import ( const ( // StartAll deploys the clusterlink operator, converts the peer certificates to secrets, - // and deploys the operator ClusterLink custom resource to create the ClusterLink components. + // creates and deploys the operator ClusterLink custom resource to create the ClusterLink components. StartAll = "all" // StartOperator deploys only the operator and converts the peer certificates to secrets. + // Creates a custom resource example file that can be deployed to the operator. StartOperator = "operator" - // NoStart doesn't deploy anything but creates custom resource YAMLs. + // NoStart doesn't deploy the operator and creates a "k8s.yaml" file that allow to deploy ClusterLink without the operator. NoStart = "none" ) @@ -153,34 +154,29 @@ func (o *PeerOptions) Run() error { return err } // Read certificates - fabricCert, err := bootstrap.ReadCertificates( - config.FabricDirectory(o.Fabric, o.Path), false) + fabricCert, err := bootstrap.ReadCertificates(config.FabricDirectory(o.Fabric, o.Path), false) if err != nil { - return err + return fmt.Errorf("failed to read fabric certificate: %w", err) } - peerCert, err := bootstrap.ReadCertificates( - config.PeerDirectory(o.Name, o.Fabric, o.Path), true) + peerCert, err := bootstrap.ReadCertificates(config.PeerDirectory(o.Name, o.Fabric, o.Path), true) if err != nil { - return err + return fmt.Errorf("failed to read peer certificate: %w", err) } - caCert, err := bootstrap.ReadCertificates( - config.CADirectory(o.Name, o.Fabric, o.Path), false) + caCert, err := bootstrap.CreateCACertificate() if err != nil { - return err + return fmt.Errorf("failed to create CA certificate: %w", err) } - controlplaneCert, err := bootstrap.ReadCertificates( - config.ControlplaneDirectory(o.Name, o.Fabric, o.Path), true) + controlplaneCert, err := bootstrap.CreateControlplaneCertificate(caCert) if err != nil { - return err + return fmt.Errorf("failed to create controlplane certificates: %w", err) } - dataplaneCert, err := bootstrap.ReadCertificates( - config.DataplaneDirectory(o.Name, o.Fabric, o.Path), true) + dataplaneCert, err := bootstrap.CreateDataplaneCertificate(caCert) if err != nil { - return err + return fmt.Errorf("failed to create dataplane certificates: %w", err) } // Create k8s deployment YAML @@ -201,28 +197,18 @@ func (o *PeerOptions) Run() error { Tag: o.Tag, } - k8sConfig, err := platform.K8SConfig(platformCfg) - if err != nil { - return err - } - - outPath := filepath.Join(peerDir, config.K8SYAMLFile) - if err := os.WriteFile(outPath, k8sConfig, 0o600); err != nil { - return err - } - - // Create clusterlink instance YAML for the operator. - clConfig, err := platform.K8SClusterLinkInstanceConfig(platformCfg, "cl-instance") - if err != nil { - return err - } + if o.StartInstance == NoStart { + // Create a YAML file for deployment without using the operator. + k8sConfig, err := platform.K8SConfig(platformCfg) + if err != nil { + return err + } - clOutPath := filepath.Join(peerDir, config.K8SClusterLinkInstanceYAMLFile) - if err := os.WriteFile(clOutPath, clConfig, 0o600); err != nil { - return err - } + outPath := filepath.Join(peerDir, config.K8SYAMLFile) + if err := os.WriteFile(outPath, k8sConfig, 0o600); err != nil { + return fmt.Errorf("failed to write YAML file: %w", err) + } - if o.StartInstance == NoStart { return nil } @@ -253,6 +239,7 @@ func (o *PeerOptions) Run() error { if err := o.deployDir("operator/rbac/*", resource); err != nil { return err } + if err := o.deployDir("crds/*", resource); err != nil { return err } @@ -262,6 +249,7 @@ func (o *PeerOptions) Run() error { if err != nil { return err } + err = decoder.DecodeEach( context.Background(), strings.NewReader(string(secretConfig)), @@ -272,23 +260,30 @@ func (o *PeerOptions) Run() error { return err } - // Create ClusterLink instance - if o.StartInstance == StartAll { - if o.IngressPort != apis.DefaultExternalPort { - platformCfg.IngressPort = o.IngressPort - } + // Create clusterlink instance YAML for the operator. + if o.IngressPort != apis.DefaultExternalPort { // Set the port config only if it has changed. + platformCfg.IngressPort = o.IngressPort + } - instance, err := platform.K8SClusterLinkInstanceConfig(platformCfg, "cl-instance") - if err != nil { - return err - } + instance, err := platform.K8SClusterLinkInstanceConfig(platformCfg, "cl-instance") + if err != nil { + return err + } + // Create ClusterLink instance + if o.StartInstance == StartAll { err = decoder.DecodeEach(context.Background(), strings.NewReader(string(instance)), decoder.CreateHandler(resource)) if errors.IsAlreadyExists(err) { fmt.Println("CRD instance for ClusterLink (\"cl-instance\") was already exist.") } else if err != nil { return err } + } else { + // Store an example for clusterlink instance YAML. + clOutPath := filepath.Join(peerDir, config.K8SClusterLinkInstanceYAMLFile) + if err := os.WriteFile(clOutPath, instance, 0o600); err != nil { + return fmt.Errorf("failed to write YAML file: %w", err) + } } return nil diff --git a/website/content/en/docs/main/tasks/operator.md b/website/content/en/docs/main/tasks/operator.md index 9379481f7..f70239198 100644 --- a/website/content/en/docs/main/tasks/operator.md +++ b/website/content/en/docs/main/tasks/operator.md @@ -116,7 +116,8 @@ The `deploy peer` {{< anchor commandline-flags >}} command has the following fla `all` (default) starts the clusterlink operator, converts the peer certificates to secrets, and deploys the operator ClusterLink custom resource to create the ClusterLink components. `operator` deploys only the `ClusterLink` operator and convert the peer certificates to secrets. - `none` doesn't deploy anything but creates the ClusterLink custom resource YAML. + Creates a custom resource example file that can be deployed to the operator. + `none` doesn't deploy the operator and creates a `k8s.yaml` file that allows deploying ClusterLink without the operator. - **path**: Represents the path where the peer and fabric certificates are stored, by default is the working current working directory. @@ -165,6 +166,6 @@ To deploy the ClusterLink without using the CLI, follow the instructions below: ``` [user guide]: {{< relref "../getting-started/users#setup" >}} -[ClusterLink tutorials]: {{< relref "../tutorials/" >}} +[ClusterLink tutorials]: {{< relref "../tutorials/" >}} [here]: https://kind.sigs.k8s.io/docs/user/loadbalancer/ [common use case]: #the-common-use-case From 2c583c10eb13be3fdb29531ebe14f7ad85debb1a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Jun 2024 09:18:20 +0300 Subject: [PATCH 32/53] build(deps): bump alpine from 3.19 to 3.20 in /cmd/cl-go-dataplane (#625) Bumps alpine from 3.19 to 3.20. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kfir Toledo --- cmd/cl-go-dataplane/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/cl-go-dataplane/Dockerfile b/cmd/cl-go-dataplane/Dockerfile index c79348063..88f4e9adb 100644 --- a/cmd/cl-go-dataplane/Dockerfile +++ b/cmd/cl-go-dataplane/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:3.19 +FROM alpine:3.20 # Populated during the build process, for example, with 'arm64' or 'amd64'. ARG TARGETARCH From 7f526e0a3a81cbd3457e5d758b249712b7ba1d70 Mon Sep 17 00:00:00 2001 From: Ziv Nevo <79099626+zivnevo@users.noreply.github.com> Date: Mon, 3 Jun 2024 11:11:23 +0300 Subject: [PATCH 33/53] Setup Go with the Go version in go.mod (#627) Signed-off-by: Ziv Nevo --- .github/workflows/pr-check.yml | 17 +++++++---------- .github/workflows/pr-e2e-test.yml | 5 +---- 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/.github/workflows/pr-check.yml b/.github/workflows/pr-check.yml index 43e2ae749..493b88b19 100644 --- a/.github/workflows/pr-check.yml +++ b/.github/workflows/pr-check.yml @@ -19,7 +19,7 @@ jobs: - name: Setup Go uses: actions/setup-go@v5 with: - go-version: '1.22' + go-version-file: ./go.mod - name: Setup goimports run: go install golang.org/x/tools/cmd/goimports@v0.13.0 - name: Check go.mod and go.sum @@ -38,20 +38,17 @@ jobs: unit-tests: runs-on: ubuntu-latest - strategy: - matrix: - go: ['1.22'] steps: - - name: set up go 1.x - uses: actions/setup-go@v5 - with: - go-version: ${{ matrix.go }} - - name: setup tparse - run: go install github.com/mfridman/tparse@latest - name: checkout uses: actions/checkout@v4 with: fetch-tags: true + - name: set up go + uses: actions/setup-go@v5 + with: + go-version-file: ./go.mod + - name: setup tparse + run: go install github.com/mfridman/tparse@latest - name: run build run: make build - name: run unit tests diff --git a/.github/workflows/pr-e2e-test.yml b/.github/workflows/pr-e2e-test.yml index 34303d893..d3daa6752 100644 --- a/.github/workflows/pr-e2e-test.yml +++ b/.github/workflows/pr-e2e-test.yml @@ -8,9 +8,6 @@ on: jobs: e2e-connectivity-test: runs-on: ubuntu-latest - strategy: - matrix: - go: ['1.22'] steps: - name: checkout @@ -20,7 +17,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v5 with: - go-version: ${{ matrix.go }} + go-version-file: ./go.mod - name: Install kind uses: helm/kind-action@v1.10.0 with: From e2110e9dcf5ed5efaf41ca2daa64fd55ea68bcfa Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Mon, 3 Jun 2024 10:09:32 +0300 Subject: [PATCH 34/53] website: Fix links in 0.1 and 0.2 This commit fixes the policy engine examples links in v0.1 and v0.2 Signed-off-by: Or Ozeri --- website/content/en/docs/v0.1/concepts/policies.md | 2 +- website/content/en/docs/v0.2/concepts/policies.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/en/docs/v0.1/concepts/policies.md b/website/content/en/docs/v0.1/concepts/policies.md index 557800d7b..5b57363a8 100644 --- a/website/content/en/docs/v0.1/concepts/policies.md +++ b/website/content/en/docs/v0.1/concepts/policies.md @@ -115,7 +115,7 @@ spec: - workloadSelector: {} ``` -More examples are available [here](https://github.com/clusterlink-net/clusterlink/tree/main/pkg/policyengine/examples) +More examples are available [here](https://github.com/clusterlink-net/clusterlink/tree/v0.1.0/pkg/policyengine/examples) [concept-peer]: {{< relref "peers" >}} [concept-service]: {{< relref "services" >}} diff --git a/website/content/en/docs/v0.2/concepts/policies.md b/website/content/en/docs/v0.2/concepts/policies.md index 3cd4d0ce0..44310e965 100644 --- a/website/content/en/docs/v0.2/concepts/policies.md +++ b/website/content/en/docs/v0.2/concepts/policies.md @@ -145,7 +145,7 @@ spec: - workloadSelector: {} ``` -More examples are available [here](https://github.com/clusterlink-net/clusterlink/tree/main/pkg/policyengine/examples) +More examples are available [here](https://github.com/clusterlink-net/clusterlink/tree/v0.2.0/pkg/policyengine/examples) [concept-peer]: {{< relref "peers" >}} [concept-service]: {{< relref "services" >}} From 86e613be7275f49d67b4ed330c2c3927f7a66943 Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Mon, 3 Jun 2024 10:10:51 +0300 Subject: [PATCH 35/53] tests/e2e/k8s: Fix failing load-balancing test This commit fixes a sporadically failing test. Signed-off-by: Or Ozeri --- tests/e2e/k8s/test_loadbalancing.go | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/e2e/k8s/test_loadbalancing.go b/tests/e2e/k8s/test_loadbalancing.go index de88a3fa4..dc864ab13 100644 --- a/tests/e2e/k8s/test_loadbalancing.go +++ b/tests/e2e/k8s/test_loadbalancing.go @@ -59,6 +59,16 @@ func (s *TestSuite) TestLoadBalancingRoundRobin() { require.Nil(s.T(), cl[0].Cluster().Resources().Create(context.Background(), imp)) + // wait for all peers to be reachable + for i := 0; i < 3; i++ { + require.Nil(s.T(), cl[0].WaitForPeerCondition(&v1alpha1.Peer{ + ObjectMeta: metav1.ObjectMeta{ + Name: cl[i].Name(), + Namespace: cl[i].Namespace(), + }, + }, v1alpha1.PeerReachable, true)) + } + // test default lb scheme (round-robin) for i := 0; i < 30; i++ { data, err := cl[0].AccessService(httpecho.GetEchoValue, importedService, true, nil) From 25c635ccdb6c8785c59f98999fcc4c4bdef33a6e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Jun 2024 16:25:42 +0300 Subject: [PATCH 36/53] build(deps): bump sigs.k8s.io/e2e-framework from 0.3.0 to 0.4.0 in the k8s group (#626) * build(deps): bump sigs.k8s.io/e2e-framework in the k8s group Bumps the k8s group with 1 update: [sigs.k8s.io/e2e-framework](https://github.com/kubernetes-sigs/e2e-framework). Updates `sigs.k8s.io/e2e-framework` from 0.3.0 to 0.4.0 - [Release notes](https://github.com/kubernetes-sigs/e2e-framework/releases) - [Changelog](https://github.com/kubernetes-sigs/e2e-framework/blob/main/RELEASE.md) - [Commits](https://github.com/kubernetes-sigs/e2e-framework/compare/v0.3.0...v0.4.0) --- updated-dependencies: - dependency-name: sigs.k8s.io/e2e-framework dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s ... Signed-off-by: dependabot[bot] * update Go to 1.22.3 Signed-off-by: Etai Lev Ran --------- Signed-off-by: dependabot[bot] Signed-off-by: Etai Lev Ran Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Etai Lev Ran --- go.mod | 6 ++---- go.sum | 8 ++++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index 542ff6c5d..4b6c70ac6 100644 --- a/go.mod +++ b/go.mod @@ -1,8 +1,6 @@ module github.com/clusterlink-net/clusterlink -go 1.22.0 - -toolchain go1.22.2 +go 1.22.3 require ( github.com/bombsimon/logrusr/v4 v4.1.0 @@ -23,7 +21,7 @@ require ( k8s.io/client-go v0.30.1 k8s.io/utils v0.0.0-20230726121419-3b25d923346b sigs.k8s.io/controller-runtime v0.18.3 - sigs.k8s.io/e2e-framework v0.3.0 + sigs.k8s.io/e2e-framework v0.4.0 ) require ( diff --git a/go.sum b/go.sum index 0c3701105..7030daecf 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPd github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= +github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ= github.com/bombsimon/logrusr/v4 v4.1.0 h1:uZNPbwusB0eUXlO8hIUwStE6Lr5bLN6IgYgG+75kuh4= github.com/bombsimon/logrusr/v4 v4.1.0/go.mod h1:pjfHC5e59CvjTBIU3V3sGhFWFAnsnhOR03TRc6im0l8= github.com/census-instrumentation/opencensus-proto v0.4.1 h1:iKLQ0xPNFxR/2hzXZMrBo8f1j86j5WHzznCCQxV/b8g= @@ -274,6 +276,8 @@ k8s.io/apimachinery v0.30.1 h1:ZQStsEfo4n65yAdlGTfP/uSHMQSoYzU/oeEbkmF7P2U= k8s.io/apimachinery v0.30.1/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= k8s.io/client-go v0.30.1 h1:uC/Ir6A3R46wdkgCV3vbLyNOYyCJ8oZnjtJGKfytl/Q= k8s.io/client-go v0.30.1/go.mod h1:wrAqLNs2trwiCH/wxxmT/x3hKVH9PuV0GGW0oDoHVqc= +k8s.io/component-base v0.30.1 h1:bvAtlPh1UrdaZL20D9+sWxsJljMi0QZ3Lmw+kmZAaxQ= +k8s.io/component-base v0.30.1/go.mod h1:e/X9kDiOebwlI41AvBHuWdqFriSRrX50CdwA9TFaHLI= k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= @@ -282,8 +286,8 @@ k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSn k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= sigs.k8s.io/controller-runtime v0.18.3 h1:B5Wmmo8WMWK7izei+2LlXLVDGzMwAHBNLX68lwtlSR4= sigs.k8s.io/controller-runtime v0.18.3/go.mod h1:TVoGrfdpbA9VRFaRnKgk9P5/atA0pMwq+f+msb9M8Sg= -sigs.k8s.io/e2e-framework v0.3.0 h1:eqQALBtPCth8+ulTs6lcPK7ytV5rZSSHJzQHZph4O7U= -sigs.k8s.io/e2e-framework v0.3.0/go.mod h1:C+ef37/D90Dc7Xq1jQnNbJYscrUGpxrWog9bx2KIa+c= +sigs.k8s.io/e2e-framework v0.4.0 h1:4yYmFDNNoTnazqmZJXQ6dlQF1vrnDbutmxlyvBpC5rY= +sigs.k8s.io/e2e-framework v0.4.0/go.mod h1:JilFQPF1OL1728ABhMlf9huse7h+uBJDXl9YeTs49A8= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= From 0906272c61cd1bc98865ec000169f5307382c9b5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 3 Jun 2024 17:21:43 +0300 Subject: [PATCH 37/53] build(deps): bump alpine from 3.19 to 3.20 in /cmd/cl-controlplane (#624) Bumps alpine from 3.19 to 3.20. --- updated-dependencies: - dependency-name: alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- cmd/cl-controlplane/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/cl-controlplane/Dockerfile b/cmd/cl-controlplane/Dockerfile index 04b36ae7c..ee402cc49 100644 --- a/cmd/cl-controlplane/Dockerfile +++ b/cmd/cl-controlplane/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:3.19 +FROM alpine:3.20 # Populated during the build process, for example, with 'arm64' or 'amd64'. ARG TARGETARCH From fc5412fa9ef4b30504c52a96d76662fe1f1b388a Mon Sep 17 00:00:00 2001 From: Or Ozeri Date: Sun, 2 Jun 2024 12:40:44 +0300 Subject: [PATCH 38/53] cmd/cl-controlplane: Support dynamic peer certificates This commit changes the controlplane to watch the peer certificate files for changes, in order to support dynamic peer certificates. Signed-off-by: Or Ozeri --- cmd/cl-controlplane/app/server.go | 22 ++-- go.mod | 2 +- pkg/controlplane/authz/manager.go | 4 +- pkg/controlplane/control/peer.go | 6 +- pkg/controlplane/peer/certs.go | 166 ++++++++++++++++++++++++++++++ pkg/controlplane/xds/manager.go | 2 +- pkg/dataplane/client/xds.go | 2 + pkg/util/runnable/manager.go | 2 +- tests/e2e/k8s/test_dynamic.go | 97 +++++++++++++++++ tests/e2e/k8s/util/clusterlink.go | 65 ++++++++++++ 10 files changed, 351 insertions(+), 17 deletions(-) create mode 100644 pkg/controlplane/peer/certs.go create mode 100644 tests/e2e/k8s/test_dynamic.go diff --git a/cmd/cl-controlplane/app/server.go b/cmd/cl-controlplane/app/server.go index 7a433914e..09846b2c4 100644 --- a/cmd/cl-controlplane/app/server.go +++ b/cmd/cl-controlplane/app/server.go @@ -35,6 +35,7 @@ import ( "github.com/clusterlink-net/clusterlink/pkg/controlplane/api" "github.com/clusterlink-net/clusterlink/pkg/controlplane/authz" "github.com/clusterlink-net/clusterlink/pkg/controlplane/control" + "github.com/clusterlink-net/clusterlink/pkg/controlplane/peer" "github.com/clusterlink-net/clusterlink/pkg/controlplane/xds" "github.com/clusterlink-net/clusterlink/pkg/util/controller" "github.com/clusterlink-net/clusterlink/pkg/util/grpc" @@ -105,7 +106,6 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) { // Run the various controlplane servers. func (o *Options) Run() error { // set log file - f, err := log.Set(o.LogLevel, o.LogFile) if err != nil { return err @@ -131,11 +131,8 @@ func (o *Options) Run() error { return err } - peerCertData, rawPeerCertData, err := tls.ParseFiles( + peerCertsWatcher := peer.NewWatcher( FabricCertificateFilePath(), PeerCertificateFilePath(), PeerKeyFilePath()) - if err != nil { - return err - } config, err := rest.InClusterConfig() if err != nil { @@ -185,9 +182,7 @@ func (o *Options) Run() error { return fmt.Errorf("cannot create authorization manager: %w", err) } - if err := authzManager.SetPeerCertificates(peerCertData); err != nil { - return fmt.Errorf("authorization manager cannot set peer certificates: %w", err) - } + peerCertsWatcher.AddConsumer(authzManager) err = authz.CreateControllers(authzManager, mgr) if err != nil { @@ -197,7 +192,7 @@ func (o *Options) Run() error { authz.RegisterService(authzManager, grpcServer.GetGRPCServer()) controlManager := control.NewManager(mgr.GetClient(), namespace) - controlManager.SetPeerCertificates(peerCertData) + peerCertsWatcher.AddConsumer(controlManager) err = control.CreateControllers(controlManager, mgr) if err != nil { @@ -207,15 +202,18 @@ func (o *Options) Run() error { xdsManager := xds.NewManager() xds.RegisterService( context.Background(), xdsManager, grpcServer.GetGRPCServer()) - if err := xdsManager.SetPeerCertificates(rawPeerCertData); err != nil { - return err - } + peerCertsWatcher.AddConsumer(xdsManager) if err := xds.CreateControllers(xdsManager, mgr); err != nil { return fmt.Errorf("cannot create xDS controllers: %w", err) } + if err := peerCertsWatcher.ReadCertsAndUpdateConsumers(); err != nil { + return err + } + runnableManager := runnable.NewManager() + runnableManager.Add(peerCertsWatcher) runnableManager.Add(controller.NewManager(mgr)) runnableManager.Add(controlManager) runnableManager.AddServer(controlplaneServerListenAddress, grpcServer) diff --git a/go.mod b/go.mod index 4b6c70ac6..f0707ecdb 100644 --- a/go.mod +++ b/go.mod @@ -5,6 +5,7 @@ go 1.22.3 require ( github.com/bombsimon/logrusr/v4 v4.1.0 github.com/envoyproxy/go-control-plane v0.12.0 + github.com/fsnotify/fsnotify v1.7.0 github.com/go-chi/chi v4.1.2+incompatible github.com/google/uuid v1.6.0 github.com/lestrrat-go/jwx v1.2.29 @@ -35,7 +36,6 @@ require ( github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect github.com/evanphx/json-patch v5.6.0+incompatible // indirect github.com/evanphx/json-patch/v5 v5.9.0 // indirect - github.com/fsnotify/fsnotify v1.7.0 // indirect github.com/go-logr/logr v1.4.1 // indirect github.com/go-logr/zapr v1.3.0 // indirect github.com/go-openapi/jsonpointer v0.19.6 // indirect diff --git a/pkg/controlplane/authz/manager.go b/pkg/controlplane/authz/manager.go index 608ac28dd..28ac357f0 100644 --- a/pkg/controlplane/authz/manager.go +++ b/pkg/controlplane/authz/manager.go @@ -385,7 +385,9 @@ func (m *Manager) getPeerName() string { return m.peerName } -func (m *Manager) SetPeerCertificates(peerTLS *tls.ParsedCertData) error { +func (m *Manager) SetPeerCertificates(peerTLS *tls.ParsedCertData, _ *tls.RawCertData) error { + m.logger.Info("Setting peer certificates.") + dnsNames := peerTLS.DNSNames() if len(dnsNames) == 0 { return fmt.Errorf("expected peer certificate to contain at least one DNS name") diff --git a/pkg/controlplane/control/peer.go b/pkg/controlplane/control/peer.go index 78abc8b4c..e5189250a 100644 --- a/pkg/controlplane/control/peer.go +++ b/pkg/controlplane/control/peer.go @@ -312,7 +312,9 @@ func newPeerMonitor(pr *v1alpha1.Peer, manager *peerManager) *peerMonitor { return monitor } -func (m *peerManager) SetPeerCertificates(peerTLS *tls.ParsedCertData) { +func (m *peerManager) SetPeerCertificates(peerTLS *tls.ParsedCertData, _ *tls.RawCertData) error { + m.logger.Info("Setting peer certificates.") + m.peerTLSLock.Lock() defer m.peerTLSLock.Unlock() @@ -324,6 +326,8 @@ func (m *peerManager) SetPeerCertificates(peerTLS *tls.ParsedCertData) { for _, mon := range m.monitors { mon.SetClientCertificates(peerTLS) } + + return nil } // newPeerManager returns a new empty peerManager. diff --git a/pkg/controlplane/peer/certs.go b/pkg/controlplane/peer/certs.go new file mode 100644 index 000000000..1463cb4bd --- /dev/null +++ b/pkg/controlplane/peer/certs.go @@ -0,0 +1,166 @@ +// Copyright (c) The ClusterLink Authors. +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package peer + +import ( + "errors" + "fmt" + "path" + "time" + + "github.com/fsnotify/fsnotify" + "github.com/sirupsen/logrus" + + "github.com/clusterlink-net/clusterlink/pkg/util/tls" +) + +// CertsConsumer represents a consumer of peer TLS certificates. +type CertsConsumer interface { + SetPeerCertificates(parsedCertData *tls.ParsedCertData, rawCertData *tls.RawCertData) error +} + +// parseError represents an error in parsing the peer certificates. +type parseError struct { + err error +} + +func (e parseError) Error() string { + return e.err.Error() +} + +// CertsWatcher watches certificate updates. +type CertsWatcher struct { + caPath string + certPath string + keyPath string + + stopCh chan struct{} + consumers []CertsConsumer + + logger *logrus.Entry +} + +// Name of the watcher. +func (w *CertsWatcher) Name() string { + return "certs-watcher" +} + +// AddConsumer adds a new peer certificates consumer. +// This function is not thread-safe. +func (w *CertsWatcher) AddConsumer(consumer CertsConsumer) { + w.consumers = append(w.consumers, consumer) +} + +// ReadCertsAndUpdateConsumers reads the peer certificates and updates the consumers. +func (w *CertsWatcher) ReadCertsAndUpdateConsumers() error { + w.logger.Infof("Updating certificates.") + + parsedCertData, rawCertData, err := tls.ParseFiles(w.caPath, w.certPath, w.keyPath) + if err != nil { + return &parseError{err: err} + } + + for _, consumer := range w.consumers { + if err := consumer.SetPeerCertificates(parsedCertData, rawCertData); err != nil { + return fmt.Errorf("error setting peer certificates on %v: %w", consumer, err) + } + } + + return nil +} + +// Start the certs watcher. +func (w *CertsWatcher) Start() error { + watcher, err := fsnotify.NewWatcher() + if err != nil { + return fmt.Errorf("cannot initialize file watcher: %w", err) + } + + defer func() { + if err := watcher.Close(); err != nil { + w.logger.Warnf("Cannot close watcher: %v", err) + } + }() + + watchedFiles := []string{w.caPath, w.certPath, w.keyPath} + watchedDirs := make(map[string]interface{}) + for _, file := range watchedFiles { + dir := path.Dir(file) + if _, ok := watchedDirs[dir]; !ok { + w.logger.Infof("Watching: %s.", dir) + if err := watcher.Add(dir); err != nil { + return fmt.Errorf("cannot watch directory '%s': %w", dir, err) + } + watchedDirs[dir] = nil + } + } + + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + + certsModified := false + for { + select { + case <-w.stopCh: + return nil + case event := <-watcher.Events: + w.logger.Debugf("Event: %v", event) + certsModified = true + case err := <-watcher.Errors: + w.logger.Errorf("Error: %v", err) + return err + case <-ticker.C: + if !certsModified { + continue + } + + w.logger.Infof("Certificates modified.") + certsModified = false + + if err = w.ReadCertsAndUpdateConsumers(); err == nil { + continue + } + + w.logger.Infof("Error: %v", err) + + if !errors.Is(err, &parseError{}) { + return err + } + + w.logger.Errorf("Error parsing peer TLS certificates: %v.", err) + } + } +} + +// Stop the watcher. +func (w *CertsWatcher) Stop() error { + close(w.stopCh) + return nil +} + +// GracefulStop does a graceful stop of the watcher. +func (w *CertsWatcher) GracefulStop() error { + return w.Stop() +} + +// NewWatcher returns a new certificate files watcher. +func NewWatcher(caPath, certPath, keyPath string) *CertsWatcher { + return &CertsWatcher{ + caPath: caPath, + certPath: certPath, + keyPath: keyPath, + stopCh: make(chan struct{}), + logger: logrus.WithField("component", "controlplane.peer.certs-watcher"), + } +} diff --git a/pkg/controlplane/xds/manager.go b/pkg/controlplane/xds/manager.go index cb2f13610..c5a420520 100644 --- a/pkg/controlplane/xds/manager.go +++ b/pkg/controlplane/xds/manager.go @@ -212,7 +212,7 @@ func (m *Manager) DeleteImport(name types.NamespacedName) error { } // SetPeerCertificates sets the TLS certificates used for peer-to-peer communication. -func (m *Manager) SetPeerCertificates(rawCertData *utiltls.RawCertData) error { +func (m *Manager) SetPeerCertificates(_ *utiltls.ParsedCertData, rawCertData *utiltls.RawCertData) error { m.logger.Info("Setting peer certificates.") certificateSecret := &tls.Secret{ diff --git a/pkg/dataplane/client/xds.go b/pkg/dataplane/client/xds.go index 66a73737d..0ddad1dcc 100644 --- a/pkg/dataplane/client/xds.go +++ b/pkg/dataplane/client/xds.go @@ -18,6 +18,7 @@ import ( "errors" "fmt" "sync" + "time" "github.com/envoyproxy/go-control-plane/pkg/resource/v3" "github.com/sirupsen/logrus" @@ -44,6 +45,7 @@ func (x *XDSClient) runFetcher(resourceType string) error { fetcher, err := newFetcher(context.Background(), x.controlplaneClient, resourceType, x.dataplane) if err != nil { x.logger.Errorf("Failed to initialize %s fetcher: %v.", resourceType, err) + time.Sleep(time.Second) continue } x.logger.Infof("Successfully initialized client for %s type.", resourceType) diff --git a/pkg/util/runnable/manager.go b/pkg/util/runnable/manager.go index 433275c5e..ee7949cd0 100644 --- a/pkg/util/runnable/manager.go +++ b/pkg/util/runnable/manager.go @@ -21,7 +21,7 @@ import ( "github.com/sirupsen/logrus" ) -// Runnable represents a runnable instance. +// Instance represents a runnable instance. type Instance interface { Name() string Start() error diff --git a/tests/e2e/k8s/test_dynamic.go b/tests/e2e/k8s/test_dynamic.go new file mode 100644 index 000000000..80fdf07ac --- /dev/null +++ b/tests/e2e/k8s/test_dynamic.go @@ -0,0 +1,97 @@ +// Copyright (c) The ClusterLink Authors. +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package k8s + +import ( + "github.com/stretchr/testify/require" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + "github.com/clusterlink-net/clusterlink/cmd/clusterlink/config" + "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" + "github.com/clusterlink-net/clusterlink/pkg/bootstrap" + "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services" + "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services/httpecho" + "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/util" +) + +func (s *TestSuite) TestDynamicPeerCertificates() { + s.RunOnAllDataplaneTypes(func(cfg *util.PeerConfig) { + cl, err := s.fabric.DeployClusterlinks(2, cfg) + require.Nil(s.T(), err) + + require.Nil(s.T(), cl[0].CreateService(&httpEchoService)) + require.Nil(s.T(), cl[0].CreateExport(&httpEchoService)) + require.Nil(s.T(), cl[0].CreatePolicy(util.PolicyAllowAll)) + require.Nil(s.T(), cl[1].CreatePeer(cl[0])) + + importedService := &util.Service{ + Name: httpEchoService.Name, + Port: 80, + } + require.Nil(s.T(), cl[1].CreateImport(importedService, cl[0], httpEchoService.Name)) + + require.Nil(s.T(), cl[1].CreatePolicy(util.PolicyAllowAll)) + + _, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, true, nil) + require.Nil(s.T(), err) + + // create a new fabric certificate + fabricCert, err := bootstrap.CreateFabricCertificate(config.DefaultFabric) + require.Nil(s.T(), err) + + // create new peer certificates + var peerCerts []*bootstrap.Certificate + for i := 0; i < 2; i++ { + peerCert, err := bootstrap.CreatePeerCertificate(cl[0].Name(), fabricCert) + require.Nil(s.T(), err) + peerCerts = append(peerCerts, peerCert) + } + + // update peer certificates on cl[1] + require.Nil(s.T(), cl[1].UpdatePeerCertificates(fabricCert, peerCerts[0])) + + // verify peer becomes unreachable + require.Nil(s.T(), cl[1].WaitForPeerCondition( + &v1alpha1.Peer{ + ObjectMeta: metav1.ObjectMeta{ + Name: cl[0].Name(), + Namespace: cl[0].Namespace(), + }, + }, + v1alpha1.PeerReachable, + false)) + + // verify service is no longer accessible + _, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, false, &services.ConnectionResetError{}) + require.ErrorIs(s.T(), err, &services.ConnectionResetError{}) + + // update peer certificates on cl[0] + require.Nil(s.T(), cl[0].UpdatePeerCertificates(fabricCert, peerCerts[1])) + + // verify peer becomes reachable + require.Nil(s.T(), cl[1].WaitForPeerCondition( + &v1alpha1.Peer{ + ObjectMeta: metav1.ObjectMeta{ + Name: cl[0].Name(), + Namespace: cl[0].Namespace(), + }, + }, + v1alpha1.PeerReachable, + true)) + + // verify access is back + _, err = cl[1].AccessService(httpecho.GetEchoValue, importedService, false, nil) + require.Nil(s.T(), err) + }) +} diff --git a/tests/e2e/k8s/util/clusterlink.go b/tests/e2e/k8s/util/clusterlink.go index 6eadf4afa..663d2c071 100644 --- a/tests/e2e/k8s/util/clusterlink.go +++ b/tests/e2e/k8s/util/clusterlink.go @@ -15,14 +15,20 @@ package util import ( "context" + "encoding/json" "errors" "fmt" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/e2e-framework/klient/k8s" + "sigs.k8s.io/e2e-framework/klient/k8s/resources" + "github.com/clusterlink-net/clusterlink/cmd/cl-controlplane/app" "github.com/clusterlink-net/clusterlink/pkg/apis/clusterlink.net/v1alpha1" + "github.com/clusterlink-net/clusterlink/pkg/bootstrap" "github.com/clusterlink-net/clusterlink/tests/e2e/k8s/services" ) @@ -84,6 +90,65 @@ func (c *ClusterLink) RestartDataplane() error { return c.ScaleDataplane(1) } +// RestartDataplane restarts the dataplane. +func (c *ClusterLink) UpdatePeerCertificates( + fabricCert *bootstrap.Certificate, peerCert *bootstrap.Certificate, +) error { + err := c.cluster.Resources().Update( + context.Background(), + &v1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cl-peer", + Namespace: c.namespace, + }, + Data: map[string][]byte{ + app.PeerCertificateFile: peerCert.RawCert(), + app.PeerKeyFile: peerCert.RawKey(), + app.FabricCertificateFile: fabricCert.RawCert(), + }, + }) + if err != nil { + return fmt.Errorf("cannot update peer secret: %w", err) + } + + // update controlplane pods annotation to speed-up re-loading of secret + var pods v1.PodList + err = c.cluster.Resources().List( + context.Background(), + &pods, + resources.WithLabelSelector("app=cl-controlplane")) + if err != nil { + return fmt.Errorf("unable to list controlplane pods: %w", err) + } + + mergePatch, err := json.Marshal(map[string]interface{}{ + "metadata": map[string]interface{}{ + "annotations": map[string]interface{}{ + "peer-tls-last-updated": time.Now().String(), + }, + }, + }) + if err != nil { + return fmt.Errorf("cannot encode pod annotation patch: %w", err) + } + + for i := range pods.Items { + err := c.cluster.Resources().Patch( + context.Background(), + &pods.Items[i], + k8s.Patch{ + PatchType: types.StrategicMergePatchType, + Data: mergePatch, + }, + ) + if err != nil { + return fmt.Errorf("unable to annotate controlplane pod: %w", err) + } + } + + return nil +} + // Access a cluster service. func (c *ClusterLink) AccessService( clientFn func(*KindCluster, *Service) (string, error), From 4a9b043423145cf3264c4aac390146617bef4ab6 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Wed, 5 Jun 2024 12:19:27 +0300 Subject: [PATCH 39/53] installation: Create temp directory for ClusterLink installation (#635) Create a temporary directory for downloading and installing clusterlink CLI. Signed-off-by: Kfir Toledo --- hack/install_clusterlink.sh | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hack/install_clusterlink.sh b/hack/install_clusterlink.sh index 047502162..710dbcb3a 100755 --- a/hack/install_clusterlink.sh +++ b/hack/install_clusterlink.sh @@ -1,4 +1,9 @@ #!/bin/sh +# This script installs the latest ClusterLink CLI using the command: +# curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - +# To fetch a specific version, add the VERSION variable to the script: +# curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | VERSION=v0.2.1 sh - + set -e # Determine the OS. @@ -42,14 +47,12 @@ if ! curl -o /dev/null -sIf "$url"; then fi # Open the tar file. +download_path=$(mktemp -d "$(pwd)/clusterlink.XXXXXX") curl -fsLO ${url} -tar -xzf "${filename}" +tar -xzf "${filename}" -C "${download_path}" rm "${filename}" -current_path=$(pwd)/clusterlink - install_path=${HOME}/.local/bin - # If the install script is running in superuser context, change the install path if [ "$(id -u)" -eq 0 ]; then install_path=/usr/local/bin @@ -60,8 +63,8 @@ if [ ! -d "$install_path" ]; then mkdir -p "$install_path" || { echo "Error: Failed to create directory $install_path"; exit 1; } fi -mv $current_path/* $install_path -rm -rf $current_path +mv $download_path/clusterlink/* $install_path +rm -rf $download_path # Installation summary. printf "\n" From 7016ff40fa1f5944f9da65654914cbb4244d210b Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Wed, 5 Jun 2024 12:58:20 +0300 Subject: [PATCH 40/53] controlplane: Change listen port number (#634) Update the controlplane port to be higher than 1024 (below does not allow for non-root users in OpenShift clusters). Signed-off-by: Kfir Toledo --- pkg/controlplane/api/servername.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/controlplane/api/servername.go b/pkg/controlplane/api/servername.go index ebf1c2214..94166c3b3 100644 --- a/pkg/controlplane/api/servername.go +++ b/pkg/controlplane/api/servername.go @@ -15,5 +15,5 @@ package api const ( // ListenPort is the port used by the dataplane to access the controlplane. - ListenPort = 444 + ListenPort = 4444 ) From fbca843cb45fb386659c8ea2e941b4bf58f82f06 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Thu, 6 Jun 2024 13:21:29 +0300 Subject: [PATCH 41/53] website: Add description for CLI version (#636) Add instructions for installation for the specific version. Signed-off-by: Kfir Toledo --- .github/workflows/release.yml | 4 +++- hack/install_clusterlink.sh | 8 ++++---- .../content/en/docs/main/getting-started/users.md | 13 ++++++++++++- 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f5ac35df7..7dbed88b7 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -48,7 +48,9 @@ jobs: tag: ${{ github.ref }} overwrite: true file_glob: true - - name: Upload script + - name: Update script installation version + run: sed -i "s/VERSION=\"latest\"/VERSION=\"${{ github.ref_name }}\"/" hack/install_clusterlink.sh + - name: Upload script installation uses: svenstaro/upload-release-action@v2 with: repo_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/hack/install_clusterlink.sh b/hack/install_clusterlink.sh index 710dbcb3a..ec0584188 100755 --- a/hack/install_clusterlink.sh +++ b/hack/install_clusterlink.sh @@ -1,8 +1,8 @@ #!/bin/sh # This script installs the latest ClusterLink CLI using the command: # curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - -# To fetch a specific version, add the VERSION variable to the script: -# curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | VERSION=v0.2.1 sh - +# To fetch a specific version, use the URL path of the version release: +# curl -L https://github.com/clusterlink-net/clusterlink/releases/download/v0.2.1/clusterlink.sh | sh - set -e @@ -30,12 +30,12 @@ case "${OS_ARCH}" in ;; esac +VERSION="latest" filename="clusterlink-${CL_OS}-${CL_ARCH}.tar.gz" url="https://github.com/clusterlink-net/clusterlink/releases/download/${VERSION}/${filename}" # Set version to latest if not define and update the url. -if [ "${VERSION}" = "" ] ; then - VERSION="latest" +if [ "${VERSION}" = "latest" ] ; then url="https://github.com/clusterlink-net/clusterlink/releases/${VERSION}/download/${filename}" fi diff --git a/website/content/en/docs/main/getting-started/users.md b/website/content/en/docs/main/getting-started/users.md index a378ec561..55c0b7487 100644 --- a/website/content/en/docs/main/getting-started/users.md +++ b/website/content/en/docs/main/getting-started/users.md @@ -13,7 +13,7 @@ For example, you can set up a local environment using [kind][]. ## Installation -1. {{< anchor install-cli>}}To install ClusterLink on Linux or Mac, use the installation script: +1. {{< anchor install-cli>}}To install ClusterLink CLI on Linux or Mac, use the installation script: ```sh curl -L https://github.com/clusterlink-net/clusterlink/releases/latest/download/clusterlink.sh | sh - @@ -25,6 +25,16 @@ For example, you can set up a local environment using [kind][]. clusterlink --version ``` +{{% expand summary="Download specific CLI version" %}} + To install a specific version of the ClusterLink CLI, use the URL path of the version release: + For example, to download version v0.2.1: + + ```sh + curl -L https://github.com/clusterlink-net/clusterlink/releases/download/v0.2.1/clusterlink.sh | sh - + ``` + +{{% /expand %}} + ## Setup To set up ClusterLink on a Kubernetes cluster, follow these steps: @@ -69,6 +79,7 @@ To set up ClusterLink on a Kubernetes cluster, follow these steps: If they were not, use the flag `--path ` for pointing to the working directory that was used in the previous command. The `--fabric` option is optional, and by default, "default_fabric" will be used. + To install a specific image of ClusterLink use the `--tag ` flag. For more details and deployment configuration see [ClusterLink deployment operator][]. {{< notice note >}} To set up ClusterLink on another cluster, create another set of peer certificates (step 2). From 7021f7b7946e3b1f71e2013335e4ae0b4c2d4ae3 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Thu, 6 Jun 2024 15:10:54 +0300 Subject: [PATCH 42/53] dataplane: Fix certificate path and listening port (#637) Fix the key path and listening port for datapalne to support Openshift. Signed-off-by: Kfir Toledo --- cmd/cl-controlplane/app/server.go | 2 +- cmd/cl-dataplane/app/envoyconf.go | 2 +- cmd/cl-dataplane/app/server.go | 2 +- cmd/cl-go-dataplane/app/server.go | 2 +- pkg/dataplane/api/servername.go | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/cmd/cl-controlplane/app/server.go b/cmd/cl-controlplane/app/server.go index 09846b2c4..306365754 100644 --- a/cmd/cl-controlplane/app/server.go +++ b/cmd/cl-controlplane/app/server.go @@ -54,7 +54,7 @@ const ( // CertificateFile is the path to the certificate file. CertificateFile = "/etc/ssl/certs/clink-controlplane.pem" // KeyFile is the path to the private-key file. - KeyFile = "/etc/ssl/private/clink-controlplane.pem" + KeyFile = "/etc/ssl/key/clink-controlplane.pem" // PeerTLSDirectory is the path to the directory holding the peer TLS certificates. PeerTLSDirectory = "/etc/ssl/certs/clink" diff --git a/cmd/cl-dataplane/app/envoyconf.go b/cmd/cl-dataplane/app/envoyconf.go index 51d2f590d..1920c847c 100644 --- a/cmd/cl-dataplane/app/envoyconf.go +++ b/cmd/cl-dataplane/app/envoyconf.go @@ -22,7 +22,7 @@ admin: address: socket_address: address: 127.0.0.1 - port_value: 1000 + port_value: 1500 bootstrap_extensions: - name: envoy.bootstrap.internal_listener typed_config: diff --git a/cmd/cl-dataplane/app/server.go b/cmd/cl-dataplane/app/server.go index 6cd2876fb..ffac48945 100644 --- a/cmd/cl-dataplane/app/server.go +++ b/cmd/cl-dataplane/app/server.go @@ -34,7 +34,7 @@ const ( // CertificateFile is the path to the certificate file. CertificateFile = "/etc/ssl/certs/clink-dataplane.pem" // KeyFile is the path to the private-key file. - KeyFile = "/etc/ssl/private/clink-dataplane.pem" + KeyFile = "/etc/ssl/key/clink-dataplane.pem" // Name is the app label of dataplane pods. Name = "cl-dataplane" diff --git a/cmd/cl-go-dataplane/app/server.go b/cmd/cl-go-dataplane/app/server.go index 1b256e6fb..7954fa828 100644 --- a/cmd/cl-go-dataplane/app/server.go +++ b/cmd/cl-go-dataplane/app/server.go @@ -45,7 +45,7 @@ const ( // CertificateFile is the path to the certificate file. CertificateFile = "/etc/ssl/certs/clink-dataplane.pem" // KeyFile is the path to the private-key file. - KeyFile = "/etc/ssl/private/clink-dataplane.pem" + KeyFile = "/etc/ssl/key/clink-dataplane.pem" ) // Options contains everything necessary to create and run a dataplane. diff --git a/pkg/dataplane/api/servername.go b/pkg/dataplane/api/servername.go index 1a41c885f..072aee171 100644 --- a/pkg/dataplane/api/servername.go +++ b/pkg/dataplane/api/servername.go @@ -15,5 +15,5 @@ package api const ( // ListenPort is the dataplane external listening port. - ListenPort = 443 + ListenPort = 4443 ) From 1b29b1b4923eba113af3b491bbffcbecdcb4c970 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Sun, 9 Jun 2024 18:00:06 +0300 Subject: [PATCH 43/53] CLI: Add empty strings to clean-up certificates (#638) Add empty strings to certificates that are used for clean-up by ClusterLink CLI. Signed-off-by: Kfir Toledo --- pkg/bootstrap/platform/k8s.go | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/pkg/bootstrap/platform/k8s.go b/pkg/bootstrap/platform/k8s.go index 42d75a8ea..316652d38 100644 --- a/pkg/bootstrap/platform/k8s.go +++ b/pkg/bootstrap/platform/k8s.go @@ -376,7 +376,15 @@ func K8SClusterLinkInstanceConfig(config *Config, name string) ([]byte, error) { // used for deleting the secrets. func K8SEmptyCertificateConfig(config *Config) ([]byte, error) { args := map[string]interface{}{ - "namespace": config.Namespace, + "Namespace": config.Namespace, + "ca": "", + "controlplaneCert": "", + "controlplaneKey": "", + "dataplaneCert": "", + "dataplaneKey": "", + "peerCert": "", + "peerKey": "", + "fabricCert": "", } var certConfig bytes.Buffer From 5a0ee39e6f03553ef84cfdca753e54e7f21a624a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 10 Jun 2024 09:44:13 +0300 Subject: [PATCH 44/53] build(deps): bump sigs.k8s.io/controller-runtime in the k8s group (#640) Bumps the k8s group with 1 update: [sigs.k8s.io/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime). Updates `sigs.k8s.io/controller-runtime` from 0.18.3 to 0.18.4 - [Release notes](https://github.com/kubernetes-sigs/controller-runtime/releases) - [Changelog](https://github.com/kubernetes-sigs/controller-runtime/blob/main/RELEASE.md) - [Commits](https://github.com/kubernetes-sigs/controller-runtime/compare/v0.18.3...v0.18.4) --- updated-dependencies: - dependency-name: sigs.k8s.io/controller-runtime dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index f0707ecdb..1afb5cf51 100644 --- a/go.mod +++ b/go.mod @@ -21,7 +21,7 @@ require ( k8s.io/apimachinery v0.30.1 k8s.io/client-go v0.30.1 k8s.io/utils v0.0.0-20230726121419-3b25d923346b - sigs.k8s.io/controller-runtime v0.18.3 + sigs.k8s.io/controller-runtime v0.18.4 sigs.k8s.io/e2e-framework v0.4.0 ) diff --git a/go.sum b/go.sum index 7030daecf..badb3fd30 100644 --- a/go.sum +++ b/go.sum @@ -284,8 +284,8 @@ k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7F k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSnlTLKgpAAttJvpI= k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= -sigs.k8s.io/controller-runtime v0.18.3 h1:B5Wmmo8WMWK7izei+2LlXLVDGzMwAHBNLX68lwtlSR4= -sigs.k8s.io/controller-runtime v0.18.3/go.mod h1:TVoGrfdpbA9VRFaRnKgk9P5/atA0pMwq+f+msb9M8Sg= +sigs.k8s.io/controller-runtime v0.18.4 h1:87+guW1zhvuPLh1PHybKdYFLU0YJp4FhJRmiHvm5BZw= +sigs.k8s.io/controller-runtime v0.18.4/go.mod h1:TVoGrfdpbA9VRFaRnKgk9P5/atA0pMwq+f+msb9M8Sg= sigs.k8s.io/e2e-framework v0.4.0 h1:4yYmFDNNoTnazqmZJXQ6dlQF1vrnDbutmxlyvBpC5rY= sigs.k8s.io/e2e-framework v0.4.0/go.mod h1:JilFQPF1OL1728ABhMlf9huse7h+uBJDXl9YeTs49A8= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= From 3fb0bbebb3c9723deaa724e6ea7b30d33ee2a2d9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 10 Jun 2024 12:37:15 +0300 Subject: [PATCH 45/53] build(deps): bump golang.org/x/net from 0.25.0 to 0.26.0 (#641) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.25.0 to 0.26.0. - [Commits](https://github.com/golang/net/compare/v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 10 +++++----- go.sum | 24 ++++++++++++------------ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/go.mod b/go.mod index 1afb5cf51..ce1674225 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,7 @@ require ( github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 - golang.org/x/net v0.25.0 + golang.org/x/net v0.26.0 google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.1 @@ -74,12 +74,12 @@ require ( github.com/vladimirvivien/gexe v0.2.0 // indirect go.uber.org/multierr v1.11.0 // indirect go.uber.org/zap v1.26.0 // indirect - golang.org/x/crypto v0.23.0 // indirect + golang.org/x/crypto v0.24.0 // indirect golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect golang.org/x/oauth2 v0.18.0 // indirect - golang.org/x/sys v0.20.0 // indirect - golang.org/x/term v0.20.0 // indirect - golang.org/x/text v0.15.0 // indirect + golang.org/x/sys v0.21.0 // indirect + golang.org/x/term v0.21.0 // indirect + golang.org/x/text v0.16.0 // indirect golang.org/x/time v0.3.0 // indirect gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect google.golang.org/appengine v1.6.8 // indirect diff --git a/go.sum b/go.sum index badb3fd30..44065f672 100644 --- a/go.sum +++ b/go.sum @@ -172,8 +172,8 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs= -golang.org/x/crypto v0.23.0 h1:dIJU/v2J8Mdglj/8rJ6UUOM3Zc9zLZxVZwwxMooUSAI= -golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8= +golang.org/x/crypto v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI= +golang.org/x/crypto v0.24.0/go.mod h1:Z1PMYSOR5nyMcyAVAIQSKCDwalqy85Aqn1x3Ws4L5DM= golang.org/x/exp v0.0.0-20230321023759-10a507213a29 h1:ooxPy7fPvB4kwsA2h+iBNHkAbp/4JxTSwCmvdjEYmug= golang.org/x/exp v0.0.0-20230321023759-10a507213a29/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -189,8 +189,8 @@ golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= -golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac= -golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= +golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ= +golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE= golang.org/x/oauth2 v0.18.0 h1:09qnuIAgzdx1XplqJvW6CQqMCtGZykZWcXzPMPUusvI= golang.org/x/oauth2 v0.18.0/go.mod h1:Wf7knwG0MPoWIMMBgFlEaSUDaKskp0dCfrlJRJXbBi8= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -210,16 +210,16 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y= -golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.21.0 h1:rF+pYz3DAGSQAxAu1CbC7catZg4ebC4UIeIhKxBZvws= +golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58= -golang.org/x/term v0.20.0 h1:VnkxpohqXaOBYJtBmEppKUG6mXpi+4O6purfc2+sMhw= -golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY= +golang.org/x/term v0.21.0 h1:WVXCp+/EBEHOj53Rvu+7KiT/iElMrO8ACK16SMZ3jaA= +golang.org/x/term v0.21.0/go.mod h1:ooXLefLobQVslOqselCNF4SxFAaoS6KujMbsGzSDmX0= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= @@ -227,8 +227,8 @@ golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/text v0.15.0 h1:h1V/4gjBv8v9cjcR6+AR5+/cIYK5N/WAgiv4xlsEtAk= -golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4= +golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -237,8 +237,8 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.18.0 h1:k8NLag8AGHnn+PHbl7g43CtqZAwG60vZkLqgyZgIHgQ= -golang.org/x/tools v0.18.0/go.mod h1:GL7B4CwcLLeo59yx/9UWWuNOW1n3VZ4f5axWfML7Lcg= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= From edfc275359be90eab3d896e3e8bb77dbe36afcd0 Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Mon, 10 Jun 2024 21:13:47 +0300 Subject: [PATCH 46/53] Update gen-doc-version.sh (#643) modify documentation to use doc versions without PATCH value (e.g., `v0.2.0` changed to `v0.2`) Signed-off-by: Etai Lev Ran --- hack/gen-doc-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hack/gen-doc-version.sh b/hack/gen-doc-version.sh index 996fdde51..72d185dce 100755 --- a/hack/gen-doc-version.sh +++ b/hack/gen-doc-version.sh @@ -30,7 +30,7 @@ # released version. Once the unstaged changes are ready, they can be added # and committed. # -# To run gen-doc-version: "NEW_DOCS_VERSION=v0.2.0 PREVIOUS_DOCS_VERSION=v0.1.0 make docs-version" +# To run gen-doc-version: "NEW_DOCS_VERSION=v0.2 PREVIOUS_DOCS_VERSION=v0.1 make docs-version" # Note: if PREVIOUS_DOCS_VERSION is not set, the script will guess it from the directory listing # From 6f02c09755723cd85522d7c197d320ecbfbc2f56 Mon Sep 17 00:00:00 2001 From: Etai Lev Ran Date: Tue, 11 Jun 2024 10:41:43 +0300 Subject: [PATCH 47/53] enable and use git info for lastmod time (#642) Signed-off-by: Etai Lev Ran --- website/config.toml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/website/config.toml b/website/config.toml index f224b04d8..047fd9057 100644 --- a/website/config.toml +++ b/website/config.toml @@ -12,11 +12,14 @@ enableRobotsTXT = true disableAliases = true # Will give values to .Lastmod etc. -enableGitInfo = false +enableGitInfo = true # Comment out to enable taxonomies in Docsy # disableKinds = ["taxonomy", "taxonomyTerm"] +[frontmatter] +lastmod = ["lastmod", ":git", "date", "publishDate"] + # You can add your own taxonomies [taxonomies] tag = "tags" From c8bb93e69d53a7fd14388eeae3f61c4336d1453a Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Tue, 11 Jun 2024 15:37:38 +0300 Subject: [PATCH 48/53] [website]: update docs to release v0.3 (#644) Add documentation for version v0.3.0. Signed-off-by: Kfir Toledo --- website/config.toml | 9 +- website/content/en/docs/v0.3/_index.md | 12 + .../content/en/docs/v0.3/concepts/_index.md | 5 + .../content/en/docs/v0.3/concepts/fabric.md | 54 +++ .../content/en/docs/v0.3/concepts/peers.md | 214 ++++++++++ .../content/en/docs/v0.3/concepts/policies.md | 154 +++++++ .../content/en/docs/v0.3/concepts/services.md | 236 +++++++++++ .../en/docs/v0.3/doc-contribution/_index.md | 96 +++++ .../en/docs/v0.3/getting-started/_index.md | 14 + .../docs/v0.3/getting-started/developers.md | 81 ++++ .../en/docs/v0.3/getting-started/users.md | 128 ++++++ .../content/en/docs/v0.3/overview/_index.md | 51 +++ website/content/en/docs/v0.3/tasks/_index.md | 5 + .../content/en/docs/v0.3/tasks/operator.md | 171 ++++++++ .../content/en/docs/v0.3/tasks/relay/index.md | 165 ++++++++ .../en/docs/v0.3/tasks/relay/nginx-relay.png | Bin 0 -> 112485 bytes .../content/en/docs/v0.3/tutorials/_index.md | 5 + .../docs/v0.3/tutorials/bookinfo/bookinfo.png | Bin 0 -> 173819 bytes .../en/docs/v0.3/tutorials/bookinfo/index.md | 380 ++++++++++++++++++ .../en/docs/v0.3/tutorials/iperf/index.md | 251 ++++++++++++ .../en/docs/v0.3/tutorials/nginx/index.md | 231 +++++++++++ 21 files changed, 2260 insertions(+), 2 deletions(-) create mode 100644 website/content/en/docs/v0.3/_index.md create mode 100644 website/content/en/docs/v0.3/concepts/_index.md create mode 100644 website/content/en/docs/v0.3/concepts/fabric.md create mode 100644 website/content/en/docs/v0.3/concepts/peers.md create mode 100644 website/content/en/docs/v0.3/concepts/policies.md create mode 100644 website/content/en/docs/v0.3/concepts/services.md create mode 100644 website/content/en/docs/v0.3/doc-contribution/_index.md create mode 100644 website/content/en/docs/v0.3/getting-started/_index.md create mode 100644 website/content/en/docs/v0.3/getting-started/developers.md create mode 100644 website/content/en/docs/v0.3/getting-started/users.md create mode 100644 website/content/en/docs/v0.3/overview/_index.md create mode 100644 website/content/en/docs/v0.3/tasks/_index.md create mode 100644 website/content/en/docs/v0.3/tasks/operator.md create mode 100644 website/content/en/docs/v0.3/tasks/relay/index.md create mode 100644 website/content/en/docs/v0.3/tasks/relay/nginx-relay.png create mode 100644 website/content/en/docs/v0.3/tutorials/_index.md create mode 100644 website/content/en/docs/v0.3/tutorials/bookinfo/bookinfo.png create mode 100644 website/content/en/docs/v0.3/tutorials/bookinfo/index.md create mode 100644 website/content/en/docs/v0.3/tutorials/iperf/index.md create mode 100644 website/content/en/docs/v0.3/tutorials/nginx/index.md diff --git a/website/config.toml b/website/config.toml index 047fd9057..8c28a749c 100644 --- a/website/config.toml +++ b/website/config.toml @@ -91,7 +91,7 @@ description = "ClusterLink documentation" [outputs] home = ["HTML", "REDIRECTS"] -# rules for generating _redirects from layouts/index.redirects +# rules for generating _redirects from layouts/index.redirects [outputFormats.REDIRECTS] mediaType = "text/netlify" baseName = "_redirects" @@ -125,7 +125,7 @@ archived_version = false # point people to the main doc site. # url_latest_version = "https://clusterlink.net" -latest_stable_version = "v0.2" +latest_stable_version = "v0.3" # Repository configuration (URLs for in-page links to opening issues and suggesting changes) github_repo = "https://github.com/clusterlink-net/clusterlink" @@ -234,6 +234,11 @@ path = "github.com/martignoni/hugo-notice" version = "main-DRAFT" url = "/docs/main" + +[[params.versions]] +version = "v0.3" +url = "/docs/v0.3/" + [[params.versions]] version = "v0.2" url = "/docs/v0.2/" diff --git a/website/content/en/docs/v0.3/_index.md b/website/content/en/docs/v0.3/_index.md new file mode 100644 index 000000000..550d0f9d2 --- /dev/null +++ b/website/content/en/docs/v0.3/_index.md @@ -0,0 +1,12 @@ +--- +title: v0.3 +cascade: + version: v0.3 + versName: &name v0.3 + git_version_tag: v0.3.0 + exclude_search: false +linkTitle: *name +simple_list: true +weight: -30 # Weight for doc version vX.Y should be -(100*X + Y)0 +# For example: v0.2.x => -20 v3.6.7 => -3060. `main` is arbitrarily set to -9999 +--- diff --git a/website/content/en/docs/v0.3/concepts/_index.md b/website/content/en/docs/v0.3/concepts/_index.md new file mode 100644 index 000000000..5ea1086e1 --- /dev/null +++ b/website/content/en/docs/v0.3/concepts/_index.md @@ -0,0 +1,5 @@ +--- +title: Core Concepts +description: Core Concepts of the ClusterLink system +weight: 30 +--- diff --git a/website/content/en/docs/v0.3/concepts/fabric.md b/website/content/en/docs/v0.3/concepts/fabric.md new file mode 100644 index 000000000..fdf39206f --- /dev/null +++ b/website/content/en/docs/v0.3/concepts/fabric.md @@ -0,0 +1,54 @@ +--- +title: Fabric +description: Defining a ClusterLink fabric +weight: 10 +--- + +The concept of a *Fabric* encapsulates a set of cooperating [peers][]. + All peers in a fabric can communicate and may share [services][] + between them, with access governed by [policies][]. + The Fabric acts as a root of trust for peer-to-peer communications (i.e., + it functions as the certificate authority enabling mutual authentication between + peers). + +Currently, the concept of a Fabric is just that - a concept. It is not represented + or backed by any managed resource in a ClusterLink deployment. Once a Fabric is created, + its only relevance is in providing a certificate for use by each peer's gateways. + One could potentially consider a more elaborate implementation where a central + management entity explicitly deals with Fabric life cycle, association of peers to + a fabric, etc. The role of this central management component in ClusterLink is currently + delegated to users who are responsible for coordinating the transfer of certificates + between peers, out of band. + +## Initializing a new fabric + +### Prerequisites + +The following sections assume that you have access to the `clusterlink` CLI and one or more + peers (i.e., clusters) where you'll deploy ClusterLink. The CLI can be downloaded + from the ClusterLink [releases page on GitHub][]. + +### Create a new fabric CA + +To create a new fabric certificate authority (CA), execute the following CLI command: + +```sh +clusterlink create fabric --name +``` + +This command will create the CA files `cert.pem` and `key.pem` in a directory named . + The `--name` option is optional, and by default, "default_fabric" will be used. + While you will need access to these files to create the peers` gateway certificates later, + the private key file should be protected and not shared with others. + +## Related tasks + +Once a Fabric has been created and initialized, you can proceed with configuring + [peers][]. For a complete, end-to-end use case, please refer to the + [iperf tutorial][]. + +[peers]: {{< relref "peers" >}} +[services]: {{< relref "services" >}} +[policies]: {{< relref "policies" >}} +[releases page on GitHub]: https://github.com/clusterlink-net/clusterlink/releases/tag/{{% param git_version_tag %}} +[iperf tutorial]: {{< relref "../tutorials/iperf" >}} diff --git a/website/content/en/docs/v0.3/concepts/peers.md b/website/content/en/docs/v0.3/concepts/peers.md new file mode 100644 index 000000000..16b431186 --- /dev/null +++ b/website/content/en/docs/v0.3/concepts/peers.md @@ -0,0 +1,214 @@ +--- +title: Peers +description: Defining ClusterLink peers as part of a fabric +weight: 20 +--- + +A *Peer* represents a location, such as a Kubernetes cluster, participating in a + [fabric][]. Each peer may host one or more [services][] + that it may wish to share with other peers. A peer is managed by a peer administrator, + which is responsible for running the ClusterLink control and data planes. The + administrator will typically deploy the ClusterLink components by configuring + the [Deployment Custom Resource (CR)][operator-cr]. They may also wish to define + coarse-grained access policies, in accordance with high level corporate + policies (e.g., "production peers should only communicate with other production peers"). + +Once a peer has been added to a fabric, it can communicate with any other peer + belonging to it. All configuration relating to service sharing (e.g., the exporting + and importing of services, and the setting of fine grained application policies) can be + done with lowered privileges (e.g., by users, such as application owners). Remote peers are + represented by the Peer Custom Resources (CRs). Each peer CR instance + defines a remote cluster and the network endpoints of its ClusterLink gateways. + +## Prerequisites + +The following sections assume that you have access to the `clusterlink` CLI and one or more + peers (i.e., clusters) where you'll deploy ClusterLink. The CLI can be downloaded + from the ClusterLink [releases page on GitHub][]. + It also assumes that you have access to the [previously created fabric][] + CA files. + +## Initializing a new peer + +{{< notice warning >}} +Creating a new peer is a **fabric administrator** level operation and should be appropriately + protected. +{{< /notice >}} + +### Create a new peer certificate + +To create a new peer certificate belonging to a fabric, confirm that the fabric + Certificate Authority (CA) files are available in the current working directory, + and then execute the following CLI command: + +```sh +clusterlink create peer-cert --name --fabric +``` + +{{< notice tip >}} +The fabric CA files (certificate and private key) are expected to be in a subdirectory + (i.e., `.//cert.name` and `.//key.pem`). +{{< /notice >}} + +This will create the certificate and private key files (`cert.pem` and + `key.pem`, respectively) of the new peer. By default, the files are + created in a subdirectory named `` under the subdirectory of the fabric ``. + You can override the default by setting the `--output ` option. + +{{< notice info >}} +You will need the CA certificate (but **not** the CA private key) and the peer's certificate + and private key pair in the next step. They can be provided out of band (e.g., over email) to the + peer administrator or by any other means for secure transfer of sensitive data. +{{< /notice >}} + +## Deploy ClusterLink to a new peer + +{{< notice info >}} +This operation is typically done by a local **peer administrator**, usually different + than the **fabric administrator**. +{{< /notice >}} + +Before proceeding, ensure that the following files (created in the previous step) are + available in the current working directory: + + 1. CA certificate; + 1. peer certificate; and + 1. peer private key. + +### Install the ClusterLink deployment operator + +Install the ClusterLink operator by running the following command: + +```sh +clusterlink deploy peer --name --fabric +``` + +The command assumes that kubectl is set to the correct context and credentials + and that the certificates were created in respective sub-directories + under the current working directory. + If they were not, add the `--path ` CLI option to set the correct path. + +This command will deploy the ClusterLink deployment CRDs using the current + `kubectl` context. The operation requires cluster administrator privileges + in order to install CRDs into the cluster. + The ClusterLink operator is installed to the `clusterlink-operator` namespace. + The CA, peer certificate, and private key are set as K8s secrets + in the namespace where ClusterLink components are installed, which by default is + `clusterlink-system`. You can confirm the successful completion of this step + using the following commands: + +```sh +kubectl get crds +kubectl get secret --namespace clusterlink-system +``` + +{{% expand summary="Example output" %}} + +```sh +$ kubectl get crds +NAME CREATED AT +accesspolicies.clusterlink.net 2024-04-07T12:08:24Z +exports.clusterlink.net 2024-04-07T12:08:24Z +imports.clusterlink.net 2024-04-07T12:08:24Z +instances.clusterlink.net 2024-04-07T12:08:24Z +peers.clusterlink.net 2024-04-07T12:08:24Z +privilegedaccesspolicies.clusterlink.net 2024-04-07T12:08:24Z + +$ kubectl get secret --namespace clusterlink-system +NAME TYPE DATA AGE +cl-controlplane Opaque 2 19h +cl-dataplane Opaque 2 19h +cl-ca Opaque 1 19h +cl-peer Opaque 1 19h +``` + +{{% /expand %}} + +### Deploy ClusterLink via the operator and ClusterLink CR + +After the operator is installed, you can deploy ClusterLink by applying + the ClusterLink CR. This will cause the ClusterLink operator to + attempt reconciliation of the actual and intended ClusterLink deployment. + By default, the operator will install the ClusterLink control and data plane + components into a dedicated and privileged namespace (defaults to `clusterlink-system`). + Configurations affecting the entire peer, such as the list of known peers, are also maintained + in the same namespace. + +Refer to the [operator documentation][] for a description of the ClusterLink CR fields. + +## Add or remove peers + +{{< notice info >}} +This operation is typically done by a local **peer administrator**, usually different + than the **fabric administrator**. +{{< /notice >}} + +Managing peers is done by creating, deleting and updating peer CRs + in the dedicated ClusterLink namespace (typically, `clusterlink-system`). Peers are + added to the ClusterLink namespace by the peer administrator. Information + regarding peer gateways and attributes is communicated out of band (e.g., provided + by the fabric or remote peer administrator over email). In the future, these may + be configured via a management plane. + +{{% expand summary="Peer Custom Resource" %}} + +```go +type Peer struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec PeerSpec `json:"spec"` + Status PeerStatus `json:"status,omitempty"` +} + + +type PeerSpec struct { + Gateways []Endpoint `json:"gateways"` +} + +type PeerStatus struct { + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +type Endpoint struct { + Host string `json:"host"` + Port uint16 `json:"port"` +} +``` + +{{% /expand %}} + +There are two fundamental attributes in the peer CRD: the peer name and the list of + ClusterLink gateway endpoints through which the remote peer's services are available. + Peer names are unique and must align with the Subject name present in their certificate + during connection establishment. The name is used by importers in referencing an export + (see [services][] for details). + +Gateway endpoint would typically be implemented via a `NodePort` or `LoadBalancer` + K8s service. A `NodePort` service would typically be used in local deployments + (e.g., when running in kind clusters during development) and a `LoadBalancer` service + would be used in cloud-based deployments. These can be automatically configured and + created via the [ClusterLink CR][]. + The peer's status section includes a `Reachable` condition indicating whether the peer is currently reachable, + and in case it is not reachable, the last time it was. + +{{% expand summary="Example YAML for `kubectl apply -f `" %}} +{{< readfile file="/static/files/peer_crd_sample.yaml" code="true" lang="yaml" >}} +{{% /expand %}} + +## Related tasks + +Once a peer has been created and initialized with the ClusterLink control and data + planes as well as one or more remote peers, you can proceed with configuring + [services][] and [policies][]. + For a complete end-to-end use case, refer to the [iperf tutorial][]. + +[fabric]: {{< relref "fabric" >}} +[previously created fabric]: {{< relref "fabric#create-a-new-fabric-ca" >}} +[services]: {{< relref "services" >}} +[policies]: {{< relref "policies" >}} +[releases page on GitHub]: https://github.com/clusterlink-net/clusterlink/releases/tag/{{% param git_version_tag %}} +[operator-cr]: {{< relref "../tasks/operator#deploy-cr-instance" >}} +[operator documentation]: {{< relref "../tasks/operator#commandline-flags" >}} +[ClusterLink CR]: {{< relref "peers#deploy-clusterlink-via-the-operator-and-clusterlink-cr" >}} +[iperf tutorial]: {{< relref "../tutorials/iperf" >}} diff --git a/website/content/en/docs/v0.3/concepts/policies.md b/website/content/en/docs/v0.3/concepts/policies.md new file mode 100644 index 000000000..e7e03816d --- /dev/null +++ b/website/content/en/docs/v0.3/concepts/policies.md @@ -0,0 +1,154 @@ +--- +title: Access Policies +description: Controlling service access across peers +weight: 40 +--- + +Access policies allow users and administrators fine-grained control over + which client workloads may access which service. This is an important security + mechanism for applying [micro-segmentation][], which is a basic requirement of [zero-trust][] + systems. Another zero-trust principle, "Deny by default / Allow by exception," is also + addressed by ClusterLink's access policies: a connection without an explicit policy allowing it + will be dropped. Access policies can also be used for enforcing corporate security rules, + as well as segmenting the fabric into trust zones. + +ClusterLink's access policies are based on attributes that are attached to + [peers][], [services][] and client workloads. + Each attribute is a key/value pair, similar to how [labels][] + are used in Kubernetes. This approach, called ABAC (Attribute Based Access Control), + allows referring to a set of entities in a single policy, rather than listing individual + entity names. Using attributes is safer, more resilient to changes, and easier to + control and audit. At the moment, a limited set of attributes is available to use. + We plan to enrich this set in the future. + +Every instance of an access policy either allows or denies a given set of connections. +This set is defined by specifying the sources and destinations of these connections. +Sources are defined in terms of the attributes attached to the client workloads. +Destinations are defined in terms of the attributes attached to the target services. +Both client workloads and target services may inherit some attributes from their hosting peer. + +There are two tiers of access policies in ClusterLink. The high-priority tier + is intended for cluster/peer administrators to set access rules which cannot be + overridden by cluster users. High-priority policies are controlled by the + `PrivilegedAccessPolicy` CRD, and are cluster scoped (i.e., have no namespace). + Regular policies are intended for cluster users, such as application developers + and owners, and are controlled by the `AccessPolicy` CRD. Regular policies are + namespaced, and have an effect in their namespace only. That is, they do not + affect connections to/from other namespaces. + +For a connection to be established, both the ClusterLink gateway on the client + side and the ClusterLink gateway on the service side must allow the connection. + Each gateway (independently) follows these steps to decide if the connection is allowed: + +1. All instances of `PrivilegedAccessPolicy` in the cluster with `deny` action are considered. + If the connection matches any of them, the connection is dropped. +1. All instances of `PrivilegedAccessPolicy` in the cluster with `allow` action are considered. + If the connection matches any of them, the connection is allowed. +1. All instances of `AccessPolicy` in the relevant namespace with `deny` action are considered. + If the connection matches any of them, the connection is dropped. +1. All instances of `AccessPolicy` in the relevant namespace with `allow` action are considered. + If the connection matches any of them, the connection is allowed. +1. If the connection matched none of the above policies, the connection is dropped. + +**Note**: The relevant namespace for a given connection is the namespace of + the corresponding Import CR on the client side and the namespace of the corresponding + Export on the service side. + +## Prerequisites + +The following assumes that you have `kubectl` access to two or more clusters where ClusterLink + has already been [deployed and configured][]. + +### Creating access policies + +Recall that a connection is dropped if it does not match any access policy. + Hence, for a connection to be allowed, an access policy with an `allow` action + must be created on both sides of the connection. + Creating an access policy is accomplished by creating an `AccessPolicy` CR in + the relevant namespace (see note above). + Creating a high-priority access policy is accomplished by creating a `PrivilegedAccessPolicy` CR. + Instances of `PrivilegedAccessPolicy` have no namespace and affect the entire cluster. + +{{% expand summary="PrivilegedAccessPolicy and AccessPolicy Custom Resources" %}} + +```go +type PrivilegedAccessPolicy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec AccessPolicySpec `json:"spec,omitempty"` +} + +type AccessPolicy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec AccessPolicySpec `json:"spec,omitempty"` +} + +type AccessPolicySpec struct { + Action AccessPolicyAction `json:"action"` + From WorkloadSetOrSelectorList `json:"from"` + To WorkloadSetOrSelectorList `json:"to"` +} + +type AccessPolicyAction string + +const ( + AccessPolicyActionAllow AccessPolicyAction = "allow" + AccessPolicyActionDeny AccessPolicyAction = "deny" +) + +type WorkloadSetOrSelectorList []WorkloadSetOrSelector + +type WorkloadSetOrSelector struct { + WorkloadSets []string `json:"workloadSets,omitempty"` + WorkloadSelector *metav1.LabelSelector `json:"workloadSelector,omitempty"` +} +``` + +{{% /expand %}} + +The `AccessPolicySpec` defines the following fields: + +- **Action** (string, required): whether the policy allows or denies the + specified connection. Value must be either `allow` or `deny`. +- **From** (WorkloadSetOrSelector array, required): specifies connection sources. + A connection's source must match one of the specified sources to be matched by the policy. +- **To** (WorkloadSetOrSelectorList array, required): specifies connection destinations. + A connection's destination must match one of the specified destinations to be matched by the policy. + +A `WorkloadSetOrSelector` object has two fields; exactly one of them must be specified. + +- **WorkloadSets** (string array, optional) - an array of predefined sets of workload. + Currently not supported. +- **WorkloadSelector** (LabelSelector, optional) - a [Kubernetes label selector][] + defining a set of client workloads or a set of services, based on their + attributes. An empty selector matches all workloads/services. + +The following policy allows all incoming/outgoing connections in the `default` namespace. + +```yaml +apiVersion: clusterlink.net/v1alpha1 +kind: AccessPolicy +metadata: + name: allow-all + namespace: default +spec: + action: allow + from: + - workloadSelector: {} + to: + - workloadSelector: {} +``` + +More examples are available on our repo under [examples/policies][]. + +[peers]: {{< relref "peers" >}} +[services]: {{< relref "services" >}} +[micro-segmentation]: https://en.wikipedia.org/wiki/Microsegmentation_(network_security) +[zero-trust]: https://en.wikipedia.org/wiki/Zero_trust_security_model +[labels]: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ +[deployed and configured]: {{< relref "../getting-started/users#setup" >}} +[Kuberenetes label selector]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta +[examples/policies]: https://github.com/clusterlink-net/clusterlink/tree/main/examples/policies diff --git a/website/content/en/docs/v0.3/concepts/services.md b/website/content/en/docs/v0.3/concepts/services.md new file mode 100644 index 000000000..5a97d3382 --- /dev/null +++ b/website/content/en/docs/v0.3/concepts/services.md @@ -0,0 +1,236 @@ +--- +title: Services +description: Sharing services +weight: 30 +--- + +ClusterLink uses services as the unit of sharing between peers. + One or more peers can expose an (internal) K8s Service to + be consumed by other [peers][] in the [fabric][]. + A service is exposed by creating an *Export* CR referencing it in the + source cluster. Similarly, the exported service can be made accessible to workloads + in a peer by defining an *Import* CR in the destination cluster[^KEP-1645]. + Thus, service sharing is an explicit operation. Services are not automatically + shared by peers in the fabric. Note that the exporting cluster must be + [configured as a peer][] of the importing cluster. + +{{< notice info >}} +Services sharing is done on a per namespace basis and does not require cluster wide privileges. + It is intended to be used by application owners having access to their own namespaces only. +{{< /notice >}} + +A service is shared using a logical name. The logical name does not have to match + the actual Kubernetes Service name in the exporting cluster. Exporting a service + does not expose cluster Pods or their IP addresses to the importing clusters. + Any load balancing and scaling decisions are kept local in the exporting cluster. + This reduces the amount, frequency and sensitivity of information shared between + clusters. Similarly, the imported service can have any arbitrary name in the + destination cluster, allowing independent choice of naming. + +Orchestration of service sharing is the responsibility of users wishing to + export or import it, and any relevant information (e.g, the exported service + name and namespace) must be communicated out of band. In the future, this could + be done by a centralized management plane. + + + + + +## Prerequisites + +The following assume that you have `kubectl` access to two or more clusters where ClusterLink + has already been [deployed and configured][]. + +### Exporting a service + +In order to make a service potentially accessible by other clusters, it must be + explicitly configured for remote access via ClusterLink. Exporting is + accomplished by creating an Export CR in the **same** namespace + as the service being exposed. The CR acts as a marker for enabling + remote access to the service via ClusterLink. + +{{% expand summary="Export Custom Resource" %}} + +```go +type Export struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec ExportSpec `json:"spec,omitempty"` + Status ExportStatus `json:"status,omitempty"` +} + +type ExportSpec struct { + Host string `json:"host,omitempty"` + Port uint16 `json:"port,omitempty"` +} + +type ExportStatus struct { + Conditions []metav1.Condition `json:"conditions,omitempty"` +} +``` + +{{% /expand %}} + +The ExportSpec defines the following fields: + +- **Host** (string, optional): the name of the service being exported. The service + must be defined in the same namespace as the Export CR. If empty, + the export shall refer to a Kubernetes Service with the same name as the instance's + `metadata.name`. It is an error to refer to a non-existent service or one that is + not present in the local namespace. The error will be reflected in the CRD's status. +- **Port** (integer, required): the port number being exposed. If you wish to export + a multi-port service[^multiport], you will need to define multiple Exports using + the same `Host` value and a different `Port` each. This is aligned with ClusterLink's + principle of being explicit in sharing and limiting exposure whenever possible. + +Note that exporting a Service does not automatically make is accessible to other + peers, but only enables *potential* access. To complete service sharing, you must + define at least one [access control policy][concept-policy] that allows + access in the exporting cluster. + In addition, users in consuming clusters must still explicitly configure + [service imports][] and [policies][] in their respective namespaces. + +{{% expand summary="Example YAML for `kubectl apply -f `" %}} + +```yaml +apiVersion: clusterlink.net/v1alpha1 +kind: Export +metadata: + name: iperf3-server + namespace: default +spec: + port: 5000 +``` + +{{% /expand %}} + +### Importing a service + +Exposing remote services to a peer is accomplished by creating an Import CR + to a namespace. The CR represents the imported service and its + available backends across all peers. In response to an Import CR, ClusterLink + control plane will create a local Kubernetes Service selecting the ClusterLink + data plane Pods. The use of native Kubernetes constructs, allows ClusterLink + to work with any compliant cluster and CNI, transparently. + +The Import instance creates the service endpoint in the same namespace as it is + defined in. The created service will have the Import's `metadata.Name`. This + allows maintaining independent names for services between peers. Alternately, + you may use the same name for the import and related source exports. + You can define multiple Import CRs for the same set of Exports in different + namespaces. These are independent of each other. + +{{% expand summary="Import Custom Resource" %}} + +```go +type Import struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec ImportSpec `json:"spec"` + Status ImportStatus `json:"status,omitempty"` +} + +type ImportSpec struct { + Port uint16 `json:"port"` + TargetPort uint16 `json:"targetPort,omitempty"` + Sources []ImportSource `json:"sources"` + LBScheme string `json:"lbScheme"` +} + +type ImportSource struct { + Peer string `json:"peer"` + ExportName string `json:"exportName"` + ExportNamespace string `json:"exportNamespace"` +} + +type ImportStatus struct { + Conditions []metav1.Condition `json:"conditions,omitempty"` +} +``` + +{{% /expand %}} + +The ImportSpec defines the following fields: + +- **Port** (integer, required): the imported, user facing, port number defined + on the created service object. +- **TargetPort** (integer, optional): this is the internal listening port + used by the ClusterLink data plane pods to represent the remote services. Typically the + choice of TargetPort should be left to the ClusterLink control plane, allowing + it to select a random and non-conflicting port, but there may be cases where + you wish to assume responsibility for port selection (e.g., a-priori define + local cluster Kubernetes NetworkPolicy object instances). This may result in + [port conflicts][] as is done for NodePort services. +- **Sources** (source array, required): references to remote exports providing backends + for the Import. Each reference names a different export through the combination of: + - *Peer* (string, required): name of ClusterLink peer where the export is defined. + - *ExportNamespace* (string, required): name of the namespace on the remote peer where + the export is defined. + - *ExportName* (string, required): name of the remote export. +- **LBScheme** (string, optional): load balancing method to select between different + Sources defined. The default policy is `random`, but you could override it to use + `round-robin` or `static` (i.e., fixed) assignment. + + + +As with exports, importing a service does not automatically make it accessible by + workloads, but only enables *potential* access. To complete service sharing, + you must define at least one [access control policy][] that + allows access in the importing cluster. To grant access, a connection must be + evaluated to "allow" by both egress (importing cluster) and ingress (exporting + cluster) policies. + +{{% expand summary="Example YAML for `kubectl apply -f `" %}} + +```yaml +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: iperf3-server + namespace: default +spec: + port: 5000 + sources: + - exportName: iperf3-server + exportNamespace: default + peer: server +``` + +{{% /expand %}} + +## Related tasks + +Once a service is exported and imported by one or more clusters, you should + configure [polices][] governing its access. + For a complete end to end use case, refer to [iperf tutorial][]. + +[^KEP-1645]: While using similar terminology as the Kubernetes Multicluster Service + enhancement proposal ([MCS KEP][]), the ClusterLink implementation intentionally + differs from and is not compliant with the KEP (e.g., there is no `ClusterSet` + and "name sameness" assumption). + +[^multiport]: ClusterLink intentionally does not expose all service ports, as + typically only a small subset in a multi-port service is meant to be user + accessible, and other ports are service internal (e.g., ports used for internal + service coordination and replication). + +[fabric]: {{< relref "fabric" >}} +[peers]: {{< relref "peers" >}} +[configured as a peer]: {{< relref "peers#add-or-remove-peers" >}} +[policies]: {{< relref "policies" >}} +[service imports]: #importing-a-service +[port conflicts]: https://kubernetes.io/docs/concepts/services-networking/service/#avoid-nodeport-collisions +[access control policy]: {{< relref "policies" >}} +[iperf tutorial]: {{< relref "../tutorials/iperf" >}} +[deployed and configured]: {{< relref "../getting-started/users#setup" >}} +[MCS KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api diff --git a/website/content/en/docs/v0.3/doc-contribution/_index.md b/website/content/en/docs/v0.3/doc-contribution/_index.md new file mode 100644 index 000000000..b59dba19f --- /dev/null +++ b/website/content/en/docs/v0.3/doc-contribution/_index.md @@ -0,0 +1,96 @@ +--- +title: Contribution Guidelines +weight: 60 +description: How to contribute to the website +--- + +We use [Hugo][] to format and generate our [website][], the [Docsy][] theme + for styling and site structure, and [Netlify][] to manage the deployment of the site. + Hugo is an open-source static site generator that provides us with templates, + content organization in a standard directory structure, and a website generation + engine. We write the pages in Markdown (or HTML if you want), and Hugo wraps + them up into a website. + +All submissions, including submissions by project members, require review. We + use GitHub pull requests for this purpose. Consult + [GitHub Help][] for more information on using pull requests. + +## Quick start with Netlify + +Here's a quick guide to updating the docs. It assumes you're familiar with the + GitHub workflow and you're happy to use the automated preview of your website + updates: + +1. Fork the [ClusterLink repo][] on GitHub. +1. The documentation site is under the `website` directory. +1. Make your changes and send a pull request (PR). +1. If you're not yet ready for a review, add "WIP" to the PR name to indicate + it's a work in progress. (**Don't** add the Hugo property + "draft = true" to the page front matter, because that prevents the + auto-deployment of the content preview described in the next point). +1. Wait for the automated PR workflow to do some checks. When it's ready, + you should see a check named like this: **Pages changed - clusterlink-net** +1. Click **Details** to the right of "Pages changed" to see a preview + of your updates. Previews will be deployed to `https://deploy-preview---clusterlink-net.netlify.app/` +1. Continue updating your doc and pushing your changes until you're happy with + the content. +1. When you are ready for a review, add a comment to the PR, and remove any + "WIP" markers. + +## Updating a single page + +If you've just spotted something you'd like to change while using the docs, Docsy has a + shortcut for you: + +1. Click **Edit this page** in the top right hand corner of the page. +1. If you don't already have an up-to-date fork of the project repo, you are prompted to + get one - click **Fork this repository and propose changes** or **Update your Fork** to + get an up-to-date version of the project to edit. The appropriate page in your fork is + displayed in edit mode. +1. Follow the rest of the [Quick start with Netlify][] process above to make, preview, + and propose your changes. + +## Previewing your changes locally + +If you want to run your own local Hugo server to preview your changes as you work: + + + +1. Follow the instructions in [Getting started][] to install Hugo + and any other tools you need. You'll need at least **Hugo version 0.110** (we recommend + using the most recent available version), and it must be the **extended** version, + which supports SCSS. +1. Run `hugo server --gc` in the `website` directory. By default your site will be available + at http://localhost:1313/. Now that you're serving your site locally, Hugo will watch + for changes to the content and automatically refresh your site. +1. Continue with the usual GitHub workflow to edit files, commit them, push the + changes up to your fork, and create a pull request. + +## Creating an issue + +If you've found a problem in the docs, but you're not sure how to fix it yourself, + please create an [issue][] in the ClusterLink repo. + You can also create an issue about a specific page by clicking the **Create Issue** + button in the top right hand corner of the page. + +## Useful resources + +* [Docsy user guide][]: All about Docsy, including how it manages navigation, + look and feel, and multi-language support. +* [Hugo documentation][]: Comprehensive reference for Hugo. +* [Github Hello World!][]: A basic introduction to GitHub concepts and workflow. + + + +[Hugo]: https://gohugo.io/ +[website]: https://clusterlink.net +[Docsy]: https://github.com/google/docsy +[Netlify]: https://www.netlify.com/ +[GitHub Help]: https://help.github.com/articles/about-pull-requests/ +[Quick start with Netlify]: #quick-start-with-netlify +[Getting started]: {{< relref "../getting-started/" >}} +[ClusterLink repo]: https://github.com/clusterlink-net/clusterlink +[issue]: https://github.com/clusterlink-net/clusterlink/issues +[Docsy user guide]: https://www.docsy.dev/docs/ +[Hugo documentation]: https://gohugo.io/documentation/ +[Github Hello World!]: https://guides.github.com/activities/hello-world/ diff --git a/website/content/en/docs/v0.3/getting-started/_index.md b/website/content/en/docs/v0.3/getting-started/_index.md new file mode 100644 index 000000000..cb828f264 --- /dev/null +++ b/website/content/en/docs/v0.3/getting-started/_index.md @@ -0,0 +1,14 @@ +--- +title: Getting Started +description: Getting started guides for users and developers +weight: 20 +--- + +The following sections provide quick start guides for [users][] and [developers][]. + +If you're a content author who wishes to contribute additional documentation or guides, + please refer to the [contribution guidelines][]. + +[users]: {{< relref "users" >}} +[developers]: {{< relref "developers" >}} +[contribution guidelines]: {{< relref "../doc-contribution/" >}} diff --git a/website/content/en/docs/v0.3/getting-started/developers.md b/website/content/en/docs/v0.3/getting-started/developers.md new file mode 100644 index 000000000..3722293d5 --- /dev/null +++ b/website/content/en/docs/v0.3/getting-started/developers.md @@ -0,0 +1,81 @@ +--- +title: Developers +description: Setting up a development environment and contributing code +weight: 24 +--- + +This guide provides a quick start for developers wishing to contribute to ClusterLink. + +## Setting up a development environment + +Here are the key steps for setting up your developer environment, making a change and testing it: + +1. Install required tools (you can either do this manually or use the project's + [devcontainer specification][]) + - [Go][] version 1.20 or higher. + - [Git][] command line. + - We recommend using a [local development environment][] such as kind/kubectl for + local development and integration testing. + - Additional development packages, such as `goimports` and `golangci-lint`. See the full list in + [post-create.sh][]. +1. Clone our repository with `git clone git@github.com:clusterlink-net/clusterlink.git`. +1. Run `make test-prereqs` and install any missing required development tools. +1. Run `make build` to ensure the code builds as expected. This will pull in all needed + dependencies. + +## Making code changes + +- If you are planning on contributing back to the project, please carefully read the + [contribution guide][]. +- We follow [GitHub's Standard Fork & Pull Request Workflow][]. + +All contributed code should pass precommit checks such as linting and other tests. These + are run automatically as part of the CI process on every pull request. You may wish to + run these locally, before initiating a PR: + +```sh +$ make precommit +$ make unit-tests tests-e2e-k8s +$ go test ./... +``` + +Output of the end-to-end tests is saved to `/tmp/clusterlink-k8s-tests`. In case + of failures, you can also (re-)run individual tests by name: + +```sh +$ go test -v ./tests/e2e/k8s -testify.m TestConnectivity +``` + +### Tests in CICD + +All pull requests undergo automated testing before being merged. This includes, for example, + linting, end-to-end tests and DCO validation. Logs in CICD default to `info` level, and + can be increased to `debug` by setting environment variable `DEBUG=1`. You can also enable + debug logging from the UI when re-running a CICD job, by selecting "enable debug logging". + +## Release management + +ClusterLink releases, including container images and binaries, are built based + on version tags in github. Applying a tag that's prefixed by `-v` will automatically + trigger a new release through the github [release][] action. + +To aid in auto-generation of changelog from commits, please kindly mark all PR's + with one or more of the following labels: + +- `ignore-for-release`: PR should not be included in the changelog report. + This label should not be used together with any other label in this list. +- `documentation`: PR is a documentation update. +- `bugfix`: PR is fixing a bug in existing code. +- `enhancement`: PR provides new or extended functionality. +- `breaking-change`: PR introduces a breaking change in user facing aspects + (e.g., API or CLI). This label may be used in addition to other labels (e.g., + `bugfix` or `enhancement`). + +[devcontainer specification]: https://github.com/clusterlink-net/clusterlink/tree/main/.devcontainer/dev +[Go]: https://go.dev/doc/install +[Git]: https://git-scm.com/downloads +[local development environment]: https://kubernetes.io/docs/tasks/tools/ +[post-create.sh]: https://github.com/clusterlink-net/clusterlink/blob/main/.devcontainer/dev/post-create.sh +[contribution guide]: https://github.com/clusterlink-net/clusterlink/blob/main/CONTRIBUTING.md +[GitHub's Standard Fork & Pull Request Workflow]: https://gist.github.com/Chaser324/ce0505fbed06b947d962 +[release]: https://github.com/clusterlink-net/clusterlink/blob/main/.github/workflows/release.yml diff --git a/website/content/en/docs/v0.3/getting-started/users.md b/website/content/en/docs/v0.3/getting-started/users.md new file mode 100644 index 000000000..b9531319b --- /dev/null +++ b/website/content/en/docs/v0.3/getting-started/users.md @@ -0,0 +1,128 @@ +--- +title: Users +description: Installing and configuring a basic ClusterLink deployment +weight: 22 +--- + +This guide will give you a quick start on installing and setting up ClusterLink on a Kubernetes cluster. + +## Prerequisites + +Before you start, you must have access to a Kubernetes cluster. +For example, you can set up a local environment using [kind][]. + +## Installation + +1. {{< anchor install-cli>}}To install ClusterLink CLI on Linux or Mac, use the installation script: + + ```sh + curl -L https://github.com/clusterlink-net/clusterlink/releases/download/{{% param git_version_tag %}}/clusterlink.sh | sh - + ``` + +1. Check the installation by running the command: + + ```sh + clusterlink --version + ``` + +{{% expand summary="Download specific CLI version" %}} + To install a specific version of the ClusterLink CLI, use the URL path of the version release: + For example, to download version v0.2.1: + + ```sh + curl -L https://github.com/clusterlink-net/clusterlink/releases/download/v0.2.1/clusterlink.sh | sh - + ``` + +{{% /expand %}} + +## Setup + +To set up ClusterLink on a Kubernetes cluster, follow these steps: + +1. {{< anchor create-fabric-ca >}}Create the fabric's certificate authority (CA) certificate and private key: + + ```sh + clusterlink create fabric --name + ``` + + The ClusterLink fabric is defined as all K8s clusters (peers) that install ClusterLink gateways + and can share services between the clusters, enabling communication among those services. + This command will create the CA files `cert.pem` and `key.pem` in a directory named . + The `--name` option is optional, and by default, "default_fabric" will be used. + +1. {{< anchor create-peer-certs >}}Create a peer (cluster) certificate: + + ```sh + clusterlink create peer-cert --name --fabric + ``` + + This command will create the certificate files `cert.pem` and `key.pem` + in a directory named ``/``. + The `--path ` flag can be used to change the directory location. + Here too, the `--name` option is optional, and by default, "default_fabric" will be used. + +**All the peer certificates in the fabric should be created from the same fabric CA files in step 1.** + +1. {{< anchor install-cl-operator >}}Install ClusterLink deployment: + + ```sh + clusterlink deploy peer --name --fabric + ``` + + This command will deploy the ClusterLink operator on the `clusterlink-operator` namespace + and convert the peer certificates to secrets in the namespace where ClusterLink components will be installed. + By default, the `clusterlink-system` namespace is used. + In addition, it will create a ClusterLink instance custom resource object and deploy it to the operator. + The operator will then create the ClusterLink components in the `clusterlink-system` namespace and enable ClusterLink in the cluster. + The command assumes that `kubectl` is set to the correct peer (K8s cluster) + and that the certificates were created by running the previous command on the same working directory. + If they were not, use the flag `--path ` for pointing to the working directory + that was used in the previous command. + The `--fabric` option is optional, and by default, "default_fabric" will be used. + To install a specific image of ClusterLink use the `--tag ` flag. + For more details and deployment configuration see [ClusterLink deployment operator][]. +{{< notice note >}} +To set up ClusterLink on another cluster, create another set of peer certificates (step 2). +Deploy ClusterLink in a console with access to the cluster (step 3). +{{< /notice >}} + +## Try it out + +Check out the [ClusterLink tutorials][] for setting up multi-cluster connectivity + for applications using two or more clusters. + +## Uninstall ClusterLink + +1. To remove a ClusterLink instance from the cluster, please delete the ClusterLink instance custom resource. + The ClusterLink operator will subsequently remove all instance components (control-plane, data-plane, and ingress service). + + ```sh + kubectl delete instances.clusterlink.net -A --all + ``` + +2. To completely remove ClusterLink from the cluster, including the operator, CRDs, namespaces, and instances, + use the following command: + + ```sh + clusterlink delete peer --name peer1 + ``` + +{{< notice note >}} +This command using the current `kubectl` context. +{{< /notice >}} + +3. To uninstall the ClusterLink CLI, use the following command: + + ```sh + rm `which clusterlink` + ``` + +## Links for further information + +* [Kind](https://kind.sigs.k8s.io/) +* [ClusterLink deployment operator][] +* [ClusterLink tutorials][] + +[Kind]: https://kind.sigs.k8s.io/docs/user/quick-start/ +[ClusterLink deployment operator]: {{< relref "../tasks/operator/" >}} +[ClusterLink tutorials]: {{< relref "../tutorials/" >}} diff --git a/website/content/en/docs/v0.3/overview/_index.md b/website/content/en/docs/v0.3/overview/_index.md new file mode 100644 index 000000000..b2fc22859 --- /dev/null +++ b/website/content/en/docs/v0.3/overview/_index.md @@ -0,0 +1,51 @@ +--- +title: Overview +description: A high level overview of ClusterLink +weight: 10 +--- + +## What is ClusterLink? + + +ClusterLink simplifies the connection between application services that are located in different domains, + networks, and cloud infrastructures. + +## When should I use it? + + + +ClusterLink is useful when multiple parties are collaborating across administrative boundaries. + With ClusterLink, information sharing policies can be defined, customized, and programmatically + accessed around the world by the right people for maximum productivity while optimizing network + performance and security. + +## How does it work? + +ClusterLink uses a set of unprivileged gateways serving connections to and from K8s services according to policies + defined through the management APIs. ClusterLink gateways establish mTLS connections between them and + continuously exchange control-plane information, forming a secure distributed control plane. + In addition, ClusterLink gateways represent the remotely deployed services to applications running in a local cluster, + acting as L4 proxies. On connection establishment, the control plane components in the source and the target ClusterLink + gateways validate and establish the connection based on specified policies. + +## Why is it unique? + +The distributed control plane and the fine-grained connection establishment control are the main + advantages of ClusterLink over some of its competitors. Performance evaluation on clusters deployed in the same + Google Cloud zone shows that ClusterLink can outperform some existing solutions by almost 2× while providing + fine-grained authorization on a per connection basis. + +## Where should I go next? + +* [Getting Started][]: Get started with ClusterLink. +* [Tutorials][]: Check out some examples and step-by-step instructions for different use cases. + +[Getting Started]: {{< relref "../getting-started/" >}} +[Tutorials]: {{< relref "../tutorials/" >}} diff --git a/website/content/en/docs/v0.3/tasks/_index.md b/website/content/en/docs/v0.3/tasks/_index.md new file mode 100644 index 000000000..0271126bf --- /dev/null +++ b/website/content/en/docs/v0.3/tasks/_index.md @@ -0,0 +1,5 @@ +--- +title: Tasks +description: How to do single specific targeted activities with ClusterLink +weight: 35 +--- diff --git a/website/content/en/docs/v0.3/tasks/operator.md b/website/content/en/docs/v0.3/tasks/operator.md new file mode 100644 index 000000000..f70239198 --- /dev/null +++ b/website/content/en/docs/v0.3/tasks/operator.md @@ -0,0 +1,171 @@ +--- +title: Deployment Operator +description: Usage and configuration of the ClusterLink deployment operator +weight: 50 +--- + +The ClusterLink deployment operator allows easy deployment of ClusterLink to a K8s cluster. +The preferred deployment approach involves utilizing the ClusterLink CLI, +which automatically deploys both the ClusterLink operator and ClusterLink components. +However, it's important to note that ClusterLink deployment necessitates peer certificates for proper functioning. +Detailed instructions for creating these peer certificates can be found in the [user guide][]. + +## The common use case + +The common use case for deploying ClusterLink on a cloud-based K8s cluster (i.e., EKS, GKE, IKS, etc.) is using the CLI command: + +```sh +clusterlink deploy peer --name --fabric +``` + +The command assumes that `kubectl` is configured to access the correct peer (K8s cluster) +and that certificate files are placed in the current working directory. +If they are not, use the flag `--path ` to reference the directory where certificate files are stored. +The command deploys the ClusterLink operator in the `clusterlink-operator` namespace and converts +the peer certificates to secrets in the `clusterlink-system` namespace, where ClusterLink components will be installed. +By default, these components are deployed in the `clusterlink-system` namespace. +In addition, the command will create a ClusterLink instance custom resource object and deploy it to the operator. +The operator will then create the ClusterLink components in the `clusterlink-system` namespace and enable ClusterLink in the cluster. +Additionally, a `LoadBalancer` service is created to allow cross-cluster connectivity using ClusterLink. + +## Deployment for Kind environment + +To deploy ClusterLink in a local environment like Kind, you can use the following command: + +```sh +clusterlink deploy peer --name --fabric --ingress=NodePort --ingress-port=30443 +``` + +The Kind environment doesn't allocate an external IP to the `LoadBalancer` service by default. +In this case, we will use a `NodePort` service to establish multi-cluster connectivity using ClusterLink. +Alternatively, you can install MetalLB to add a Load Balancer implementation to the Kind cluster. See instructions +[here][]. +The port flag is optional, and by default, ClusterLink will use any allocated NodePort that the Kind cluster provides. +However, it is more convenient to use a fixed setting NodePort for peer configuration, as demonstrated in the +[ClusterLink Tutorials][]. + +## Deployment of specific version + +To deploy a specific ClusterLink image version use the `tag` flag: + +```sh +clusterlink deploy peer --name --fabric --tag +``` + +The `tag` flag will change the tag version in the ClusterLink instance custom resource object that will be deployed to the operator. + +## Deployment using manually defined ClusterLink custom resource + +The deployment process can be split into two steps: + +1. Deploy only ClusterLink operator: + + ```sh + clusterlink deploy peer ---name --fabric --start operator + ``` + + The `start` flag will deploy only the ClusterLink operator and the certificate's secrets as described in the [common use case][] above. + +2. {{< anchor deploy-cr-instance >}} Deploy a ClusterLink instance custom resource object: + + ```yaml + kubectl apply -f - < + dataplane: + type: envoy + replicas: 1 + logLevel: info + namespace: clusterlink-system + EOF + ``` + +## Full list of the deployment configuration flags + +The `deploy peer` {{< anchor commandline-flags >}} command has the following flags: + +1. Flags that are mapped to the corresponding fields in the ClusterLink custom resource: + + - **namespace:** This field determines the namespace where the ClusterLink components are deployed. + By default, it uses `clusterlink-system`, which is created by the `clusterlink deploy peer` command. + If a different namespace is desired, that namespace must already exist. + - **dataplane:** This field determines the type of ClusterLink dataplane, with supported values `go` or `envoy`. By default, it uses `envoy`. + - **dataplane-replicas:** This field determines the number of ClusterLink dataplane replicas. By default, it uses 1. + - **ingress:** This field determines the type of ingress service to expose ClusterLink deployment, + with supported values: `LoadBalancer`, `NodePort`, or `None`. By default, it uses `LoadBalancer`. + - **ingress-port:** This field determines the port number of the external service. + By default, it uses port `443` for the `LoadBalancer` ingress type. + For the `NodePort` ingress type, the port number will be allocated by Kubernetes. + In case the user changes the default value, it is the user's responsibility to ensure the port number is valid and available for use. + - **ingress-annotations:** This field adds annotations to the ingress service. + The flag can be repeated to add several annotations. For example: `--ingress-annotations load-balancer-type=nlb --ingress-annotations load-balancer-name=cl-nlb`. + - **log-level:** This field determines the severity log level for all the components (controlplane and dataplane). + By default, it uses `info` log level. + - **container-registry:** This field determines the container registry to pull the project images. + By default, it uses `ghcr.io/clusterlink-net`. + - **tag:** This field determines the version of project images to pull. By default, it uses the `latest` version. + +2. General deployment flags: + - **start:** Determines which components to deploy and start in the cluster. + `all` (default) starts the clusterlink operator, converts the peer certificates to secrets, + and deploys the operator ClusterLink custom resource to create the ClusterLink components. + `operator` deploys only the `ClusterLink` operator and convert the peer certificates to secrets. + Creates a custom resource example file that can be deployed to the operator. + `none` doesn't deploy the operator and creates a `k8s.yaml` file that allows deploying ClusterLink without the operator. + - **path**: Represents the path where the peer and fabric certificates are stored, + by default is the working current working directory. + +## Manual Deployment without CLI + +To deploy the ClusterLink without using the CLI, follow the instructions below: + +1. Download the configuration files (CRDs, operator RBACs, and deployment) from GitHub: + + ```sh + git clone git@github.com:clusterlink-net/clusterlink.git + ``` + +2. Install ClusterLink CRDs: + + ```sh + kubectl apply --recursive -f ./clusterlink/config/crds + ``` + +3. Install the ClusterLink operator: + + ```sh + kubectl apply --recursive -f ./clusterlink/config/operator + ``` + +4. Convert the peer and fabric certificates to secrets: + + ```sh + export CERTS = + kubectl create secret generic cl-ca -n clusterlink-system --from-file=ca=$CERTS /cert.pem + kubectl create secret generic cl-peer -n clusterlink-system --from-file=ca.pem=$CERTS /cert.pem --from-file=cert.pem=$CERTS /peer1/cert.pem --from-file=key.pem=$CERTS /peer1/key.pem + kubectl create secret generic cl-controlplane -n clusterlink-system --from-file=cert=$CERTS /peer1/controlplane/cert.pem --from-file=key=$CERTS /peer1/controlplane/key.pem + kubectl create secret generic cl-dataplane -n clusterlink-system --from-file=cert=$CERTS /peer1/dataplane/cert.pem --from-file=key=$CERTS /peer1/dataplane/key.pem + ``` + +5. Deploy a ClusterLink K8s custom resource object: + + ```yaml + kubectl apply -f - <}} +[ClusterLink tutorials]: {{< relref "../tutorials/" >}} +[here]: https://kind.sigs.k8s.io/docs/user/loadbalancer/ +[common use case]: #the-common-use-case diff --git a/website/content/en/docs/v0.3/tasks/relay/index.md b/website/content/en/docs/v0.3/tasks/relay/index.md new file mode 100644 index 000000000..e7905bab4 --- /dev/null +++ b/website/content/en/docs/v0.3/tasks/relay/index.md @@ -0,0 +1,165 @@ +--- +title: Relay Cluster +description: Running basic connectivity between nginx server and client through a relay cluster using ClusterLink. +--- + +This task involves creating multi-hop connectivity between a client and a server using relay clusters. +Multi-hop connectivity using a relay may be necessary for several reasons, such as: + +1. When the client needs to use an indirect connection due to network path limitations. +2. Using multiple relays allows for explicit selection between multiple network paths without impacting or changing the underlying routing information. +3. Using multiple relays provides failover for the network paths. + +In this task, we'll establish multi-hop connectivity across clusters using ClusterLink to access a remote nginx server. +In this case, the client will not access the service directly in the server cluster but will pass through a relay cluster. +The example uses three clusters: + +1) Client cluster - runs ClusterLink along with a client. +2) Relay cluster - runs ClusterLink and connects the services between the client and the server. +3) Server cluster - runs ClusterLink along with an nginx server. + +System illustration: + +drawing + +## Run basic nginx Tutorial + +This is an extension of the basic [nginx toturial][]. Please run it first and set up the nginx server and client cluster. + +## Create relay Cluster with ClusterLink + +1. Open third terminal for the relay cluster: + + *Relay cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=relay + ``` + +1. Setup `KUBECONFIG` the relay cluster: + + *Relay cluster*: + + ```sh + kubectl config use-context kind-relay + cp ~/.kube/config $PWD/config-relay + export KUBECONFIG=$PWD/config-relay + ``` + +1. Create peer certificates for the relay: + + *Relay cluster*: + + ```sh + clusterlink create peer-cert --name relay + ``` + + {{< notice note >}} + The relay cluster certificates should use the same Fabric CA files as the server and the client. + {{< /notice >}} + +1. Deploy ClusterLink on the relay cluster: + + *Relay cluster*: + + ```sh + clusterlink deploy peer --name relay --ingress=NodePort --ingress-port=30443 + ``` + +## Enable cross-cluster access using the relay + +1. Establish connectivity between the relay and the server by adding the server peer, importing the nginx service from the server, + and adding the allow policy. + + *Relay cluster*: + + ```sh + export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata + export SERVER_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server-control-plane` + curl -s $TEST_FILES/clusterlink/peer-server.yaml | envsubst | kubectl apply -f - + kubectl apply -f $TEST_FILES/clusterlink/import-nginx.yaml + kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml + ``` + +1. Establish connectivity between the relay and the client by adding the relay peer to the client cluster, + exporting the nginx service in the relay, and importing it into the client. + + *Relay cluster*: + + ```sh + kubectl apply -f $TEST_FILES/clusterlink/export-nginx.yaml + ``` + + *Client cluster*: + + ```sh + export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata + export RELAY_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' relay-control-plane` + curl -s $TEST_FILES/clusterlink/peer-relay.yaml | envsubst | kubectl apply -f - + kubectl apply -f $TEST_FILES/clusterlink/import-nginx-relay.yaml + ``` + +## Test service connectivity + +Test the connectivity between the clusters (through the relay) with a batch job of the ```curl``` command: + +*Client cluster*: + +```sh +kubectl apply -f $TEST_FILES/nginx-relay-job.yaml +``` + +Verify the job succeeded: + +```sh +kubectl logs jobs/curl-nginx-relay-homepage +``` + +{{% readfile file="/static/files/tutorials/nginx/nginx-output.md" %}} + +## Cleanup + +1. Delete the kind clusters: + *Client cluster*: + + ```sh + kind delete cluster --name=client + ``` + + *Server cluster*: + + ```sh + kind delete cluster --name=server + ``` + + ```sh + kind delete cluster --name=relay + ``` + +1. Remove the tutorial directory: + + ```sh + cd .. && rm -rf nginx-tutorial + ``` + +1. Unset the environment variables: + *Client cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Server cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Rekay cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + +[nginx toturial]: {{< relref "../../tutorials/nginx/_index.md" >}} diff --git a/website/content/en/docs/v0.3/tasks/relay/nginx-relay.png b/website/content/en/docs/v0.3/tasks/relay/nginx-relay.png new file mode 100644 index 0000000000000000000000000000000000000000..110b308c68360848869d4aa36b7b9a4739c2253f GIT binary patch literal 112485 zcmeFYd03NY+CCh+=uBOhX>D;~sa30piin6TN!l_lh#D0Y7f4hD27(|4h!CE{zD{9W z0A-amMOj2N5ilTxBwB=6wjhgPOJYR`A)64$L-J(#-NAOIGw;mt{q_C#9)}((gy+8R z^SZ9{I-m%}L3k|q@DL+u~tgRlJI*auGFk9nlSyT3y* zI&Ea&A0LH&yZ>7hsw!=v;mpV2-+#QY=L89b+WOS;d*1qUCx=idbE(f?zCCgYuP{<> zxauGYIsCJA-~3O+qi=4UKEI=KXI#yREuJFpmzn1r8jf=h)_!%dA$e)k9fypsHu*LF z{m%NspH}>`^UrIe%P2`rt9D*p>T@f7{k4*}YYyD}((Tg7X=|TH>-+B-t2zN*cdI@}*Dun*aOte+c{^ z0{@4={~_>y2>kzzKp5AKqEI8Dc zbseno2JQ3YIj00ox#@drJk78YJaaONGCMIgY85DHyqkNTRyFy<0(vuEVxYH$)}-*@ z(6jluaFnYVmc*+P7okv3*A$6(rdJyCqVqLOSIW>8qjCVPQd+rEL`_Z z)3ZX)P$wv?0=)FI3!NB6vm|SOH(qb!x2CKXny%4pDL`c5G3}FlTB#z|O3MxrI3jqq z9t1Qj7OjN?1JE-ZYoS|RirK$t)jU*-Ur6eLn%yo}GEbD)?ghYwV(R6PTNdY$B3XI=9!XWa+m7~fv@b*5 z*w~=!82iCQ)GY?Pd1~xZK*45qLwxAKf}y^H04lh~+}ymnF6zmX2A$}7<&fDykc>k0 zIM%#q?)!{K1_`mU^~RLq-dAATDMJhKll|D)uDdwX)warfD?GrkbSi4eUX25Iy5r1g z+(zV0$AUJ~aqWw^(%!@D!~tc~D8W3`c!9uq!qutYEL;e7vB8UNy&X9-@L4`o6==KG zCq1SGZI~*@ceN+ja~d0rG_n+nx;%jPyqoyAeWP;Lvi9QphT_p=W|Zli3xPWRW@1bd z9@1gfx!0@%D=T^wx2%O7m;psrFfZ|$BUj_RqvV9iF96UGI^& z@g{=Ek{La27*gkBW63kMI03 zhTrdkxt<+=M$JNX% z&L$suZ1=S|*vAmJjGL_X|*{9SehLj+B4_X)E4e%dVD|8kxsxy3B>uQQOV%YUL*zK*{A&o;gxwG*Nq~ zAVP2+$+Uv-+E)bVUCvRl+1c!!jd~i>`ruGV>Q>6M;mSHe=JIk|GC1|)H~87^a`QWo z8@NEc$UyjMxWA3usLExlk?R1-2{KKq+KqX)(#nLINj6wLn}(oI#sqL+1)>c*~O~2V)Wa4jTMp3IL-QIi#J+(J}Uv6i3HZJ7A z8nSv;=Y*bJ*&|Bty^#`bepH@m>IXSIS*<%%wv8^V;(WSFVh-)mfrC82b zy_pfomA=tyuZJCbaLBa#e+c?)LRd(blh>@z=4#qVftxIM^5)wcw^#3WNuID;ISGB<0 zrnV$-v-J`H{IzPgY-5$;Rv2%%V!*GBJIG*w9q0WBI8x09yoVd{dZpLe8)mPVKkwNl zpNHD$lY637ksgwlvY-a%@*!*FCfKLygd`?blbjc?zIt6gWOnYj!cK&(&?GpIOjD^u z_Qs0^t{`tt|Ghdi8>&$hhS`fopW%lTb-{x3{&-hht8d=AQ34Kke5!+)H~~Tn=~cUF zorKE(*B^P7Q&lT+!qhkNI4lKIxxo=2b1Pwd;%eP&s2Zl39I3J4Gx%-x0@!z9*xl`S;Am8!&=Cp5$nAkpK9 zgbsMOVe+)SC5rlB$tFq^`vWGkEk{HpYyM&wo&IPoZOjSmZa4O=(^CNWmZIuV(}2A$ z>q+w0R@(j$`g#9TU_INH%`+Lin<0!8HLZ;~%rOTMu9Iv~Pu5D?sS`uDe$4|Z$fy((|N8LPB!QHJxnakI8U>-`s`5>rk3QvHZ zt>%HYrf#|cd(kG6X*$*ZG|0L02H4k@@@i%_$5mM)=utciwvY55L54t#GVzfSpUXNTQ=#@Lxfpb1h&or_W zRl)TA6jg1AlZl5-G(HD`7)5%Tgl@E{HxpWkuT{d3y;(zlg50t)>`MF!B;xDLR;r#) z8~-etCOW1SCFgPK0>1~{{BikIR6ow;XtYs_lerbEkcua@GRR53!RkI?@6xLUtSfJf+!_rc^;Vs%z)}0NgJeXr-Rg4nH=7 zM@}^Z)w75F9i0H;DIozepqDJDn-p+WGPWC6y>Rj>*!ShN@goTD-RZRbh*I)vM1w#Q zumS@lROe1yiW^qDGTQV3>WQFrQD(^9JM9OQ%K&60A5Y(*sV1?#NXj`Rjfli&#-kk(UuCiFzn97YhYr^K?)Ix)eh5ybyTBX)cjDCJsH#$_}jEd1!#t#?9Bhelvj9jSUU;ov_Gk*P1RKeXPW-QDFb+N2vZw zBS@Ta<=H5cG46pS5&uYeS!r*+0z^2W&1JC0E`F$7P1RN~TXYsyfAyoHIjzy?5*;O8 zkHikt3}sQ4yIN~sjP@zb!<_yUtg(sSXwhnKt^gPEG6~c`HCzVGejf^t?jHR7B-nr} zh&x8pIB;)riflSrPzfITpnXqR+yEk2kDc*BKJS=T88=f@2P8T+$7$*W_T7l-AQ00im@q)=!8ab#O5?Hs z-VLx)uXs%{8P<(#{229A?OM~luE-7hxe9RBn!>VVMv?wih*r<+-gm6=Z3n#pk)1N1 zT?U>rz5%&7kl%v|%!Jqqk@6D^E=P#tf&)zMXI`38b)*@0C0+oZyI{}JRXJh$lY5E+ zKqqK?7+3)kSasftGyRN*9-)nQerJVEpuj4E{$jyh6l#{UpyqIHZUgDa2<>>g`D_Eg z9jydP#H6rr(z1}M1^5|;pk=uQ^4W%JY&oZ4Jq7ko8#d*}eU3e!1OBqF;ociu<1ZOx zb-aQ6=0i)NArBNy!A=K|i}KLHJ^-~0|Ek^7yerfc|9Y_+`L2L&d6?UF7r(sH73N)@ zk(MlbYsaNdxt=j4yl%EMNGEC+Z<)nE4$ne2D0;#Ko5A6=R~{hSZT z`idZN6Xs9gbH}e#Tjv;eC_5g)EXP`K4$cIS6w)RI34U=3Yb*U4g(_mU9|$YkP;<;P z&az^OT?6&3l<+s>oZ4PTmDP>3TUJzPKjTf0E}+B?)?KeotE<#K2ED7%J@A`JW0WrI z(j@rEqg}zc1Bm9sG%>d#kY#~*r)2ttpP9W4PUOpr>XFwjyp__mU`e%ihvG=cy=|vB z?-2T|q;4b!ly$1&m&$eKBYE}L)TUzG@LYpvv@C6-?WrTgKO&TK*y}0r>^ry~0CnR= zXyna=(&R4!MEYZ;qAe_Kvt2_26vn;?D5MYq&4zx!Mc_#_`9>f30Cl>KocPvPkY8!{ z*APaW!BR?p4tzfaN15XC>P@{?<|$DSj<%^Y3@rH#$12f4B-8Z!=?P+0U?!(gAywYT zL|O%+Cm#YfrLBWT>}Mwl*R3rHZ4h7KxGQ^t1>8&1+Easlr$BHc!2JHRb$PtB4?8)q z!+sv>!tvbF?i)u#m~o4MhtPf5t|4Ipu(-&uJM5L7%bRtK-VO=~mE?%T4K}q=@!ILQ zVEc052TZTb|I8dJTZx&fC8SvY0emJgaDd)}57a8F$#ssPWpK?@QNI`0D~Md`Aj- zkl$8>*UrSnV!0NZB$N9RHhp{@3%=XCff6&wY!izZufan?{axA;DS6R&vmZUp<fmj>ThSe(^r7pRpz4$U#L1DioAb>0UwB8UymS6>qGN4!2leM1Ybo#;FT zVnIL8cLtezG!AsI#_gSbjc+zRFVDo0mr9nZPSj>3WlrVm)9G`i> zz{hK*MU*J+r%o0``yVQLf|&=9xPhjjtN;+cjry|6d~D`eo;}rUJ+krrdV5{`bDIaN zEEq0GcG^U!QCZij^`zBZTXMazS-0 zi2(zAPJ`yiWD*Rm>Lh??fWZz^q@Awj)m0|)z??$G6uw&^*5zKZ#PSg=|*|uoi0q%RY8kx0qXKEtvC7#*(=ODJ+m!kkF!EgJ!dyPmPhnB zQsM`NZFUmIbZj73p{3;?m=n(816H~$%d%9MsF0(W45ml< zBTcRgplV9`8$Je1St~7=8ne)zN{OYaBm854{Ajj>J6C0+vCz*B7$DlXl^83vG+e>s z5es79fe+iqn-;}V;JfzvhdW4?GnM;kHIM6ZAa0-%hP&G6m(~})KC0G77?Qv@pI!)- z9H3kp5DXm+1en|ctDT1sM32r>mM6%v z&}gMPm>K175Cwwb-UUCk=1o;_w+;$&sw04xA4ynPIY+> z#9=q(aJYe!q#PlrK=i#!A_~@?2PzXlQ%x?18Wl2St7V~kAXl&hg@Ii2Cf3Eo^WZI7d>Y<}%9XK2HKILg0 zM=P6MjC%EUtv{28l;hw1#2)h>)nu9*FZHl7pQio5+JOBmpPm+o^&Aa#v|WfI9?_~* z2LQQ^@!UXJe<1`EK^rU#H2oArxItP{@EQF0cDi82W#`P?O3&o1fd{%kFwD)JI;w$W z8Dx7!3os_uj?j6ieI22HLOnV)AOw{y8f~LW4Dh+}&j}{P37Bp?(TC!9E`+~JP^UMw zzIkcL_PV$6OWABmLCr=-navMa#ZS;9;tBN>b;0G(EF+tpk%Q(aB83btdOd$cU^hAL5}stfvR+4yY$G#h&;OG1JsU> z>EEKBQiG$^s;OXJlnI&C1!DBit>SpIJDHazR87A;M-kIa-4mN-vy4IPyR)}`87_rm z-_V*rK@}YZ^^{B|YpR)U=0XP$soNkWE*&L@=!)DDzv|p^Au_M-(&zBwaiRi16zf>y zPquS(IjQmd3t*)ER)M0%Mo%ZSVLW5bVQ62Bx{Y%wJ2JWaBNXwdR$I*@ll9l->+H{) zJg4d{$vLXB#&e84lh08kM{lZ*z#sT4H3f~+3EG5Nydw$igpIa{)xDCDI3Ipf{T{XK zdG`_&fox0_c!fGSwQ|Ja z>w?_oYnI@8gWw*MEF|c!)1G*ONx)byxmNS})7!1+AA?K}i0)$g${!F%A2Tm0>=m6GngzpPrrW-+TmZENMVKx|G$bMG(F?9pPENay zg)T+~YxlaG$nDFMtOmiaR88F)a*LIgL?O@0M*=?HlI@@p>NR&Sm`mt0&q0|y2pkw; zv!MYvA6W0T7Lp|MUbC>eflFI2gT!`v7S}htahb;I-fY{W2R;e7b=h&KOAL!sQdC37 z%EvSvv?x#+9b&UgKWPpkAJYXfCzJmu?P|1H=dT(lBi7@(M-B}4++jLniKz9hi(VDB zwLW-du)75Ys@NRRrDdHyOe!pWv+v^z52ixB$_+IP;*)g?iRskN&p^#QiVIFttF*zA zC=+FB0GNlS1bCs?BrqTITK=?GZ~Eb7kga}UxoN8+!M5ShQSnOrT~RkF`}kn_D75GT zwG|^4y#zd95QB*#kCV)#&(?@ejN-VH8PSY0Cp+4l^N~cSRcP^ZOJ-_n<|mPOnMg_d zX;J-y_V5mhW%lX=HDO8?xy#O*yf$w6{-RZ}j^*MGfAx!L4|e1#9SN()Eqas)QbB5V zAFNip@ygrYn6J_r5v%J~UPcx`(m}a$V|gYcIH(=_V>djO{A~hW=IYa3b$l@AE40+d zEnP1$BFHN70WiG zCEy3M$AY7b)1;8NZf~$^K3VMdJx1ZP+W&dH@Nx5@}PU=(0Ib*~qN**`ASomcUnGi9cdt z-})##l0k<$_Z`bT@i5-XGSAmqTt#{39*It$#*hEx^|B6TL52gZc6;E57pUEWOK)8M zE_;GobOFQF%m8lTfhL3SetDz6O5-9{42em0>C?*o0 zgN_S~hgJDouf$5Lw^q8U&j7yYiv!n)zo>4+j0&1QSnWAOC(q=Zc^`klmt`XufXR9! zk<53ZnB4;Kl0>jSyZFjc`yGWC8tU8zBs=_OhH z-i^whNzD8m1YlO{d=m#G`H}%pxiuE)u8xytl1)G5{I*Zkkp1($Y0lBO>)f*02j5eN zokqD1F2~869cxZ%?7-u6e^LhT!8* zu)l}4n|_xA9FqVYA=;eNV1B(DJDoqd1{EZCIrud@^3#aUK+_3OUxDBV-fk;uOGn&s z*@Yve(hXIQaeQRi>qMZ?*p)d(j;?SMuwaNlc$?Ey1ygfFbJ*=;nu|v&bXN|Kn)cC= z9|L6t(hX>?rPiiSG#UQG_UF7n@~(vISgVCM?u`lVMmoZ)yW`UX;k5s?^;V2TH0i{Q zG&Rri-k5^ybxM#ARWlstM=#0sPFLhy>|d*+!rm=7^6KYmQsZ>M+H9@tlKqDT%<3F6 zIo2Fv1&ZZwdG&9xs!j(r^v3ah?j9;tb9ynbx0|ZRSU5dvX#{Pdi!~NnD4U~&Nqe_{ zR%}INUUo&h!gdIX=>*=Y;bCmS1LL;)}DU735 z3`2?vdr2^2di}{34u;aed2ICF*0+)sUDBfOoZnJ59!*HJw1Ux^BwanB74i#tqzX6| z!upjOfWk8+0C+7^kksKaS3i7ehqfq8htJ2#Pny)zu1|F`6QjR5^iMs}bP(K3h|3-! zfc0j;wJO&b#WueVrR&4J{`mBEH(@XO+=EiC=L^-T6^40PvDFG2C$#_7QpiKhM@cIc zqoP2LLfxT`0QGZ%4Q_AMY*H)GXzX8c4j#8}n4p-q7mnDJ^0S|m?<8P|x9Ldfuv_qa zd1r9&F`~H1(!x3bxMjUkQ!0}DO*)mZhMK&O51qZNsnQ5uZa4GQWb?CR>@C~}Te2~3 zA8&7cm)q&u8*&>xo%~GIV}H;gt=_9BkaqK-LAuiSREahaSt?AFEKqZ z$34TDP)DkY$CJ@;^BA#Jp4oXQ=&E`o4;JmxHN>R)4F!-II!gmqUoeA#pIzxj<>_pJ z4%kKv{#w?)9^-f02V}VeLi8}*RBcY&L){Lc8qO4JTf*} zPHJBjs2N+4Y)C`_zWX+k$}>F6k{YkXd~n%0b^+ac%x1V1J|N628OhjYIDZ3uFLM6d zX-;^ie0k89qS(cwhMgW|<25l63Qxk;x0LCZa6(y3dyo?0?vw5b7o1{nymDkVJ9VM_ zMt?V{WCqh_3#h_4M|pv4#pXm+&jdDoterO25~+3qN?|RK6x_lQ1`~B>Aak~tL|?X9PoGdr}G0e z9R8&x*Dn@9l1eoT7d9?`38Ve(NFKQKRGhZ3CV+4&wUVVq*SMRVnL%`EiM$KmY~E<= zmhC_vS}{5sBgV9n9?Mt6p?OH^z$ous_0U zt_2E$>uc`_oQGaL(o=7bH__$W0O$Ekr5xqPsICWP0UbydtW9mu&H?=+^NCT93}D~yxW?W(VJu%|ri zAP!3@3eS*J@Q%x?x{IQ83qo$qFA7XuzLeh;+O>~Q?kiIFkNz0f^?*hE@!_FVU+M5G zs4*yjrkqLDc`3I}*RS;LAResLa&lAmw_qe)(grQGz>2WI=wTqvVb@) zKlT2H&l8!PM3FWc736~?Xehfa5}^A=ddTuJVWZ8_)PYkyx;XccP(rKm%2$OhGD_NC z+xh)RH7kU}D+hLr>4xrsC8SYwMk=m*GLR*$jHj)E#4fT~NMC^53`~o9 zASBaQsU;Zol@$=p`g~=4YAuf|kZ*rmf9CeKd{^f8fDFTZHHY%_y9O@_VHTJ>UQzMgX!%Ou<|UA$ZU4$&+3{V(m0~BE!wrt=3(_58AwcrK?&H6L1(A zE(yvgZu3)WdZny_iBM;B%XDnesh@(Jk{Vk^2Yc#k&tth5tQ+6Jo!`GsDeq9*S}4@l zWp7^aZ9JeF_mHUl(xKoNT<&9J!A=kV!xZ|tR(?NA?F`qaV&KbhKGOZ&Ret#suI|%C zmGxNGtyJ}Ww4_Ti4=O5;F>HoZ>^>kPAW^^G^kCm)|Mg;Kw1>ZNjEIxYg;LTq?@A7jIXmW>43fl*GMeFA&^yV6kwX@Cdx+`YQPR zftkF*?jwT9VyT;LMlaXDyXxmIB$mYtmeB8F)VEH`IGNnY2*?iUrq*7MVK6#`T8^n$ z_|c3o8BQ?*@myEE(?P)7oiTkR)jG3`LKrCwN*t(_pN7{Hok8_Zm zJ)RQW3O5z=GgE!S>G?4cTVvwyLyT}H`5HX6t04&>S_vtnCdxhaOb$~`#a?zUfjBzO zY71|N?eeKv)I-FlZs02($`aWnBk+kgmTX;D!2yI&*v;h^aH%;li4XITEzLIYk+Q~6 z{}OfmA*69Bn9nQdQjFlJ2PSt)%0)Qc!W$2@@Y7y3`|JyOL1)iVZsdy@H4^yzNq5An z#`%^Owy9%xSn3DM?g)&9XHyZvP<8_D&RG2LFj+>yh9L>vSV=Ic{ZEkC8a$kv7XNoH z9BBcvwHLcmMu{w8;#*Gbw!WwPDbqC3aNS(zEiD0EcSFuAoJ#hZIhE-TM zZwVPSG)DplHZS&Mw^zN5>Dw5+(&8zciF*;GCOza&xcvAkz+acQ>dM6wZ$YpJct%_2 z?J|5pO9Gzw47P4-%OQk?=yBnT8OPXZ;4%5)X9`}ZD9z%XxvT8abq-M+&yjO7Yd>Cdg zY3b8Hd8?eVAc%o>4-_obi!<)kbB8%sixoE8>fiyvB<4USoOk4Fs3B zdW1_$b$u&+!h{6p_4*Rff@T>YX{OP5JUxcX7?j&i*~#uAMoTous&K|9bu+*1fjEvV zIelub>V0MczY0$tahO^DjiuzTe=GTzsU~?Q#n}3Ekd7mbivU?^bGIW_$xX-Ua+xMCu2i!YKvYUxHpGx}$2Bfl|EO-UlWg{U`Rs{t zs$kkRs8Z*itH($C)9Ip^L}8a)0@pnKUpaYSchu!Y1$lI>1i;92oz~Z>f0(IX*C*Ap z!XD7BYRSkD2U-sEz=qSfp!a1&P##_8g=m?5jF1k zA6RmELtPn3S*P;FpOFz;jxj0_sBR`+lEvIFbrkHROPT#JW;i?5)>r9z zq4D1v(geGlTzEDVuSZ`EqBlXUBLgP9j2Jh7VWh<{*4VNS6|rO5H;=mS6OeTRgliI! z)|4l90G(U!-omB-Fg@cF-w9I8(x9X@m0I`8dND?QcZ%gL9uYBd1eaPU4U$TLt4QVE`FPg(pzd=hk})rG51=q*W{pQ$cPl^i4S%DjZ50c-}-AoW%eOGuyld4&#z zOFUf#lH=j{WX5z}rrF$8uSJJKMF`l7QkNuVPiPuTN9sz0(jkGQcB-GPW~qdr1FZ@# z5`uP0VoY5Mr*Xpxni$3LaDs1Y&cD&c!5|T74@_@K=MuIG(N{6>&-w|xH(l-ec4nE} zPSh`Sm+JKQA&E_;SLd`5c1U9*hF^D44sv~pGZ%+*5?7@kqIsW)+LLXt0#AU6OE|~) z9xz?`m91xXvD)+U`7lA9;lp)DJ(pUSpS-X{=CY$~E)0WwN$5ef%aKiV%4|NYgY>?@Ae9t6FMiT8*QoiCuS#h)?rleFG>%;#V5=)1|Y1^#=qm=R8RY)eE z~5>>@s-77fDTKsBsDIFddoP)L${Xu(Z*Xw3Z{ zI_?4g$XzWYglBxDd$b&Dx9;d5A4yu+zX{?CH!yJg5(nx1fe~V16Z)z%oySl?cQEjk zaJBRKN{9;=vIVZ%cxEp~;Hv(?j~n7w0X@7n2rdPfBCgN@f_u*q)Ggq#vS-%t#`$3(BsdQf@@z^wki=k(DUmwv5RitsRm7ws{BZK!K z)sn!dj(B~ebz1a&coH(9AHU>iFLFCriy^^0n{@5W7@TSu%1f88?by1H8Sj4Irorlv zcJS=Ds-QEspvjW_Ihp*7RP~0m@3O(=43Ja_O{+D)SFR> zhOY;?kjqNhV6Z%syw9VFLdOb4;;Ti>X_zWthqxmU_?Py zAcEahD_H{mO_)Cx?+$oL*~=X0Z-vM4J)6C$m<9n!pO>O!4wZZD{x!W_Jo)8U7V5wJYRUJq{x!Cg z7ssz$qDwE_)lR}E_FjugeR~8b5Pg8g*|%j3oj4w=rV0YXX|H;!elC^|mTE&aT{Eu= zq_?HKG(S#Sk1jn%v5I7}+i+m!wNpNw$M42sKq%@;$G<}?JFcc`U#_qUuJ zBnmQ8ZHFAhe0T;v_!1`n`DwpJ+fjjN{in9$KhO_fanG@g8k^n6PNp5Aq>XgV06NCN zyFpj=h}$4T)|Ef8I)V&-Zc&<=#ZW|$=#*EpfcCJ;u4k2>WHkd)_EZgZJ10-;M1nb{ zc=)@~ISB)=(66YzLT{U6=k2NWG@&$hf|Pb&s}qwu2Z?f0V4y59Fu9X9A&iKS>C^oJ zNP4c1l2`;rH?eT#2!tc47^egoU<4{0|Q!ytUslys0ZrSH7r5%NsS>vdxf_% z`MIg~eiq1Bmf2HXJaw{66AQ{1!*fiF=LyG2n`E;qq*In8)8^`D=8f;AaDjbYv2CgH z;;_#q2Hg(31LHo6}>i1&2n+2GjkF5vA(FUOWyYSHoXZ zSdZa&p0(fRB!F2t2`JW+Ac8@lY`fi22R2#s+WLQ)Y72(DkiD`1ACKB&0E63bKr+Uf zf1e{EjaO;wqmF2f^Bd~6mx1;HsBk5c5H9f7aB(e-b+=Oog2!iG?PxZ34+=+b(|;IL zAF_+`psu#9X>%5~J+=&CA-@~KPAR!!eKt zlw%=N-WA(9nU_*3V3m*Xv$;`+8J@P1tknJCVxdobREZ|GFCfYzGKa*&6?A5m1QAQY zSG>|>PXfawIjErDuv@0A&ip=1vGZP>^D+`@KmXNzjyOiq(l0oZ-kH_AIi~!0TnjJ{ zk~gy;3v#9h*RCfDqzx3R_K?EHnCT#AW1f^%GQ|W-2=dS!UsKhKG3C)~oJ+s~8?CNeMy}Z9 zat^+?h&Z&A=}VU^0EzcL>`F~;swjJI-4D95el6S`gQamd=_l!HH(OdrqTPxW$fA+* zn~dlng>;LLbZ@wIgQa9ozL_ag7Y_4F(YF4T1+?*?@jN~BQfjb#as*7SsFrL?tG4{- zk$<-819yg9+{f_E?v^&;Jl7`rFmzYi`eK1G9a4WksOb&px!uwzaZ9RFfyB${nNeX`|mTt1mxG`gJNwJIp`Tz8#%8sbMy#b}Y?xIo;z-wBu zIR0AMY;m`a?8A@ph)G%LYxpmD)!T(EmByHV+{0hP zf*&t&5a8VX52 zQa5tqc+kKy#CdIyaZ%bYT@+8-i~@aV4%Jedk^ijC9Y1t%cw@qz^{u*Y*r8JfG>deN zE%pdq%~sblIQ*D#{4qM^=v?#}!iYt(6&c>@p(;yiSy*QPE)th($!+~YwxC=68NEjk z_dbh%pG~9vXJR)7IYqOsKP%=-qz!Z6|Dz)PoIHa(n@Pj|da;l4e=<||XIDQK7BO48 z_}gZXe)X!$VWq%CLW4I{erFQJh!MttbvxM^yaM;tJb4k5>Z z8@E?@(?)Yr`%)UZYEEek!ruw|;V+N^^-bXRuPG#MP8JWn5A7n0;Smd7x%tiTaLA&3 z3aJ(=tD|<~2-qBN_*14s$=ueqg#+%Ts^yrVfvn_V_Be(1<~=?o0`#2}Ka4dS4>K@uS{!&!hzN{^(0wwG z^i|B!KS6l&`>78D`wgE`kUCdD1h#{npTCDNUU8sAcB$S5EXC9 zh}!=Th>Y{HNGr1W)Fl})jFPxw&u05;%X=J(M!4PZfpF`Rn273Q#_q3vOQ3V9biza_kPM>VNrv)^X=lgj4vzdXQW>+bt(fx1 zAjvv;&<%AVx3+Su=qR1&ZlM7ZH-rz=f=z-Xb?t8~#E{q~_nH_Jh(RPpo=c~JLN^7a4Li!` zWN3xEzq1x9KIzBVB?=;{D`L|;e*hkwk{JIWs7P9(O`OArTX!50R~$l?l~eca1PRnV za~_Z`=%MfL4}Qnf*{5G{ngS^a`oa1V`OtPPRE^PlNk@rnz7S98FlJ%|O}$G3)dkFf zs&3b>T>)2H0?k}!n_mKxQU;MH-gF*viT*?&XzLBf=fnj3zQ88fjZSASOt@rPtbhfu zx3}(a(rBLWum5OZ6*+}fg>T$djsWv9^bt_BrJDgbLfOq26N9x;>B{t>6Zi8!Y+v=i zJO7ai<2#)C;^z=zjwq!EaRzOiU;GL%L=GYkw zHx=qlUyM?E#X0D63i{7LIL9#j90g%;s4FOo|A-PWu&g7-XN<0 zTyfv-ec#g@vb8A}Q}~e_J9e$_lVfp~>9FCS)8T2KmM2d6-0dfRDw0Ss2R%&3 z`HALf*FYAyZKc}jca#&ApyGt{ij_YS&SY!di{z%EKaqGL{5`_ki#Tb6 za}63 z3})0{3|%9&f57?!f?{_S4sGq0Cg4~%e^U|_PzrnOVrL6jvy<5w^Q{i zN`xzG4GL}i;_KdNGfUHeB~jiW7~b8$IQK_kYvjKv?~Oim$`@=EtpLdGDs{%HFxHl+#xQCy%36hj(LtOznDml#hn`9G&Z@QBRc?ep1v6MLq zm+vi0Oqr+knX9;ZjJLR@YgOVRi!Sok6`yI`!(U6n(3%~-_yzPgZT*hA;F3MU?B2$@ zmYZDi*)SToFIm4NuniaoAoEvHt>)dBo9O;^C^;gdINcL02`nMSl#Zm=tHD^(knRl! z&&?uqP-fn$dBAuatrBu^OVBvK2@M|8OkoOUER!Y}PN?r3RGVInlWcc-@LL@0dU}~{ zw?T!GOg5zB7Po8vKiw|hZfQ|7gYtkQ5mYr6Y%oIVHK{v* zCV>2_-x4v3F6JyrYAK@h_vi$eg>~nb-qnQwqIkK z(e8uW2mV6}9VXGM91BBp@(RrX79rV?4hc_Ea3L>ql1fKg(P`x2Q3-8~6jG<0xp7A$D`Hxck?4Wxn;gFOOS(eq50 z6+Xkk9ienU>bnJP9L?Is5bv3Kt)G`bhJ4KuTCT*sYJpJ{eND;`5OY%{ zFEU~hx5f-CKcz&9cZ{LIgznBSZZ1R677-|-@t-{x5I)Z%cm}tdy#K}7l4MDp-K|VdD?WmDn6~8R_m-a zR$!!+^=VID!tsvX@hxDePoJAMleqSJFB-p5+H#ZOAk&zBnf|w_WZ7Ag)n65X@g(sI zHWzjmWsH*uj<(qBX=D0D4vTq6k3o8i>JkGzOh>Y(@)iSue|N{2G)R+BMyq{kq_TTW zMLv8Lu+U2d>e4&fgtRO!`5#Mnj)t=L2(t<%T<5>HTyF1_jGn!_2nyQxMzo3~xx8Kb zsoU7)A?e_l&3C`*O#!mbmfo%#j+AS>ti7q(wGv*k`0kK|NFS6kwV~zrpa`h59vDF7Q@5ri@_2P4c)U2xm`KWCt`k92UftCP0bLiPK9=E@&%%;>kQ3TSgO zR|VlKy~n7VuEzwj7sYorFlV?(I#e`dY0=1o_e*XnBu0Srk@cSkI{h{r1>VPz&HDKW zGdBJbnk)J@SjQbjZUbbed&laddeCZ&%x0xb4#r}7Yd`#R(gexYwX9Kcsy#j}#DO_* zQ076-1=Fuq2(DlfV4dw@#IsVZE`4)EEdV+`MgjCXIOL2j*S(Ka=i5+rP9^{7mE&CD zZinV`7*)w_P@Avi-8ypL=0CC)9tDG$b0l|>^RMYybOku^vRU^M;!wFZw1WL0s)Q-p zL!kG9dm{Ky^f=3|`jPcqa`I0CtL#Z&_oQogeO$fZzc>r`d&Qv&@J@Nj7zMm0pDPE- zp0RF#Ob8Kj9~@R)EC-g*iVxrG{`L)veWA=}_ZMAeCq12_=P=#36{t@skd2>M+Uw3U64&!` z_9Yva8<9ow4id1w-*aWPoRbb}-9fMu9v)J2FcNqfB(d7*{x!#`mLLyo;79lp=%5Wx z$J?LJ4CVsS0RV8o6#7O?3rt&<$@M zO%%b;w3IJ7yRQw(MSCRWX%{0pVq;^OK{>_DmF2hob+Z445R8qacGw)@G8AKMtww3f zWqNxi=!!s!jWw)0gamvcn{_T#yC3nOrEt~B-s&q*EAer>sI?xvf=3XtN`w)ApSs<{ zvO8=?P9Wrfy~=CVu6qe|g(trj!|d3V5a8WeqwJcA#9x_e?8DSD28C5)P$59EzzI#r zhrdZkTZ)KR|J)@lT`xQsf#>H*Pv523vTaDvuY&ycQRQh57UV4n*#ZDi-?xnr^!jrW4crO^Wf%h2!HI|Q(ilTUDRF2Vqw>b~1VU7DZvnYo8X-|<6-&y!k|3$b z1l7qYnPxBBAzk|{pZ@tb=r*Ke^8A%#zTWRJU*gQO;a z<`H_RlWlA$B`rnpaLQUvh<7Qmu$3!Z3qjRto1r8wtp%KxtuFRfOBk56nz2SUHEnT~ zJoK;QL{h8e$~b>Io`Y)1v^bqx*wB^!3dO0vAOQq0pT%7kllKP(Eg6OS7PbAVGCYT^ zgSc+E`i}~xj5oPv0BebWl#z;Mc?Xjm7fPdr^Lh)k8_WkwY959Q6E7%m`ONV6B&IB=kD z{GwXc*GQicr}fu4B)jlcfSjN?9K%#$f@Zw(`)8;}hAAjVJx%gIO6Ldq#b`jlv$3F5 z?FjHY6|V->9}Xx%ZwHh{DIa}RyimT{eL13kr%3NHYS{?nihPD+?DttxbW(`MMQku% zu%)pLb%h{t_QA@g|-OP`jhl+RO=#&%x3l*YcZ8tPx)v_uMC-w%6pt_MAd`X+mXRw2yj>W8#$0PM}S7Mh7))YX(KJsz?X_L zyP2+*e4R=jZA9-eEF97~P5zLK4J>G~WiYumktu`R$qex?y@sXV382yp=$G$1W4KV7 z(d{V9O8K63rq5r+5$c3@52`O!j~EVi@B9LEEgr5CtNRcQ9EtzP8v}R}LwMpx7}5FX zPmSbofy7IEI!qZ6(a&Vna`7iG`hssJ(j@T)f^@`nFSj566spUZmNmnT=%~t#6w(x> z)d9svx{(kWT6LRF=O{- z-ak`m2dT{jMtv2Ze_<1tZ87VJGk7?!wY}st3e*K4X*H%_iiP@ui;H}BbfzMLt)jOf z$3PpE<^upk8QHl}0I)sf4UTbncnk(br+`W>co1X)BwWd9&s;1oCqf(a0r}Ui1&(wv zJL0+ny(*kq`7MFi^w)&euG7ZQ$$oCz=7n8i4Jz2O{=hKRNHDg!v*TqM^d`%+<6owX ze|^=_xNTw5b=T8o9h{FP;}a(6I|;N}M#>5V$Di@f9ZwlAM^$wSZg#M2ReTOWL&V?= z*=s|@&Dfd;{xqB%eV`Td9fWG}zMg5@5G~3zZfFW1etD50iJwuDz|xFMw?|;@$+9iAYW=nh@FYV_n|Cd;L=BrU)CTF1jo7u|@QfeO&9z z+I2Su#ePsn-U}{@`8L*}g`boY0i;!n<$r+~|D+VHjhp0wD5^;mzq}1Ua&l7iL;Z z1gYNT;(q+#IPyu5X`&&PJ>%IouL^W`jl3d}lx^0-zG2}{6*^28Xi2R@tVCiT=y}#^ zz(t!Cv!Um`oL^syQ+?pG6773aeOF4Rk7y-f6wA;xR3G(B>dsD6e{#Q3?SXiDE%1tn z(S8&SjI^a&U=F1@a$!&V`BqsztQF9>KBAMYPXY5~qerC$MgwA-i(pEQA?Xa?ZMoHA zb}k5ve~09U0b}7tC^BLdf^s3^ z1!m!YwqR3yQ#6&%R&$4^_I);b872Ou6AtGnx;{Y_V4X(HEM`4d8a|cSL~7c0H~)j8 z@XX*A(b%wAc}HBB5Opw##IF`)vd=@ahn}OCBS}X`HGD{{&gLGm#na0{>)El-HqNx> zjRCH7UYK}wA5is7T6SxOW}2uPEvAbin)s5PrnF&>M4*8l=AdqlTMSu~M}vB%51f34 zs%chG$FbeD#xk$$Nig)3Kgb!3MzA;=9b@w&i0NkOVG)X1qM0d+yqyuR&)4gMG$w$!5Q&}!JdQ+&h|4p8ZG4SBE_=E0;iLX zPpm`8ekM4kD2NkfHF2NzoD<8k#_==`)bkDX9@hqwl$_>CFCVcbFg|gtS`j;G(3(V* z^omFN6O`9I`44Fy-5Oe$!vQZT_;9~WY6hJ~@q|vJEFDrCRdX_iOI@I>LglM`;N{7e zK*#XqK@o6jUKa_87W}8?YfyBBQ^1=Za;l(7>SZ^zSIG@7fY$KRud3eecRAX!9is-+ z684hIM?f^o>iV;-3LcbY0Oi8qKWl91 z)sM>e{S$8BM_t)ahzkCr``sXeNt*_n@!JUvEHCLs3N|D4G}UfFMU#22g%gMiGuq&f zYZ4X;MBpU>AAo2|u7Ly;k6$<^@~BlPY5Gm!WN_Nsdbvt3l*SUEgSIb6*T64k76KS> zlNA{vp(r8&EN5UpbpGLt;E3YKAn?%Q`SyXxRVpP=co|dwXU84rWAp%2)dW9YyM1ze zk~;@AB{QL>qzbo65{}DCTmHxxT5jlnyWE6KOmC2Y?NvOl{(Y~7iaFHfCu9Fhm6{_* z4Q`skr(8ObdG=hD7mr5&w~w}lB()akCC>rUD(%13YAH;4q<=x7FMjyozy2YI(mI$s zBK-CADiw!q5Ef4Sw^(qJtf)3 z_krR2!0>%w_&zZFdjmry_=M;`LbDpG)ll2?A9$v^rn*1e``K}0eQ)i&bKYmX6TST$ zK3ntcKX07*`I9YA{*m$djqoKa&1DuwZTD{Js@87_-So?+J`X)j?wm%{-TX4V%VcBr z&Ffzeqq{va1rAtbjEidw*+nQ4XVb(!mw98=r7hCON@qHDoywl=Rg7`j+!x^A*)8<# zWm!v9j)@P49LWxWF2!GQ)-JXLUn5poPj2XLngd^2J@e`QaiJ7O`uXRde?PX*WPiCY z_$Uz3IhXF63tjTsO1CRZ17Gn{=`g4g=qbRbWRCsR6GtSa8-V-mayy)QF>r&5O0)sp zFKS>bxKx+DPk5-j5d7fG^nvk&WN0R9{j~-!9G{&GZm62&Tv&7zSlsY|Mt21M7jS99 z_b@7BY&-b7WAi>8BfX z%l7PR1fMq~WCzx_1g{2{2D*x*+v$hEXWTkmQR6oB2(Q?ns0Zs1m?MZ~w>J0|By_v7 zqT@ym3UrqD)YiX+K^!9#$kakBy5pxB{prV58g>f&uh{j@xA1$W>)z%zR-~sUG0t4n zV-tV+F_lW(y;5H+CqcykPdnaauW1rQh=kvdcP`!$Mt3OK*n* zDW!aO6vq%x@Ba4NZ*RWr~vYn03ux;n4S7 z!E{;Yaky-KO|`e7ua@}fAijA;hvmd2FOgb>a{TFv8eUPd12&Pk?#>=PmFSB9`CfGfu~bcex>LIy zak}gH*|jQmh4M&; z39mID`R#3}q~1oXJrsRJ&t)FD4f#oKor?KUUzmd+@(Q`P6|GS;eRLjt&F$7di-mD0 zZ6qL%?`w)SZlUi26zf$^9jmj4E*V@JD|EI4-+sgk3gRc-8y`ZJcHDNYO51k9%o9%p zhi{D2UY_iur~4KW#pxS3XE}mvsS0Tz0^x@hy1w(X|2ry(!rq?|g8~%@^&JSC18Rda&nVQORuGw_92=>6x^C>_)nxFS}!_u1ewpf6;R4Uo6f; z!{7TYjP=$AoPr`QaIsyZoxrX4Sq*k7ePo4Q!=2oPEsaHY_L#p-BxcS|CW%7)>EERT zPNyu>Rq1#fc8|+zZzkO11{7fH+6~vfjZNiEe~4=pR&E@7Y=8<0CVg;Bqtlvb9+NGx zrV5C~M48y6XKWTo(0SZwH@OZy-9simB#pp{>Pyr*_9PGx=4 zE=PmzVPBY|U_L`Ht+wWEu$$s+TigLQw}r}r^k=;1ZMdR)uZeSD%Lm6id*<^w5}gA+ z4R8DClJ}2(FYy1rWbQr8-W%6{RatlhvDyEtlybR9hP&YxgE&f^YAwncj*msJJm{Q5 zhP+ik%3C?*UjuJEjQsG@DP<;oBE+%E&#b_e>X0A*D6!eYpxjR!aDMdkUO`L_w^TRv z!P`WGMu%5s=2Rd1mzMP*>QqZnPQEocXE@T!gz&h&d_dSZqcl)XJoOC7Kt1Z5*ZS(y z0EGi^XvKza6(sH{zjnBHO$zguz-XE;eWGOBk@`zqtrI!N)(x90V@h+6)8Hra^q3vu zpTZBk!yX!@$gMfU(TBj=hXSL=T8C6}^K!@}#S@j{#AtE=2|bS{vG?_<_r+~(j474 z7QRi6bB0j?CRS0mzgZnD8LuvT5T#kXK{Ojr9iDCo6y2IP>hW*; zx#%5s@iG7O9uvaoZ-LP>LrcqQqgEH!;^&L|bM*mE;WqrGDc`;WKF_xB&j&^`cQ2=w zgF}NQd-_5wO_X^v?qRWqO$g!*vDjnB-T^_XPj0WKPoyPR4BVYz9OZ`oGqr+n7*>(s z7jT{^*P%}+b_!s>efs2J0&usCtwUbezc z_kRz{e%vfb>nb16bF@4GvoSfgdg6M|1rHGyvWF%hdMk}<#+ZfhfBjp9|C{OGmGM4~ zo>J^A9|&=To%m4?^NTR?{sqfrrON#i&GZSqbuRTXeVWe?n=iZrIJ>P+0&s?hn^?u7 zO$g1Q#`g9%d=77n>LL~7#g99wbzJCJX7w)U7v@I(4U^#SAdQ$_Ef-Trb}!69mBY|l4F0RcbRrTs?C$h)C2shg@34=?;h0zx!sPX{Tl;6$c{{Z|gKhUQpje+Q}eYUL4RJ;0m)c$WN+cy@Dh^Ce+s;**}w;@J}5 z3Q9GNHA?v^f$t#sdn4iQz#r~gzOts&`1aJjyOLVHP*P3Kuo0ZYBb;ge--UDDCR(|M zK7j{c15}5PKQhS9A|K1iF&`*U3S5uVs{ER~oT*a;YmC`Mf7Hf<*@D| zvb)KJtNk(l6Z!A5{&4;%a8Zuser8!6UCj?K-#zKjO}_Mmv!Y_aCVP~Bf%xnH?zF=1 z*_`wRwBV$cb1W=}u*vxNF0E0k9hh2iUvl|?-1R9x!^hx%m(G*RanHXsAqXylU~2yC zvMzO7_7=7?sUnxl8Yq**i1Eo>)@Y{0iqpUUe~)X;6Q^S1ObC)+=o1f{mt%a5O+0s@ zM+sIgImZn^JW6iJu~|6Hno=?y}srwE=?ZE+H0^;K3 z_BEyITWD1K3=M6wdt`Rcb5D}MO267qt(*44>Cmry-+?Ap3)B3a>pk=<{k8$qg_t6r z6Hn&-7$_2z(r6#q*HZTV?{RbcogvI_;27`HC!WSG$E-9p*-~qv#?PH1zLaqjn{&C`JE;+2sLpTNA}y0Hr@RA#evO9= z0b=Uj50_&;oZVwzqkCv$(SgF*yMl^={A`UpD_-95*>{Le_DqaG3Mc@6%wwas)6y?> zEiK^4^SK3X)DgvOLa5=pIDyTPP+l!6)iA!DMV`H~QR|w$Ap55<)F~4JeYaQ{0X zXYu8F3y0pJKCr$Yn@>1cxVHd6M>SQH$z zlYf#Sol(aE=3AwP6ta$}h2n}4wM3McCU995BhvyFMrz(Fmh-F9?|m}@S7OMr0v3+M z&&IUIiO8ozgtyJQ$^04D=w8{%GUGc1d6+OUfR4+ucX029XPS_4_kVILTGqbFQ_w%= z6H07|>vbf_BUf`h7Td8dz|Q(yiTH`y1@X>xKKq_c`tb#Oxe;ac$&-+ z_+BwnqZQto&fuJV7}>5e0=w?lAEfO8&_b?}o@2=~TYMv|ByzZ;L+ENH zZ+W>|%j0>T*&Q!{p>fQ-p74fgmz?~(dW@E*n8KbEaaF%Dnt1lg{XV3-rPn8u&x0vM#X-l3} z-z)Tm*#|itllJgoCJ4hbwZ?5;g3Dv&nD~(4>&*mDnUmNBSlYIOO`Al66c=m)m+MNF z*pZ#mPt-#OSD`PTa?OJ&rUhv^e*F*rAc?a(Ra7E5V3|T8>UR^O>uc?NjWAaflStoU z0E_KwR{1){S4&S2kmG-?A~JN5G*yCW0>ux?9}LXXdiXZb;tav!VjP4R)=a|~T-j|v z*UKSLUErq>!}Asw$%!p#71{`*h!R?ESC`)^=PG=MHLQPRU)vBJ`O5dtJ^}-ZRn|{G zu4#_aK4s58^xAM2g&{!7`^FB@tYy|J$56g7Jwaej*tO(Huh*89o#{-W(@)zK&%by?9p5G&U5Y`;2fzfk) zQ?*BCF6_J|suWxZ8YI0o2borSxC^`y?+o?C&&9Y%*rEm9tzgaC(24x{U&ibdasy|$ zOhEql>#BRnzGz*+_c?HNTtb^oc>c!+ouv7tB|DdWmFU-Bq3t4X)3$MG^3$?M3)0c0 zalfIEr=$i|9jJ|kLE1?9s0a3td{BGd&*uDOk@0e@>=vkenxs>)@fpT#X%4T#O)XC_ z1FECe{K3SY>?F_L=+|b2PR@G@L&Vkd+FdPUOfsQG7SBS+?-MX2DGw{locnyuEt1X) zn>tNzLb>L_lXsD`Voj;!n*Em8MZ+F#V4NbxnLmFU`-{*@Ma_lNwJWSdGFyf`W$Rh` zL2m|{TgE%TB;~0=8;Q4#C(jwyv7H6%m+=7}clCEwQMw_+4ZMK@fw zyp)_Y*KD`%O(lE*HdGmwtJYjHIktG-_3>Qu)AnsN=~OZ;zTKmOpTKZzm##mPv;*R; z-h`u^0b}KFJ9wVue~w3Oq@n1;wGPXVHSWTc#0pp7TJ)-2e&FpRPeY`z=YguMe=J8S z9%(TVoDdkGd6aTBc+>+Pxn-j`*mFCZbG+@gy~-Tb zs#R1ziT!=ev4~vjr#awTH;ZOr6WODD(wL~8B!05OE7rgA>=L&sh!gY#AvsCjS0c3* z!DzRnE3wRU`PE!s%qCG31NI4{;#NZuD<-hzxJvwps>dk-4ec*n0tH8F0+b#9Hqu@C zM^Fb@dtSS3wF;BY-NcxczD7#sFfYvTS^DNSu_6|C+M^P+75(eW73i!ip((h^lNu|pb)87`Re9wR z<4%64awh3HFyYVyh8QOX1Re4^^J#DMiWa!DBrYrwqL%aFF&kePBG0F*HJ9VFcP$Nj z{ngAQMi!x%b{9T?%-St|X-&Z{tbTW`W87_FIPv3yT~ERYd+l9Szc!js-VK|7z<1`3 zD6a=b3w>tLtL)#@jP*}Ro=c+r^OHQ+AzdFX9@K1`KBVP$biSLf7>S|8t$Vp+;U_xd zZq}Ide@wah^=nJS;-zP4vn> z6G!d+Yt>kdTLaa!ys4T9(xb6@l0Z97O#5>kdVjEUO)sFVvug3?zQhvtI+LY&q2!W0mq;m?Z& zD2;ClZ}_<_`=fURLbo$j(jG=DUfOU#F~(EoDs0%m!_BO(l=kqQt;`Ur%rQt@)Q}y*|=@jnOA9?&CL%~ zf~h;nqb*}b2J!tozy29a7sdhSGxRq~ovE{sI%vlVoGJzy(`i2DAODBkp9mSGLQB3t zJHF*p{2@hzgv%4AxR$H+w%e;n_X&a-lACXkAHbRATj0!!+oznfp<{_?bY->GwsarH zf5@S|7#TTWyU|-6ys4C79dvNQfh z^*7FJowqPBLgV?3FJV zz-r}sX^VbyOjWkFl7h2G(x>+dy<2X6^ff%7VA4|iwIcv+)4!4g&fEna>P=AHXNh^` zh(YS^{1YqU}F7+-!N$DtSIi&+K@|c(obXl>AUsE3&*#S z9$nHtd@NI$=jDobtSd?&SSXHQWotaCR{MVq0c=U|+(#lyp zNHn7a;{5XJ*ykdN#?04w9%mjCXU9pKOz~^Aqp7=Vf8U|BCX%-Evtdcz#eau{OSwO6 zxMdicDCd%A3+WSs6jz!o8&@$RQ;7SNk!F&7@jP1i6f@H^Y?=|cD7Xui-_A5+MZ^fR zv~L=k5|%`UkE;!RN0eyBbXY7~-|Z51vo`MGs9L&xm9{Wd@PXN9ZmBc;-sn+zr>q2! z_}{{hJojVEPNw8~E87i}Cxkzk<-svZiZjHPR5$-USol&e< zk6^#0{dZQfB7*WF&J+qNai`=Oa? zb#~rTjns8$D&tNE-H|ZsQ=w)feCX!fsKd;h?3HEF4#;jl*#CEMi33f!>kKL|QLh?( zFzWVF55;DM!YV$x3yu5A*ohvPJaE?$h#&Wu|+os}HD{hEviJPN?3y z%h|04aK)8A2D`dh?KbBY2q2gFGv7-^aij$Flk9h0J;JjFlcQPn6 z*@$IcuJXI9a`vlnV@3q=^h-)!7oPEb`J&9@X5(_#!>iSAnI;JA)$YVD@6TtTDG2)N z0?F_FHBv)dtfy{Cp9>&pjAUp=5c~ph4Q-WKW#z zB@@0z6GKs4XKiyO8DfAkB#|)xTseoSNN*m!*h{VSpQKnWHYby|4+={+tWE3VCP*TD z&W>%f;fFzer91yFZ<5F|#r9ZN(dJ2Q!rzhxXksO!&nAP6g2N}-73nu6I&`tJlQ6Cq zolQVbA&+P$*bZbz%#b3lEmHH14QtoiP-{*`7AWbspHs`Qr$#}4Vyf@J@lPAeW=Ny* zUT&6rpX_ev-{ikg8*ahO+SY=C%p4O|Dm5U>4+xJCy5Mo*kmVT7Xu6_uar*meb;e#I zX;|9Ivi2kl%pQl=%MP0lmb7u{;Hi`^Y{ACNNd)i_v_|FS0#iFV(YL0$^p{fqtXNesJ_wx znd?`WNjA+(y%SJH6GgTm~QQ+IjG7&i@mS;WzWes0HRjoW^kvgWQqu2px0OCpqU z!XIJ~Ym-Y5JFy-x_@wIq%Xk%j=id`nupqg?9yX@qU{6Z6xw6DN;a=pfsZ%r>1}VH< zD(2_48YKS};MfO{g2DciN9QHu*nh1kDLf3V?2yPRHED9+-{as!yHuz({<(bGHlmR4 ztekFAddM9JUFzs3X=ll+n7-PJ(@T>!hkNVLqoZ^dOzhOTva&t{Vj2A6BUEd0d$3rPazM97Wy2&W}Hz64}fKTeeNa8N7l zJm7-39-7+7+)gx^3A2zAe1|T+mK1VpmaR&t-G6p91|Yk zL7H=MZBOiRM0HSHv`=;Q9{<63t*^2ZT_QJN?`LcFgwG=-`~NIFNu&p%Gs5>5jty&= zuVR$^b?s^P?qk3Hj-s45_GB@n%Dg;CyauWgv>C{JC#z8j!gUTd&2d}Xm#fVv} zq3Ol7n8!@)`sBh|9oZETc{}?6TkE&%)+g%QFx{_32VAA*=0B}rRYuTKL7NCuy{3qf zf_Nk8vOqLIn4CQjm>6>Kx{fJ$tI8jh&5277!Hzm#>}VR=(eOR-w09Hs26_x5|F?XK z(A6ZrgMh+@aPsSX6kU_NwSPAv$KIr^4rkm}_e;AZxO3%s6lyXz6xS`8a7?!^!b~@& zr=NS^mMOPqZ)fjhA7sM=leACek6FAHj1?$jpfi2HUtM?!jCsxJ{HrJPRnD}%s(KsY z%pf32{alQKeIDA^F0KrFV=KYFQsg7wNkdBgPj3Jnpee{?=47OsQl;V%hK{Ytg+>~1 zUQ!&(T-kb#iM%gf2#I#4{v2+nfqFWZ4(74jQ&EYbO=Vv~Hs; z>q(b)(o%Lm=+BfpuyqzT!`sB*Y<@*S2C27KsUNabw<(NV1`5=|9TS)IsWqv!BkKFf zle6>s!1odgpj<Rg?tCTaQjDT6fne{L-Qv6Urrvd zHJVV2Nr$x=iEh-I;4kDyR_$ETRc3v2QBFu4OC;jOfpI_l%e7 zQ|n*UwgfCl*?U=)y{gRhE>6MFOCm_Ltg7bo?3StH8#|kuPp4MRb8C@EW)sVKp$;gz zYYFXpcwyfY3;QMoX*m&R?5fM)zA=(fmP|NiG@>|H3g^#qU+40biGe@q9Wludt>1v( z%hbDs=RS+H*=ExA+dxm)7M&GcI>tKH>{z65rvEFc8C>Q~Urr1mnScD@1Lq9DA$tL8 z9VqM@k+Huwt`|tMAbyO%PBNFq)&yytqK-eGZIOfnSNh7AE^qPLw!I^+>T{^CFzm1O zg>Opd+3U$%KMU>BlMY%V+}(=nA6tVjPIJg>z^{HK@gOPeejx?e8tPyuUH2|`3_&1^P$XGO-j(aX7?5#o3Z zP(}Gn4>pmLeB9931oCjZH()IF?Yu#MFxjvmYystBcd5QQW%>u}l55SM#?2)JhoVau z>31Kayuj3g(?YqImJX55A0|Bouo3(c8)YZ@9J;g)2lX8qWyK)UCzwma4%%93tNXwW zdSPZFiRBVdD!QUPvS9O^eG&fj$3Yt#URZ}Vp-yx~tu*^o{uf05wkTDpTa=z25ser{ z^~lTyyT><8GQ)NNq!B-seSZm1_f#5xOgyq^z5ldYmE&#F)`OH*MIKt4 zc1{u0F3N+)0j%;(-1&VQzpw>{Nt{Z?=m(7F5@bLU)AiqBd1ch7unNRbP z3lM>%h>s1-w>OhIxv`TUih!)wS!Nk?bm(Bw=oRe808Tq zCkWAfyDh{$@n-nx4TQQs*#)Ftrim?P8E7W&3IZfTcrr8_F+b_*4434>T`^ClW+s)7 zt9z;CYGihVN!y=io*t@t?6k+0U42$=yi-5uTU|avfyhqmaG|D*vWi14HrVtQZvZY& zUd-yW?ELN2LHqL|B!zT&`8Jr9+0Tm13avHCSIc$+m#s&+rjZ_*7aed3!du=8yiuwr z|1H_&OTZ_M^+cz#=f2ir)=n(@-82*(*{qX;S5IyrgR78@FeInZkc)p6wftZY1ZRuf zxlmxIly}9IRrroyLd8EyrB+$`pSRb39+?jckT2V-M#?Hk=(AMQTyJ%|!OwD`W<>fS zmcPc4eOR{TvhhJ-RLb?qc~*!2L?=mJ0)E$5QR9doP-U8r#oO|l@(UQgpbcvl0cJxf zBeMB1hVpsSnB%ZLGi8PXzLC-&dV0Ai?p$;F~u<=OpY(iJ>Xc z71R*EH3G1d9w2AJfY4Nd&<`&b31(_QgIKn6QX)leKIhrUya*Kpq8*teR9zH-A9ovX|+Rj!bl5!O1RA+ zy&C89SqS-ukc5uvi9d~T9l&GZ3uw7R(29<5g_lbwFe0^bcmiLWfo*+}WJ^1><06ER z4+CujFA!H8{L?16eF3{{DxrXI{CaTiMh%m;izgE4ss}Kx#oy-J44@Xh}SDGp)A3h1FgXB-CLp;K``R)Tj1O zD@2&KnMR|+wu3C7z6Htx*4|?&bL?DkbC3Bq;IT0hA~fKL;$tv!EFQ>J_tLA$MZf-L z7-miMr9J&^oiYtHl3MfSE4~W0b$46!py=kk6=Y7(@y*Xl5mGQNTo4zii%!ek)xB#Z zQloQvA0=d6Gyqa~#tTnRHRR2lK_YpAp7D*c`Op1pCkW3kR3v%2?vpXi)428vLj(dJ zNH#C?vQ-mfmfwUszhrvD=}&m4PycKTw)x{1#4w);3T!gsO#!7{-+-^ZRMHQ4QpA_)v~_*Xhp7 z*(PzO%IeloCMChCLWEi;%Y$vx!RF$zAozfSBz&2!ZhGB5!`OB+VzV^m09@bn2lO3v zPV!IR&1ty$w(`s~*iLCqmS`rv3y@c#IcuS+oVGiA>n-~t+Cf~924(q7-(${q(Vv+m z^~g*+&=U&|56nchyl$1zsQjz_!p0w=BsT`g1PNpk_j6>E=eFOEzT}+8DrVAhN!# z4`Pl(#ya|OlQBasHjl5Qr>wlHEy$s?U+3P0g04c1(o?NH;X@^br+;tkAALz}x;@kT zIzQw@E+GvaI~wI3H!a*9hk!h2(<-y1Iv1{6ZK@A@6N9(+FML5^i^F#lueffR))$)E zjE}%i4%X5rC3`7FwKT>@pYP++=9?fu_n6rNZ;0=C=>%?X!<5PyCV};si0e6gedCy`pi)X3kp8wHz|E1$` z>4kNZe*M+_rqpE*qeR4Hdr{eedfpS$+$jtrqSUy&WU?vXnc$|{0ti<{0}#dx6y1FG zgp1Trhr<=4*iTm!pbm9l(+*g8@ytVe55b-uop{A>%MhWf)*!+@$zGyKF3y{HMJp-gLO%-}lm9 zPXJ_XV|=uW@ew58MW|vQ2+##O?_H-W(06Nd7YgoVvThA76oQia@cOzv(ae)W)jk>1 z8FB~4=biAKO49hzneE^TN{8P#5r3kiu8QVSGLYU*v!?_OkW8bAIA(+SR7d%AlY3M` zXM9OWi7vIi_|xY%irA`o55y20DgF@_aXH>GY*7DlP2jt_VxE^`ocYw7f_pKfqei-i zUisJfVO2utB_DuC<3_*!T195>EG#*PQ5~Pw5Wobk96Reb^%QL^Ql~naOd$t zZmFkanvNw1FS<=kmn>SVe6G zE$_Swt1Bb)OfWON#iC&a5NaA53fsYG(Ca*y*b3x2`!YIdz(=|Mif1OWMYlXQI1ca| zIt-MRAXN%#EWGLPov65#QOCcwp$iWSyW$653pdlO@Rber3rV59Pj0GLbg=xUiL1s1 zXrFA(z--Mnc4oJWiVEJZ3V0r5sb#_m0Rj<_J^q_ZP)}Xj0j6lptWw83SoRD*`&ZKs1Yv_WsvrpVlqKGocjpmlx26GE1Tx^=ReGHr#=wR+rwcD7S$ElLnt2kq5v>R!Ub_vgM$>5{TK z&~D83=fF_~9i053m6Ld&rN&F)UfW&k8T|vPzb~d|BH~wADGe{~cD#gdh}B!;(>%s%Q@&Oo(bdj9g>rXq5UiqpwC9PHL-MdYt<8;ZTP1wRKK zI;x(2I{{-+#Bco3IlII%=0w1~q{&mmje%6xzI_3?9d#iLcVuH#Z;$ES@^Iby)1IbPPvN^!PWT_#tfm^~v?cAsczZ-9{V|g*-r_ zma}_B!$JuxF$*|U{w?!JqS!0 zVn`xFjLXW}+CB@j3)oMsAq0Mj@XP0K3n8Ci$grY{`>MGCiKFHN4g045?1WiCov`cH zpIOMNHL@dHJHvG;^%{(s{33qPSen4LI%M5oa>FnlXtV*Lip29rKZs_m;EhJ{P@VCS z#813N3Ma2iuQ#M4NXhyOpzath3BEL9;2!O{DxEN|T&I+KCsS@fz!pKk&LlPM4p89L&h%x z;f^1z4Ve$fZR-^I!h7)u4X>tR49kw$4@R#mD(k%tGS^XWGSd*!O1v1#Ojbcz1LxERGJ9rELKRxc8q{iFc_ftisbRI7}#Uz!6 zTtrm56D;QyJeIgN5G$*`c4=0T+!MWf0O)ns!>ANI3YdD&cyhIm*A~>sbr*_1J*>Qo zl*uRxNy=Ez=Iok2um(g93u>;q^NB@N1~d*(<5|6%PV(_b@=q_*nqrP-*QX==4>7;n0tsgZ?u zr#zNJ*5Km9uUk8@gQ&dTfH)*7>>?FK90Q?bBzV#QJ?cmMbd0ohcK~6JWWK?^Ht^nV zCO>>TM0?YT=$zx-8~-I@q{%on zG#-G`E`&~s)(k@;auX@v8tGEp-QA*5&!3-P2RAZ}giV|i#BTn~%-Y$)fj5`QK1w<} zE7vfifLgFIsjbJ9nE|V>9s8m6i;&)Dac7TR%2i?B?8nT>d&qnrdVf>w+3iH7cUPKY z#HDNVtXaO=0q)n#aK3aHR~N_{9D#>8Wz+`8&G&Nk_A^a-rh^+$+CF59q77?R&KRB` z{9#u6*_OfTLDeotsw9JbTV8;|qX{q)qOUn*72h9{BEUq@!`+C!)BwV9x)|*yO~Ho{ z&|+X7p-kvOk(z=Xxf+7w15rQfb7#tfRBb^8We}sH@_F1|kg(#MG+{-O(CUWq#;)>4 z1T&{91aTR^Pg?EXJI;ujllPMOn^FEuQsM)*sD$$kRLdTV`yyh~q|>?(%FdA9y*O6M z?3JV{OIR1$DTHA-h)#y?zj5hTp~>RJCt+6BNefNwR8YyanJM79fbJIAi$8oCR7G5R z=2557@KrtQP1>UGH{LzPC;_$ZWU+1!P}4@6-e^Ro+-v>-e%yFMD|*gmFcPI~M_K8E z2wM#lO6X^kXtC|t?R&Z%6+>h%)FVW8`;)|^9UwNrMQXs=^;+x5+3dhEwPrlxdjk-4 z0gXL~_SvzZu&aFgGaHayr1v=z$~5|F-hMI#Y2 zSOVBlzh$UsAJZgDUec1F-G$dYYSaVk(H%t9ELC!ntWIySVacv=HL-gvvb(TSO} z9XzPzNU(=hDda%LEnBhRQNNPv@!=wR40XuGa`|8%6G4`_M2F`e^XT!{%EAZC+L-l3 z`vBK!ybK&pEbawe5RyaG+;=UNNU(v1K>NXH&B73yT%eyycP7j##5GO)BsvM%6bSDN zpEFVqk$J%4!>f)eBABp7lwCk>R1v%1G^>HyGKf|O@QIV}N~W}SS==`zxlp#Dkhy4z z?E#>zo`YVMM$-*j{ExZ@CGQ z8Fl`JYnohN6mlLGM4xd_W@@4l%(euCrsCcnrXq{x<+&nDIfQr2O|V0}65f$i!lb;{ zv`be@pwGRv6^kXYXIt9VHf_iSo+%uhBb_{q${%}vEz-TMXc8X)5nk%h3&QKG%&m+_ z_G7scqkCy58c?(lN^^+JMRw5(n}ITK_0s1z&(T&X3A0+MQEAj&AwCX}r&?EK$*CZT zuZJX5q|u&mL*qhIz5zWFw=)4cw6F4oKu?br^faDBaW;_M4yzEK)BuwqJHV@G!{@w| z;TJk?Hc@MMJF%*F?hhr@&Vnm7v2q6MxCLlwYLt1-; zRwfk%B|xf7%8Wpm6BQvpX>D}j3`||_&@4fb3 z>srIUuHmQ7786NwEK}UmJ1Pz;Yw|czfQ8B%(S%22tClVgG-1G+uxzt{AvB*rbn%(J z7G?SYY++HTSSyD<@oMH7=*80&CqU;XV1(V^~Gb-Lqa9R@Fkp;xD zyrng1h&1e=PfrO5!ZpbVI9Y?)R*9CDwKLM~SG?apRv-n~Lh|w!#+UYeZ#`JlgDLNi zRHeb%OXhlseWyy{Ub%HIOT63)&|CkwmjloHi_x2S`$uY#`Dke^g8j0wF(9S=(~3_% z1*n1zBtZrXC1$;&2mM{uw)!`tPuHoTd38jY9RL)405145`4y5Wr_02U2rPlba)_0A zVAC9#Vp4atOA&vrd}k}E2h&nYk`l!)B*=0}%6*LLncal>hXi-a#dCUAQjn*$D)O}S zkvcHr1Uf~%YsvF#=khjz{z};$lPE~CEYBgU` z|9_(k#5PLXZ1w3y;f4;&2_0Q_OF2pHo{4i;*tfRB_I9}>D<+CeuMm^lm<_*==wO5M zr^g}`8^w|iz3*IyV4jy^mQ8YMSa&xKmModFWpZGBtsnJ>NIVU^H_TK75Oq_|7XW%O z!T)xQdbNUmYGQSs$Zm8r0OuK`Vas@&tH^GF&%^frVEGhSf@5}|Iv};zTeH8tgzD_O zFXx@T>Mg}?I4u~`mDgMB|+ z4+i5U73s-JOO*DdboP~@3CNLfy0jfD+H1o!?i--gV!vo59jUY|qv{8kixjIE>|0Po zB5Xb{n%QxU=wCKSTtJkY)n7nRhw}j%H`F<^cy#lWVjjXy{2q9489kE;3ci{h3fhFp z@+?Oy5zoq%KYdo=@cftDr^Go;hG0Zb6V&EH&P+Fj;59lc2-W32{Um8q7kluDYU%b& z+1w{MZksmkL*~mY4`a4o8&O)79`oR!veUcUAE(A#y(*AfN=OW@%DZxHsNyz} z?6TB1@q*|t!pjCnqH?`z9_vuxXu5M`+17H9g@}(Z4YS$9k(;81srr5sr@E?CRF2&n z<*iLU$=OucX$DpJ7LiBB+zCdVCo^^r+5v>NJAAymo^#!)DKpYeG|Smnn=VQfN_Mu2 zs^FAc)BTme$tW+z4kU#M>!?M5eYA&@kH{5@CDfB|bt^RQ=%N5oh)=JSR^hH+Eg z2DH~Prr{~HYxixUcqWdtN#V%c?nY|{pF=+Yb`UA;&<&=E;eu(Dvy8@<;pVi|K*3m% z9JL9gAnt%W@^iJ}wn+5)eKb)rC}Ox~yV2vI0aB20)1pw#S{uL(@=pYFZ^>Ow37wc; zU-!rfy=z*cODqe$-3RU40>&HmY1x9AxhvuPQAqrP%kr(MWhJ56JLB-e##Hd!Q~?pN z{J71Zf!#uDG^2+!zNU#%F~RQ_OsjL}(8iw+1~2WSi%18x74Bqr@FuXnN1Xax0hXL~ zV>Q%FJg#s@oYcWWMHo-R zaH>HfDI7e+(APZ;x)oTKxQZR0`(nTt+CG$RfuI!P+NZWLhUfDd@zY< z)TA#=}i>Vf#(vtSS%Y<1cP4XhOqlJ>%i@+ zUc=7FKBKNEDUlVfDhV@}U)o8MZt2xtezHC{Q@vlDPGws>u#Sev40rS3@fPAb#k%^A%!90V4~0tP(`UCciiqA}z)%!;3)l z*J$s$A(R|qE}ZDd9SMKD;-|cx8s3AwjvRrPl#xSp1Ko958RmoUq#dtB74qOl#L(Q| zI-^=Pa%+@(ZUn-wetBF@20J$QH$*JCS^-|wRaDNN4myXvtSk)_qBo+HAS||g+N@(` zzneA41!&EoYcx!XH%}lO?6_9xRg0}(@k7^?eh6%Ky=Mb`4ZTBMW!S6*hJ|W!-qR)*@WUpHLf9FKKEdrig$omz3ZWGPX^@ITponkYidoY_n z)JkH@(TKW-r}D5ad&zTG(iDbcv7nOn2 zglCa8^b3HT8ql>SeO*;nFmNUt73?EFkSf8@qT16qT!iVcsTX?b zqqtxMH~gASYc?vN=5qWr{t(vq(TQ5kHt!}4rXznC6OT6DPV^q8jyxX)jU#0 zP?HPzgWS1Pq5zY@_ujpdynS8=HfFU{ZjrmzB$cdZk3y_P_2SNi`oL;u}(!L&Zk_bUq=aDE>1Kz(kd8+rG0^D zLaTDG`^u8$TJtWAl=gQd)B%s|TnDqO3`toqJH(8-uBb&BH>EYAQvl>DLE~9ZOTtV2 zq|>Q3AP8UkF`prBe7{#3Pgbthp`O9>1=d1I%jLgZ50(b|OHZ3zkL4Kxyis4lu<^ zNq9joCqc;LB>!~c`BZb58*TP$O0@&`;nTs?NHRfEl z$Ess#NaeksN)(^gX?3c8c)G5KI5L#z7{~%{pP_Wv3b(KVExm%(y#|XexwCc?SmPGc z(qYxlfxkU=x8SOemh(98%&)W$KSZLx0E1q*EX~6GdLk{m!^Lm%eC(RJs5~mD^)hMRfA#5QvNQm59uk?$B=yDKwltzom;G=tqT_Gjtcxx7)a|;Z5 zSY$+=q!|TmI~@ZvkX58VDx!V`QOxvEyjAl360s-eu38BvnvBBZ8fS`Uv=9HxF`JM| z{iARP=4{Dg-rVj;0&9uQDur{?01qPr4(HSDX{zRCs}p!LXoD%U)WUXNNL)-jA-}d8 zUOWxaWcc*g;*tu{|Lm#)Ttu1dGwR7*9i8s-uIk2``Ir2LZ?>9AOB5qf7s52~Ua>iN z4{6RyRG>w!Y3iR_H>M~IQ$cm0Y2-7%<|BC^iSF&n@jf-pu^-YHO!Q{aOQ^xY-wS(S zMHLjEtBS;}sAo?6^bK^J&e3a{;cQ!hKWX1L9=ceuw$^l|bX8XlvP3Ed7;wd(xf&Fr z2d{{OTsfM=Ae6C7DQr;m1-b^oHBiSgZZ3OE({9E`j{*~yhC>{(eeoYyTu&6T2#3E? zw2FURZ521$Zw|^(j(r?K^9nuHdH20fVH23A0~C>j+2DS$EZp)7G&xl^*rz|a7wh8% z%>0N{C=OQU*QxnnkS2iHM!rqd-EtbCaxi6w)-+nMmdwPHhMRfWzzcZMMUU|URbcB30zqYG=!hMKsz<3yK7rd%5jKA>(=oBBs@gB(^~H44n2B3ME`8gYipT*QuW=ggceyf> ze@)VLqr!MkU+=qytbHiXbZ7a&$gFDuQw++!8Poyr-5 z@F<0q{h1xGzA!Z7r~(KB%!`d=1%8B7Tv^!^kuTd#s!^slJ($P%9A+sIxrnGkcJ_~l z;9OU%ohG^7gS5!X0V9izZc6ehUfv1p%ZIy8SqXt5gQ-In%DS@bDiaDUYCf&#gX_gI zjGg#cxpeZAX{erz5u!nT*VbTI_Fya_KSCqhgYu;8x@d2Ys=1bOo}gUgJW>}1!FFd^ z8N*@|Xo96fMu1>cH(JyYzc_AE049K3&GMLGPa&|aX!g+Zlv#nH2sA|inrAGdw;EwkRNxO2JvBbMVBHhasX06GEDGgl%{~@t@Z`eFvVqsH}jh6D#1A;5r##% z3LiYxhB-$J&+ngGsNT5xSo=Cbs+fbphJ(Q%SK#6=D~2J}%BM-b)v0wJca)efAFbKv zy|E7_LZ>#PQFIuyAr1V9db7k`UFi&dFYUhWsqQr?z#0Kyf}{*7d=3C7$>7q4ZpD&| z^DnPVADyJ8%)DxRE@Y-rC&GjsPgGsq0C5^%m*Gu{9(M`m!&w z4kgVuKBag%$g@XV$!Bm&lu0>hVQhf|NOst_{1Y`_Ge3)oVDj7g{@2K*$rsSn#_f?u zyR!YsEB+QtB0YjmhgDy9TU<=`YU5DIiw2(eYV;n>(?j=A_G(hlW+fZclOMI!`Qs!? zLJQLZV)B!AsNSYx&{xFQQFUol~F+$m;JE4;~>5yl7sJc0Cy4bxcv)1PR%4`U1k; zRx4buNivk2f1?mEs3~Z6d=q9AWB9@@d&9Pwf{%zciZOh6EtI$5NyqpOCiDWhxv`hX zLi4(aMs7VKEo9wv;Gj?MQh5z2NZl9IKWQ8zm?a@|pd+e}BMo>52+?EcBA50t7~oV? z&X(Qb`5sCiQ&{&nbz;AgMNs_j$+aOSl}&O8%dy^HL?z2p7#oSULbn4m$=%De5QK>$CB)1$tAQzxFOo!qB!el4NgQBH`Y2A$Am)q23Vj{GSly9wL)0+3sL|8lu{aR`F zqZ|qpUGov~lClzX1P!^T?#~2u?p|<*!E^EqsFxjct~vmfHT?KR4FZ%H%LH)VX|ayA z6WZ7S*P}I=e;`AlnR)gZ7}$%#V82#88PGh< z2wVqlY;v}dox%vW(5W#eJV8s)cJKN{tjHh4t$FOPf~>oTnwm;M9+K=z^IaAWrEYH482o#KzvZifC!V0|uh-HKs4 z5ddFg-=u$YTV}bsv1UjOmcgzL$N@E5v^E`P!}QIfx^S|;jX?|Y(5#*@Url1SuT~!{ zuY&dP>txQ%%~f9wyd!|aSy&XBa~g1q7Jw>+;_(X7s7-5pIWkj0WcCNI`$2-o2qxc* z6^ASqHzC7ii?8^y*tu(QsIwBit`6$8>{k%a%9tR`Y?Q2ZvRKa0>&qXqDEve;AYU|M zn;-wfG1!OW8g1EBs=C1NpbUY2;rb^?8|hV$r7sW9og(_*(&=w=pGAaJz}d9)WH4|z zn<(%a4R^abTf_z_jH089b~Uq$^nid9vYf*NHh<$LY&CI%CT5|t&;)pCPvqyVXfn9R z#AJDDEn2%&gc;~cab)V6=RE{w*^CYTKB6a*{lmbl+hGE~6jV!BN*$6#&>JF#T14$x z9(9_3dr%!9%;C8eAFevscR3Z)rppst=PFONY`R!D*pj@g&8QKC#&Tu@X2`bxmiqoUaQ9y0=3v9xIFSjmf zPF9ij?VE$qvV1UZZWkTNK7&^X7ms>9@!m(@9oTZG`!rGKG;zHxj|`wT(J1T^(Y3aL zP}+AZnFcQw`=uQ=(<$w6)>Y>;F^wooFRcO#rXg3wwoGI8QE5#OOFg;LN>{TKI+$jw zolLM7?Q5;5o|m+}6AwB`mBd*P4=RzMY@>MRG7xDn^)dCUVldQuC~Hm1?d`;diE2CtThGhByRy4xq28hz+om z8y|_l^JzeNq4VSo<;9|gv3Jh$$Ztj(6Ivkfc1@dXS&sI#P)K5Om`6 zri*+?9rhG+oe%;2xA7@7xqH?*p4Ilq1XS`VA@)iW_AR(~2I9qTktxY=<2_A&>eM^z z#xzF?Nzqh~%C_wA-ssq;-`S_p z*=OR9tqZVxO3Vy$wcV1;ukJPhodzvpLt2o+cxY*3P!2Dcp&@OPmt7*FPLAlISt)y` zsdum;x}>G5h_WU>b0}6fp<_%O*MG-r2-LLS94M~=PUVh}9+9OQIMpjJyu#g;qwnBd zL|nnDpQ+v*n6nVfKXzJPe!3FlVm|GO0DsIVkl7TX*$L0*_tT`)lP1d6bjP&{FI}w~ zBJ2Jr1A4SFM57%8u2+SG)TF_v9e%Qc{D=UYKVSAHbl^<_fj1EjnDwiu)Z5s6ci<%O zEThGFZmkoC?M}(rYI3LH`;8V#;^zw4ZD|QPmN`zT`GgocY+6ha&xTmq%VgT5CP@!U zetlAHNtiyFUuDK8NZSGX6`fLR*qd3zp5K13c^v2jaOkfZAAR=vAW41tF*}Z_GlNZZ zQwtC^>=Lp?oExaaF7leJ2aA6=2Bx{9se3>XR)-eiz1b`9oaK}YYW8OThmv1>v7U2l z#E&_b{`g3N+cNOfo)D$1{!z~4;x)&2hBf}43ysLa)laoiJct-Fp?qhPS+@|?+8;bprg709m4_xys+fTkg`F!BeTwb z*`_7@paFeFM%78;Su4x@;AVEhalO&W2$J$Xf;TmArJ(alygCDBo+>t%o$C9T+kmz` zjY2nN9#d8nZlaNOY2LhUV%^w&o)!mDtC%xTrVGdv@7ZhRk_7rJKr8fZJjHW>r7B{m znxc=OWV{jxkjrNVAnhZoY2h(qBOG<5?|>pKftoja&~_omo9#}FWcGyyD9Qz072<^| zga`)`!@=&-wo2p`cb3To-nLTWurNN5neZV1vpmQCU3geq+UPR=nB4zk91yMf>!bD1Aps7CP3YiSp;VKaHfobam$R@o^V#Ndse zVW`%r>`nDOPaFF$HkGUif814NJs5etR%UM_%{W%Ac-Ux{tHd_DuqlneU-`DY{4VH2 z*}oyU0No$clJ}DgXTmIKp@W4HOG2ZkG0NAs;kuEbf@f92>s(AK6hfmb#$dwLjSDu} z66}SoAKi)0_sS3l)J*!^(=pz2YM|YDYx+P5>svDtC|2w^92&m6l~g?`p!3uM)x{^2 z+=_YWw!ns~udDlg49a-#4^O{3ec-e&K(81V+h8x8)AT{fV^3GgZG}IBb@qif&j%Y1 z$SVRTtV>CiB*kxd)lJRAKv!BHwd+Ngn>4|Y3K`tb~4;~9OU4D`Cyl#f)W|-_iqGKD~e(10bg3^uS|H# z^?JF+IL8Tl#Pm?{rnH`jop;-~_edfG`r+q7L zeu{b_^#&mDiqY4+@a|i?J-LT@J0vU7Ybz3}8nP4X6N}961alojQ_dKDOY8(J-9eGN zptpuZ7!t8wx12FG9vS#gOKXQsxL)9Ho1N-R3;6BXZ=LzoOvhmB-LB|Ke;M}6o-P4;p;Vt9Y1=H}8B-1_2KWn12k0n{%t ztGu&M7F~9j*_Fz}+A!f%;x+r;2RM=DJA3jXJ<2n+RL9S?yKGsZPj3wyO>u=Zc#82lM zw({b|?PDZkO37j~1|9|2HoyaY)?p_HMf7?7WOcpr$*TPypRC$)9GtvN`OB}Dx?mw% zhMCAK$1Bkht)TTZT@v3>qP~VJ7!1~k=pFwSOW=F|dPns;ESBkx8< z;AIj|1X8#FX*)FlvK`d9i~oB z@idr4QhftS1a8Xk(gz9yj;rD2@uS#)3{1#CLhs=1#bzORN`pZPj0XD>QiUVqR#7p- z9!RSY`vGVx>uX;82jIy`XIh*SniZHIe@K!8r#u6Z$I^yyiNx};0G$eSN=&bqE`oOw z_sPpTqfQl6S4y=GD*_yNj9d_CFm;V!dmOyme+*sQ%mKnx6U$noM^clk_8E68giXyc z2Y^t`u;8I?u!uf-at5?Lr^RsRJbFlz?Wv5GFD0#InZl>q-{r-&3x zr@p=Tvb~+6(moyVXjlv$ckr@HoJDY((;(|*_Wg_wZ1f%+F?QjY14MJ06aCe)DF|Ln zpA4+sRu(Qp5T@?sJ}0Yfr4|v$4}H}X9fW0c5x$HrngBcfoz-H;7VnKUL8&JJla9MX zgbRD-XbquLq=&Bgb$_2XVIY1625)vj6;b_1$kfmfk!34?R6HYHsqAJOfU+S)Bn8a0 zOxeXC21%ukUV&gk=d-bP{J8m6DamT;O?^U;r%XcuqlYGD<&v(FTvyp;0uChZGKQPk zzf+6?{#BvSH9Mpf)P4FP&&MoRd!OY?WUFb0?d*#%KFOgMfd0V4Jr^{gBn{I)cWAo40pM2_23d z__Q|rqnv@8)hnR9_)P8xrpcf&>e2i2Z_TksYD}SAgXD>ZAjv>!2l1hA#IPKlvWB>C zIn!U*_m;Nloc0C zo4GOy=p5pYkbfF$FutcHy27zE1)LvKP6+Ks2z6OeRs(1Z8@%9UvrRup#0bc zR`R+~S)vHyD!V}hRML|hR2qPD@ttJitie>lY8_#PIQ%@8X4rOoVcCHE{oku~56gth znp^`lhOnZd?R zn1fTHA&L8lm}B&~bjAC$>gXlD!{#7%AS)sc)9s6k4f;%$T#1d>P3b6^3fXaZJ4(4S zR|`&2^zubnCFJGETNMjijYtK`g_(fYbSn-d_MkF;92M}s#IF=cUN7wXnRN)+^1@#3 z4^wk@>6&Sn|E;>(&?NH^<*CX?#*eRtKksN{629-!x<+_MZ4C60zV$z5`6#j5F}14w zDd+^}2Yo%Gv{niL`LUIlLb0X+c=~zMlrOFZ$et|T7?Of0Xh+S9c6E&T1u3bH6f^)y zXQN5`Af~jyj&~WU!zl<<{4T!L8+XNGhPcN`Sfo(nlU+b!mE>rL0`})*-i{` z@MGV~zQD}>6_ihZDC?@i=cVHl2nU}4&n9XGE7D8ob*#Q5bRf)z_9hLzs&)P>)p~%e zV5#eDphYKrKQVY4b`SOrL8m~T2iEkObLElIx!Wkh(Lc+fT)GA}E(1c(MOM=0%*9s=t9*h&&A2a-qAPWt@75N|4WtIi#?vs-O9sKM^ zN~dMCNNkW`-bkWBA`e0tpDKzUQZTk5CDvK(9W=7Yo3J&x;Ko(iQK@Ukr4Gu(yY|_EdMiF? z|7nJQp@rUN(On>Srw|Cu;Ak5j8u>U|$28%%@xQ3UOV0x$b3RtppehzZJJ6DSmK{d+Q3-PU;ArAG1B;-m&OkRf?e3v}zbA<%yBs19@WHlYhDF z*aI{?*->pnT>^epd{&g3TbEYDejG=`{0%%(vdvC)hr7>uSC0xati+X0+R+T}YQMp{ zSB>KWdV+#c>`e3D^NE614Tr+eSLJr)Keoex3wXO~_e(pS zc$I2HKq)B*|2DEzPN&wMjhRsv<=MFw;Bz8I4y^SFc>Mgu&^J-x3($!Xi9(b7>O!+u z!Owqecg4=cL|0M}-+>0gPM5gU-`xqzm9dHQp|NU5M_SA9HiE+JqnTgL`32sX1 zWv8jvvbn1a$K3+5WJ~qua}E`Y^OMx!SR37&FvpSl7Mx^A!J0Wxu54KSZ)H}5EqZZN zn$%k$JSybYUAvAi?5#drq`{}XA17GsXN9{g1@{O3;jU_wQx{%wIy#=(#g4FU`Kaan za{gq9K)jcahgVJ=9<+*7PM`g+Y2x96&V;zLGc(0bH4b(bf~=zsYj@$BJraUCQDZ=` z)To(vdD@Cht$u??mg#1Dn&%fFl=Z#oxCLLp3tjmt0T9V>ie+Pdf77K%7_I%Fvdkw6 zUY6To+Vn_TAl6e-<3bMqO|e3}K`oR8w5^FiR<=Ep z{3$-$oI9hCaQ;N3kh>Olr=!i@sH1n3TQ-DsiU}7|gH-U*ooYiLb2QF~-)f_lD%+?B zdL0Mgtsx67uNwVvCT8#OA^ugx7w!Pcr1R%+e)J(Agns{LvQKOn)F}3jo2d_I0vCY_ zO1+h(wy*Qf6}dCzmkvjbY*kXgv#sK672Nu>_bV5&`)eisSqVXNH;VIiB$vRvo0sX| z{jE$+|2<#2fO4L^x4Cz!!nSnap+c8+D)k08BMB4%io(h!Mj~hL=^vwPJ++Dt6x{kP zqr|vUX2MHJ|Ks&fDAQK)$hnUCkNmBq&n^Ur`ha}jMvkUoY-S?$>4#1HN2qiAC@)02oNA<%k178&d;QU+ zY8CO)+$c&S>__;BP}Dr8(1%cV8_q0u8_F{GRNC6>L{G{M3_!t07V$gn~bA>5LTT);rd>@P5Eh(s74_ zBG;OfJ>KHoy*vEy^@Fsb_f&bWtG9`L3c|q~TK_0Jv>^6xew8qerE~Oi{+=M9`;G9` zGawGlpl{mY8Ird*^~P-vp-Iz?;B&2^GiV4kz5VLKwgYyno34oI7Hf%T*u!pEj-n zB83*}nSE4gNfxD}kGwFTRtkTCw!cBfDQfJo-hPKpZg%eNI71I=eI`?jxV*u~%5WmRzZYq@D_MG#>mEA(%< zG72G{Al9!6{dcDtJoJ`N@#I9$Q)jcUjJqi(mUc|tUbr^XRFG|HqV1(&9%Q$A0u^eV zu`!PXc^c|SE74vnK}4wIQiw$C4Ma#%-r*1bt2va&vu(@NcmCU2+5Sq&@zjFC8-_;N zX#*R>y;+^xR)ASdS=HKi&eiyva!xe1e}k3LlHir(4l>rN&KZ*O2O9om6`UTbZHobkQeyd+D!c=x%VHAr!-2Rp<1V|N54YdN)6%s=` z@H6-JC-STP8Q$`v>={LQ&yMQ#(--fk+@;3PmR4LytyDU%6uZAA!wqOU_r>h9q2a7k zH4YFBLuK{gqS?P+K>|BbKkvic6IB#YV_arA=@6vKoZ7;mVn|uND()_bn?WLkH>YM@0}W1AQfQ${nKh1 zp>#QWUsog4{QtdA-nb|b#&-NY9uOSV-$p_Q4jc^dzp(Fn_9X1;`hMDnag*nOpDv6c zoH_IMwdv1Zc||FzqNi4k%QJxz8KvQG+mxZ7!wdE2o-g<=c?JgjF#KH%72j<>pHhzm z>AcA$@w!8wz3uQ^=M=9Am5PXTvhU*Gb!AdI);a+U2(MG3ZuaEH8@DmYam(XmMq}?( zp8~!>sf3MRtP=F(EA=6z=bT!;TnaDx{B7p`zpPvZZ1Q3#&e{m{9X`Ny%{`52d{~#9 zA(&29^zVU;yw!$7yWif4mmmKv5aTxYABRc}t40(y1ls3C~%8BDsByv)y$&2&X?h4~uaFnv8emnb7a)TT+++ zDRAf|1F4{9*8pmEDD`pHRmPA>6vd%*V69W4_~%($K~y%USU0stp$2U69dKCdj{pCN zQ67d1x_q%5D_J=CpMl5~RaxSS9Pr|vx1<8&SPPr0wLVzU$<>Qr!n~z_`^`toO`=EF z$&#l2593M(YQ-SLCuy0qjQ;YEATN`!k zQ7dpI^%mUUX6p>w^OkI@1af(oU6~I*TC!-gt9xX<^S+i8SZCg;H!pN(ivy| zdn+2i-XjXBwPcQ+{Tr#+ElP20BU6>(D=~&$m3opUbERalPzuBqzp(_JOUo?B+cTaBgR8`9nD{Cwf3(Fd*9=)&5#1_!Fx}-qDZ{ru|>&c#_)Cy z(8)DD`DXD(dD4<*FZ8Cn0p3f;1d|YVjH_Ged-RM1Hn>7RhOX9sWd9%(~z3K;)a*o$cBg?>aFIrjzbOZ`$%kwz0=!Tu+@O_yDKV z_d>p<)#ANd1O5L4+pT5h3%d8#q!%BEw8$rrSDapzHUWt zIX%HNRBPMMwRlTV{IV07+gz%+7QN|?#k3>>U@R|vP5LpGDEwW?HI-%l=zfeaQ1 z-da@bxcbcPJjqwaIdjxCFDf4Ad&3cd4re4ZKHLbo<{RkS(f7wfrPMf5D-6NxeOFZ5 zF<`(^&UR6WaGG#P12-oYF@>R6vU$5-(apLbdatU5bX$Vv!kdhd+eJJ)fq zLeo92FXne6H0oW^MW!PwG6eH5*)9*rDnCJ9KinCgJ>l}=>s;42KhrZR;Q|r&6Rj_J zGq)S0K41c zxG4iuL4d}dffcM5odv;Fl%V&B0(8XUQ@!toW`{f`0RQvJHwu?c? zonE(6jXKhO*>+Qsn0eMW8WkFYpD}Q{+_` zffR|}p>rz#{O5y1p1-dtxck$_FRH$>y|nX%=U3IlSBoVH7|Hcwt6^ksF+s2>v8XhD zdH-|YlG5f9sRc(`UYuu&KkK1;HYH;5IAtMO*YZ%Om5PeGKnL8J+&M%Amjt!B{3XII z+n%P|79T%0tF1R8nGJ!u$FMRw$@}B=>}l|JIbsZ^#fP` z^wJuY{tL#9?`iNYSkJR(#%>?dn(UPWZ$4!I|cpb_8}g6#xi6)Bo{%@zIOGUO;Iu2J=X28p5SQ4LOu#^= z;&)<@q4#<_*Q(r|rv;s&z-%rTxMWV`8!C>;?yeKun|qI6f}Ymw6fB)w+VP`{4TT?( zL;juSRl5kdHvaW9jg$cXhW{8X<+9Iqz#*08Q)5O1|I&5GP`l9dA1Xf{^iMnrL57|S z3k_)U!|48RzT+a)+0v%*+T4u%*$dg?2(|nQE(;#q)n4_lU7BQi=+aKx^qyCM2T^6? zQK{rBu#jJE4UT4etEg;4Wp_>X>)2d2?4-qCnb@tOvZ-+U*T{%=eWJ|OqqDt&n{b&8 zy@1izeeze=ml9Du8A@#Ym>2>?MkTJ^KMnlFzx2tKWLGKVcHZ+?IgIhDu@ARl+C{Hq zUGUd`(R$}bS^cs9#Nqdz?f#1g@qMP7YTPGF;A^5EvEa>1>V8Rm@`|B~s}u)P+Nk)3 z98;9uc)d(p?$4^}%S#s;Rgw?k1hXA~_26LDuU7x85iI(HtPPklQmRtGIDMRvWHG|Lf%L~)(V67J;9L7nsb*1>h z=Yd>S0-p0cmQ=C*uNprn?tp(&`koxlx~o{<)?XqrIMq1)DHsYT4;-A3)-7GDG2>I~ zy3Q9c&3NaFxunFt`e6|-N`LCmUtX`fYFuBkHMR2^69=f;pru_{+a`=d90WCNDlZpXz{okwwbW2MeAm0)H)B zpEvk2(5Bvoh*dqvg=f$cwsya|Mh##RIwQ0X;zxpiZ?97Si@O?{67e}pCs^2?U_2HA znPxr@u|P7PpQF|!$l8)m*EbTDDCqq75W4IuE}KDYBnU2((%1#*+Gad4a4vuSS9vnL zdB_8695_tMd=)rH0_4&HktH9+ zf@8cG)A$KlFZjK`c=6M!r>&DIrZb)$f{Sp}ormWyjt342#~vZ{%D{hmYL1`|HX-kH zoxg$Qk}hYESWJsu_{E_dv1|+1i`wZ-h6L~bi@$vQY#t;$Pi$Sy!H!OzZc2aNqW5&@ zR3m?s!xWnCNz{(ZeicoZeaU4fqi4MEgsTVw2Z7s4qvT}?7fg2kwPV%3hqeyO$alLK ztp#Z>l4=c#zU}v+PBaO(me-1*hzyBed8dVh)wr!2e{|LpNT!TDFo2|d%PPH!N%+;p^ z1M|(ff9M({Fg0#{89|p?aF<}M>Ac2UhQq8UI@C}}Lq@~LqM^dKVB5CdP)OTWru!7* zReRMqp^$lBXYtsYiv`$Lslm4Zuh9MrUa2lbLy1EjMUZoOl=Ih%O)EDT*JCSa*c|I7 z{ttAH1vV&*Cb$$#C`cbZ*+jk}AZ-)HCUl#eddF0`5&A)w;1ee0ZmOn z#R^WOjyW*hrvKOOpwC7&8)*)8zw7tZ?Db#W{!DQ9W2pzs6JsTWBdB%%I8f2yW$ zW*UoYg1b~m@mu)d%*nQ7k0eNKh1^Bb|2YJZVuxQA%>}PlAmRUZR&#z1t7gL>XYwWr z$Y1rgSK)04UzkpZ7!L~<;M}+gPiNwlcYcZM*Kpw`u#2agu$SSH(-sLj-42o+iv29c%1vwDIr>K-ZJD?cxo^7; zgs9lgYZP!#`! z+zTdi!<%qppz>gPWhK8ssw)p-N_Fd}S%>AULTU|(zhq_JDR7)=#%Xh5GHUBI$Arr9 zL#IgJKVuF~Mp`>CQbwlF6~j&ym2XIf_}*#+xI&qQ3%k4QRa7*N8k}r;ap9Az$$RP- z)Q?IZQHrF;>T@5(xWAtbUy3cHW@v~_U@}R%)V&-nuo(J~8#raFoslDwaUQ?zVp9U5 zKRF|RuW89j?-5$6ZEo0dakiETX-gmI3nh3(H%H*>mZtl}PNa&68>1&0yIw9y62+l| z)5)WhpjQ@oK6p+Ho$3^v1&5pRuodzJ8&6N2#t>C=JW#37(( z5ZYM{+aeABaM`LVaQZ<_FDm5K2blnCwE}D-MSip3f_+)2`a&Z{34gb#^}arm9XMSNXqkByajyED~X#WzDV zH`BlOPn`0G1H{(PM6r5LG#nNNk4u>+l5guJw&qdETybE=hHvW^%sOhh?WJ6=3zA4?j0)-^ zX^hKoo#BdYBQqe@a()&`b^xsejC+s2?umOPD|_eO2l!O zM=6@(9JP?F;F`?5{5gjE3~rz0NCeKhiTY|*#>|mNe$n_~8Qs=>!4%wod#ytCb-mYP6>wUSiIpW@s4kT=A z+lH5E%~h2zSQSU=K{BIkl%cy-bHZ(V@qI6oDML_J`sqb(!>Q z8agpCeY${z`rE65_PhlI@7_v;tS#E`DiP9) zj%{W`N#>o_v27bv>1c_h-f~*h#te?_dG?Y`LrW5=7`bQHDBG6GcL{nW1Luwe!{%LB zqA0&J{8MIpS6e!wo~u5xR6sRqK{p|K$Xz;+=Xn!{UVDb}Kjj(B(vv-mTN?xo>EuhN z8@3UT5cFJ7LycSpl0;~uVseDD4`E`{OmM!SiJFLakika1TJsstyQA&d+@0iC)iS0= zRL@9JJx@I$mkYznn0b*>bTBh2bAvNoU=kmMx6sD#fHSyK;_VuXk-AYYiy5z^dZvgs zJkh8DH@0pX9@MEHo)Cmg{@<=)aANEU8*12c1lRcjmTnAeFdz(}BT@g#Swik#wAj^n z1eYHoY5FImEluYgdk`C)D{#%8(KlEx$L_PC?+*zhv2B$G!$p-iCOw$kg_d@QT^f5? zy^G7PI4se?$Cn>$qu_(Xa}c=h#Cqm)U$0y8S?eZ5HJ4G_M@EA~6Q9=oYEEtV7mntS zH8#@NsFtgL0vf!}+W?_TJ;?~YUjVT7?7uC~>^TJm^klq#;li-U*NeF!toRD4;&}@W zclKQypu?+&UJ|Y4tgQ5Dd2H%DzcML;t;xhAk^(YJ>DImG-E;%36}c@M02vMdtxey&r*c_|MBtghYr!T!ngu zy4rC=Wc9&`oA(fU!BadI3EXPt1QE-GM`5|=`d?9LXlmQbxQrlaD_M(ztr!h&dMJ+d zi`M4GXu`EQa|U%?)VU&wII6Xq&0F&f)CZf{f95K9J}XGx+DSI=JYUUp|1fTlwW+D= z{8fh~L_RYNoiL9UL1=P^&2rG(d`i3g%*I3J6lRXY+yo1`9?oUixu+>XZOt&*8 z;9g=I5lwy(85<;t+8Mz$L~x>r*qJ_>7s|HY%5E2SW~^XC7=4;LRzTH{KW-=4Ze}Bg zUl=7@b&4x+QqK%#ki;PL@+da1*Jd?BUi&iZBnkh`KFwoR`ZaeYQ-7R*>YdJAJ&s~CId&Y>6Q=pm`MQbnPZP=a6hEs}AY#7yBGB3!411}n@|MJ8_Ljc+VO2=*;Pmr?eBedI#P*IZp=LMC&@-Dw)O5eucYKDsB`qmQ8F68ggv~@Q?bh-}~ zSoC^PXi+I2hFKM3W$EBvxv6gWMo$4l zd;bg!{Rs_yrqNae(aoE?|44AwUN6y-x9-6~F3A|~QX^CN9u)N1;0`O@88cWU9 zd07B#FBr>~+|vb%8f}~iEJ}turD`=}f{%Aw$O{f@W)~1{sL!^nJq!EO`}~`DiK*^G z-*7CcXdZt^Z&Kl@hM4`rbF_?On0s}}9(wOIaq@S#`_+SDT3hnQq_k$gN_nUkQaB(t zy8uuoMEdff-*;l=#U)cGA$!F{SQM@>Ee6SL$E0QPPh3C`f$ z8Y+s%a(UYmy6_{rKp_FR*_=PKx9@XijeiMd$Lf9E(gS(m{yf>EQrvh{9JAHI?PK15 z>@KnKHv=wi>PaYM$0jSzHpl}JpS*Gw@bDM5?`9M8Q()Td8%3|2qhnq8?i9+4xO3 zeZ!}A;a})IYyDkOtPwUqo2wQP7@TkoD~e&1Z}TBn{nCeGTZ1M>Hg^+)uDq*#FnU7VtYsT9{DQNay;#gCP) zQk2HCj`L7eevG#d4ME^~;U-&dF|wzZoGUNifbw-NC%5_PRzvffI@ z!UXH#`7d`^4mz*ACKCh5%aAbGEwQeJx$Mz{KVm}D$$ z$5fc%M0Vu5kubwIAtR4SXl&mfT!QJ1NXtn(lEWE_i0EBdvDsFD1NIb^S~)ZV119Va z;8FlaK4L5_PYuD{J`bZh^Z;wiIy$7M3H#T9AV20nRu@87I)LDvNZ4bj?;1Tcs0eoC z^DM5SW^dF$?iOkt`#vapv^jDr0{b^-_k`tbp25)|%4ti44AYN)oG5QOAK$|7={#@+ z#Nd=rMKx?9su7UfTBjt@?$BAGTcA7bKql$KH^ zEt!Y!KTw-I#Kfd3LBK*6du>+OP7*a@e1+&QiS;o7ilrB;{@o|4d3HBZW~pz6c1gJV~VfZRF@l+Q_bhU%|!gaU(+9#;>eDX!(buq|KR!leIekRB8vbf za_Hb*Kcny;0{D=MjsIo~{Bh|W8??%@G)Z$9p@=5E)U!ghYIeoJD14W<5nf+VsgPP|?wrkm0>BV!kSx3(oL z20LX*pVg(jTum-5`77t;AK81n#hw~|>Z)QzF43DTTY7QN-I2NJ_js2jn#jt|+g=T-m`P1cwD zt08LMo2#@p#{XMbPGqfmz17b%D>&r!$cqb~R}h9XNeOiRAL}xAbp7aMb%^ysV~1ZG ziSOj%apceSE}l_0cd&6E?je#!CS-)S`ldlA<=_*vA0-@Xky1^5a_dno8*iFmcyiwM zqVQM~lX8^w7gFPh@_==0*2iOhEQS+A!<|anrnx~HSPgwO-**kSCM$`_waG~m8X{r} z|0j>?pTt;_qJTu4dTlNaVq=KpRbY#EM?}*e2;eo>5Uf@?^}PHmZ|NUJ_)(Jf4Kum* zSR+l$wSJ{n2B|4+nXxg>XMij-X6T}c8C}xaM={Q8Wb9a;^XM}ehT}YZz!TE%!Noc_ z984RDvdiI^Urnz5XmY3#-4>xk9kA?TU(q}Ud#I$}qqNJVH z4q>OBY}33q=6`bN{!#o3*=_{hYnKzMcZgh1)vIz+ zg5lYD+mpikE$Ey`A4@tf(ukJ+h<^T~N$V6&7&H0W=Y;A?HNgQ{xbV46OPIBBG9%2+ zm=?NcqSX6i5pOdS+wZMz9(-8oyQ{1Hjp7DvKIvray4deXoU zO69#>KYL>4I0hd$L#N8FV&@q`nE6I1M%W%>A|tHUD7Pz28F)8m!AAqmjP3}cMx3Hf zDu0(?H}x)-Z>4ufk`ZQfZX}uN`FQO4M^o%5Tl6+4xzz(t6i}1RHAPURx8J;CXWOLI zku!Gf#VI3UcCMV(&RO4&JqcfXZs@nMGl*o9^h$-MRVhHJ)}rdIkRKL)=bn_wm8B;Y z1aoEf)wF*_SvZ=0H0@PM2fIxz3EVc6-Of;bHBO2@HM1Fp6Ao{t$}_M-hR^%eudIb0 zst-XNssr-L$Z3?VAax|v&az`~jH~%yxt;$+mstbs`W(BR5g;X0CD{zxg0PbhxkFlPsKQOu;zRV&$2%3T*R0G88$N z;hfZZ4q}}bPBx=c!c}JU7k_3IyU>p${n%0Pv8x8wE~6~IMnB{}{!B;p$C3~D*6(2)^%Khfo;LLf z<^MGq{}Yw}w`849g5|&EA$$@n|1BT$lVJHISUy=S|4}&f$zu6$rDmUu#Q#>v_sK5* zZxw-`T%rG3k@!ild=f059R9x-Ab+yUKiTDzxN6I6ukWuy#16^@hPbSG!*=&O#V;m z^OO4g|Eu~`m8rBt)24*UgjkSF4%%I|Af5WV?TH?Lmg=M*T4Z%CVkrc&Vp?$v)U&*8 zaoBY)nH;qcOg8+<$)HRdpB5!qt{Iv}#m)_xHrazTYe*AZA{r`2N~C|5&sYit+VY_+ z(F~nn6to769}g@U@^KHW{G%!6PGLL%cQLb}RG7!t&^c>0i3a$g#WSt|)Ww+;N^DI5 zMuYsbiYE;t)oL!dMo*s{1Il1crsu-Jn2sQ_=}PdlWybHY!6z4cS@yEzc61$0}`T_g&PlDrfp=~|DMsXJ*6zo03$A={Ry+O z95cPR!Io=bstn9J0~&_9rN6iu;hEn3XV=iEC@E}r)~zVYAGpZ+3m0_(Qa#ctO=22# zDB;ib{!Ud4&7Tz_J?-ol;0$YLH4K*pz9Ja4_Q$1U+eKZbfl6vloiPl#=y+M^FCmm!aNHl z0ijMaY#`vVarQ7-B-siK`8r?-)4?6A4!DGdgkyDuh)R?YKg2O@-$vx3lbdS)=U7ub zi~b;UzDS@s-e5aXX)5!|dIo@8l-zYB)byp{X~mi}%Yizam_4LV5Qvi-|5*NP1c|7! zRYo6`Na|kds_=UC*dnf$b-{hBiH+=J)_Kqxmlz=V(dvKBx6cw~i=DmTs$kWt=>6ao zgAcdhR2QnDx;;PLmjcIwvJNVa%F_{#n>FrgjAYUm;W9R2YIHaKSO85-yo=jRDaxF;w_-zqFo{RQ@#p@+PT_mQT?{lVnpg1YCQ< z9x!eMHjjdF{935vXB%R@Fji!X13MAd)q$Pte#;$*D`-gPo)F5$j)5NP;lu2B4&Q0)d4^0i$O(<>k&H~7P{#($SUvj+D>!#KN0tgrz23)b{7&& zyEd#zx8f&oiDbwUB~jK*nw}4m4l)`HWGs8r8<^!_Fe|r=Xn+xwgO^D zD7ZQg!R{l@OfR^`jRLty+O4=xc@B&Ru2il+dKBj{-9U}6J{Yb{X!n1u$%0t*;j z(|$4lG-S>I6{Kg=P|+87w`!lwW=I{4$8jDF!?9Y_q5p2w1UMxqEu; z)d-LDw#j?oh#n#BaZ1#hcGv1C!r0*AFt!eBA>^|BG(5hhA&o*UmeDZ`zQMpw)$^A` zeLXM=2)eq<1Sy%V`%On^ETBGMYudL<2_o;X(Rq+TMT%#z4KXFXFewP^c+DJ{(xGJy zseYcK@S1Hxb*r^Tvbwr7NQt6uIJY0rDfEdwRw~3>6ZU~YUtBXZ$hc`7s8wt4gUR_e5x>8{mDSM&XiQ$$ zo6F|HO;q(0uDY6r;mtY~T?Lm+RS?rCgJa}=|j@I??PwQDbK^*AE| z@_O)Vv;%0`eQf*y#z{rT0P1MIkfZ?QgNU7aYf5M7kg*EIVlW6!{VawFO82_Z9`1nJ z%|Q4RE5_wq%LOa}dGEs_b+HkWnYs)ezEIXQm32e(Xr+*7*AD9OvJmN|S0mtKv}gY5 zU^}kksb&WFZ(5Hf!pUm0Fp2Iwbf!ZEDb0~uJl9uQUazh0G)|=v_TY6}eLm5G%$D%W z;+ZKvw>M1ed)EdvUERJ5rtWnfUiKN}IV)v#unmUq6Fr`XHV3=u+zMz^YY;#C^URaQ zxe2p|0LJ#5vgCY6%9IeD!l8N>JE$&2=IdmvrGTT|tI{3H#gPpo$kXm7(;bu1gLt>c}7pS@1$+GNLGGV4niWVy9RKF zcnY*@u1WhIm{dU@b^yDRw4B?WRSnb%)t$Zr$j(aY-79?KvhMVZM^C(*C8Zx#V4O~)jHT%kl*k(P$1 z6bqP~S&5T22oojM^5znNp<*_ZsMDIuQ`ENn?f`&G4B~|jm#6G;ivbJSLSxz*65SjG z`j-JmixcRC3fTnHNtCs^;5~TVQR4uO7UB2xt)GW#P!zmV;3p%dUO(WQj*qv<-Ld6_r zI18NJ3|oxj*zSZ|psnD>G_bWuV}8Hn%9u_q-PskPZpoyPW$*J~*CK#*49*}KCduRG zWJ`o)X=!S5@#ILPyeiH*-wmm1qG%Fz_F;*;o0H*K(w54b)e5Nmczk zQ}s{`-nHyGf+Uw3M?Le9E$au8B&|Riq_X{ zb3kaM@~uVXQ|hV?+nflgaAgZ(`zu^c|uh;gjLYR9N+gZ6;e z4Pu4berYNW-CLblmdxx{HNC@kF20I-)eEIgBqkaVG`N{xz`W;eg;NayP*GbL{E zA($Cm9qczd#rIR%})c&iE+wua{2T0iP z z04S51JrF-!8<$3zNw7IQ4>G-Ypm45n{p|4}wC+7?yDaJr)GpRIHLT0UovgAzAyw@! z42I2hPF#e$gj;5vkFVRJlz?ek7sQMFBITuGAGV{6*#Hga!wI~^M5UAh<^t^IPt^wq zk0_@*&!VWu;HLp}Rz-QWR(6Z^G(w3}Iq#YWx!Q95ZeTtz3g|rS@2`JjQEjSvtaH0I zw*@vP3b5FjDz{j(_DBG6yK{h>r82xX4=&NVvZ{HL0|y5T#lzVHYOS~ClEKOjntF0r zpro0YZU?h{_6KxWC_{SV61#KA+Tl7oApF{jLslx>y0RiW;(+0}6ylrIa_(;6lbwgV zOF`$vL)ImRWQiWE7A!K)A+N|MdhB>&vXiR8h68x>TEW2UP2@F~FoFiTuGA{}fnkoCKu2z)$T z?WAzYs)RsBAGPj3?&Vab`qz3e5U0vBkt&!m1dPq)=*#UxO``3q|a1WoPTp zWNx13s)vrr%?0SEc}ypski9IR1fuIc&vm-1!(sm_{I}70=k;1JEGIaSa1eA zL&Bn#X;U3*ieG{Kt*I!-O%LpH{u+ZH=Y_3Y0I6A>!rI)n|3G^BqznkmM@_x8{7G+} zWpXHEM1d6NkStBeiUu2<_I!Ten53_4?1{RXG{vxoOE_Ne@x*~5oRX%4N^~`-5E`LX9hxo<>z|>c)o&F_dQp<$wPCf;_5U_gZlI%I_(^Hh$W}7t77RIHe^-kb} z6t!tjDcJp?8&%~>IZS8NHam0}|1Jm$ILhuuXA}9+EIqKT`vC#D7P#}Q&@Q&ps9x!o zimxqhw{{zd!wtTU(??30pq7H@>2%qC-HPwEIC)7rD@Z0`+`(V!M1ZW%G+64K{jjP@ z9Xor)VzrCITmc?lw_HCSn7wNO&>hE&b>B9LoL7eyKKNb8(3)7qWAbVgqF*A&UkSu1 zW+Y4J4O?tg2DAJbPMMN^Y6w(A>ksdHfhs5vwXwK;5^dZa4MT{B#CYU2MEf&vRe2n3e+@PuvMi&Bdxk69p9!r%ku7W!tJkKoQAI{ zZkinznHeB^9mi>_b8ikZ*$QQ1HBr&1*|a}-)PGW19B zt5R6E9Rz$X@>B3-Qz-#77hKRgeo+9isGtyRz5SVYlxdxHJE|K+5H}<;iGYNQ7eG#X zrk`vvzY~~Mxl^yZ6tv(uWW5)P}dzaY*Zk?eyxL4<=;wy`bb&T}@M80TLAS+Rg(Sbt66z=wQ~BrkY?&NDT4l4EB^8aLZ9P^QYa!z0O2 zCdwJnk*|VH`LQ2vnOZt2rVZ**^;-3G@=60IRa|2noLHl(RB1&Wa4oeSA@olSCz=Wc z_5DP-53fVoEC?PaI!^EwUR;@iCr#l4v}QP8*5a-lY+V^qKrETcL+{q%N?*`JiAPog z`DQRnjvDXjTfP#`p7}7GvzYd;p#8ptmf%e8?sh+QT~M@fq_}Q8IgNBDRXwxbfP{3$ zmyij)TSSddgNp@J!6hs?mnBj(T~3gAOf0^d4}&s7V;g-5KmK`Ne-H>1VucM7`@ygud` zHNR5p#Pky*Sz_Et(io+T*Kr&vl}nHcTI4ACw2CS|8-hut}D>1`39{aB6DY&x*Z}cKQ>>;pn*L z7`X>^_2X)CVOpw)w*XEo!43|_Cvt3MFFNC;i9)DANT6sR3@bs5`9YUj@j}4VS(*bR zj(6Qow)q7;MhuVh%)Vk4s(Kb)CZjaPxc+-bmz&K#J@pDhhaqd%|)?nSkes1?{2 zz4F)Phq(s#tz5dlZU1)WGrRr_71D#FGkZdtIyt2$!eI z$_QpBb0vmQGW$z6iA%B~@)zeI$pUo&3Z>NK8o0D5Vi4B&fyJCqmWgMiNELNTf;nvIWF66j2P+l}SE|y;ZB^hH5&8;e>+YL^XFf z2Wp|SwTj8Wnn#hx5*ndVH0sn5H}l_hAz|o-n>l4pzi|Y-CtSKBF_)$+ZS2sFjxe;} zV&vj6r1t$VDM}Z@#L%4TusX?wj19_Hii3o)k>Uzt-$l*>*n^RXT;r zcax}^u>quJk3><^;_m}vw*x`m&DC0V&_%V+I*vvI9@)*6dplyda`O&;%ld%`M2=GA zI3B63k40(DO7Y{wWP-+;Cne3^7T`Rn`2|A$B#x}1(#}0o);@);uC`25=)TW!m^0+~0b%c^>&jk$-JJ51ln#2l4Fo>iXP6`vn!ekNdv5LKw z7Aj$jPf2m7#kk*8nz|yT`UHWL!igXeI*u1`;2$bOtK!(B26(p z^oQcGXGZxgGiRgwCp8$`q?8es;(e7v5B{NvvQCR2SkN27eSA#T!z%5NsUxGu9FwTC zT^uC2K*?eFpTo4`rTK6=TV!pUIx>wb=njiPw-U5t*95DO$z$C2$fOx=4NX(2vb{i* zM?6ram#C~BYNC&7`?2@SRS}P+L5H;+Swnh$ji4pza;S;)(kjr1M~afGQAvln9-c`9 z%<>9D`})P&GQe3{;J5q)l-H4dH_Bv$mJ#BFBg3k*ACcKVMTyGlQwJ>DZnrW;yC=oL zT?}KQc+ovE8-+|pQZ3(XHFUN_#TS$IrZK;ECpnT(L=VcM(pbbp^1+$f2Bt*|Zxt7P zgB#3XFwl>{q7Nf8vMNWWe zdq)+u+HTO(G$@t1lP8^+-rd?Q4O2T>a5$ojfH0XC_jqDoqG0R+`JgBA&J@kklaSOB zeWg42mKhiQS}gyyfoWQw+ZL{QnAP!=Wa}gqHwCQ>MdqxCj*=^eMU{W3!|gY%R|AYc__1DPNeixwT~5g6c5#YG{!x>>?FG`; zJz%4r7E0Y@^H%9&XsqnX?`zBh-nN{%}$<;d~ffN%cal=Tlt0hzP1%l-h)BF^|5WyV#17A7%vX)fVar6ijPm?%sw*durWDM2AyF{nFVA5q>cmaLbPHjcMGU=Mg9~JY zh01bR;^2O+x{Nn6s=(G!1^2V8yij*as$x5)DtU(Pbe@>UDPTT_)5TodXCf*~k;Cei z|JDH0Th6gos$>*Q;4o zDIqWm+*uZ!H~HJ_mS!8i*zFCpKs}#(-UK zxsTF(vlV6@kd7@;#tr5T%%d5-VHk8?xaDa&mFzQKtrF82$&{+sky@O5^cvT68<#0C z$OK<4P_(TqfZb0`bdd&Y4N8NF&%03uY4^HKFBfbaot}ROlLq^4nMmJ=GW0NO1I)+IZs4af>KkFP629QD_K0yX z!y5B$Z~ihT_1@Fyy>DZDW9rK79uUF1&;GL2aI-P8@SVl>0aCnP>*Vkg@^d5n_dg(g z-9vx0DSK{?m)GjKczP9EY%gehOnkr@`p(<%9(V1=x|7ww+=G7WO?%+yxQmpQCeHlP zoxxD-qm_{huTFE*!$W!RGBGfXoh(^Vc6dw4*(JWom zRv^AoA|R3iQ@Z=hvQWj!#uAb_(sS$?_+Aoow5XdDKkE^F^xW=zF$#-BI8bx=Ik*|w zKWjLU}Oss&VwX!uE!> zIU!~GTaQ-_HAjnY?vv! z@c>Q2wl|Ipz|1xu>#2WTX??B7@I?zry<#WN`bMf_klJ*Qt>WAFXXPf;&)qc$R(#9y zpPexB-WDdp1jX9h(*6@QlpFL=MNY{b*_r!PY>5IiNVb>Sd%@vJmt06!iM zJ_csp7P)z={qfqLNK;D!ETOXz(R^}iZ9s@c#6+d`A}Sq0qT~=|85#vzC3f*a*27{4 zsO@h9ii^v|BD8`VJjX#%HgSLm%X(!QYW`bsEMA5W@; zVv@Uqrx*i|rnVi9-z-ie~rc5 zQyTtItc^o++tzb;PjM{~=2BkwTDMe^C(}_iJ-@kNCg?NY;|+TE;iTCjZNp72w!&`H zW9FYS4Dg}yv2ZFRMjX=v=GMb_KsDC2e$L&)b!r*gg8E*r zV9Xu|W~|x;wt4f`a+mZolUSf6H#p{Jo1tVVbi~MF5Yf^0Tu3FG1i-f1JF|>ioPG*a z;ehiZBF{yzSM;^LEs(w{X3u7@SK#^^@tGQ~A%od7$vbQ2?b8qMz~uHd;<@4)0aJWn z8$;R{rL89}*Tz~3yeag-_le57fwKjP%1K@fgDDAl1*YOnyb=A?F$RYSq37^7E~we> z1w>P-bre2~GlcUeQD9X9MuE1*OMUpOI${+;zf_A%?^gQpz#+O#(U_yXIR8X_0G?W% zuhqO4#f(%kji5b^c*ypuI~?nJJ0eD+*9*STv4%hH?(RksMh^cS6L1=_o7cIvGBzsZgtCT8k@HUik z(y@Z$B?#r>;DnVe9iH-NOPX`1u%Y}n-1y=aq!;|>8v~Os-^U|Qu%E|9hiXli(YAs( z$c58d(?ZLO1GIsKB_L2^SfC5pf+(+w9^DRtM{eNqv~6d@yRd^`(5})OL_Wa`;hZT+dl|9k#Lvg0IDQo8%flK8lMm*Pnaqb zyrmLw17h`i$eqRuY$M`NX+Xlq#gRzdhxU+aw{M%s{rG;^q(%*!avG%xL$n5UeQL~rRvXG7akXhgqy}G$sKv``6f@e2epD?-K-4CR zqFctya;sxC0)Gv0K&xR6z-CeRJ{!C=ApyZ=zJQiL6^Gt^7sOB6SlUx#EiWW+TL`JyejT? z3A%%vaBb*S@z8$>z^;_Z9Dr)|6kID8-87MKtI#_%x^9&KWe@! za7pK2LZz(^*s?9F?&5w6X`_s>BkJ>&k~v!xG2hu{i-MSkgS|K;-iL_G?<6>*IKzAbtb>(St1-~Ys{xTY-KP~EdC_t7e~tJ5{~i7cD2;a3SOHoIoouu8t| z41co9^v71aOZTK9!|xs&%kztChOgTU2$UNbRs@bIhJN7_NmVdr>qcizC+&cf9eCoxp*$Le8`sne|PQq11 z*Sp+(+6@s-?KSk0=(`dlh3ON=OvU%(CXY0bI^}AjDL?((h{-w4KDA{eMr6BS!1#LY zjp%%TjM<**pF$ow-O#|SLbmyb+{ZhLw_dMLvMqiwdbSt-R?O5tru{-rcF#qoD5$Sr&2q3Enos=iur?>SK{CEef>I}7w1ALi5!;{U*88gX9TJ63IKNJ(^5}WqT&ARMRdG;h41i}ct z;|E|TIm=JM2(3{Dr_HazTCgdbsw#=aUvXy%H-on{v0Sv@G-M5k-9yevg16kQ52P$n zmgK_Dom)~DDQ)6spI=Qg?33;Fe^3-r?KG3KnpWmqyitbqr7{2Ay_9l1b^C#8h*KBUzL5yMp?|*-lD$9i4$_KuEAzh zxBBVvGYa=?yULLC{)cl$DQuIp?qR>=UkD!b#+Ldu=`z*sEXy^Th%3XyblkE zf6{r+7`-ggp)TAfLvfc@27fs}-pgPb#V^nksrKMk^#0i>wH>xjm11p7cEJ`K_n0 z$8ry?+p8y|DG;*9Ik+get-NUOd!wM&HCF~dL#(a!i+ou8HNxr~@2gRp6t&F}=2v^Z zN8=~Fx^~C6vFbFfm9KXAXt1r4O4G=5`-iVeRWmo7`?r6@l?o=1bX>ls*G?%&7#;$+y%53=Qp%D)awla83Y zocW5jB>%{cUIDTqx5V@0rI(AZG%vOK%EZj*=)H#*Jch2nG`iv#HF1LV!220m`NXNH z@5#O3=D{LUqSKPoMMd7*7Jcp7>u|bd&&!Kfl9#Ui`e1+e+RBJ)2}=#WE;cv+*(0Lr z@pl#eUa*l99BW-Dm%FQpwl{d;xC7x)?e+#?2`@+?p#XMh+c7yiaye5D3 zDwmz-0*bERcluJzy>5J?&13Z3_o36yqlIr^zi7tE>)d_z z#LLD8l43pJK4Ag=Df*fH&a@YsN=sE9FrU2x)BbOvJMKBBn69-5jJ*WEW@;9@-^g>u z|BCmu;DFM@qdAAwo=uJoSVpqNWw-5uFvVqmr%RSYua|Z#3wrBnQ#OQuCk*I!>0Q}Z z7=Ef@^>vzY{Tk=0F9up`GRLirJQbm@p0wP0Ot7d!e|YU4_9n@;?B)K%;nz(|ocC0` zKfR9odQrnH;has`ct=Ci@T05i*Jj<&m`8%qHP&7^=(N4-{{)Q<#Zp+e+s#sP+eL`sA$X2p_%-pKts1r&_$pT^*2-A(aQ)nX$ptAj<-1-H4Ddc=z;6RhHpI0`wdWjJ}sH}6iM(nI@#l8O~9N* zKIKx~P}?!09>Os+%26BHJQIir3+UQ*RhcAQ3IEEV1@=b0wf2E6%lxza1Flz%<}1HI zH6z1sqF3kzmHrfYCpxL@^MPXnkh$+WhUL%AzvbNCGM86uipi7m7OW2WZ0V`k%yDNm zYHo7^;lW1uz3&cwmN_MuExds@3H?&+t=ZDk3ix^ z%olrIPTY0zR$Ni=cgw5wkzjX6oVQ}1ZB*<|*(7{=*;@Tm&tmA0*H_HsrH;cDs$X>` zG!P-po7ST*-aBMcebECis)6j7_sfaz&OP7o{q4eees`jtxtX{vxijDB_;)T+(XQzJ zo11yJ=?i_s4MM-Tzm@hKBnUjCidTFHlN-u@II-PDv$*f+FDGT2qiH@18$QqqmtJk| zxJ8>y@by55YBu^V>i!Tm@0F0k2Fs2>_s<^oEQ$g?O<|c3Z?(~Y~GUV1@lhIidzv2Uw-K8)Lj4W{pQWdvs=HJ zI@yHDauAR*972%OrOTT~hWviq9c`3-#6$`|Vq*P4*nNjCetA2pd1Um;rr;wcW3%gf zmwdjG;CqhWe+|9z=x4(dd-X~`54$|i+*Vs?R^*A$Te$)Am#Yyr{Nc`l9ryf9DJA;a zGX30R`;AWy%U)e|m~#?O>ZxRRJL--R%r%q5r_3)mtoZh}%6$}@C|?%h#@RV1X^fTi zT(7>o{??794@?e1ZxR!;@$+)U1G(R6SM5o@-*xz*$wP-D0OH=-wqcuH?KR+eXxZV?^Np1fE}9#)$DCMJl=g0G@yd{F zzn-nEgJHG!b=k^zCqO~52MQ!F-kY_rIr-&%Il>z2 zUH{qmXj<9z%cOqen%;fLN`^7&9tU9hSVYN{CtlnI~2I)ixv?r_I8RvLgx&I5d z5*y_*!mNv6mbtkW>VXT-t?A*l+z>MgHcGo|N^>{WlG-*(ZU21ZT@`Ufa|*Qld={^P{RW@dv$O_7d3zc*}>(W9}e5=kDFIfKp#4R9)4!G_w-sgMi{{UD1u%-Y2 literal 0 HcmV?d00001 diff --git a/website/content/en/docs/v0.3/tutorials/_index.md b/website/content/en/docs/v0.3/tutorials/_index.md new file mode 100644 index 000000000..1b9f89903 --- /dev/null +++ b/website/content/en/docs/v0.3/tutorials/_index.md @@ -0,0 +1,5 @@ +--- +title: Tutorials +description: Guided ClusterLink tutorials +weight: 40 +--- \ No newline at end of file diff --git a/website/content/en/docs/v0.3/tutorials/bookinfo/bookinfo.png b/website/content/en/docs/v0.3/tutorials/bookinfo/bookinfo.png new file mode 100644 index 0000000000000000000000000000000000000000..d360870b52dae93da89d79ef832dab061a81590a GIT binary patch literal 173819 zcmeFa2~?A3`ZsJ}XUfdK75a{y+R9cdY5{?|Acie=m{vmsEQ=^>Ekz0u5JK2P+P_n2 ztB6JdvZhu7MI;f~Odv^Htf?%Ch9w9jsbUgCAQ1w@?C<@AV1?5E%zMsv&i9?~m~(op zKF@RC*ZsTJd%3Rr>qCM2*ZtSV|5~$V%{s(^e+I8v^IwQHYb@sf@-p}rFEnEW{P7Yx zc>h1vkl8j;@RvU&ydUuXnl;VEujr2b8T|dPCl4G&uUWJ62k5_--u*OUY|R=Y3GvVO zL$hP1__(t!60A^VIQs2Nqu;H&YZ!`^Qg+kInyp_+{78S`fm32!q^hU=);E78?JZpo zE4ldYpjGeJw!LY=U(%ep?`*mM`kimiz4>+R&^~?R`TzRy<*0A}e1f#^3&X&-|Mm8t z*l;J!D=u7dhFy9fy+U7Ap1@|8=dk6nKX+X?(j~cH-qI=TJSF*2fcsy7&i{vgA8*OR zJ$@L=$TMC{r7IWCz%dfRsbV1&H`AD<@Oe7Q*IF4W5_`&a28-5OJ zJ?@E|euj7xZG1@eQrYRO_(p9p`Gf-!?||rNomHk8(X(iZetcZ78#lhYZt=@~D7#$N zdZU1!XH_sG*Po$e85PDo<9{jzH0-`7gYPdR*5jIW`F!K|<}ZGDL+}I)C{toMRj(M& zd(z}=-35BUlf%@oPF+`Yfn1LJlf`1|zn0kLYPSjVRfeBm$&yFw+`0pz5a8wd2BTX0 z<*YK>cm=lj<{ISFGCQn+*afKP2&l$inYJmzY^hc-|481Xz#dnvRTv5UPT!|GIm>t9 zp*$%6(U*LpQ5HL@Co$C`{(fKetURG6KVH?soI`aMvg? zI$5o5*HuLp2wdQ*QX;OG^P@=eU>2RG$)UE-qI1;hE?t$t_>1KtO+Q3g>7QI=8rgIU zW`;`ev$1AR0!zcl@}+98z-Op@oaX5(oh-vBvXiJMQEv5r3Wg@#bGX3Q6~3*ESk}ww z7fB>@==sn19H~e$Ka0*YZuT}0H$-sK_#-n{CD;aE@9~M%++8ozBLFJygC5R3kwiL+#;Dbv`f=>_ybVN0I7Fy*VGi?K4^OSnsbY$(M1e z{lszouUcHZYCT-rkqK8xW6xOAAr^^MXaEjHU!^~#Fp78u@<4cTy?h*Z@}mN>0(Z;I zv~O?b86Qd%Muz&%Cm4_b*}*cOr}|!k4N26}LdTndl6GG)Ew^gfk%=`hqe?L)EAA zIcg!q`DA^u!3T&=R2462vd0+sT+s!_qphp2bS?{~>jZ2@{2(?ev8+DM12adkWSLtz@mQE`}LhLt{WQYr+6rFc$0XYb+Fs6W4 za(o}#|K0p8d#pOo%J2(6Zv)OP^?b~f;a4kMlB%Y-U{lHss83I34@V`E=;oA06oz9V zf?dXaYW{R@5TsytZaR*tSjH(gg|FsU!p&wU23acxkbNyLFJw65V`oe;!ixn@3HgFND^-!d3pG!8>TK_zg29$yDva8T%mX zYElP$;l%jc1b}IVHdh;0Xyq@!vVH~xyt(u$For)`-CD;c+Z&X`V(Tjq{wb+&T?nz5 zV@2{9&wFU0-ETH*pC|LN+HFh)TYtv5IXQ(PJH=tG`v`)>iSvpS_K;0KV{GwGm@Bq1 zw?1}L>$kz!2TZ3SW^`n`@n12ce}Rap#irt>?mKh-MhUCgMUxa6?wJAq7z>nCmo$1; zcSWI%8ee02`Itz($tS$z`PZVe1cqNZwMBg$U6OK0RGrh5ArR@O36eq&xMq%!a3(lM z{@5OPSkrGfO(Li_`~V3L@xjYtd(FLVS@!Bjy=fDs&n&}gHXH7G5p`q<^xeB0d;OKa zEPnZelnsB_}1m%^@ij6AROr!(cvrb!=rfMz*DSYkr?1cm+F;GPP&O7#1BCH2} zAjeK>A7E~~6q2=5501x6>kDR5S?dBJ&?_Qy7za4EqMTaXZnKOW8Me=6)yr#(dbIXW z^T$U_M8r9B-jb@Xc$du;ziV!=x`ztxfBsNej`;`=|$$OU;% z;FcZ^xH&iP4Ib~A0G1Zi<{94fy;veCF_Qvwq zbJk_}!H7c3{g%#!jsvnSh+r>*#=H-GW9Tx!?ov(F3Oo!L_ag~Ui6sf`2+}|Y$2z}u2@y~TuxT#%f-NAUvFiX$O&7uf` z-$j-GM`5ikz9cjx$L=lPDVf3M$bXU>&M~RghNolWE@%ebabOiP*YF}^w&=Bvk zSBYUgN~8zU+}7l{{_>fQS3T^6x@|yA3Ha9Asl^8aY*P+W5HN?FQEpz1KV?Xr(U)4> zRpp1rCpmMrbSDsIb=-cl@&4Qsc)sitH)oy#b1b@vaFk|t483%RJD7TLu??CyA6v_7 z)b~iTzZSkDJ6{W#gF8=-lDwH2Eq_|1<6Cw$h*qpqynSaMYp>U|J9~d#z9y8+SpyhBI=$Z(=G;cr<<%dRF;uZu^Odgp+iua zAKs&7HYfUF$Nm`hCIrM){DT(%oPuXu-B;bIaxK7uafIOXsjL8hL`RFv?!1%5dMn#+ zoQzEi9)l#F0*Uhlq)5{b%!=J5XwPkh4qJv6LYS8<5iz>l=TN1pepOg{*x4H^W6xEV z;8S9GFb8+l+o7-@_GZ4li+=n$vM40nsYkx1hMsO?=%P7}q!sW+>p0NJ7}@SA!TvNX zChn6=(Z^vCjqQHSqdWEYu~JUG4cD|Il$_#ysUR=s{Gia1KYG;(%j&GlxaUhuMsBsRWYkuFN zB33u?;PwQIcn2D6IRC3lLwLZyX)FIg+p8O6dM~F;Y`QZmAZYEpZXFg#^h5+|aPZ-# zJ9@g6)wtZ*sAb5;wOkTP5Ia#}@v0k6cdT30s1r&#q~R_Lgd{r0^)27V4E;n(>5>lo zwenorDZu6=_7@%4#OgMG-o>jve+if4vJXtBiWcP zN31>C$sUd!ujpy3o)w_RaOWiXXPbFZDNku31mA_rlwO!Z{Wk@rb}S?}<~h-5m0sGj zZ3~|)w%a+Txf?>O8@#+WrkjfjVF|MO-@h3=Js(4k3W<cP|%f0U=^CCc+64LUq8 zY?#L8Vfx#m3`(?bDvwHrMKcO#jX}8MM4(N9VXO8+n|ZIX^>zgb<zcT{XP1-PnSr;F|nSizZ4qe*G*oU{8ZT$CWIp7iD-a&BKuf<@GBNITns5?a?>3uv4y%09;4ZX3`Z;K8H(P1 zeN?^s=^+F47*JPQ!0K(tQ^iQswP+eSboz4l+-T@4g$}*$wLI7RPvYtTO(81AZLAfN zm-hXf<`~QH#zCMW{5r1E|2H+O4Pdpx9Hk3`MW-i|tZZv&8OaTm*$2|u4_apv2HeYx z*r|%sIA47^$+}ZTwQwk=D~^dcmb|^8x%%L?>|d^Tl}RH)Tw<8DnaI_UySJl|!}Ua! zkzK;_=+d@^IIBFDoCCnXhUGO$fBp3dr%&%$J~nvl7Rl<73q8h>p4o6tPxFee&$imq zP*M|4J8GU_9NL!bS=WyM&VX7djDSs*fELk@ub0GBdM+5sJROc|)mQ0?-f1iP5F?Y# z3(UnPqkY*`9FC!&{;ED1FKt5-d+zrnd-o2uLeWrZpoU6210kBbTqil!eEk4 z<7lA+@2PjcdK5vSG%dU**rAHj<{&7xr7_`SPu@OGEz@IpcHAg00B-fwxvuBTdg8#2 z?a%;J?*)GO4;12;O?P;}akrg3a3iy+ zxBB7rN5F|6)fUw8VqEE4G2d{0e&av+I9)E8N*8%12`@GTVu2iMP$lG zIyn~=+>~z>-2o%o13B$?^r8F~IRuVo9IU*EPYL3G_Pu-rSCJ#-wvkdao9gK`VU68whkoON{5U%&a7%w z+U8!Og_MyULHAi6l3$NGHZ`Xf`JGK{0`~d|pze>qbeH@vz;Ln3Bgqg$F>ydqFXh#s z*I0>oO(% ztE8JL8npH_ZsAZHZ;wQPvdXtr76=9bfdXFDscaNItI(fUGOF<`OJYlI|DgsP_1Jpu zs?A@##czk^^k44&SDcIlCtmeAzen;zc;aMChxf^d5Y0%#+-NU}XX6A+U+eJYcJH%g zYnFUpFOC|_?BzB>ZlRk~=sOpEx zU18JO3o(^omT?lzTyWs`t=3#wFEJ=)z} zWfkrya@N;tzwR#GdoznqDQXoTsn51QO~~_y6pa}1knS6X>Z*?;p}@i#i?`o?$7YaH*nEJku*!6l*-#7poVGd%~9rq~!-=NWeOq0ANU zG(Q+v^bF8CSR0Uj{gqc2ANVpaZ))Yo?WMQMsQ&S16gJt4`~wk z)sdhjLTadeXEr9wxRw*6zJ#w8IV;=y_`8d@AUq4X!qk-!8D)$`RQJfE?ehy%hn{BR zYX`!_Ddo&~%X;;8L~t*OIi}hF)?)LYCz@8tV(P8nGqd3Q$vh_s$t@+oqQq$x10HR~ z-r4c0$NQPrJ5?hsogH*VODEWs?UNT-!T&C_M{s2X*imtOd1ZV-DJ(*qSkVHOdzx`2 zOWQ=H&;stGW<9l-*{TJ{NpeJSujZ-ZI8(mLN9axpcIIck2UEk9j|sDLR5*PW#UF1O zD*m**EIy@Ftv=wfe;}V8RnCfct@57r#~&XUUfvmRD1}vtmBFN9{)(78Jw0NNOVp9ng<|=+VOG?(^Zg`Lc=k?GW5VC$wrCTu0qo*4yPfpSs2?OyqIQ z&htEungw6zlkW=h?a+>Hh#ki&8{MWE`gJ%f1oudMU}?@#@$qtEaRz#KKd+aNlYdos zsLD5H#bn~nXb+rPHoV=!VPNYuhs5e0n?NPCZ+bT;Qckze+ac&>c=*f?%Bl6SBUqU% z_8Y~Q4v8`GHKo)DF{+|d+mj!N)_RhP2UfsxqYmsC#j6e-q3A*=Or>=Jx(tk4U4)ks z1+Hjq+y=^Rs^Alm2VtG!n9xsgb0pe(M$OMEP!gWn9l(>rA&n}{sT7|~lnUQ1_x*4s z)fIBwulJo%gmr@u;%OJd@U1EFcJ-%R;tT4jf(SEDGGS{b71oG|df}uv*)u!V2~F#h z?Pe7_V$ZIGKE%iHbs6Pah!Zn$Yw|UJe0xKmXg8(TCP+yq_(B4aidF2I8SRB{i75y< zPK#a67FV=azmiItUnz5e^G_x?u}38V#oi*KH=*{n6Pns5dC=#I_U>05&gEH93_c-F z0DG)Iez>9U!ER#jc%U*rC2{(9^wC8=;kZs7P=?n$g0~>X4vUp1kC(w}#Cfk^rydUR z{Z7eT2)N1ltb*>;^Ch(|&diT?>IcT+Q_9rUVolX)S?Nl{ac=&C1tsd}mANJg&M~DH zjJXNYs9x_Nbb-u-FHdvV%Qcg!A1ZsO2Q$m2zKNY>g)Ou`#crNtcT~&Z3Z|!D+Yi@# z>5TH89odz(HVi-bp+YX6JDFLQQzO=u0)X9Osci1g>xspxEr`*&on-@ECLl(+6)Vjk zi>4YG5Zj3iQg0>)%Lw}>p1nqr?zN+ieoS|wzbPb4yUd!i`6 zAEXC7cYaMTFn+<~dkGG0AC(Vu-;WmOR45}&fKqRBI~WF1b`ZWc*;xtepWewCsG>MF z@qUMvhj0EoaY(!15p&LZ`4}7R^p7`?qb4>N`>a>&Uj}4;KzEz}@8I%B)vgOc1V@qp zvvr3dH|NY#LEx$0K_p$-Szp-F*{9qsL2V?5O|I}yu)`6XlP~$>N1gf=yCofy$qMZh zK?Yfrv;X%W_ZP?Lw?#75Kw{_z@T1@NNgqtPD!~>|#c34bcLGQlK44vkuWk2NdiATD zawocjqN$8j+=M9G?Rl0n>K;UpRw_gs$(?(;!J(mvs|!RIG;-%j+Vc)=VHLteHm_uO z)j^g%8>CZoEX~RcSAIZZ(`t<$f6@30SWGJMLRuChS(BxBp+qli%L|ihnt_?4cE}UG za279o=wkfz!dbj<7B2$P%8M_oqWHzk;{Qt%u8c|z1t;2&DKr+^J5@dncQWQR=*s6J z68~TZSx{p>Ke>2P0_bhuZ08Bv|3hphA3!;mXW?r23G%Ze_ z15^A;k|YY&HA0VW6m{&Y&a!NN9JM&%_ve+#{b(*~?FW+JrFlE!ws`^i`Ek{t!5Pc( zaNiV~U0b!^_USO+ZG{GqACRnfAw!fi!=xH6_CUuL9Mjx(1vi4v+q+20^PqCkDUE)3 zQu=l0=!>KMn@Sh_6<8hS0|4}%W{ZzG8qS#Q4dUVpNm7lCnZt!qX9{jfhjRc2DVejVn!ftU7``Q;G zO)Dc1P!m;=T1DAx*4N#5No7tQ+7apk<&VO|PM9TO@f4tqbJK-ANDlQ4OGK&730Ses zCyCC);_k(TA(9NKN@FL_tB~rO(%ivJ@zFd22La2k^0gq%(lf1b1*Q%m^zzI(Jd3A? z?=*8aCbrB~RieL&Ke4$>*TKbD{#=V0IxIq0`DQiD<+@aISPy;^i@lO|aIkpR`Wud$ z@*16+NA#!A??5y_Hc{?l&iYv^DTFOs9WEYRdPue2z7+4$-DN z_)~ZZA!qm*IwoSFoNw@3POR6@;2f9P zKWI9flzy%C!_n;@W zw$GPo#5g!4$iDxnTlMUF$fJ4LOi{;Kkl<7kv#VaOAvFuX%TdR<&ZJ4&nQU+XmNI=8 zRY-w>gX77Qv)U?wjB7xBtGnNNJ9LGJILD4jQ9Ilhd=44L3|dEedtEDew8tB@H1Pis z{jcJ$fg^!QzPB%<1oj_RDl2*k@QX2CLl{f>l(pBTo1v^I$&-q}w9me~mhoR^p$j}SfRcJnwydcX#A;RG)9KKWW1_8MC-#a76hWshIKO9 zQS7rw3h%~}qCirta9l)bxw6V%FFbOZ)Mn|9Ins1|py1|*(_d6j3vY?Lvf2V{OT6k} z7Pi4zRe?SnpHk|m-j=ktQa$uH~z_QR-y_G_Fm%Us%{LE ziO~eksLq_@NG47$>33exs`jVe`__$A_Oa52Q`_Bnc{xQe`%c~cUVGUgw56p@&N7a+EW7$k8XFI=H@&1|T^>3(8hGpgB>M^(dQQuep$0YdP z3Lc$2|EQ`s@+$1oUdi1bFrD)E?}S9jswia{0pMWgU<1=o$5;riH{{;bHFI&Xv^c@3 z!_S+L%3W^f)P@?ZK5OdLhK6z3$>9=Xr%r*)AHTsi#GUPL)w7}|C)iE-eR@nCYAjZG znP(Y`200L8!8yd+0tr_{Io0}Mz>?JcvFbTnKjx2xh24KI2z3zi*?PEp+vQj^U3=;! zDYGbyybIfhMO(hQeQ40#Ht7t?wv_Bl0EhGEdBQ=S;JUkDqE}UA&~p2+RuRuz(iFCh z?}FB=3G;nJ@ka*j<(!WUsyWSK22Dj0ujXpLU7cJ{vsGrvPYOy<2XU?#H72eoIy@&8 zoUCsn07*T!Aak1+aB^^`*9m3Ect<z7l6g``o8JK6|jY`pyd?Z{WznYwd{4F5G^Ke?X7e3 zc9S&4jK~tZuIiyIdBmX1UvKiD5dK=TEnRN}^}w+hG<*mbp@g2f5?y9Ul;W}34l$Ne| zUHKWChosdVYm?b}X~kU@()p0;S=q6t^z(*3ZUEL;9rDZ)Fc&S-vIwgNjkerkgZ};# z_Euc~`(MI%ULCDpz*TP_pcYoBKsZUYcst4RO&g`5g!d|is!Qehe4Ca-Z(bUT;)+=B z(w`ICLkvK@>NSVigW(j$fIJqa$v5f_FZ$>H^8NJ&gKGe`xSscFqaNK|Cs)vv3zyDe zIg-9=uwjlmGR=#fuCmMQQRhuk3Lp5?LuphvR*m1kAs(2qovOD#=zm#LB@0-mxO}Yx ze)haPmCf=JfBSRQsn^_PAZ5jU9#*Y|bwx$E0i~><4f_o-Snjbnw4itk!;VX&j#tSL zzoVGANb%XZieqSBf#|mHLNm(s-VQ}xM8Y%|v#wFE@j8CEEYKY%$rsv;J*gQW9;H<< z<2a`97VM4<(PPbUFArlvY(FVSXMYT`5k{H|EzbG;>B6ul@PnafVSVdu1{Kw>&bLE< zwXNwQkm`P_pm!)Hum7cUFhzGWuM?a!&+lietLk~eN6&{Qf%B?f5jUa`;2xFRA8bow z#u)=6uTpUyDOkFe+qSNeJ`4n$5z;4wpc)w&JEYeHmx@ z*|oGa4?Qc5A`Q2qp}BaMgbow?l%uOjtsgqk_k;nZxwb$<`4&RCAxyD&j|8=wZg@Tp zg?S`wwN>hU67<0l6Ce!-iFs_`5^Wp%N_RK*Xwbr9EKun=jf+2T&$R{p-pm-{CkD1Y zuU)53x9#Q}Q!`n^8wJX$JXl?vaAMI5nC!Lr$iZ@Z5jpu9Qk@##M77pw0ln11edLxM za7i4A2T~FyO5l~l@Ae-}m_oybt2n8-AY=w!4rP~@sv_}7Ny8D}!7L)VvefitZng%2 z7Jt+){Ie7T<&Ra#+=kc$m_o~FWnYgQcA>mgkkV`1qY@NG$0$JNL!`3pLPV~CeUuWa z^|>gJRpiBJ3@O3S1bTJ93=GB*TKYPWbcUvQG<_5q;pQ&USP_N;ifm6Dz!?rn=iU2r zj03U|kU2KGc2jqf)x<%pT~L~W2ZH})L2 zoyAP5>Y*I5=DOjkJdltig9N$}_THrmv|c*jon$0C4dpXjJ-UV~6on*(V(V`n#XPfH z=}}(mb*#wSdyGBL_W9>YDy(pAlBM-Qcge|r=3*W7_Xja#)$4^@A+u57ixQ4Cs7RUB z)Yt*uofTA9C&h5oG-28BTOg-yU`Ix@=yVeqplE!YqJ7!r$g&&kfb?H zg_seZScFCUma%kvUy1%*63E$%QSRVHMx+~R9_EQ?F8>K_)=FgN)a8FI(iUN#AjH50 zRo$=hU>~P_D9JPADX-_q8Lh{i z=$Y5z5`sa3Ax-X$K-oEgbfI8kZ?ibRU_e#~XapHd2PyCmkOEWH&?Cn`ni_j+N3 zY+|Ys{pKdyPmjuSgG@myLEgEw&>`0CcQ(ki`a3F4QtKYf(|2CW+)CieB)bznJljtO zDLK4y9Fo&A3c_7D5gDzh!a#nl>N^tL-lO5)2vvyvCys?yWb|jBi~v_ep(HmIwo*A(`y|$s9Bsn_G6E-W9!Mw3ao78z)#Jm57S9koji@tro;aU#g&!MmaO_$EM zHO;3j^ux{`GE6<0M@1_fEdy$KbzCejH1F-iNz@*I|5b{<#Bckud3xopveKy$ztqI0 zK@5{MP|-j-j)?HWJf4VFDe6Z{YZBUGabTKXrNrW6a94Hu^qc<%D_EtH2h&{-McHmW zqP_bQ#%lAm-&9Y=SyhCPNKCbY*%~uR6HX-d4+gh5K+qgH*#v8uV>ReC=mR??iZYW0 z{2wNT{r9Wfti_2%?vb+hejF`)gRGtTjX~OZ$@xbz(hBD)?jy-Er{3@AC1FB%db=Q0 zU1`OZwp~x|=IxazF98F>RRYIni|_kSq;`*_*r&GrQ`5k)R5`XE)_1a6=JWIAuR*lk z%%d-Cy0gU+HI(@chZSI25b1!nUy9K{r@iWC>3im}UiEW3faqR=Fi;)#T5Hu$h7nnF z8WSIQ*r~pl$y{D8P9pw~z<@8DAH}z2AGreZ?c)M@KDUDXalV+}D2m0kDW(Ut!JUDQ zbi?bgTZ^n+T)kscVanXAv-)r+l@gFw|=V!IZ(C+=$jiTQ23>GPkaNCERrdvAZdj*!9@=f6^T zWZ3)Q^+zvtHkW#i?@g$t7XCwQQ#w*Mj3DuVkvALy5iWO7a#EgmvRYB5Z>u-%Iq@{j zYUn4GqT>1Kd-a&m4djgS%2lM+<4N)=-%v=q8R9;MEZ(F_Y7Gc(VZ6wz`g}I^b}y+d7;U+0I7V6Jt_q+tLC&3@ ztxQCh)jb>XI0EJxrN5g+337gQC9*6k@Ok z^R|xLLZNhCE56nuwgE*!(k?!mRpza}pw{8H9ZmM27o;Te{rY`A+aX!=j|QUO z>ZkIusjV>}MYVM|rk`bVe*2J(Z+TvmC?d%>m6XAE=k3n#KO&v?X!~tJ8>ocPtxdSC zetUHIRk1exn!B(JX75Zd0K|ZL2R5Z*9rj`j{Mv$`m@eKz@qqlDDl6qbMYLh9eH-Z% zDsIT`<4fqL--dR|$H5s%)!ghx(WpP`8iv}7ItL1=&?sHUpU!9Zd6ro-tQx4q4y-g6 zDZ<(35m97Eko_r&BY{NFb<&NTWu(uxx$8E1hrSY<2ZR9-9tY-iWNmYjBwLLJpeFH5 zBA?Y>=YG0BhX~|7LX$gyvIH5!--yR20yLdh>i&)0M|<5$@9_g9S#0tby@#N(A85l2 z3T{>RR&A0{3sZ0!ki0Tav5)TuB07@iv7NpUIutdqBQ~h;s@~J9@cAi^xbMmvj0Gs4 zJK^WcVs;l;!~r!P27UU=cZoQ4?v|K7;Y*3t$IuLeI6p8bjb5LRbL(I(jB0HQru+z) z0a!DY%->}Oiy!tGYzAV=e1$7>^T=Wk#weSWyPMokut(c0tb%~^%f-26 zsb!ZJ3fypg%_&@=t^2=OT&^Aiiw+=w#hr+>^}gow%SmdSRoeDwTDxDhrUK=QLtExt zo1Z^~4nQ)pJfzvVyRClUWb-Pvi@?BSwj5VN+1!}T_va6Gu3)X)80_axt z3;~!TBrz?mqO&wD-CP;O?2uRNLNr{yy*FSAY=M#$*B4?S(QHN@H1VySf%^8=V7 z%vF@DY&hQ_w#)Yv<#kaODoSakRK@$ftJouCKG8;*3&#BsGi2~af}gEeC-AtvklbeC=lDSe$*yV1~rHZ8ZA?C6Ho)?-tW z*6Y(wqyJ^8s`Kdp>%rG0h> zH-eR+&P5Ho#734;NOd6vK2Z!qMd{v;(S3P$O_IK3N+k;_Bq($cEZpXfmJX6+8)bty z<4`HtrR`Q!)OVodi@Q)bH&f`MHmJ2*%79;VTf}X}YJb(bCv%!Mlqfd{^W@>{7GIL{ z9)H(WkR9!giUGxWX!0K~XVAg_OV+UPUXb`7n{vp{v$u#xC4}!yvzAC zD8Cn7A16?{P*myCQA-S;x_1bVvnQ(!b;a-#cVtIrwZ~F?g%w#BY0jI7x$(#db~c4th|c}XzIA7FHvo2!hWCbBXS3uo|or) zWd2fjw3EtPRpAy+KK8Y$>Dq;1V$#>eeA9e=2|F~y9=LGeg*I01V(s383>v~57dTQn z@@)sN5y`ScEcUGm_H?dWnQvxN14qRbdPJ5k?Qg2g+fUEWn;Sx%EY|cK`)@42?ASG3 z$817}=j$12&&ba9P->~(-JT{0&YeD76sy=h#Ro%5CVUJ_L{8 z`Ln9;vSJ{l_0~M`VXi86tvkw!lHkPC^83{r-&K_lUm61@dh4?(!z$_!{Or8-;<6pg z0Y-r-c+Dm|idjtc!@I{48)PoBzG&d4Dbt~`R^2j6;@2qHkgETU|51va5J`>8 z{3NKlAmh14hfkIi(tD2mNU~H$%P*ZubB|U<^vVh?JW#`3I*BBMFAE@QLt+%%SSy}- zlrG2Ax-WXIKU`u1N8wG26{byT7B$>5-qtfJ%%I}pnfxc71lgNjBS*>eB^C~D9*|sM z=q$TOUUNCb2*q7ll9xEQSDW28Uux9Tc#%%_#!ISfVQ{R4D2A&Eahb7e4TbEeDPG*Q zie5ZB`fb_?cCzSOD13C=vIpeSZ?xEK<(^rsMWLiWjE{i!hezj2ZiB)-5~1z zFh0;KA*);hPF)T?(fCVao;xo7yUgo(P#EN7d=V%had7-Ski>p(Gr!7?Er(Log;za% zb%O+y4$4bcZGIk(Rny0ZL3Ne%_##u%`^_{b#Ay3Y;=s5m>AMMl6K<7W018w5oCj*^ z|CTE6zK-X#G9sv&bWu=q&x=5$d%L1@_BQr{H9M@#clK9;GcQ5|)jd%JnTitBQ%zL{ z+)48***QGdd?q~0>dX0p$5^63&ra>Kfoep^=B#jf5l=jG^20LlZB~WMWv)~er7!vv z+}81BcJcEN)7u~9{w~M9jSj9Ci9Jf0Ggx8wKouf}rKUn8rYaR(1W<@!Qs``|(E?TO z$R@6Uc=WXFcRByV3o{9f8+o!kieZ=%4Er{u;NsEIP=Z5s-OjAu_vd$m8Yiw$F^$8Z zWQ-dN#ms)FoGAe1E<(lW6}cK+;}IWsB0e9}s^=CML| zT6kzXBJyxST$<Jly5zKfh`#=_4VvV^)xWM?X_JN4y|F+%qR zk;nc_;fEkg#)bYNDA%*Cw`)7PrH_-9wNy?fkoo>>x;;+p+}XkK5XLRCGM5P5o7Klp zKjhOd+DS22GttW>F3Xnx7Hby$c*=zfTX55j;pTdN{H0f4Bh+di%0TH7^v%AB-Sr$& zIyiGHC<8@JAbxuPvN@UHr=&ILGd)rV;LXvXB*-0RDwp-RK(Krt0OQO=)afqfS$$c& ztr}?@#XY%5vHg6Cf!XSe+taiOunZLRahZ!KP#o_FJ_kdCIvubuM*hahYKqUdv_47)0}8>S7j7P7h>`_*d-p zKgk7g964l-U=Cl8#vB6^fm$4&yXX9=Ao9W++F7lBvH#MG-0ilZFU6HTwE}FXVYs-7m? zY%6GEevm$*P}7;5R6?kd098jSZ%H)NT+MLdtISt7Ee)>{(&c9wxO0E6cK|xGi0hsyG;cPo zF&6+&$rTZA$r)P$D{z^S_>`1HxqT(mtybs)YL*MPnOVB@uAivGCsX*wwtjf03&MjE zV$+&+C^(l6%F1Gk&95$sUNCK)I-z3=0_`=Q`cCGN;11I}sVm-bsn%Ra-uYq!f(9eO zm>JRu{|*w=*3z-wrM3c0ZMnmaxK%Tt=PTP;d>*3i{6|DR^c=UMpJeEIp4w)e&!#4* ziaP^ter~S2%)d=by>PM2m+a6mt=l!m%>`w9<#|gty(}K<=O&*h!ubzvdK2NvWvDWm)zi$Dpel6z1(F^Dd%)hmUcp-GMK#IbYK53^CKZ zo3h0CN9WHv@Q1lwpzc}P7MO#1qd6*Mu{?bY2V!xDv&AD)vZ?pw0=b30*n5!W0sk?m z47kE}5r?JTHGG?j3$QL>fj7~BMzzS^+CTsV2<^b`+ z2VTS-k4$QWgv5Ti#s{Yq;qXJpHXYq^Hc*fksJKw2=-Yra)z ziC){3-`E?m6|fiN)qzp1Up}fL$;l#z>a0Y$YqaJj{`P9G1W1Vd(pqKT`&kWqP|sPl zl>fGP5&(f1fOJBtOWzlq1eJBh#;zDnwB^_VZVo{_#U&LxH;`9*1q(szS!{EnY}5@B zkjUli2_-@THp+PkA@w#5b@_hgmVQ)Gw?{!`t%GfBMWcz!IVOQ$z6*;Ncie_|Rs-cq z^k#k#>`62YX4!#q1VRbi05UkQX-B2)`I1tTv?eSkCL;=^+k=24dMLpaDS_^xVQ|c^ znAKSNl$P3dQQU#>4~{mJCrfoEqRqD#KI0D&HyVa-Pc1q+b(hwf4{k)z7MLq#FI9J! ziHKQ*+T_TIjsq3l+*#9@%+mi%>dJ&4Z$*rWi1$m`9T2pf<>+(g(FgY9+a3FRNl5!} z?xKpDZ`WI*#N{Fc?{NQ17-^SD0L&V+;(0HCUuSx5w_Ju?%KW_Wt$~TYCK*};EpIt! zttU(zPPVK?dR&3XG_D}?hDnGrU~fC}p*svzK4$G&60+Wo+nel|0m0nl?fu=Rp_t&v zdu_3wB{%{@RLR-1To$3WvO@TdBmWxI7IsBj^>-kP(kuGBHbZIR%sb1+Q%z3)s#%-q zoe0s~ae23~6jH)^lQ5iFKCX;WS}M!LX_^!=wEf#5@hx8Yl8yXA?SwPa#iO6T1w9fq zS_eT8WRCmHzJ)7=M30+RPUQBRT;9bMt!Q*xcOxo@!duR!TKg*OvJ*K7gzVAvVuya` zeyRx`Pm{EmZNd^f!@Y28X9Naz#FejukfE<^LIkP06d3B1{g6N!%r0X|ATj8t`+<=_ zCrbguttqR$GH+5K_WSA51(PrMXmJ!v);IzVX-;%MkfluM(ga1w^5>&l0h95J&fP2% z=!^bj+av8Y-rx>$fYfXisb=Wk|on$*HPOIZ58?E+9MZgTJV#gr@v zDaNwA;o90gv(`fn{5svs(Fo|VV8vsQrh9;!?ATuqvD55w%yw^youK+9+*w*D*>|%x z7&1dRvr?JOP(u?4)9@+jz%!1TJmazDASAgJe&xU)DFph6nsh<5#Z zo#fdccnV{{)cNzZ1IAi3dI0N%?bK${E*`nD!|pG~@S!(evfiHa&CFkpzqjY(PvYLQ zyY6!N^6pT?@Z+4CZ71Tc&zz{+_shNmpTGLr!$1A6>^<3b!_kcyJ-_THhIU?EC>N@7 z3WAOXH?SLmr^6E9GNm9mG=qUJHkE)LxC)zGms-=U#$SD#Bm z5=7xrk%A=(>1AUh{*%M6rX#OLK%-nLg0M77g~&e{Pul@d*ct*oM=v)245ifLZz6&_ z{@%11v%!a4E7%T^V7_zu83|b^R&vJ`V3)QIzV;8?A4y3Y7e2BxuI#jX;_ALp=NwmAPY}o4LS6_eSB^N|FT>7 zg_7O;@05i6DYG9BU7zgvA-G{lv&|2QAHC4_VX*B75@+>Oi^W#H-#VIc7F>=uAD+_b z(lFE(SN*kJcrXrP@r3>2E6<3Dg@`#UcEaN)od6*y^N62~$GG#BL&M+S`l&QD0bmzn z?qSUmi}mFI+pAu(Z)!cy02%Z&fBsAcIZ-nK!PV4%G}Y3e=MgKP3rBYhKDwI9{rGiT z<02doswgvuo>9R>v8K}imj~h0Z)$4~L$J=9n|d}9Av6-{r~86~8zFV5SlN|vG|e5+ z!7tUcJkF*-YBdk3Rm^(xdrKoa^-;_l5fwnS^kt0){FCCLcg*JT*{Dx_e(ajERjZhQjHG!ojCX=7`H4c>bvV#(E>Cv>Ot$&w(bJ4e)tbd#+Jv|W= z=wZ0S8tAw8-N7%>4yJ7*S^K8@(=>4jwY`Bc6|Q@+VWaQmAC90--k7^H@9{7D$g9o( zpw_nUCaCi7ScKi#K&oVHs6?ZtE$ zHcN3|YUt?l)qh+JKM1d)BXhIaWNGtdJ~v1#FRi)MnWsrC&}JLvH>*|lzu*krm-z!K z)zd3;^bUrjeqwClfN(U6GB?z1iRiFJpP=a;ZAt9EfgYF*E35zD^tP}NAk>aMVGdi` zS|CK0_-EG9tKp~Lq3q2;XvXi^`{UH-^aIQ(HEr%d+{nzm%K<)kXTJ+pBD<;>az z!+FvQv=>NVo7rufC2(-qIXoqbKKorpKkJQJie3I8jeM+-TYNl~_Rgno#cW@D{@#-~ zKH>eIo%_@mZ|@T`8R~V?N-f=mpEg&>g%P7FORIB8kr^m|NRVf)S>Q*bfmP(0=gmu2 zF|s{>o&dP~M!g$dR$bKifnloWTx6%yr=1(^FWGdCJ{nQpch^rFQiD2dwD(%m?%Zzu z?9?$t7>6+7K!`eeICvYR9l2%=Gb`$3LQ_QAN17>%b3wuyCU0cl!wJ96w&|n`Ax*Ca zlzwrCm9Aon)=7h`;M=3KH3uSS;)*Bek_r^-WJx0n!H>=uA2x|O=EQp%>z^oJ51wm8(5Bl5qd#CJ@;?BPA_*GtlJYN(_|e;8FaL?6 zdAe<&Lrjk!IQi*&M@A?1cc#)BN>w)w~peLKLxuRdbb_(y)N$;kBqT#+C(eurJPMo31x+#R0J^Vtr zA3r)$Hn}BH5r;7BP z9z{ezD`*t~6%ZLyU#k!hs31mW6%_+AhCu=(wj7~#0E`UE6lIh#GKK&lRVoMwNWu&u zB0~rPA|wn68Gic-!3u4^x7YXQcfJ2|@$6^swb#1Wz3#Q29rq-Btr}g@?ZE1IN`Lv{ zJ92F_b1GRysb4RG|MOnz#Dw~A&$-AX;07W@;4Z1F{u|1ML95Eu3gMX|`WViPuoy$a zYF^xcmrZ^^MJxUd-6UeYFI9D{UFhfWL&!!ZrHwPJ7Z;aozFvG+nARAj){gGU=Z(LDS(POT zb(PB|3y?21M&csV!9vOSkqfr^+o43cZjtlN+tesaZ6Voa<2&P0J>}cb_UI2*DI3*1 zv;}bVNXC9zYIgf*x;aNxq!WqTARR1C<0T++fP&QK$h=BMLCAyegL-nbcZFU-_ud+f zSE~7GG$q)AS?i_S>az42{HgF1OxbGzmrF}kx5>(MIUDfKt1&L2;4Qr9fZ@>Oo4B&9 zJZO?ZvLZ$1_ETNk31pwC?E|sXvpYB7>~7F*jTTT1I6L8G3MS&x&zq7qswbH~9M=l{ zMmcOeiS|-XKT5KF=e@a7>Kv4~<^e0V)raEg4+Hxcjn}h22V)j*F*g&n6q%QE##0p*3J>(h#9wMY0De#HaQwXlB`Tk96EvO;Kj^!SD?>04$>3A*U@Z+ugQFlAlxEs1*tTJTmFh1++?x7@!_ZEw- zm#^MBqOGIZ>P^gUVge$I&Vw*nM`nQ)OunKWIE&PvE-nf)c^40V*6W?{qEUtsnj*Eu ziMPe9r?Uf*Od>2fO;R(zzC5t;0f>$~f)cEs4*lT;Uukzs_OUe&S(GsO#h1;0zZntR9!pxbd5Zix;xH^xYv z9{eVnst|kp8&TJZO<9z%<#bjwf5bQ))-(HK6U*7V6u7mqS8ia(KP_o*_p4g^UJ#x; z{^5A`cG!dQEtI&JU3|3T^WmHi3~Lb{-~`<+P$;c0awM^dV(4Wm*EUPpduG zS-o_Nj`(_;56g+WU_n$By$I|(P>-zt(1+wFYtUW-0*SZld>^Er)zzYXhN*cW&1dO# z9;;p6;9F5kI>Az2Zu(fTeRGrRr2yc23aWK#(+g`!1OBKugsi8LF zU4y}4U!efT8bKIKz5w~4EXA(INs1Bn#dU$$$rZ1X3h*^J6w?|y;Zeu-lFv_;v z>XsU{-PDl-px_{pKIGdc@~JWTRk00cnBx9}EkWwCvhruuWxEp#N!O-4@5Hth9ox|x z)10tZwJl))>lPx+Lv}UarS9MFU-3GxGH|+rCH$g*s=%0^;YF5QYH}e8^b{#Sjb=w{ zT7+qZKUYV@g*)&^+Fga`Kx!al-7el|@EowfPEx#4EhGwCXA?MN=uAozc#vr$tS?3nLACe0I zyWu3LsPs=?J$WnbnX}1UV_Aq5U}O=<`8%5j0sv2OtR0SmoS4RF#af~RXF&228tc29 z2sm9m_2D+Fb#uSp7fm_iS~e_lrkl;?9>%{-J?62+`wmp-*EiwNWGIYfg|V#QxuPFD z5^Zxveuf|IUXSf~ciHG5;(2)Zb%?PulC=zij%zw%BbJsj%MJtEncJA-|g|h_2gPQFnVkNCwK}iY_&rt!zW^pXf`(|dNF*uH(^)i=MCo< z-=@0IHUM5rR2B`F2Tpelqox|HUATSak~vQH(_tpH@RW_N+~LYx11Q%x2{Hrcss~;& zpS^h9HJ#P1psWDZ(`sD~Q1r;E{*Te)SSuFr^q0vdCc0?_=55VnWRDfu-wt+=e2F$N z%LrpdG%xFmXX@QI;A*l{VwP_L4pRNWAQY_eXli{)tLKY`6x!!4lDxgdp>*<6-r&~| z5wYCd)7Wkk`2ZmGyn^y3>6TVxK>l-5=Ax_q15|SosJrX-m>l1~<6b!OcmR?2tCL%T zis{rMb!5N98a+8Bk@UVDDIO~!L~>+ zi@V2o-QHEOx&;)g$WrA@S(5R#tvZ$Yef90xM6HO!Y4NIzF11ZYCtYk!3=-Am-JvP1 zc2ho1$#)?v41M!D(W?GISuh-i9w}!1NSsPlF%V#L_P1=?3*+9wyX+~=c?wFflMh)# zgAoJS9y8%rt0jpH8+|wN({e#7my4RYUtp2pB=ayUw< zo|*ju$gbHeg$RaJ8|rL^5LQK2V##MNU60SF2Emv3flkrIOXrh~28RZG+8xrV zs@f7C$LB+u#7!n(XL2BMYqAG=^SpZ9rTVtR<5=_FzICa5Q+1%)n!23h77*Qifd@P5?Iy-%d4lX;KJ8*yfH?$~4D z);kNggu?AwO7`aAp=z>PR*xDEYl8v5ZDfi!f@A}si0U$mX^5;Xp#Nv z7FpWGOiii7)TdC(LPYL$%ErXUFHd0X5U>R=wa-e8ci2dSjm>A$=iKvwSJ(sL|@K2G}JQ`s4?XDA1}ire}F)9|60)e+9izDNVS<+iakL8 zT3%@ZJ;p^Z6eDtsij5Cx_TBVe(wjKPQ4%@!yD#U)9R@1FU=V`qAw+A~&e9un}r3R2x7dzB#1Jq{+k!nVve( z<$_R7POkx1Ki=JNbhp6}H^gUWypYRYG;e92jM1>bT%1Qx-%^k6?i256D%lQuJbox_ zk8<6d6I53!ZhYvP?*8A@Cy<;`A(IMaK5)YfUpKXZ#j3%CzT2qQ81~fvR1=ImK8<8G z6x0*W8ZG}OaH6T)Re_gnHzAFE=AM%50unmj#w-T2SF_j8ZppYy0!S#YEQ7=3 z8QMd8e{k*Ice!Bu{=sS*kcfJpT-5#U__z1vEQZQls=4{~R9so{KPcy?J%P=MJ}LY(Mu`_rfEUcM^@IaE+fAV zBTAueUab4Z1nXgZ==jeG1KzM8CLzbhCHRJ8`O`84&KNTv0WhtR256nnk@DVOqRwXp zzP-cA`hxAd6;BWt5$WAepgzq;5`s|z&(rVD4fTD9K=S0h5ZPp2GXaO=2 zSGE9$5O%v0UZrjtmaV{viT+|!NuV$3r%^(dlGgIW;vI2^(n`hM<%=KWA?@h5lXsP` zFO)K=DM`lh){bE{pgO)jT{Ruds{-P49gGdBACrktFx(Rc24 zLB#Fy9gaGx7K%;Q8FWG54kZ-8Ri00Uv3T>K>UcJ$UI=NwhL2BNg!fCS=;5`IRa{)d z+sh)=xUr56kOIDF%H;?X0_(4;&NP4CLQVJezn z=R=Cib1|&|Nm(8c=qStd2pQGH1TZZTt?+!&S(l3W`tI_61^~;5? z0)PDg%hdtUb`e!!0_f7>r8JU7ewM4WM=0yFulj~rk88kX^kkcxmGr*rLI&j&ZbH=SY7$X?Zb$y88J!ZgR;)My~QGPl4KwrW?F zLAqJxoD}bGx4Vh(Pmj)9)+MPUTOj97U+XNi=_zj885n7#hqhT6Tm-e@&eh{{!%h-A z0X1kjOG8WWy(4B}Y?!O;+vL!bwOQ*z@=TrJ9WT;J%;3E@8HkNYofxq>r&t;;D0YME zyJBv%Z!z~NvMHqKDeDLl${D3tD^#NK<7?Ux`*vNtVFq}Em9MMd2SemE6S(y(*Ioh+ z6--8FaB8GrUM+HdVkc!arPaD%f(~6vfn_d88!vILgus-K0e<70CbI76np zcHg+FJy_vlfoEt%aP0_I9%+5S8OVA$aC|3cv?_;hJbIO80VisEyI`$>5Jzm3n zq3_n{c*JG8ChP4shBYvWMcH=dPbV7AA1j{%K0e1GjV;{rczuSP%yEJDOH2PAnMGur zgB3XZMGMuPC>HeodRh1T-h9^^#zPSZ)wo1_#q@xuP36(4r9O?Jr1V-5f_8VHO0)nZ zOU_buUDd0F?=|0$J$TyGLG@Uo=nk$d3Dir@bLNIu>E6RA+PhJ(9j`JtWzy|HOGSaz z_WKH|j|NtY@`p-uhdrzbMrG(w2Y3K_%pFLk-JgcV*uYW zzU8`USPvm9MN731k_!P$=Wa-ppmsV6 zZX{PsfV8mF>8v7d5Tfa{9>%{+_sgAM$vD(2cj^dIy?rtp3d*{*L1{NF>X2e~U zE~iSjd~3QuVwfwnyQQc~fT!COI0m)dX1V$D-3~-xmU&ZYgdIw8yMBYF@QWA6JWLC7 zyu!;o08kE)>`WhlWEbXGWs0$uRs-~OP-EFvKyO_=86&Ii?54+COykmPmpGa_Dv`62PM@#cM-hQ+mxlQMED2(t|9G&M%}|G zfK6a~7629unF#KeU#zzpMs>XFiW%^!xs2Zj>5kM!;0$*{76z{tFs?(|N3(oTerMccVUTv{Zq%r$w?Stz>xo1L^u}nTrIpT9>W0i_=9!$t zX=iOnjL=Ri8H&roB)-rE<5u0t`qAYt#!U-tBe8yTl5l&ssbs!rFtl{**NPC?l~F%9 zAa1vxO7nUVl|mb`#Kw&uI)He1FItK12?>V_!>U;{Ie7j7;_>sjT$69%b{Y^hTRkPa zatg~J;aEClpNX)KVWB;=P5xN7gMccw@?-(}q3$BlAH0qRXogqt=%*X;)e zDO0X}&!G+ud)O94t9i)7@I$4%&xr%)9}V~v*a1ebW+hc{SG80brkyjC{FE@Qh^s}_ z#Lo<9(F6ELkh2Z!m&}Z|;E6lF6-e{BzH%mR6OF}&DVxeSJx}gW;~2n`T_FHjUI!q7 zt~JGK4pC~9AEMM4zUHNtMmPBk;;aQuK1XK@+*1i~!+r;z=xhimUu`K+om?TmGQ?_B zSb#LL&m^4fTM-U^$E2>-rFta4p7t-|z)baapQnS zy`)1YK#t0TqCl|DQL}@X-e=@73&a5(HGcLq6;V83q)ERSwB+1LYU}kwp$Dj{IRkWd96^PCN|;tevqlBJ`(^7m7gv{=(+qXTqu#PQw> z9;~`)YKU(-r1~JC(XCwN6@(yPFUJJ?+{=&L;EtX2?X6$b$s&k{)c^s5J?SB@lFS8#^P?a(E1;J1c&x2LJO z+Au?D73bXfcz ztH-lf8uzK5SPVUdwt2BuT3Ysim(T43Hjm!P_JI^R zzj1`LbKGurr`ht7B}%X9#X*6}Gw5tso*T5Qj_c>&?gAR!GwF!RtLCMH0n4Ns*9XO< zQTO6~ed?fpn&Mq^=St{3P~}?hFC-bkD)&IgZY1XrbxCS@Vki%&$ZG$Od4PI%dzJ@B zlu)?^p4i-_CgG1WZ1#G%?J`xZAjB4vG^R(6km+sZmrN$Z3GKpB$pxGy-0+ORV`TT(|=vd_1)@%fpn!^vzG^Y=e77Kmf7&k0NH`}w9C!Ez`H9E0J%j6WkuJBl$NMw zYqzJ`1>Nxzk4LBATqk}7+K91FHHS&dBubV8srJ#?A$SAIwRUvbE$?OFyLfth9FRDe}Cd^GZ%!}FF zZq6~nUKP9}v8mDR1{$I!1CNj@;jXr7HD${9OrY@bVo)v2fXijo7u3baIL<4uo+mIY zmbGeN(g26GR6!hh(^8?Yb;Tn*huteT(RS~@91u6N8?=?Wc`lY@M zUt_G&7@wN=bk^fA*S>`Uc%5Hkd#;BM!&Hv3lwZzGSk6fcAk=v>%3CUyb6p6qi&N7D zT+VdBD-5!Sn!sN)>eWI}OvvN!W3&3|88)cxBHhKCW+~)}{mApk*@wus4=B>nUHo&j z=a)SWo=iSziNRT4KoOn13)IK1mV)+X!Hkw5q!7Y3H zbt*Fe#UD?Mn6zhgA=+#R2RGo(B7$s;+{lRjY1{}cB~i4y)lqnE=9iggnTUJ$G!PGr zFTPr&Di(i}w;dde@N5IEpFA=I7)5IVgFTsEz<7{O;UrPH-c`w?Ie<+kCDN;C?ZvzM zHcW?ixgv>oV=aAmiZOuNH)(F%&~<(GYh>_Dm9!mJb0^CYp?LXC%T-I%!1U1TTOiyh zP`D+nSbMco>{4W~n96(SA+ZExbQ-Ak)sO@GAm0;!?mAVZEQH2g(>2zc+ThW$um%_>a0!XhEAA;Z7i{k#K#h3kQSg9wa1PA21tcU z-5iIi2=I~8m~|c?XrvQAIWcwk-cwav8xhH2*M>i06kgkW(jSck##oU z_iv^LV&%*i-b<%OnjVMPiwb6UTKPhExnG1(q+an&WI$xf=ML~mXJ~TzYbPhD*(2%; z^0n3;T)yMzQ$q#^d|*lEhpOEBfj}av++jMEum$cz2Kn?1kR8%u9Ok>qF9M~lKtvjw zdpE{##u-XavsteVf^GNHgU$+ioG|*?$f4jGa9Aa5c>(#V_tq7T>xls4lQ<_|0C&-a z!sWn*sFZrnfjnhW{=19YM9ZCz3!*Kkl~T5~c1tSrTeJ;LO#B-=eQR8RU?>8|3TC&! z8^EE4Bho_+2iQKFDM?wuz%L`A;=-2sXQPxt+!j4vjIc)k5J20eM)cZZz93#P-&Iti zY09`(gsw7n+=L(2;9c3p-_kyh72j9SGm@f{OnYAyl`EB~2tt@8dbkb&nwa-CeUqXD z-0$^$qZ}8(C{Q64ap9qmw~Qa+BMq;0A*esiZ&d3tAx#Hk6+~gu6FFH)QcYy_r&3m^ z&W)g!_Udn-64^$3P-phZC^&W2a_frX!Kow-^B52FfU-p6gz-a|Z8%JZ%hO$s}K8{6bB z{**LX9-z;2-6L@Y;IN?xRNJ!~L#0(c?d|(U+SyF(%$K-HApF-X>Hr2xmmaA&w|2O% zp*~kNXEG#fKj-FELvErnO$#x<2)sw>^`53zHd%m%=MWD?9E$Mk!spypaz0^8272%U zrlY48BI0V^(zDZfE2FN0*nTJfVZZ*7(vVz)TiAEU&-MDs^~a4)0ks!Mr&io(iC)Bw zR%PHgWu$*y;*2w_an(c!3k%i`lWP<@GKOT_HBAhz5!`}OOmzKi5%t$Ca4pO?t%?{e z(EIz|@bW804y~KV3`gF4a*aTGjFu3&{=|gMEVT5|48pna2h4g5%L(PwI1U$8Gj#<6{&5k zNglrwl0`2zIy@Z^gq5>g;DQl>ml(9Tn)^D;nWGn9yXDY&BK8UO6VV2UeN$;dC%Ek* z6eaVrzLAW}O*NRJ7j7ddcWurPY9`~#2j(|rAr_mSUx96f+~geSK9b6r)M(o9UdEZ+ z!WvCYXH`j!g(z;oCj?t8H~=S~g~;fIwR@bh{Mvz;y+O;^fV@x|Nh|J|hJpUHYhCpf z>=w9b2`jPR5^o7~87p6xxr=D#YerTqf$!|NL#SjSO8mTbhMJu@&)sofzy?&@$NhZSs4Ay~xIDKpvMSdL-21O1D(CTGm`n z3Vbt3$g%5*Xh_(r=SNCFEsirzOY7Q4I5<6E9~p1tRuPi46>j?AnkMG*jjsY|(`9%#K){;3VGl0HvtOy6Dbs=-C5z|ksa`E$D!s7jfHA;+ytbOxxL_~#i|nwe!IYA+X*8{t z*Ixijk2-IX4y36x-%}PRTigZ%Z@&g7p;6mm=?fQx=I9n&GQqN)6R3Qv8I$uI{x zBOhYwIbSIGd@0e)I`PJvSOa}ql{F;oYG|(A4;vsG_48)Sw5%G_P(Fs^gF3Y3Hn#^u zC~+yTn5&RmuYDnY`HGp*)Jq-eKbggFj>CKcFTI1n6zA8AuA<#$o`GXg?sF)^g*Vzc zUw02av~~z_a(rj;fchUus=K;F2YrKV4C?&OFo-0PWlc$0RizHM)7BDRcK_5KUslfi_@L&_aE>^93BjYppGoXCg(MN3;u@;unVOa2bKyQBxGx?S?U2G zu1r@YUs3$+o=Vu;yLIQK$0m>hXj=63)1tL+lN-PjNNV%JP$-A331v9k7Rq4DP@PV8 zgCfXn-AvBehyn(tSz9kxy2t4V`d9LtR4Wi{Uinv90=2+R794e&`-QMZ`z21 z@-`NdaP(y7k!7(Suh!*6pa<847q;S2$|rFnT_!h}vL2i)Di6T(DZzvPLL1hHk8ymw-W?Vg z+4pzfnh151E5tY-iMrZA#}|WU;W{Tc8jLwPTkZt`$I&%o1~%~?xPOo4;XDU!!Tv? zyJu=hG~_ZavUA-ZJ-Nu&us+gWi4qi~eDblKGVY#f_m%$eiyeSL%!fcul(OE5*wPa& z7u{XzAEoWY?II{|z=55LQ>&1|+Ulo3D|L02TAoRT*9SszM*AhF3iq$drOm<>?&-14 z`#S4DtV~zmNtaYkX77PR!6&uo>Gc*}=(3Fb&lu@6zP@`hTjT2Kb~jw-5M=v5{%5yF z{eMJ;YAn#h>V&?+3{6|V{%L}OPYpfxByNnYViE#h*%L=utLx+D_q%BWiK`UhJk|KZ z25!riddT{JGPEPQh5DBwT&5QGz=?f{Su+rHvGrEg`Ej!*o$1CGj`!_pns5)a>p&%{ z1r!mxrHjN675zv)-<4=Zga$B~2F>Zh@U{Cy{(ayY_{(_yLnZ@9zJ@ zyX_Ren;LiWPC`eR2N)9v!E&W)rL1xFak-vYy6L&&t`tZ^Llp}6<~T@{4#VUQyyigxkZHBdR^K(!HOJk#c{l=sVSYXKt5ydGzv3Mew z;EHM#xh~|WyvarY1+IKyMtfK-MtQ5IVq$z&xAatz>?p`bxV0iQp&3fC zta?3q=P-7=S(V@k8IaSgNPDj9EO^dMbo`LP&mb;P$rE2D;44YL2;A&;>8^4 zdpz@I2ZPnM#S8%3pChOp#o0apGy~|4;9eK$_PVcoYiBptMsT%S)h&2KX)$_%K?T z{m0m!EHczP)Mp&uo5wt>62fKfflszVK;5?fwL^vUlXucoaDiUl@JRynFGGL$m!h=D zJ0Ric0vm{=L@HwCZNDSPa)tZF661#PScfQu%*BGT@y2+yd1<3&dA%(poz0h;#(#V@ z5itpnLOo9cEwWvY5weEA{9B%5eYZu z&r2usyK%M@j3zK~(@bN2l>N{O}fklkGp7r8L74UB8SU+s9Yz-@lGaOERr`NTok+l_`SYUPx zrU0j2gdznmtH?f14S85&L+nH4;M7U!0&vMvd&rROqdjzct**T*srBU>U7a{0;69tB z?1fXu0bKH#GPwM>^|pD<&quFBsmYmH;3}b`iCBzmW~(er?Y@48W>t&dl~Bq=djksW zyln#5c_B)sNPdigJW`RIg|)6U>dsX__Btfn&X30LcWkZ4jUav*6fRo$6lubxcTI>N z%k-30r}e}(YSkT8zB$N8PSBNT?5!?I!9&>j?TrPUR-$gn5I6hB*Aj8HyQ_fXCXSW3 zPxW*#9#lUqtxF85EF@l~YqEXIwJbfc9Y<2ImFrg1-}7|8YkV;5dh7PaoX*YnoT!lLuRqkU6 zQh|Tp@dVjQA6;0vH^kF>Q<~q4?Tr~~d0_|Hf#n!WPw$S|-ar1Ii?7l6pk~ZTP=ecc zv#%>BbyjD%m-ru)h1KB_*)1Z;`>KB$4YR4)=UqE_mldNOp%F2y#IR%t8^H~;p2dZP zt7&w*YZ;<{wr~5&MH!Qvis(Z6`z0l#jrv0>Tc^RV*f=@F zW(HLkGCHcYKyS|d`5)$e&Nn*!X2#mMd( z{!pZ+mUlJ2flb&c6~}+j7#{~_C^<0cRN;RSK9-% zpvqEHGj^~;duEWUE!a)hKY3=W@r+P2re-lc*y{MBz3|DHWyaMHZ7BU8N#!SzRY zrZ&qQ=ggs%dqBE+-qz^*_z!wl&D|aTAx16FsQ+e{0`Wpk3tFX!3kofMmByc}haD7M zJn+7%X2g>5WpscVoPO3JH>GE$G~J9{n_1%^GZsxZshjbhU zv#+_+&AFbIbk42AzvmBLnNOIgx*O#WW@}%LkxukZOie?c3}$})O%@cuxGuW)fcc6u zfZITp3mp6t6M%_iT!L1yt%xCAK*mV%9~}iuL_u!E99La%T@=$D3WsL;C#C@txxNE< ztjj5ONO_wjG*hoX?iHBGsrutM!0a6t&y< zW*!J3{x>jJQ{ew%*=PUX;2-Jaf(Do0EXAg$iYJ16cWG#|oOHz=W1{9|GgUL&`QJ0h z7+zL1t&(1F3`z?6p2b@X$Fsv66P`BZZ9NksfAzE~v9q=Q(O9TAo0QbdjTIN`y8fQ8 zYOk#;O`wbj5)rcYXl+u`lw-;iAin7*i4bSqMGB=Gj^@pL`+K@@C9JDOWU2Gq#)Uas zI<@(;OT6CJ`S)?7=b|j-;GRyu=cAVOQc_->G|_Zi)PG!{YE3*Ci6b9VGw34V_rSeB z$#$XSby|iDlXt%9_n&|L?CAfaXo87Z#s2f9Z3-<)pY9Byzn zgQ@4L52~5Qyb&Osc$nJ`e3FX)%C;MGSz84OpEnfAEhfMU4Bzb#iDs+4!MsoNdTFh3 zS(TsAky_V3-(I=XG{uFKsqfqr!F@{KUE_r`uq*d8RfG2C-<|)xIa^~^^R0`#lKjew z9~&nk8le#;{4<+-@(0pu(8S*f%3p2Sq@?*fSJylHy8<_<=@a;#_d1sE^>{%djS^NR zO3nTgj`mkO*C`phdegSc0JLS7o{WB9|07vf%^{sV&haw04f-4VCvs72XleP7b;s1Y zX7cu)rt#s|pq-!p2~``LUniZ&p#B*p<;^?SJX(Di{n+&?e$mVQ)h~1Qz`Op2@#jNA z|B-&Kqw%^2c=Uowrg84U*sFhYzD|n2wOIO%HOIN}{QDiW<#+DHo%@94{Z&+*l^lEg zlx(ZNWkwMfzXfiQ_O0Q}U)q!+>R=C5z49BXykmO0Mn~-cc*gK-;NV}H0^8E!JJx*5 zjd!F!Z4bODF01nw5X*lRGa?7Ym(T-S;MTB{xct8n+mr44jMVxxPliRQsh`AwFS}xD z{>riEJ)+bG0A&2Feeigw*sF0$HS_DgVHCU~?gsX6I_3Yb|JN$mywoXb0XR{VTDJAS zl|%kAyxW_8533iFW~}O8b?PJB2t5|{hj}o2G?M~KwQ=pbKA-9JN%P+*0ltnA{*3! znPM&N7Z39RZ2S#h4tYo(3(h6%L;lZG!0JDKL7r`9{9=0E2rTY@idYE*dIe!W`-b20 z%h0gj`9Fh##A+dNU?kxwWn#oC=jfJRp?~J0Qv3hB-WT6$zp~5Ua>4rQ6M`_UNY2BD z4NfHePZ*@T!|K{C?ffmqiLoUW!*z4Z*#~3GiavkAGHq`o*8r#XTv}vy2g+8W@UcSIBGk(S z5Lu(^6J9{LcRYPh(3kH&@W^KuRCLg$(odpC{o#2XgGx! z(iSWi9RgMix_lkQ&0uDUwLc+)t2ZyN%C#~wSRQ)y+vmkUHqDjf1gQTOyw$=wFsQ|P zi+2oStT|<}{S#7K@v-0jm}};CV<^zM%5>;S{5hM}M&|=Sgs42P7_dtP`mD177Wumvb7Q==V>wx0CDtYU)L+c8#IksBuBv72 z98>iJP~3)*Bb(*m7Da{pM;dE%36lga4oG+5V(Ra*RqwsfObYvv}ey-^hfayO2?P$eH8ER5F` zCC3uic!(7xz?2$Zo4`$MI3QE@z6;V}-p5phZU|U-p!Bj;I>(!9%rBiI@DF}%w*Qwh z0{#PitFMxXu6b-XYCoY`Yw&rzE;(aF^HXIl2Vo%pOAIk!hY`lq*Hc;=HeV!mqUN;KlzE7ZCmv<)vXBQ?nP@{ zm{yP)t*h!a7M|@tDQheo4mEnC8WF)geT9{&QTz{mEp_l!)rB<*XC9&rmf>EMO3<}- zkp)@Kwei{qzWTnWXC(WX(J4vb>=4&~?F{omzB85dzQ54IJaFG`yA1yFQjTD|W?PkT zMjVP^TFFe5-v&Esr>AL~#iz=vZbfsbnStcGX`X)iAgGIp``0^f)TQ!BG z2l`k2ascV6r|Q0T%N@sR-S3VZ!sFw)@eK!LjpJW=Zyv!)i$mS#DmD@1^pB`p6^*+W zuCb0?L28Jus{5EQ%RhhBIuSaHs`Qdj_tL-*X4+fykE&bMP3%}>ohC$^eujHtDqh#x zUS^%iBPj8NUTexjLO(Xh@M>4ka8yoKQ&)KS8e~8QByjHq)ctdpdylU5+0{q6o0f}{ z|9Rtw9KNcd_F!kcE-7O~<5PdxhcIaWIffA6N6a_Ge*As^o8#i`0osc{dI%W zAGGW5%$0q5`B~Z#6h5J*4Wy(i4YEP_fUJ}&D|SDlUmCB= z-%=Hc)T^EiyE3>bwqhc8<~w}#d7+V+2C%FYoB_q9D2<7 zNviz!jc+dSjWhf$R#!+5engr3kKc$6k^N%piOvwt!R` zeM;%+$n?Cpj0Vr_xt)>+F2+&b3oL}rRpN-O9YbF+V(i66XjM7a5rzWCZMH&}S*86S zPUg^x#Iv)!dAg#7Et=S&NJ!PzwN_pon|3dBZ`rjrC6Q100mptM#h&$ZtlKW1LJRO& zR5;kW$)e`32FDX#RL#CmLH<3dgq21^RqZzMrx692ML@Ows2x;hX!6X zp8KDM1F}T)o$mnu$u&2WIJn;v>se

Ft9% zYghWw^*wd7WWU{JWV^0Qq*e{AN4Vcn!K2BTebp9M61FrfV6LY5xUMt^ZD9ykN>a~k zYvW1OSojlvPnl8P;z2?)@f#O*ET`z7(dXDN7g=nI82u?w(9f>7x{#+DMt{n#JidKq ziAn^$zJ- zxtfx?N!R)xvbPBa?mZnNq#nUuad{ppC=E9F@CL&CGaiLCdj{hZ1bGwP@k#{2<>T>S zGr?KMrmNM<77{XJ(A(xZX?|ks)gw7k1sS2GQ#@xKtmN>a9*xY zkLmf)j<5~!OX{~9ibbEaN;~BfdPcagz@c6?cbp>{pJNLtsU*(p!3In0%Gz$d&;tqS zCq=Fq-fQT>{euV(G-!3-PN}rj4w-tdQP1H^yWNMuX4OQ^Bx}SPnI2a(A)F}gLi|#Q z!zK+83fnuXSC)k)Er!3R>IRNReF&=~HLwL=+)on7O!#jjviB0vh-D!#rOHEP(g^mcT&i+AbQ;ElFC z=Psv&c~WSr7W-y-oh7-O2C0cHl`}eyGaJHR9PxUQS4{4dCw9f{_tz zbF=htxFPSi&9vww=LFMI;bh?ZW-sc20)ka;5y@0@@Izj?pD365(;&F@0^A1WWA0H% zHcflLNOTIr*9FrCN=d?BbO<)N>7g!y-zM4AJd)jcQQ<=~f~Be|=@_`g%3E+#ytIE@ zV(PC(X_bgd2F(yRCjTQ`a@VXuPmy-P!tD)BB_W|bju*sN`%`dSik1{22n2|q8^(Xug1^LIc?SV`H7V?$y6*(Bk@*GV~x zqePCW9!2zF3bt7uk9&dF}wXbAi%2EsoKl`CwCxuG0ow zP&K43egBzEDoEKJBDn3o@RUs4(U)5w7=vTJs56$2-HeA0d(uTbCk{d7$J^}+LjMzt@=!{Xp z>nxx`@$%!u`}i(yOFKu%7@MwBHPx1!@jZtb(#MP^HGAEf+d1{?h(6imTZ-$qgl*02 zC~8j6KNVugIo}dAo4xMMIcC<5AFrL=nN_>}00G|Y^>A)ycI|mZEIr^#ac$xOf>(p< zfUYb19(lZp5Yg=QB-4T)bT$-x`m0CHA&ynhdfCrz^XJcnx^&(2ws-Z4xIVk>evd0~ zSM4EQbzv>$5TV1uoiGrfdVnyplQ7ooRUKg85PP58t>Me{`gP9H?`o*#BR7R(uBwUF zZTHE+H)>;y^(d8x2pTP3Z)`?;9*xak3ym-o3(9I$G!uGq-3fM$u7vT1jj90ZOq^5- zjAQ4FTiEouYGQb63&kTe&CXT9j8Si|L1<}VbLR}nvp5-hxMet1)!wZCCKZMr1{5Xs zm~Xd5WT*5%6gZIc@j%q(=+I7=gco(!FBB^%sV3iOe0PPvH0F;E-F@_+z_}|TVZNfU z3}m?Xz12!ZjnpTdH$@SbLQRHlB4r0Bztfn0-ae56@ZCY*edskz-}k)*_z3{HmU&C; zQgJ~edj1463Ln!YN$+IIBJd>u>7{y5 z>27ZaGxxwOKOC2Hg-=EaLO&2lbEFt~>(g#`Z54ms^`ED&$1CUc9%RK-vVsCbO~_TX zbw0J7GM($mz?wEzaay@_~Fd$qYX&(DAdx-T4NJ<$DM68~9T&}(HK zW^A|tMZ^YEraW9Gf{|6*ecfJDB$qK1Ca_AVmPGJaxNHL{Nmnq-w19*(;d1O44693F z)bkfr-l%eZ6fT(Ydth%QqDY|!qg^7>=0`{jBxKH5>@0n52cII@Bk5x>8IoWz*LPeZ z9uO|pJ&;ThxQyijA)E4sBIXJgy!c1zk{KKuE`gFN8B##Gz2=0tiOa%JAxwrgLO|L} zVKC>w@8MxZeZpmYlPZ(O7x^@+N*aW0r8&GC9vk5;7J1DryCg6;cZ7>t?UXuQ316Kr z(rTt)FNx>*{KfVbEXGy>Sb=EEWMv!1hSQW^>D`Nw7Bj z)M4ysFL(%ZeOzd-+Uq}*pULG=U=+sk92e~?4&$UB!HzxPk5EvJ5oTN4+u6?_FBcoC zNyw2Qy&=3O!bS1T@w&a6rjGkwp$IX{ZK*_0SMk#F)L~zqH-C1qjq`um`|_}+uJzsY zc&evIk<->XfQqe%S`kzT$QV7;0U@HGAY)J@3}Gm;)z7`rfD^-T%Jrq&2< zSL(yWdB|~C`M_aeLxstEZ zFmZQbc%itH(Ssh-Y~5@pZtUaP^KX(l*V>&X!v(k!QHueN)JN`-jNk+z5)F|=iBat` z#SDoM3nfr=)SJgw%gKj!Y*wHsoEqY2B4Q4yZWi(HbqFKxg9NzN)o8nYnuWobqzgu% z8IXDTW8(VEJoKB8_2k!I8B7FqYf=th=I=xEylcrt>}PpsQa;b1(nW-xMF{fw+zds# zN#!BtvxzbCXlY|>OuC>zXfeb*#xFY?6_teBIm9&Omsv(tC*ib)m~eiXRg^dhXEMY* z%P%_z-UjrzbtI#yH7{Low~P&N9!Svm`FV`u+8&$c zen0VxyXk^k%@mjmZ%^rI4Va4CDAAKJ#qqC?Jl>y)ZfR*E#1Av$DiGo{0h4ENKw>T# z*v2G8)Hg?3Z%;tux0lx%?~Gv*o_+2@$QdT+ACic}8(T+!NpB^w+;}ez@|o;49NIe* zK65svsdW|(t2l0)z?jGyf*y$?$1&s3mZ9EfoOUMB{>c}$yif@_D(3Fy^c>@hrcgKH zbNIBTDMBg}hDetUdfyVX9B%T!p9Bq1a@LBVvr$e5;763WHDPPKqsO_ z6yYQ}S<=rKVn_r+HX1mO(@_Eoek(bZF(DSZNfOW^k?={>9f=@@FN(S=5&R&GQUvbm zuw?F)kQp}=hbWwyZ5Q$y*{>c<;b4gj!a99p(EUw5jn8FwvAfXw;^G++@UHAsS{a}8 zwhLAL^AMx3a=QlMX8qC!86oBHEm7%bS6kIzyL^HS1%bnbPp-YVJqYpYr32ZfaI2m% z*vH?Yc*{o|bS6Z%r24kP)7XAnYG099CuMPg0Rq8L_=%dm+Z0;3-h7UC#K`?01Yuiw zKesF4^cMmO1TvlaiSj@5wf}XYpTL^n23%8JaO)7rI~XxeBvfAGe$wBvHhM^e6-SY+ zja%CrdH)`;TC8!0ME7gyY_Prdw%4*eVTy+_ZrO3!{foucqsMh?OyV`6eHY2XgRA|{ z_;c357gh-&ke8v?zR}%>Rv1wlMMXFc)Nd))iS*rqSy+66K<<&&9rMW%a@`}Ydk(R; z$~UXO`9^++N35oeSyCKh`bxKIyuL@U2{jo7cIR#JjpTpBXG3=_trhHYs=$VEJ55;^{Re@z7NA?F^+m zTX}6{k7=QHr`+v81C#ju>S(w!Vz}@QF0seAR6|a#4CJ07@1UU?hfpVYt5t!AmwQpW z&F70#TgW@qwQme8Z4hT48AkR%e5|p+M13Sa^*lO05C{Kc%L-*xiCtfF&R@^cNa9v$sBa;8+!*a z<~BOE$N2YDm5Rw$!!zBY$lfbmlV4NXx1QKfS^n|kAWa2!@==Xi2QGD1`iL@OD%GQh zW(Yey@bwx?u~IQ}rl)6CpRBD-FJvA{i;avd4X#`oL!?iHGuYN8c+{M1&z*#BlPKv{ z5B7B6(W`8_n;L2NS?snO-=YiZZ`anHVIa{6E^=8XbM)Do%=<4kYX5xy1=2;L^Qyj3 zb5#O%FCq?4uqc3Y1^3(|G&f_syq7V*mzw+r_vGTfRqcivO=K+r~=f<(E8?33SQKDn=H63?amc>l%8F-dSNvv>Br*Rg%oSkmk1Z0DSBI?Q^Y*7B$r)2a7;*Ni^e8cPF5DJ6h$wXtkQ$eUX{nMn|Xi-zZ=BBV$7$uX$oVwJli!4i9 z2UwqPChZEG)o@2CHPQ$ywjN7Kzzm*5>Idq7EMz6Ai?m^V! z2`KIq88J{BNUA$hc)!7#!~$>HTXZ4O&s_Hu{hUnjDC}AD%}y&()Vt@cBrNpWKWYiw zzjFuAI1F&0+Vt*ag1sa+{;REPN5a?HbL=-jH;LO@*omzb_V71X#%_N7w2pWGegkz` zk!Ova&=$^gLLhU| zK=;&ollb@>43okmwBtkUzG*I_=o2M7AWn*$IC_B3UC|21bn@OCxgIH6)kv#iG47ZB zAPPCS@`w7eA1WTxK}8UrBl3hsT~~DVYsm1RA@-|1b9Nqc%P=~RtUFe6z$Ct>m~_+- zK^Q+UGx25q+E^HJ-xTe|%gg(Erj0z75xn4XfO6*YzC#=IJTwKy(@LTAp#i`25o4CtiRt@w8H?$XCtni}{ryr6UcBzu-@K78 ztDY~9>75T4f7<<%_nLs!m@Si8|M{8AYqd6fHoe&-(Ys;E_UHL}?VBHrIc+Zo@oz~M z?cSqow%cM{s@WjrGr0eWi+j=j?RPdsoAIhuz0TNo=vRc=$0E^XRm$pNw)@pjJ>8NazMeCMiA`- z82lVG{Z2nLQU}=gO8dEI1ax`WBpzb4o4}AZe1CD@=id2+JCVBy6D#gGKU%o6%59YS z2=Li;N!zmVwVY9fz=2!XwKRqUzbrgq-ezW!h`oWXb}wH7B|7RdurX*g9*p2&;{Apa z?HMkYvv4F464pHL=`Qd2B6|I1L79(xQRnibKHA2Q-S!lH3dVQOYzSus9;;n?4p1TiSZai1YEI^e5_oiBp znj{wFOM_5E!-4~k{%xKako&@bHo*F&xL9*fsxa_eP0}DPYXx$1$@Bd|>@JOxZptGU z?Id+H-Z~rTv;M0f-p=x@xhL4HBGLfSvwAfR45U zypTPoxkaATX@>J8gsf9tL2CO2xgqxPh<(5eT(M3)N{b26X6<PHRzGL!)ye;;b3S+U*wG`dcS18yq+dSndM^H~IowK%!*XLB-c+kh zR{mYj4mj#J5?7};DES@; zqK3jVH1!sO#y}1J#V1HBRmcH62HI#&jh1`%RT_N*-LRN@q*6lP@EN2yN%E#Sn&`0* z5S$Nq-Pr1^bL?-BFB|a<_b<0EWIIPn7q455JclQO1*2UBD4oUd*6eCE2qEtM0-KWj zf`#q5TRXAethpe<=pYb^H$H*XSvV5t+Ila%7BzUyJGeFz9>1y)e-aW+eZ($H#;O%0 zu1SP1B(F!v^bLBiy>9#+l6E1e%B_8H2iptdHL^r_YBNC14WL%j@rZpCtF|V6@yQwo zonz|{W54+Xl2+fW--~Z}=DG-sxe5^KE<7G&`Y+@GGyh`o`nq4>i6%?BbN_Pj`mGl? zv1&Fcv^+Z+vbvfonAhC_HKgt+kTY5z(M~7$F{N?9<>D|C0_4e9l;?msQF#U^nD2E7_7_R;YezHPlcgadVp(%gb#)nXDBN4YKzwC0rzkI&H51=E>t8RvS4q~ri&zoY<@gPi~+a2hFTjJAuIlQanR1cgZf}8Wa++3kR5PZlmp^Y zC?37^A?r4)0zO<|zDbpi<%m5L*4{kmR`+bt@Q$DX7i73W)b8T;0;VmHxi395y3_FT zBqUE}#t?HodFY5so#w~OvClbdY48_zYx-auY{hglc6T$4)>NI4aqYlAi}2BZqof$IcS56(+m418gY zjIcFk4`HzGWseZe0>VX0-4Y}^=p4FQ`sgAJ5^EQ19RgXU!wIMYI&E4|x4R;cT4vFZ zTWNCOZCaZU)rS=5T`~G9e+cK@)mn*_wr9Rj&_Zj45hfhVIsyWBo_rf5F8M5hkpMY1pf zfjo4oloS<3s2p7a5HRzl)IUHV)E@!r9>Z+5vFwmc%}+}%2{Qe|vO=Owofo+W+59)` zvg7cUyZ+@(5MI3YO%yCB{($GwsG?IY0O6b$$m)D}$z&j3UV1f(H|5<~y%;(~QWXSf zNmI78FDOQL(tBN$%rgY$OP=pK;Hb>+L87-%j1tm&uPT`@_=kLL;@+Y%kCAJE5*ZaC z&IPk7;~<*ILSeMWvUcxj%7wx)x23^3?~%Jc(7Na>AvG7S__Yr~J4n71YSPQS4JjlWk?>D*#Q;hg@AxsbiIg zEDb~%U|`ZH1NtQuy9c*t3j!BYjkJg*bAh}|JO(5wX~8$(TjpoB?#iIyc_V|YYxoAR z7CTSq17e_u@1=1KxOO$<_$pw~3*?G^1%@zh>X0L5fOJXoq>Dbf~p$$@_JrA9c);x5|(y=*jmy4?4H#d>vCZP9)xawR^ z0Yn*J+l6fgfIDVNc4TXW#UEqEh`#Q z*Z3;&X!f@{+E#1k z26+P$t=0g)Jz};zILNi-G^pnAGUp(-K40!A>+%x0A@1oiZY1;Gddm{I%^Q}-T=SFd zGE&ZcOTh~Q!8gm0(oAUU1Vr`)N4I&~ z@(3UWQ*ZGN$wT2}S(Z1RA+3E{wQTbc=UtO_S8BCMn<~L1?jP1hst?OXp{eJ^jcNA%KJV;7FsS!153HL5x~%89p&g&T{l1NSZB># z^QtrI4C} z^%K|j%ZSP=kyLcBcqb5?)z1G!O9Ck`$`!n62W{Gr@$-%Wa=;XsxeovYY;&~jE&-=| z0t_49YA3qqBr#{kS>)3d!nF{Jx|(wc5vC<7)&SuzlP1kwt92W2et|mh!lO*3TZqyY z>AS?!95j9Y(G1&MV9l&mtD8sY40Aaz0V94yF-(2uH6q9pjEuN}JhOGT^+<$K8!flt zIGB|=`v*n+!veOBV&8*K^D>sClS|?8qE=Zh00!ON9B{J*UrjEbD2&qS#H!KWrl-Xw z02edp5!IE>scr41K2k|oNc`u6ASAjP8(R2^-odls4RyBSfeGIw3nJ)yfcrAPJsfja z4>p}28SW4e$~@rW0VX~Q@vB}e^tbu?u?%=5I*8#U^17r8Z!y$uLNhy+Im~DP0xCxw zO>BW330V)__(6|Pih=DuE14>5{|-!y#M{9awd`*iZUU~@E)Svn*$MvD6=@7`5~pM| z+uyHUw!XDPIOh#=1*6rEbH8_oZ2;bq%>iF~8%Eh-MM@zH>DrN|{g^&&7QfNijvti4 zmtJ>a6O1RuiDy9+*^JgwUdK^{R@tE6v$s}HNZlBPRxF*#O8?AL=UoMY_M}If{($+D z&M6?^o2+zsTN7VvCsH2`Qm1bK>G`UxcZl~T829@j9#O8gZ}Tk$c#U_SPPA>udT zB>C#F`j_QY8ay!r?Zz?`y^wKecD3;8X|H5X_XfRR7Dxc8y91ZyLOtw&Ut8)Q49x6P z-UZr!v*r0}Q-vL}>-Se(vGZXZ*n*`@r<8z)AY8YobqblQlb8sA3nM7%<#pIk?&gcX z+#wBPs>3v2;M3SX!)U(i)xj$zjgo&8IE48rdo<~#p-Xy0z~d6PETF=d6=y4<)ztYU z7<9ermEqO&27H4KQg*j$IOe(@%)z~DK6?T2{|ky1CLK`GOr~fQf&cZT7R9fWjxTrt z$gP|fUy#80-_4Ww7`g#5oGeA3{IH0`wex9f;mF zBoiHdkYj>X=~{CA5G$5!g`+h=`<+)3NJyQ8pX?XCyyvs1!e}sWjCYlaY)%}G`B4w1 z`VhOQuX}4iqVvS4u&k2f3f4;TrlKEe6dn4k(b1k!&Lml;1bUCM2i+gx+yXUo_>-cq zYZPrzi3<)Ha_faO8;VDeL3mCq)VS;D&59cBGvw(_9sqDu`k?7(pUQT$`#ciJ6mZ%4MefBrzW`AzbOX z?(qris~kcM_3MCBB6lREBR>NJJ?Cgv;9vzsyv#`!^GH%m^l_l}0cBG>! zDGRxun%{sS*%FVl)#u+X8GVrGj_E(|ABWQL)wvcd9)TPXQ^N;n;+UV!Qpw?O5?tLn z(AAGeu7;X=22fTX#j1KIB&p?Z(npnq=%Svs_=7bb@VEI%Zs_g?atnd6+BrfbN^q5k zUCmO*s=sfTy2&)WlA3+JfPR2JJ*SAnZG8WwEhhwySZn&FrDGEmRgUzg*$&G>e>2;+ z)M$__vhdaSde&5eDCuiT-A*;sH1+~Wx^)&3B)kj~u2UkQ0U!mAA||;`kl=|n`wfi` z(lE-yUc)fK{?gY40mx*#vyPet4!WIOE5{R1b}rJfGoLV`pJ(d}Wfvkvo%yx;W`|af zxq^<`KQ&85@0OS)X0tqJGSqfQwqKsBFvC=0p#4oHf^K|CF_Qns(9p?0w4EZ~R+LEKZuR^Y@{x*Ngi5v?AZ3N`L<}N*N_~ zVol_w7N(m}4E2XQP;SFk7<(d=v!7z*(TO2h5|N(N(-fsgwq{V&Bcyj{{u2zznmFu9 zbzEibmt7k7W|LjIi(MmuLfxJ+L54{=3v_ zn;E^SHIrYHJG?7K15Q;AV~n3NwhaB4H*p=ClzmpG!mo&eRr5|5MNsZ~4xh)kz$tgl zI2I#0RdI&{ko#w5EO6&^7~HOOs9pK^us2fS%DWQ=P5+h(?fmck7p6MVG@;^yP3pE$ z7b|V{c)W(s`4(cmJK^y<$4|9E?21Nz{Mk%V;7+V&t8*_Terl!|ju8FUNABL_pXtQ8 zoz-YRGaE)1H9e?{nYM(1PYph)4jg^%r#eR3h9#*Ix{jS z(x$+25l|EfNulQVwLuSC@N`kNEr&jrij?-RwTxn+!rD{~1HIlale7ohw8~8%(*H%T zG{Y;zCurXo(I*u0^vz(_+@f7g;wS($ZCJAxQTnPUZjY~^L$~QcwZC1&KQy_-44$E$ zQ;g0@&o4Y{3GYthN~r~Wol`m-4a(761{C!axu2HLY9k#kMaFjJe`U~gT-yJ=oQ)c` zGx49Ar7-S}uPTHS`dJT)=zxj4Dl#tgPXZobGB%LdECoA!qoZN>fUKIYkfd6y9mWltft;D;v$ zbUQsZK#332P3!N5jWypoMMRY#>vHJ36)KQb9r;)c*NS-ClWIjNe`Je*vh$Huo%t0Q zu1$Fg4WAx_aFhh4{HP9`*#K)Tv~+kaIS;Ys(boWq+YV!PUQw>zb9K8(Relkrrmz?5 z@Y8vv*wEDBd)hBBBu8Q$Hr%Lg1oMO#0U{qr{4PIh8&a`*97xxE8%YkelkAU4rGyxP zX#c(F)hfV2V`1}^xU-3!8iKoD5MuDd^tw-N7+>PTAC`7cE5EZY04jf ziRk@jz`FH}#AbI^)5q6jDiKU?`ogVo!0m9vQq@NpJKL8C8U z*?d6QGmPYD#CryCMAyi)(TH<*f3U=QQc;xhg1iun7xibe_2Cw7xreuHRM(p>Sev3Q zfD3;N?4G+?pKP&JD4P8_<=U&;xY;^Snk$F;f`!Fu@0Vs`K6lMw52q`JV=|{h-TTaZ zx&6&m+KM?BJM%>t=<8loKqm%znV2%FzEA4KqYrYXcAZdn0B-Vpz{FUN&K36OLj^b? zl0W!$=30wzgNac8%Hv*lwmPury$^|dEfc8`%Db@XsmS{sb;kVUiPn53Cdje8-^2#W ze%{jN_wO-cE4sv)HI-G15I@(6YW?TL<(wz(lL8wF7Y9h@Ev!?62V4b|z^bmHJ$c|* z(R~Xw?1YJFdyJy~NWpwR4lVbjV2W(*gRRr0kk)Lz@c1&_hZWV|tJ5>oRn*kaxXxPx z&G4Qs*=bX88UtpNnSSnlD4iX6w}9Hh(2ZS&ZpB|)COXV@_E4J5h-W;*DRYIkOiI-- zpz#ypPj}VVmfDN2&26d-7)M;`>@TXP4Qx82bd1o)f7e;~%B-+LH_9q;I(1G2l)gvF zoMMPI7{HXJ4&T>)gdx4@We1)P(Lpu0_$EK)QBfGRiYPD_eJX0piTv)wQuqq^GbWU-wW|-9u$fXW^|}eO>{>) z4HK#pQU?vtq?mXlMP89iwB&!qbRy)@QtpooIdQ zTtrjos0XdiO?kwEmEK$4Vc3eA(ulV#O!o+2I%o;n%#+lfAdR~6*D|ZuTNl{c=&+kw z{!Mh0dC}=q&$$X+73+S%&%Vz!Z)va3Ve3R{C3fJ3!)5%l!2|$X9G>K8+}S;VW47P{{!~)uu-K+3bl|nUO9)qE~SWZ&&zU}*Chw8t%D|g!`>hp zt@h!TGUQB7`X0DHW$ycN)Ehz`yQ#coR9&y<;RVb*vey z8_sA}RlpCgc?4GDqMj<5d$i<)?uoUcC>7b7;n}6RwpW8wvZV6VKq^7PYLFcsK=huT z+1VJwllWAUcjr)hE^@=GrL>4BxgX@~9o62`i(v*)>O+ycxz=(rg;DgM*lM;!!t=YE zQq{u)+*o@3)ZzpEQ>*HeSN*o>h*D`A7h9cfYH4p4paH1V>q$+ZU=BIf!Zi{8UF$Pk z_e#Zhc>)ARG_P0SmsdiGiPMzo{R+YYLkQfT!&Dlv&?K3*ReX2q>Rw8@_Ln&x4{bGd z*mcw87ji;EQtl0~TzmG1kAZw!q8IU23tcw7DYfoNpuz*+%LyISd4kY}9S_q;?nJdQ zbqkq)GQJ>9PU-}HByxEB)KtG_gh79OqSK>5xU54|_l9(FWwtm8>4>@BrrnvZf(^PH z7MVZ#GA=mf*MXrSGey2+|Aivqc7ZD7yjanC`S2P6;a}YHzFh)NH-Mj>rUT zrQ0z&`j~j_sR4)b%y!*Frk<#I3WZbbh_(TERJm9OdR)3F< zit*R|gv9dwjtExSHC0lXHXE1{z6=?e?z9i|Vw)PA@-T}C966*7L?>;jP)! zgu1FgxEy?*CKL>((_q^c0vlP-ZCfI9Hz=TK4M`6$YRRF#fS#z*F9#y^b5q-(DVg9c zRai5k2IV`CZWJ}0#cj(s!*V^`a`hwD_D^<7!azQyh_~?Jt+I$kZ^>22LFq^COw2Ax z8Fs+LSlWA1Gg;+6)JGHyi5MY6kNO+P+zZWlJtgATev{(69VnK**)nw5DxKWn$w_E? zzJ-BV=NpjW^J#6nN$n3TZlFKQ-JipjZywvt(%M~*s<0&nrx2u=)tCWQ@>AO*OW=$U zb3NzhiUU=8xwJNt+&D~O7+YVTxqg?oZ2ZhnOG401tf>y$*`Jt5%?Ydm1W)J~1^{WDZELK4 zENo8ax`mBZMzHSN2c;C&K5=fCY9EgJ0?iw)ZOn|x+zt!Pg;T0Zux-mcAQd3@cjucj zofwpE67iM{u^-8OGrvoM!DJDJ3->MXkV4HH$PU`?=qtyB>C|A@yN=mz4{9EzoJ9O- zTRz}i-#3w87(@xSFZ= zeo7h0T}r6=aGQ!2wo(c4u51X_lOfVcrzfx7Q--EQjhh$PCy&<%l>KU2$&*UR$QgA8 zuJa=M!Iu+7ywZW)-;$mL#>y)MMd{XdWK!%(i21U{^`+9(^@e+($+d5W0-9QW;W4tK zpuBRFoJnSPqNrJv@<0;ptRp!%rReGtXI9YwGjm#D!s&CEX>I|fs&0iM%8;@8Jbhnk zQHJY&DYee*cPR2=m+@X@i?3oM#PV)`%2xG^9XQtqeW%BmrkPFS#OPWa~YJX8BALMcoAWM2NlF7 z4>5+z>ZP3}-X2r;5TKs5xrVnG(N`O6KfC@yJWNk2cgu%zmy2)bk-2icJgV|)xUWp9 zLQ{)n*;sFORB9nvPP-T~G7{rtMhPn+7LVaTB3t1)bI)6;&P`_`eC^%(E;EI>aqqX3JJA zSF@DJsPRamLVwtImGx>0kIHDyR$+zU6^Rk&zvuzujZI~V9 zVOqWj>64N{H^Abt;D-lw7RNoq0|;FFbgX$ox1B8SNjyVDW`tMiqIy~m$v#7YZ((TK z&P*`-wNP_XBzaxgN0SR{u=XLt^)$s%-~?Qz`)DkOSv8svC_)227+@jc1&w4@b4_z=m(rk=JW?XTCkN8js<6 zyXE#&q!1N9~O~B>2ua*xeVZi~6EXRht*)dy4TwHMbx~ zPdby>Tg47w*on#ssNNR;dGPDdy>A{%n7!ey?6L1JR1(w_%FAe%F;#HCvPVh?P<^RI#%s#5iv-6U7X(WOz zxp1cY5xch~p;VWK;V)aag&i0EJ=@eBkUVMGU_C57sfBq6UeorG z1@t{ZG!YsEOe9wc0-|o-JKo-lda~Dub|+gWRhsxEP^*z5eO^#!et>~le}Bg+Ln!ZF z_%<*n`Rruk%U*WX-sL2sExRM%k?F*tD3ua>=iLp6q(x0Y_;4ok^zDx#GvOGtqJ5#q z_)#8gE@tw}J8JjKXyNky%rw`wqNQpW3-xg;n|BH>(pZ}wP8dy$xw~;9&-AL#IH@a1 z-sK>OQI2Dz^R%*zUB1572G094956|3P@Ehl38!e0at#i_u;Dl_u&fxz>-cY~%-i-j z<+KL;6boY^M?upZcJ(p>tfB|Z{@p!-_gVu+|5bgxt8{V_%@QrkhviZLxFxH6LSA33 z0TYR1=)md+A7iqBa3ONrt9gAj8#!C)muc_KK^e>GNj14puxJ~{%PmsIADGteag395 zb{&o^Qm5_IjM}yX85jNxubq9n&3NtE7n2I>p7--9Xax@F*mR;WWnb`+aiks zVQ{76tT1!Aw0Qg!p+IWotZ3HnZP`hMeEGO=u6*2}N%pp@G$od?Z8=U2Bq&Ds<#g0C zB!_10*3g3utO>m2VR?!cF?Sq4!_BZlUohsUYbMbuXB4zLKQFZ-N*p;%3=DT$D;J7p zrq0c2!`U>qgj2F9rj*ec9`ljme=gcw_5cMEaX9_Ws4odSv1Mo!t?NxLE948RJ7~xf58<4-oh-Y{t!P8# zwNUzV3s7mPwfgv;7!$0|-uU@$Ix~>KL-Y$2jJqQT-^b?!~!Mgi36oY=SB^ z`SVPd33=MrWu5iD<7J19jg|J*tE%^i@M?$Tv|XtjqO$Mhe1K@IZ|#-!WhJd!>pc<& z58ztY&rSRqq$$hNq_*8z)>R?T?^n8yDCoQtW5Rhel|-e%Yrh>8YL-%UZca_A%VsI2 z-#YSwPD_)8nscV|CjasZ)sT_z;<~4RLhSJCmSXY3Fmc<2Y-T4jrHt!UW^!IcDKhB0 z*vje90=v1ij%=Rg8JM@GP8N{(53K<|Z8zXIkI(PR=8L6!q@m zp6ODZAc0p=XyIXjx=yk&vc#`XwM?Vcoi*{FJUB%^Rrm3In}{&|LdI@b1MF z;|J%6HEwVbi) zt)%xG8s)bVkC!)lNP8ola!b4tyiVaHm?Beu;eQ)xc^EyL*2>JJJh3(kp3@@6wyQNn z>F`kREpuGli{)6hW+lsnBPT~51>U-tYD`FiCnzD8U8~l|TSY$JsyHk5%oNslDU>-v(gSliAv> zT2>Ue?$1=;Gurcz;LX~cwI|X?s)y0w&-;^ZXT6dw=(S06Jbl>$xAUVFd5*hwXZ)-4 ztj)Dvmb#|FoV%59a@F^kQ__XQj~d+5PQPL*bnWZFTW6Du$>;1h=J)wnlm4aBx@>05 zkGU}gvJbkV_pl)jthpwzI5>h|*y%#DY&AJSnxT!Tk2>l*>pxjYPTSvm4tx_)GR@EF zE=oTTIgWY+hdG;DEkhOXFBf)~*8oSc?FFj=QQpM%2w6^j>h9RKgU8PUsRydR zC;bbTF9-d{!Y|+)qoz)+A&9__z_gNk+=_^y9L(hWGKnkzr+;A6u8D*0>=`x#@Mm)G zvcHy9qJ9{2zh?(04nFmIsmz2YRQW_XV8I@yL{9vhD=o`O$MiN(Yhms%!$$vhogCol zAB(lM6txC1wMn@xA09gKxAqo%?WvT}PsT_NkRl3XZZek^lT(8a0>>|wg5I`y7PsUZ^`-O0=R z$p3&Jp6q{D?v0Gf_3KfktLL{f%kb)TJy!GUMN*!Oev~IvsvEH{qTjnuRZq9ot>^4= zB0zkyA`z$sxyc*k)xp73>4N^e{gb7x>qGm0N>`;bR+pLj>$G~-SI7ZzX(bRgrEBZ` z?Ea2#)r4wbRm15!^lp>CbwFt>)8pF5!uj_^(;~8hOXB6k*n|Q5WpO{qTWa%qV#6u_ z*4pN_AqVuZURlBW&fQEoV#{8u{6T6Y!y7TTb#**w+jhBa;>w&z3dJ+GtAl~b%TP4^ z0Tg;dOzVSOzJYeDb33!LB({eE&!n8X`Ymu!TURRZ`@8}70_6vka2Cfbb}FFcsQ;W? z#B-Cjf8hLz*_L*1IZ&GhcB;G5)Zd4wtk%w~*2!`443vv?t5(LkUcl()Y?oWVPM75z zC$V0xZeYpf!1#{>&Yt04B&KU_bL)b)sWs5ULQ{IQH6*8ti{y0sLE2znidC(5eUoZj z+ObDxd-o2#D!G6sJHr2A!1DoSY*XcvWM1FqE@zZt?uzW}X%O+Dw^Ax4f)!wmTt0_I($e=vJb9FM|nfL-!;0bFGz^caZu>`l^S3oR$3QA(|4!w$aBQp zUCY|8U)--|Bjtb!_$W~2ks1c;TQ}?mvjSz^LQhw@oP)d~=ZMAYr<4~mZuZhYo#|Ge zVMb|iptn@EwuRD?6cQ5cmt9UisJH!((DdruiQ_i0VFUN!3Fr)X!hqGXoqeDR;3uCm zyhIoNjt{wHj`rys7W4T^)7O6YA^Yf@JRQ6oyE(Gz$PcE#$3dE-d3sK*VVZZr3b4!K zkH=G}j|qEZo5m}8?QM*#R{5LL$Q720Hg(O9G|vwO4`}<&G(%ccr@kUu>EEoNoafbU1rY4RF=zX`r(N6 zdzCfo^Ol1SUsh=K!D>ZT-tE{zHgC9+?pC77t$c!Ydm$2UUE|0Ju>kx?2ASF)Q_!95 z`t{*o>OkH>oJDYj6lSWpcMnET<`F1IcD0X@-Lc9zajZVz+4Nm4Q5n6?HSsdJ^;Hx3 zYHIhHM{ne^mvt+h+q=A-o`bPHY7J#;`+cG|Zw{0xL|QW;Tf-oNBMYF=dF2|$arm1rA>p+m5+lrCo(>UCsqeTq?^vQ3RLEf@5?RZ5~S7WCsd1#XE z!Bwo=(P1}`xOPtt3}WKz}(%G?90ePA)eWM7QP zpj9+WCzuehJKeD14)R&F+e%MZmxa84;Y8<)Q@Hj=IdNMqgWTU+&a1No+Dh>Y{;Jqo0<+6s}+luO#C@%BdI0WogT7r`(O)b1WmRd}@Z_P9S z(~M#2H22jzIiLQ)j5`9QGtfB+eOzwquzoHe-JbY9Q=pd()4kEg#J@^fj<8rMRv#uC z5lY75JU71#U+E9MI65KwB**Hd&h(_@z*Peo3 zTJ~50vSQ~C2wz9M=yNI}7vzQmk4=j_;H|$3`nn?=SIjtKS7g77)i-~WI*|DlFHopFzULWOXpRWI6j5y1a*4+QOGEj%5r<&+tLaV^^+g zX3-fVHYX_hx&nJiTPU>yKw3ztIIM%SAoss1(nAE~2X2oIk#oCEE5gWP$;*oEhZS?l zhmlY$AE^s3^cWZ%MZ+rg`X?~9`OQp}j?mO1;TUUCxpvM#u8~~2W3nRMS=zLd-!b`X zPvC#BC-6T||6*Z+Z0Qq~NF`ej`G0CF=r6T%&it`2 zUD~vkuFw4cv;+8;+W%7f$7B3oPX4-VHTf@VUs_53KacIyo;UA13;FirVTYr%Kr@}q z#4@cy*{Phwj$U?k?qX4ke2P}GRDkzE*#>PqV%n1?1PAq$$)NhZ$8(t8yM-n@GGppF z9>nPQSQQ8O=;cD;(MliQ_Liqo(G5jQ4=ng16FN$6>v6j~kq%lsQ2DKvTSkktS}fs_ zFG=xQDvSM~B*h@=t#Y0tM|zx4hy?C6FWoJ`?ax{(?q4562sY~Xr`1mKK~?t?B%YEH zKpB;7C{ZH4p>hQ)eTs%bpnQ%cJ;=xCP|e*B4N#|)BFE=v)7?v@Jc~((Qw8T$o?~39 zLx^ug9e^```2kg7U6Se%Z(6raM7i!oMbG0oS%Io%Bs_R;8 zd?Q)X)md2YVo{-EJOI!QZqB6@>dc=$$sWfDx;>XGbu7W5q$y9DsF{QweLks-RRv45 zU|X*sJ;Y=EsYL5?@qZMrY=ce4g4P{*dPc@f&^oobxGbPBJrWcmj}(>>3h>LNFkDPV zoZVPAu1xHml+;l)+_QrZir>Ft;iLY9?5rf__cqkM4`}g zt4>>hqN%49OV3-2)25!(5gC9$$bmgNhBNjcvC*BF(&NV94C&IL)7e;yglp%uVlbhB zQbLycAV{T=b6CG|(?q+lZGj&Y2wBggI@lb7 z;=i;BR{+^3YS<1F(x#f~wDv=HpKSdxeD~}dYi8IIJcka&>K4V%A;JolbcoLZCpyc#r034U9Z$9$rZ(qEPn4>`tytVwAGnDPGMFli zcjb;jiTf#rrflo5s^yjnfbo$=XRrGvJ)eM;R0cD%n!*gWR(GS0NreC!>uAnF8bbrV zrD%F6nGw=InEYgo0L4r9e&r9+SFB z-xk0n?UL$a3^Qgt^_*5M#x@0c7aX8*3SYF}U8V`ye)|hNJ}{$L6X{_aI` z%d1mccIIHxJDUr&+f})x(@ObT6|z6%uUPq4s_<8)^jD(zSDw7Q|NPf9;s4q+A#-}Y!{k-}_wf%>v#O`KQxwtq zeGCRC+JB~=r@kp_dSGbE(fCYZ=!Bp#Xd*PEOzb$Ju3IT^9K!G_OC7jk$HrYMLU;b; z37BovY(rIJ$c%pW){xgvhF%Ua_O96$6vm%A2)o5h(6;S7+s$hWRW;?LQj3E!Wyd8K z&WC`l>!1#9Rp55hHh1uLbbX?1+u6`GGW2;|Nt`Ott5PhvC^Bm>Y5!S`EOws9wPiWv?b#$JnjlKN6BSXa?oE zWEMi}Q95VDh4hoQl8O3~2* zMvlFoNbw!xlKfd~V0PPkgU^Qg6JwG6TZ%$Ay%@%!Dn9dfM76Y6?sMg|B*H0bjrFBh z;APxx1}Q30o`j;euLqh^ZDu=rE$#ENiYIb7CliJ>3q06{(vsiB0_zbp;xNB{$UCIh zp+qP$ac$Z(u4ylv$t-%VvVBjFVNeMF_0KkSaq%6IA0bFD##;C||M2Wj{*y5<>+!ye z6;tQjtS%Ex{Vrg?K|D$i!8#O+n+@PP6PNfebZ#~(Omvq;@HhyKn5}$~ke8rc^2e0q zl3ODughkV@44P8V5l_9uy;WM?lfOl}HZdpmPn2`7%^plBCwyeVrM}AKwqaESia$ms_w9+OA-0yXaXaID+m-Vt zJfy2XsuYA>J2g~RL%&d`)*->b6U?@kWqI-I5tUC({W*+&M2WkOX}^D@aI^CfE`!$!x;CeK8c!f=FLPQ_$z47~e^yNJ zz7r9hNK=pAKT}nB6hK5&8ZN z&-9A4CiY0XCV#B!t#;L;Jwru`QBQ^b92<*XgA>)~T0^3eXN?Th_cFTKsG+9*^Oj23 zKVCSSV`MU_?s-XwxD*PSchXka5$QVF&P%3bj(H_i($!3J5uImgGJl%2zW~K<9+wuy zPCgWjGZnv8tY*ay@hLj$tv|tkYCl8VTTI_NH@K0Yks9^h#FakfTTe*Ne)DHwV?q*o z$K67bDTpiC>S9WymiJ8aDw3e7MYr!?u;Z4zkM?a>A~YCk1kobXq4G_0#a=w+YCBbn z%YI)$(b=P@K`*~sIdN#>+r6<*^-l?8t3u*7Oi7EC)WI$kFyqCg{)vMYHa-avQ`JT9 z8U#&mAE@lX8hHbxTR(>MhK%oP;)lj{!Il6Dk6VxZ^g$I9oFeU=>Kyv9Z)VRLmB`~l zrBg}~f8r#4k%@yQu>CjDf_KfU8#d{hiigET9IoykxgEpNsM%c2r23Muf{?g~>zM@z z9B|ukgNbvuXGe6KQY-VawWt43vv$he%voa7iRzHhP?{|**|GKMqlh=k%;zv&jj_;1 zzH85#2@GAej71pN#ER#{MYFkUnvfR8XM(FQR_dCLqkCM|MJJp`0!@93h-58!BKj9GpTpS6yojz}K$%4XurBH_?_ehM^I<6BJ+= zug8nt@lQ12$(3ki*5I!U!wVZFVN!uZgtkL2o1C12XFI)J*)Vajmq1X_css7oE*K_B zm}Tai{g~^jeTRz4uq;IL44MM4-sdvYiJza*>;qz7$m|*k8p_tBqGQ0v;VxK8{@7y; z^hVWZI0hSW9!-t(M4fdCWh4UWx9=rXGOtfP!Wml!KYcwsdvM@ovs!~-LgQv;c)r(E z)0##+oE-XCeXetri-=tU3m@SR?I)L2sqckp_K5kJMPuY%`;vXU)KVjV7k+QUfM`4e z)ABKHzT7sM?3-lJUt$vK0}>I|<4?d4h{~Gk7G6(G`NG|r-%dCLThMuF0dHw`mgqkl zh|_NI(@K#L+qiGyO14)ZZNRm*M{I)R9cLr(?jEiy!#x3%?uWCp_$t^f7zhVemGO zv4>abft?b*jYUQU1$ILoLV3MvYny8AEAK}jIkzr8Fl(;nDG?$dX{ZX}QimbBk5I>}a;asQ-k7ZA15IdmD%T4?xBYhoA%jeA$5t()8=3zxeNBYWC=GYfU_d@y>r zBwSc2*+p6Ii7dXZd$*nSMSKWI66r59D%2eky6q9#1Tmoy(Mkh*`Y0MI>X%B zm$jqm^)29T{E?B)kB+%TeeqrbSMRu(pKWhz4oyX{#xret*HFPNGK=lNH&0OD;NVnE z2j4=FIo#DT1UyOW@&drGTG1tj`$FikA+xpw3~P2GpijY{AT3!*>=D|j-gQ|(=}Smr z$e8z-CVL^d+#`ze4#_5oT%nfnLJseIvC<^VML7vNj?rp|c!%R-jdY=`#c}K|DHM~; z!gl&E@UA}?dwl+gDAfgYto&y0V2GR$+&p7pZ8auZ z$IpmKGU{=Lvtc<^hEegnux2AcDSn19cbaU zB?GplrCE_@jIh&KI$6@)5d+e|)08%Wgfxq=bRc3cOTOwUoa$anTjituJw>fC%cX+j zB3W$*&r{S~&psxdo?HgkUxKnER|$Ehff#S#K!NW(P~uoDVaEVmzsuY@8O8@4Z)+1Z zX;!Zk{?vKkSvL5KFR}Ikma-Jp;%z(!I8ELAPg8~jEsVWRXa$JW*ZbR^;(D|991)|k%B6cxyx6=u|Oc1`X%``_DcnPk(6G9zu8nBZ# zsjkVoAi5UVl19^;XQ+)&oA$vrmcw+O%3LbsVggI~bHZdd@M_L?ba!QmSKGzYs-aCp zyMdn>*reHF?&T14J=&-TSFHmb?}sk{zzjn@b?5`TCTCn)VT|ul;Ps9T%Igq>Ih{ z)E-3NNo+*(*YuRFi%o86nX9qsRp3Yc!OA81RJRFPmkFY!ysp~aJ7dcAbyP7ngik>M z06)OlSjy#VJ`PEp{g-$EJ4^y_E)AcJ5Eu- zK1{l9fz+9+%<&|AV_CSqP~jpGk%pc6Eea#+G!`fy)zj6!In{)wa@bNjA(ug*mBSWN z#8d{6h55ZjwE*gk^w>-w66(v|)qR503lhT40hs>beb|mbe8IZAjB8H?W{A_lek~!Y zbU=Z4+Oum9QusnZa0Q*S9L31vd5VB%TDgE@Z%@BG^%Wmq*h{ji)- znWSRp6QGxw(YhE2cy z?JjBfOTNex(-x>3Q{n-u%9$J&uV`e@+{@%O?qul~O~c*&k^!K=qgD#MMO+Elt*QAI zq^D)qBj2AQj>t!KkX=^GdhhU7b3osve9wmpqOO&U86J_FJcpz6}6=N`%YwmL=i+A0SC_^p@ZA<6Pu(Z$SWUF`*mdi=pKc$6Q4UAtdYxZ1Q2!!gA5}X=G&bb z#l!q{PA+G`m*zGugqGK}dK{#-$k~OxJ_hYS{k4lL;{KGFYMNsQy?jq}iQ^tFMU0$# zi1dDkqr^py413s{+ObFnX_q@1*IyiQ35vn!mO420yUeg*)O={Z%kMXDSFFMvXuT`D zn$}x{NF97e^Xpf}G!i(#)!k+t>ybfgX} z2y&=|)RU}n2@mP$hQx>!49}c(v8lho5+*FdrRvC+-5vpigHw~Fp%P^dU|ck=7#HCS zUO?-OT`;BZIEhI`sGgYGXIk0ZOn6r{!|^GvLxGfq?w6r%!}@tG$y3=GQ2&}FLrr++ zmo_;LR{L5Qw-YpT>&tTo6{GZn^}0>_=+`v)N3p6vUx1D*nKa3Zd}yoghPkC3!2H5ki!x20 zndfqxwlPXLB?$C!G;mA8UwE139HlqW-&kSkz03wwP(muJoA{Nl@w@Ue_WVNbhTd%f zbj*y!248ONx1s_$gnl_Ko>xe<b$<+o5rT4t;7T z$QcSmN+As_6r&weWs4boKJ-Q^Ou>NyAJGZaF6zP;$dKI}F4k4G7-iOpyWQ3LF1lUT zhtjPuRNu_SAZtQEefu-wNSkPt)y_VAYVz;&^Rz+_mT))UjALg{A59{Yh9!bu!_vYA z%Zzkh{+Gs$`DE81APr0H-_OIdp9|XJf?4_&PM?|z3DE1CyMgwq_eOW*n&#oK-NakP zaNY)hsnbOTq-7N}=P2OSY=UUyq@gKgo+A1(QPf9}g5NMJDsc1gNb^~OS_aH>*5=^cIe?nUml(C%U{aLR~h@8l}zj>60z zO2Xu^S%ntM0n)E_Ci=fO@duj|AMJ_EeZv=%1I0g+`^IeUdMSWc)uzY?5};bTuinrw zHBgO}+$BjDA_rjSVPM6Np@mb^Qzr}P@5qVZ!aN6!k4oiOm=~FZHtTUe01t#H)|uzn z6bAF^Cwha2RRY)&9ibn6Mv$h6j58c;2*hDS87kmqV_ik>R0^BASc}*_CphoitZJ>T&nQnJ)GUg&2t-&E|IV-wv zu9w#5fMJ3_A5^28GMR)I;@?8NM*2h5a}Rzp4ed33rU$ZoWY)C z-j2HkPnsw6<{7zoy=h-fZs*_cI+X3F@&LoYAw}0Bt3U*u125}#_jw4${3y-Zepxdh zt;|OO+55T;n(GUm4@2`?>ATEzi`Ufy-7&blo08?D=pj3Z1F==rPU^8!;@uIAMBBC% zKG%T4I|V01vJ`WmKBO=A423q6Nm`ua^c6Z7+|ZouPPxkBWVq`2W<1{WZu}Y5sGA_S zm5zyJ5z*A1jVgI|S<^ex3{%k-|D=N7Z$WVwupj#xaRU}Twor#oe>xoNSInN1_K`sb zD?ZiA0PTZp(MtOu7(k?84+wdAkf3=)FTvqqkj#lcfY+M={~5A}md+Rudie~UzRb&k z1o;_6O$KjOIl=(#P357zoi+8`-X0qZzr3RP>SZ*g)YP8d07K48`v7G4GB#q19{>=x z2VUd&N7A{03Mf#;qt?2__X2}^<*@d1(EvvNlkB#a!){Pf>JirDOG2J6SC&C~;lhY` z%F9_NRfn_YGrIxn1ajW7MR%Ls;XpYIF9BKE6PtAc{(dU>TrjF*Rrr)p>QES*fMtp1 zl6!r84fQPx4?(>Y-?1}!fpGz7IiH$mVc*mw@=NA)I^n6%%b~0aihQ+G4yp)ygvECq z&f(HgQ7rkiJaQ~5o<23O6AE_3j=ijZ7Xfh?a#98a7qevz{EFH=@e1x%0$aP0-=s9a zRopKPZQ6wZ!?3J8!B8n1@YM<$yd5FIY*=R1{=zSy4_Ocn>y(PRs^Q`pXB{Xx?}+EJhxTrF(~8m9f?@@Q0|dJQCgbynZTWFO?x)9V?4aleBHHBW za+8}{_R!OYU#rrwW1t#;}SZ zep+O(zv2hSQy96Hr2JTrvnq4xm{Wn|9r|v$v{{`h-8~RXc>x7{GU?|mAd&`@JoC1L z2tW!ePs2(k>kD%Ww-%zV-c5CV3RV*FIv~2t8g-$}0j~qBCxud9qw+PN$J6s${y%NFVB-GA`{c$F+PvlW?3+_<(-yy_3MQH?$%}J)O}< z>u+g;B~BEDcL*(anvX8Q{0)>9@tf4rGTG%i2Z?b)HS=$iFx3Ek#|(Pw^ zzrijG2l^^wRjg+qx7c{GE_@KM(h7r8Q7rWwt=G|)>raVUIq+6A%Pzc|9pDfJ%vt@A zv13PNTJv5Swgs?hGy`pAqIl*Ie-)(pVaj}ZgH@n6_|i-+KeRm92Q05P1_+yW;TiF7 zo>-@-m+vBE`1Yy#4D*gwH-*^*HufsrXH$u&dZHRSe{mU#w^##QhUz-(rNOUy@=>7|I9G8 z^L73ny{n_`n;X zG`0?xyA%u4tA5Z8L|4M{V2*D6S~A+Vhhymgz;hqzg(o%$7_)fsy-g3g3?5T{BL%O3 zb{PRb%w0T<<}R7$xfAJB+qMY+n3o`S0Z=Re%mn}z*ECj(F85UPtIc@lkPr6*6r)BS zyF=;^yy~pRA@)`aDP?+bcW-)(m9tJJdm)ATAPr;Nfk9YXZOVc<#9m`1K-ZL2z5}R( zy?7d6zhgL{rTL)l+Pa7_XIT?41X(bos^muv5i`S4xbG9fq9e#*a!D2*h^n^RkgS?O zfzOyYOX`&}`l zj_f@KNhP(&x(Impnl>sdQ5D2zn2q|3#gEwm?)(@~-ikziYGfcV;nF>nwM(J?;EU>9 z{*lN)f26A2o1Y5gWs4GkSLNMQI9h-n2C(;HDEbWu2PGx2W`8T}1vxZgfbplM5K2z< z0p{Ak=MdEkkVaC<@~^-UVPiNIh%^{ER|+I`kPlejda~Dsp0Lo1%G0WGk^0P8slEI8 zcTteCn#)q7UW8MsH(F8cu$vCi@^T^2d4!MtfVlQd+?MFEAd)I^)DS;Q-)UOh8mwZv z8NH%+oBC*yZ=N)&S`IC2%0W*IB{ygF4-#CgC%XW%3YPI z3ej#U5L7dW%VWwBfW3ATT;wLV2#{Ir8@lG1s`)M79cQI&AL6%0LIcL+Q!#y-15#_% z6{}!>aLUX^en(_rF!-$DSvRoYQ66+a?vX|KmOq&f_Sd|k3I%6z)Pau$UGkIJ=iXrf z4-B`b7X`d*|OT!0<1gJu<#sQPVnS|<-jIQ2QnV?Ew5a`U$fI_Gu^vl)ZIB8HIIHfE151f6S z?#s{?i1Pt~?s$9!pyx_~|Dz_wrY9mhoa`psx5B$M47qSme0@= zrr?DqryS_Ah4VUrxBHTzZRR=v5p-bMvkYkdEX|t^x6Op{lus z@@sVY51P?4W_X5aQrp^?>=j9E@M0|jszG{SaZH`|9K9gVDtkZQr^9lB#^DxV0PQU{50?MX8>k>~|6F0!Btg zZ+Rrp#($ErvhDA(zEH7<(9?>83qQ%4rW|%as|@zan4YhIZ@?Wa7sb!!hWAWS@$lm2 z%9bSoWG65!kJ6IgNtG@SnH*4P_}BJP?Xb;@ygV#??C@ULx3CxnubgFz7T|)Rncq%4 zk(F+oS`t!uiXo%ENuZTnYd#}earx7=z^k_^O@S|1red@$52F5D$50!{D67-)=TLeD z&rox!84l(MKlms+d);TD$?!v;_E@L2UkJ@kSaapm-g`eELjPoKz218LKTm(o zx4pgYv+R8z<@|lC>wR7_!rN(W31Z;f(5qAFNyQ1^$iJQPO&qE}f#>)RU9H$tO)LI% z{l9+UyY_pQ%?y4r8@z8Y^E2fw`2N|%m}DV{shC+}D$1PTG`AShJ)t2gs4GX3d^M=+ zS^W_Nb_7|ZydiG+p-9|vAy-TqAA0^>?=zM7>ErGQo&TSppM4T>SLAo?Lv48vA<7Sz|=M2JEtzinte`@9_aFzDOFD;6qI z2l9CUj(u1P4l!yqrXSpEeAUPdB4?L7{*JthV!RfL zUw{k)-~bYOe$P9MgPt&_ z$hG?^gDu{YN~m?d!0bdbF~f%?0b&+1Xj#|y)wA22-A4L5G+a?gQB5I#8OK%t0Sk!4 zn)U12yfYO4NzHXyM)2ss4i(+>vP6ycXzJocE|~IN;WS1I^@S7f3MGtw`jh zFb>?DH{QPa*N=*6Ib|MVqYqBs`FyZhxbx2(J5zR69u^I*z9n}Zpt>MZSnM%<1w3v_ zNZ~(m`Up<1_md@ZrqpJlS9N_q;ePVrMA@}Qm$7yZJ5XCA+QMkI{s<5wXrnB{x~OQT zno?wz6eEh1L+f-xORAhmyo~t{H^Ylr%#|=DdvpIK z;Y7bLVOMK4^O9;gA1NczMC1E`SV<3Vu>Dm#8i3bXQuQU&;Q>{#`)80xHMX}wCyFmIN|+(fU44E% znpF(lfqr^#2|4@AQQMrqZ9(mHf0Mai?=MRn_S*Wd-*2Z%XL??xoc}T?dajDwXj3K5 z!NL37gz!FdD3R*-GL>PMH#(QML}9oSIpC?{dgGXx0A2wTJ5U=bzNFzipuv^Sl`ugk zf|W`3WqBm{a55zXdB%2BaBUZ|h|X5jB#&j%RmI~aMMIR{AVfLmbv_gQ3?>YmGCST+ zif!N8bMZk7Q@l8XDk{GDvAzxcM9}*`+FLD3aBPFUb#pCfRgO_I5LoLO~9Gku(OT)1+;Rfb3~(iU;LBp+|Da zNd<}36~>#6udyP#B$Xu+7c#a*o*!UuZh!nTSVBl1J{2=`wYv3~XG2EWGjgaU=D7lD zXt9`Q$|;5zPrvkGLcIy%TS-i<-%E#_6pb6`b!((ZDYR_JmbCUI1%i&mZea246aN4m z{O7~XF-X~~HA3mdi{hJ##aGaguJByLdd|%StJYSE#KtEj>ggLazaxwID4Xo#Ue z>63O@iw6S3kt072G2zpJ!hnyc;YK$)gHCvrD^`85*iKAI$&aX39~JgQ*rXEm54|YKlcl0d9VWRp&@-mj3cW#ge%709b6^t*gERpA4@A}I5?QUt`8gorlu(L^uy}1f zRGE`_nW1AFZ*l{N8?|ZfT4=knkXQ`@+mMU8r`Zx_n!OCLn zYdy4gAdgxnD3eBabOmZktxoLOk~~CahCJ%_sltu+%Z<0=n&yrQy&1gm!^ozSQFE^_ zAAIoce)-#c#_Rqqts@ExT;Kd zIr&*q{iBbZu=pn(HtN+`h`(i6Qweol6vAfq{eEBq2=vnNbsFxgNKw_o$!SN@BL~c) z@mkvOvwF*t$ny?ZqaOBV3C|q}ftsinaLU;ZIj=ZBm>A;5f@&(sUNd&&BgNQJg)(Cs zG?~dJqI=oIrN>vZESODDx+2G7ULgk>>HvHFz(H8KeQ*T#t5-c@0I^?{zCrB3=s;l& ztH;aynH0vV?jE_e9ZDY9GPH(J`Gt*0V|; z=6NJ?atKFjx5s81vHpAG%qRi+7AQo-r6`vp;g#0`gbM)wU9zL8a~-Dc0lhLT_WlEu zypS>^ma-JtO!k#2LzEyh_zpX0y92SByv+t2#__;1D8ytgrLe7wqSSup7Mb3J7_pa+ z)uzR!zB-UuXhXfd9kUQhEU1tOoa~l2D{2zsGqxS^s`hL!!w}iWK6UBxpzbSoXn-8l z);3_pt78mZ85CZnCks>go!A%D)S0CrV*5=l3s!@cOI^i3^HKhS>%cPQ74?7M+dbZk z9PaxPrO3Uqg=NC*$R5kMHio9ly860)ZCwGsrgZi;hx}3mMLLjJx}(v@ilLbP11(E8#LBrr3UBv7BXPiw<}DfapbdHY(;5aw^ay&GQB6{B zCQS;a{m|QTp{|T=|NPc}q}Z1I6m~FSiB5^D3*K?(sQDK9Az4opy>=qByVuGirWy9+ zJJ3B|Vf#l%8!py0G?*_|Sd19f3HN^Q5sfupyTeR!w%|a6)dz<{qE88H%KxNh|I9zZ z$gWV;KhRD#8^;J(A)!{JB7|QkygUUK;Bnq+)EfZaMbw?OQd`xvh9Yer0i!77ghiqLobDn@tGETFH|>34Yvd9{nlF4uz%^!uIHTfrbWnMfbZ2C_k)|BMq zPTp5(7NDu~9HC_MNEB*n0j)4Bwr=`~(@&riYf!l@Vi#aBm3Y3qz&}3V=0b+jgNDo( z$7?Dn;$NxoLLf$tsB7Bq-RM8WG=L zTkAgRBbPZeAc@uWsnAq~e0A1RLxAR|#LM!HAW^=JsNVTxwQwnsNu4uka=Euhau)yQ ze&Z_Qs%a56#EGt?OD_H3Anm$M$7lWq>=V4`DlTZouQN}r*53c0Be>i9#Pe4wt93uV z^tT4UPRW(4M*1z19IR+JPsRONWij8DEsg$1`^(Fd>>m_AU~i--8_nPuJ^F2pypK1X zP~*l-Hl6TD;j+inHDa*Jo7NanJbEnD{xi^k;;!sBxYLv!N0Heuv9;$eyqq2d@`Q%E z0-(NsaC)k)LL0P=?ze^2vN|hR%8o*^tN0XNGag336B>A}=MYz>D+kWghJ#6~f|J;p z(0FQM!MC6hI@{zQSKlg?O8R{yb~a{URDFbew6JG&ntwJuZF=7zk{S|vBJ#?E;qvHf z7JJ5XFPxdg__on!f9l*C1-Y1Jao|Gr(r4Z_1gF#2M@zqX!^j@b;iBa4IDc82H|jio z?to@+L<27u`f~E9DPaL)pJDz+h1xdp-KA_IKpm5Ox@m)<#m|q{0+QF7czN-Nxrcf6}PyA%3dqk!P0i#UMj-&eWc+U53s?+whNk; z#b4#Hi^`{pN&>Y+f_-k^7S*;?!H>K;lkgVRUUIr5_O=jh07rms0jJ(DsMys6M3nb1 z3ORibU#)xOS(O9yLYf=JyuRJTLX8`Dekd?G_xYa;yH*{_KKP%&+VJz3_n>I9ph=;r z!;LuVW3#9!KLm_vi^Yx;;h;+7{ghK*;(Z6|w{<~aoKyah0X#+0{E$PAb~a}iRtVzN zilr%BlY-^nmp)^-!fbM9K5`=tkIW)pw$NZaJb|oZ*3_p4qhFZlyGxLgSDo!xFWiS& zZ;LL~%`I^!d;|80If~83))3%)gTSrc-`;rWr5sBZ{C+mjdbZwtxx!);0KNn9l%?wC zBVg>ktj0fC$&l3+!83TwHZ;&;;F>U(Q%a*jT;GGTt0bT0|;up9!7FzFZ|C4!E>Bqkv2=cuCC->Z& zTVUfQkE*_pNF?DB+8!V?b>B}KQLT2hKTb1_8iN9d! zkD_-({9I75S7dr7_6a6wFb23MAzWC2n*V6*Knfn#0ZV&HidXAO-4XadT^*xdm^)Y8^HLXBc#6miGSm_C5Tem(#K$fHfU z7~awSYqkyj!K+hbFCzL$CQZ#B&&*0+>lY&?rq~32%`$}UR&%#&S`lKSl`5*Rj6s4XF!W+LXp z(j?oqf7oBV8{HaRejc}Hu$t4D&y0Euqb;hPnG5PBD0B?HiKjjN_^CRFz|9uPW5MASCEbRdZW64OT=o}Yg!Vq*wHe5nhj z&r?LcTG-Q>R=pit{!}@G8MLI!eSMr%YiSG3V?LnRJ2yYu$Gm3a-fIcEQGCh6nh zB`U}!_y^OmHhMKd&YjQ9mefVl&ZJhTlG3rJ zhJ?i2o_yq4eJ5i(2oPbf?&UIL&K0GM9V)%}XAe4=dh67mbmiQ&H~P!x#)08}itej} zQE&vB!v>rrwg-km0lba`C^kVyX6Aa6L`ofbY$kx)a-C z^OimXZEMs#;1QF+hfcmNo6Mj|S*i8&!lSf|u#1AH;cytE$__Tcz&5N8?Pli=QTx(- zYb;aK7{rutHT?Kz?d%wA~us`|lZ+lO@p>byiE0YRziO*ap9XDfE z$ZvhMO?5-{SJfYHI<7vNP4h_V?Ej4J2X?Gf`?xoZc?_^;neb;-cA$HX1JF~t@{L=+tOY60M>gADs1~e#~n*iLzh4auXV@^ z2B#{Zp-q3+mB$RJW%V4fG~N{ZQ1JJ2AN^nHy)x?gGj;wMe&-E-md=pooE4mpJp(v} zyoT!7%M40OND+L09Q>kFPjwLw+SdðOo*av4f`)n8lFkjU(DxV4DFZE@lpxT!SW z&V?8mZx{TG%5I5^tv&t{`$yN#r)P)No{qm#BwP(-p|N$*xJMS%A#oFDGs-qDkG&aF zH(k^xdR5UUnvU8nda!x-Pph-d+>0ZFx*7FjCH}V8^929yV~`x}Sz1>Iu0DvlJiTNN86d-1|oVwoM}w ztDp_YRbUEJzYI6r<^h-$+4?ZzsbCEdA?H6Fwc2xkjZncK5qc20bO}_W2I7&o_qQfBuCo#ec_L|J^d;i z$*vDa&eij;WS0|bv*#oxn@$xc_iYu%b8Us*ed{hgkADlWnB8P2L!TyrzCLBT;r#KD zvoN%IfIqclC=m40;+%};x@qsFXG@A#uUcVKFNwXl^#-j;mY4DpF940Yb*>rrkX*8+ z?Jlh(OlX$_8#Zi<8J5Cnm!`LZhGIeeLNe28wRU2uk+iv+i1=~x{p?L zsIK{>BsQz4O8`c401qOOTwS7B>U8=Ub*trD!Q-9@S2*-CSu^*uRa5Kp{-Q~#Wz&;> zS;n?B97~ZmvYDEmVLi|inqRahF~0~b25PkY6m+7IxX)*#XP=a`A%Z7vtNA;a#Bs>J827-@j!3V~HIeDH_VhmEZVc4gD<{^4l* zJ^U*~x&HxCFIFI`L1E7c8QsN6dVHRP)XqI}9ImdTI8`-3ammc2a|1az#z)u<1;P6V ztCPjQ4g>{*2Afr#SxaVanMk@Nh+}UKHM769_Swg6fas7+>RX4!?0lrEVPZITswgJ% zZ*+xKE;szUqC`c8EqBU3|C<5(f`h#N*6_o%r)94$giObSbp6kxyzs1@nAq zlU+uv-4HnzZo}k;VvACVnk6oB%-vWlt4o>u;^kX5qjv1B(vT`IX@X1CRo8K)9@)?@ zR5N#_?2(1xkzBjRLwgY;uj43Xu_wu$6MHamMm?>2sfYFw2cND+&j09?20GY#^rqdt zjtk^aVp4ySuP1)s;~gkT*1tqL@KuigBYwT#zL_?iI67Q#0CeG5D5bM;7)D#DgmFHO zxjRHzN(9{4k1It#vuV>)yLjh>Q;Svkc>!CTxk<~!{eal&({1f}k;rU7QWBAZul=l@ zBPfZVLpaU97Q(sR{g)#A1^`-t5`ROjj%v5qN&G#k(Tc(Qkc1x`gqH|^_HTDZ5wL?F zfkJyH=eJQJM>pEuS}XYqup|!vi1cwHit{m9LZLfjixP+y5G5D{bOGUOODSYmpxF4A zzM#3YZ+uky{WXchqUF^33uHJX==sdef%CKWS(q!6T}R;}#dW+Ny?`^_-thDF;Ux-6 zU_i~^o&^5+P%muwkz<~>c!e7yOitJh|3l>Y8ms#RmViB%)B*_V+wVD$gQ?F#g7 zmy;N0HL4Ljj20IIE3YGtD2ChSFs;253wjp@B=Hvp2iaH!^k~07M*sZ;BQ2#Ql*Z5Oig3L;pssj3)s@WgeKCMEk0R%hC z4Vw7KZ9u1f(o7R0$kLAidkYv2V2hLK77$@9NXb0Pj9Ke6bXeYWGU$z4+F9DwAStk* zdkWyl<*rir7@``-HZj|w?^?qgwKlEV&UNA|BT$G4Pj}tX2BsYk9UpG&3Of5Q;A;2_aBVLC1GrwFap7jxq&}ip(7KQonup?p z{Y{E+j^eywX`?2F;UbI6eMZN#X%XiiIb2>e-jH@QYyUnlbj5GpVln1U3J^aVrzQKLGCh!{eRZXo+5dR9oUp3wqS@UIhIv*3`Ll&qgM{C5gdw%$G`f+BRmQJ3v=xHgE@X; zUy&apjse{C?v`lUO>_^^g8UL`sktHk8kwt05|Nn=Uj-1(n%Ul}Dp0Go_FTCCU<&YK zBK1?0Rr=w5Tr6S2(rZ*1tXX5#3As|m+7 z0VmOr&50;zax+%I6MA!YZZc?-GxKB3Jc2J^ixTlbg$IMcep_azf$J5uJl|t%u(T~` z348g|cwBVyq0oQt&Y8S!Ij3p_IALV;nNT(?%AB@ztxwcD$KfBo|E)*s%z6&?g3vZg zbaCjIMmXfThMMF%uzE-2e}lmr4D2gK+hH3wM zv4fTGbuITZ-OY#=HR@Is{WPf@1O=va74+dbT&b`zA30}nGHvy3Y;gj?!ih)YWd_Xu zXfegWKGHY7mLf$zX~d6v5?*EcF08xz@& zWzW}@@ISnN+1|YE`GH03@cHoD)4+gSxaQgAhvGk+@$2S&i#oE81`@uKpenU)} z1Kbt1tH`WjfHMZ13wj{A2`Y6~uLeEUX=;Y{@uzmV^B&HdWT}nQ=7*t|y*W z89`)Mype#7Xw$!0k##nJa8)TedyMs#_BofJxbw^RET;6DqfX(GhizX@{5#ETfprbj zm9tvM0a)OxPcp3W)VlIAaX2uy$TIqQa@O0W!$e{z5DybKZgMlxKFi#)cw?z%a*A>W ztqtD~Um06IKD+rZ$%pBwq7_;P*~b z)(b#xJZXwI|8erIlYr0HvM7XSG35BE?^Kw8-A|O!|xu`p^zb12rr2^Tn(N z^Z6@*BX`OatDHve>xLT*_a{8Eem^K7{n>fnaW@sd03p7ww$~(z6DUye28!p>1_kYf zCk=~X>f~O?!%H8j8c%E(J-pK5$IXs219r?BXwc&omI3zKpGNG58KezD38iEc-0i@L@3aYY(i%S?!HQcp_?+|CNS5*7?hl;|DP8~3a0_^)L#LQzgspV&0R@{>)+EE|?T1jR4# z#HJ@L*Xp?wCoN6DSsK~LxdyW9+Z~PQIg<(>@Dxz}xi2%UooxROiKWH9^k))h9Dy6D zp|8a}R#=UEo?#8COH~?uvQYsWsIca5R&`?Or}-N8#rt8UcDJ4h!N_Yx2NJJZG%bc`I` zJ}w*o&^GIz;qmyPe|4JhzjT_JKW4=*{LfDD zpT$~gM}IBNAgVPne?cNCh0(t8?vwWB3 z%%_ttPrOJ>Dbgh}U68L|ri*IN73hhHUv(>|+yG;MyIf#1oji%$CZ*`Etl>fNvn}!vZXHm=6;Y?9k|Ku;= zhw4{$a6UQtpV7|C(+2%RH`zTN`%BYmY-$>9Yph!5Qd!37*_Ov?KVntak?s{INfh$NnP!PqY2^U%ghguk3cxm+OFKA))_1ZW7)EbS`MZHc{lz*vY?KvW)mZa;snTb$`EzZ)Dyo}!A#L-hA zq(p~&4irXvU(4v$9zKqBxX+SIcc*jIuRL;bM}f_Wx3s`aDu3o?1}CNYux<3ILOVpL zDfh+LnpdM+O+Tq7rfaNjb*!nPnCKvR8J`bQ3>V1cbTA;l{j9I|PW`t7U2oezSOsY6 zAE7%~eSLO0YXJq#>gOFjdvHv6)}=4g9Er1VdPQ>@H#?;dAK^0Nrnjg*J4ea*s=wh# zo?)Bc|DeZ7*k)qC%vJm&-*e=ChGm^RHrU!KUG$bGiUBYwp|h(K4j!vL?n9-vzg%@*K+qdO4rOKUBPow`Ev+d+dKVhXV zK07U5D?9_@=C2<;Lqdt`4>evOEhs+%c{4r%$7+Zj(mJ{cjnmYvKPUo&LY9M=Xka!0 z93?okoYUil=jc%`u}1cQ@nS03s%o^To%~o3rx88_w%G2>%Icne^_DoHh8!*ob+nI? zAC9rjT6|1%Ljo*+`y=4V)aM_@RHtGMXE;e`LJQbUx-HKUt%;UR_r`gjuj$k)C0Lm3GhxRT%6}v-%Y4`h%F+SPkTKsLw z(b}(NPeM>lL=>lDTb_|sXSHQo@9=Ch3;FsttY3OKj`d-mm8nQtAF%MV1!>$0+5TW< z>+|hnC||r}AeX}q7oxxcznkqZv-A74X}r>Lwz(J?z9g27KR=*%Ts|ZG`t^pG2HJ5G z^|Q>+>e)&2RQl=r|LsnW^3A}=$_jG-e)8j^StUYIJV3j@qMqK1a_RvO0Rvuh87k{w z$Vd=U_6r~Ra(-j`aJFkz?LJpqN}nqu2X~k+M{RWxQ}|3tu}vE2IMQ!ZGP8-R`LH&6r-+by?m8`6 zg6 z-kFyUS)!l~VwXc$d{2K#1jnrd&zaqpx5mSw?U~2sy3u~~hDS-~c&6zc@x;R{$M|P}>gTL%m8L|f ziygzBCCWZ{5q}^v5N5RQ@*sOjbJ{)vwq6IW1$8C069Py{x-o_nKH2xc!6;-V>X5Upz zXo;N2>iXlrLRmu{Fj))lIEKqMy1p1B(i=@FSB=#w+Ph&Hoxj>W2r}TRJ8}{J_}lU{ zxku%=y}4y0X}{)qO&1LzLt0sGDShwXt~&X@=z zRQ4jT#FBGorQ`fMaZ*&ZJ_90vL~|Uv!fspJH;2U5ejCr*=*hwCE+(Y{W{0Fwj^C{G zAK0^Rdg_+lziW1zn5ilTB{gvl%jvqBv?uiB0;;BdJ6u${8N?b{z+C7D7XKb3q;n&U z*7*Sg!FmZW=^_Q$=S%8n-V72@F?9*9w(*?cut-|!T`}N@-i>Lk^s~-S#N!I1x|v*r z0j1nW;R%el6G*9&q+S#M`P#&mkj#`Vq|SeD`O|imRB|fib!EG2NagCAcH0dvI0%NW zhEZ_o+B!>@GnQXx3Fq{#{DCwAS)m(iKRAyw4HN=~T+vLYr_Z`!tFPt7v{%5BY;Iy!;GoPd%|qj76` z`)pKC`*@b}8`k|jbHG|kHeuli`b8&84k(U!F__ar=i&_lh8rUIW zlhMkp@@0SN#iqGOddI7u9(6O^|9Mk|T~ zcQdZL--Bz%Myh0>Ba@8aNX(!bsCSGgjeM!{tlW4wR++Rf4;zU-ySw&UnP%4|WEnTG zx>C|kvI=G;h?Yk*CKT%>^ zWSZ5H3Oezv1wHZY3;H<@>*lW(m#@ToNWe}|V_eqm2sj)U^Dw;QWv~1;D&lw|Oi7m< z*70acBE+fMKPdQIA0N%>YwczvK~Lda$3d>(;G1XNQjW>qBR~k{zY@Y$zc*3d>?)uk z`)QzO#wZ|Mfa~hL2QB0K>Jkyb?TYYL=QMIG8m5D*`{2N%=lte_yPofG^}2_{RoQ5t z0kh8IF6#Q{K2`)jydgEpN{YiiQ{cO5Ns`uDc zNe6g+QQiybU5Cs*FGl85Y&=zD3gc_!j;An-0EP^aY!$ z!)qBfaWUEUE$2oK?Ra`;|KtYP=zr`$KEX^!>)E+#MvLn0`fg$?CIBtSE zI_CX0{gAdFxNA>d{i<^H!tZ1lD+7Ie+}?S}H=YX?J^eJc-tpMQlRx$FhmGmdB$y8m z)Tid!;Za9e;+j#xpr>=q`s170U+gSqRdcLcY>5g*uD5GOYg<=ugAW0-K9iFJ3sMP>zM zM`$OaVsafhygQKX%wZ8zlbZ4C^}iNp#3rQf$cgZZF=timfn6YRw=Sk{Z}}gfhKzE* zebP>E#q>)_#Ya71H&c{hPxs`0YRvoo^?Q;nt4faq1rI#JjK23)cr+N*>y1>Pt-z6h zUyn-Hi@3*nXhl_vi|h@4-0`x~x9Y)*`q#Tl!dq61w6xgYYWMtmhyF>zCN%y=@+FL! z;Pi(;cG0QUZd9@n98d9&V?7)y&2}9AG6pR^rELLfX7d6J=n5kswD}l}=gPs3?qYskk1Xxs`yL4=t zUu4nPt+<#M>VJ;9>2Ermz@tr`4ePL~!ghQZ6&orU>C)2=Xfj6g3{~I^6L94uBM;o9 zxN>^7hrt=crk9Cv##&!FC3VxXdZX`3+0WA{RUmxdh#!6i8p?-Xs?tMFfsQgj{&rDY zme(uu5yKm}`pcuAVJENUJJ^W?`G)Ro4N2V$QJ;_u2+oJ3EhM=Z9*aU(2jPfs#|PB| zOl`*xAQWa^xox<=^-+cur{}{H9K*T6(&21=PzdTew9g;MA1zH5y@ISR4ONVoWP`_e zn-U^ksd?N!nfGO&m|hnpi^*dgM;~t{+100>6iCKbc=*~Rh1_}x9x&L$5_XA_(Rz}w z0k`5AsTJjmjCgyu^E;2oz!|?gwaJ70+MeGhx%Oj$yw=ol{ZZQ0QK^Qs2U0!P8v~8z8&(k%iAqOZcn7h!S71>av)iT~cQ9nmnSzI9LGcuO=}FAKQBG zJW!|d&T~fnr7_|2j-jRuSsmbKlU&I#<1CNuXi{ zVbG%)zd`mZInL;LQ{!*ntsjVM9|ctuu%Qs8 z30c3(!$PZCLVR-HvweT-9}q;hq|}<2fP{RUNqnjoiDB$b5*q7=zhBy!@71$e-C@1> zC!0pR+xcymWvJ^z+UT{5Qr!xDuu9n^4H&Fh5G?hlymCgs%b{9mQbhoui#Wp<-ajIF;3$F#{=H;{FhkrC{U zR{9m8#4}4c+C>nDl^If@e&Gahdf7*1Z|I2G$T`($rHYqdMGu7%os{FOs%)Q*;*u87 z3btH)qwLSp2L#?{#hN3x`&`Tc7Y_p$mtPLA1gZY#VO4*@*%t#o^@e_TNR&AvFW)Nv z_;-h2YQs6wKBx@!F7T-u5Xq=$mS-ig>5B3-m4=)*m0n z_#~U4dO%IDqK{gCEL;cEihRw^9IpvF{jJtCsCp8mUmx7t!FKA4Ygr@hw$-tDPG$D< zItAM+psuV-ESWn3RdC(c%O!k=f?H-|1MNH z%@k%@ijS=iJ>RHyYqA`BpxhNF1seDy*!fS?9a?Qx7+f^?N41zjl~&HsIDZ9B4vnD) zbIr+BBj)7X7cbE6RC~I;VMbMK1N9j`cN+Mb*d2r|#f zKAcb$cwDV|&+Tf%Gr$1f`qnKlsM=dKShsU9s{8ppGP&tgYPW}YLtIyBYV8$%=_SVc zd&xs4@$_lrNxB%~1ajTXDsiQY%l}yxGhlbm54^#Q=BIb|l)A#u>6khn{aTd%IY$uF z)>U3%h=!U?12_kWA>)5?*_aR3-D*txp0PmK&uh#CFV*AI_*s1VFs8(heweFs^740x zUFz3$*B{g#a6cJ^g-;ikj8?C6SpVTv(Cw4FC5p~hmoUi9Bm=L=hxyHZAcr`@!>ilUm6i;pjYxNA1gV3sqS*JeQtM7)^f83+ZuVXN zrS#yzUEgumMNsfPbGGFq=J>?A;5RL%J!5vUx1)bnu>=hh>~Knt6mW6ZeoRW=J*qHb z*GPG`WYzzK0<(+&Rsgy51r5~AqN2gp^M1pOir5Axs#W}+$R{HK0|8{u{vwc{|7-LP zn%rwYN(yT2=8{x_6BzuJ6FAH5eQI*Mo7lMVgS~Ke z{_sv`=+jHbet(D%K-OMcx@X&=?@gbD3O1EmV@?;+d+DpmWiKA8uQ3aDI~5*;>{pZ1 zvNn=DW7g;G(WU6e#n5Zs1d&U8bNN1#x+{0kp)NJ(8#8x^UbEBC z+ZR2!`CK)uKcblK_#|4n;t`M_q10uJsX`Y+l>;>qK5{#~cwngw%;?-hjRR+u>{X1Zc9f*JJPQN9q_Q=rB2n3wAq0!1JVYhRHXXqDYi;aaG zdd-toN9yff!00pI^xv;X$`5#f~)Ze&i zxLe2yYwB^UQv>05Uv804YOMDqnBwj?O8~BUQD+Pdz4OSKoBKTU35dSnbap&lFEW?D zYw#4gaO?;xV=X*=#`hWQ>eDU6O!VLfn)x8{VIJ{ zu5!YTSeJ>L+LK?CLaJcadPX&1FvS;|w*wlBNdt|1Bbic_8+Z}-r7$TMQB@II@W zY5x(Oh%(k{s5a#FKTi`+JdR5{eib(Ui2pH5`|_3BpxK!0_P7WA_+&c0o3(62BLbv6 zZ)piLScShObxXFpRk`K?i1dDtHY=@6&#zw@!EetID-O}r<}YEi{IKU}!R+9iHT)1= zN5XJFxvqCx zIYq3qRA~C-?`I(0P{rV6$m^Xg;h*}rJNU^XOtDADiV^2rq!nb~QwaZkCmO`Tr}0vC zWc27%3V$+Q^CCPAEnQL}ujYIzA6EZ|k>mmAsyZpAK9}DlB%zu+z%fIy0(2wtawz?VC8+`CA z)eLvHyo;R{_9uU0*s9a_ne&ss8(uXHtgC$86TTE$0m=;iM$c7~BJe%>qJDC`>(KI> z#tN0uSJkV4|KUiX6;4o>oNU}C?Rr~p7IoHvh0G<8JNv#jN1sHc=o^frD6=aFD)4A= zSVf7to3}Hpyc1n_Hl0u8Pk&SaEhn}Q@jrO#dC#mQzh&FQrNKPE9SrzeWA7046@jl* zg_Wi5qL}Qf>JZqYaKFdjg614~l~q{V@;El*jC4#%)o#2**>6p-<9Mt0zIYc>M{M1g zt^Uk(H2u*D%-ab46D~+zdxx+RH03@CEq#_z29-i|#wyNKrt3`Y)Q7)^j;3wkIN*P^ zDCQM7cH;R`;vvyLVtrn~gL_WCN?pd24=6TJgdEx?3MzyP#pG-AqgyOjtq3%|eb9_2 z2ps{124~k*)6FKR?hMzAiqcIW(Rrcr+&aS-Y}|*P%hKtTCFM3j$PxXb(ogQKuG zo86c6_X`-y{CbW`qa=*HF!U=Uqj>2$U$Sdmj}_*0Es&lltfn13(L0$OTwRK$`@!J) zz3hkdvs4vy`=q_k)k0L4QMJ)2>9}+i#b}?e3c#~FQUEE+gTY6RHvY6;Ttt(lpSqeC zXu>MY0AY=rul~6RO^3BWIH%*t{R3Gf9}5qPg0&Mv26W=94z|4x9M#r_dXsy>{CeiZ zLpYMW9l)Uj&6fuoJoMel+_&_re_Mq9{0>2qko#=~J4-Ojzuw~`Lo+y$#2=f;9}TD0 zpMtBqN$-fAM4W#ji{>k%dTNcOoAFCpUT2@qlh+;F%lnY1iSC1gVAadx4!DLFd5o>e zCOc=Kqw$1 zcMrMWhJVL}Sqs^2nM<*Z#~k?K*-FtOmwlh6m*y6^q_jm6SLkQ8Hatn+gdAn8|1k5u zPJ7gpiaQR~38|-dbC-?S7`hs*xt48VnJ$064+iV`}i+Jr)O>?C$0KYBWO>5C#8PA zLBOfgz6?|B8p#aKJummmk+`09!7r!8d@!j;!RTdiRf(UQyEBExP^#@j(0syQ$nf}2 zk0?D$QU-B}0pN-HxLCJsLXmC|paxzA={%6BN7w^LQ zpI?yL%}^KaE&jVpAIENk6cxjO(@HIoG79GTEoCIdElX(wOIGBeYz6Z1w(Z{GFGkR; zcVH->B?FfYMjqce@mNniM$1!lC&srz?;zS!_MJzqfuvZAd?BrQ@wmP{=vZm&YRL9` z-@U?4`Kv8uhI;kKgnpZid`q#t5gDR=cW&cB{!4SsSg{niJBZq5q)=<0FmR z6YyLg&C01lcl?3e(hM%6S_D@S?j$F`(Ctk-65>p`4~eOr8vE15AbSi%D>TZ$3XQ}1 z%P*K8@mv}xIKA@14qM=0TPiI8WPZJaW?o2#hQ9wBS9AwZRY*dpii=XHAT<`3`Yx3X zrD9KEVC<|QdB-h^QW``IC{zC)c7+}>xSj0v9E0w73TBz!1$73;SAz+i$|iJb<#U0~ z+a=udiPgn@mhl{~CF|uym#O&tGC?;XhdziD*Kc1dF3WN8^-}LyXFlF0Q=op4j}Gh- zdIOr&0f!&aucoMJXQc3K^h+Ic>9`8OVhx(RMD6+3j3FY>RD|56`qHK^$|zMy^Lnbd-L@nUT&r>U;YgnN9JiKNWrsP-dNxy^%sU zu{!KwP;rL0*v-gZ^xdi8_tV-w(Xs))5q*4&whxeNvJuJ9-POyZnSQ_ga*5ONYQX~IbG&5j?Mnf>LA3?5n}-Qa>) zcVSQ`2JMcwADw(+Q6i<7oE9o-8QfO&f-XJiR+)Kl`V}IgFO{9}lW6Y@azNTVETO-= z#1}zA=)YYz;_%`Ja&f^V#%#h(6_O)b48~fJs1MI0fkgcziaEY_H`6$q_R;q$ijNdC zIHw=2-{qhr%Ar#Xsq#EREu*KJ5Jo}QeALvZJ;#xUHhDY)GyrDuwDe*1N0+?>$H_O5 z4TxRi?z}SI^x#i^dSjV=q+@08Vi;YW36!cxJoJ=&5%n4TLBNrTKBbQFA0Y7}rBiPy z@`!dqPVTLhoFkl-$TRqcaQ4767>%3Cnx+;&*U5hcxU|O9`ZuU#( zVk3E@A@pR|%!=KYz(~g(^1f3a>fX-sjjyv8hkTe-jGy1JqYSbqNdz;auVEJdUf1if zd;$h&xWJRZUh#iu78(KKv2_{C#kUFvZ%U1Bq2he$;YPkx3D>RFS0<>?8~lNXp}~h+ zE~9zxlSlC+EF3B$e84|D8peCwrQLuU3}`}CXZB8E;ema!!Oz|!9Z0l$0jbZT%o(j?6%T^%Dz3@|>b>L#*ss>#1=vcm_aEIIzDUa`32U)JgA%pW*C={wD&!9nNew zLbq58yCQ{R>`V_F1Iuycjp_Q4rPLzY;Z#jBZ6FBUp*&Jt!jo~5MFjEAgg@Z1OuYM$ z!hK|mKAhAPjOxO^Qc`C%MoK63Bw3M5eR=&uV-%`SpDVABl%0{`{R8=Y{n4vo`jJP; zXk9WZ=n@Q7PUoXg4yy$il>i)eSAJ$Y6ff-QMqC&abR+wYzu8XqKeUvK47sygr?S1H z$Vb|P`UKTgs*7=WU}O7kYtzSpx-*)*=4Yf) zu}z(1PEmgqgA{RwQiN#5+OuLKkwh}P{v?_0}TZ=Or=iXn99;Bcny}+Ws zhie=dUSl#pm#h?_OB`}bQZ#WAwAc zRF}Z}jcF`T?^$FEmwklrMk> Q1uRv_?`>q(=3PjUF*1Th_6!2-jr@rx9nKV%YV+ zGHPJ@oBD%sV67%aQrcWk;C;V`a(u#-XG3>pzW_Vuic=apqZ@CZ*;fl3eBeplKPB2i zBv??Ec|@^cn(op)M@4H?kAAJqoi4$4gc5EwqV21xqp2QP4~h>KykGG%zdEEN zoWNtJKVfu-pb#W%xaCsg{xXIn3r49DuA?FOvHaHGaR(;oe{p6=zcPLlS6+%JJK3Sn zZm=1lCue6=*cBx^p=gCq@MT4$|GYY+OR=<-mx8P}6sZ{8f{NMRF{etgvl-U-ul%IW z`02yeRm>>0M=)1Gu?X##^)04B1(lq<=71gm`bIP1I5{4O|+8nm_JpIstbm+mOTJPL9Hhz%sAdC(D{!Fo_RVh zz8U({8Z+N9X|rL_9I?h}>$)9DGg}WK%iiHQXQ_CVM-CIIkw*WKHlIe-!jLKsI~R%ajU6!IgNhbWS)ZeQkie7F7|6VjLF)M+k9c)^La-@bvE|VnTvgHndIvIZr{3k&WmJyLmipvqm<+6 zrY+?S(?&jUbti9%!AL#WWx(%K(uT6!$uk;^gtHgleaQSXB>tRGZb!Un~@JStkbFOVa? znHiEeDXyd+DF^;n7}ffc$!)OX4_hJH@h1%9nz9*S4*ts6_HLdrFON6PO;;XKO`3^# zaMJ2L_((3;xD+g!(5a_lqrl&ZR_41Z^G`H0T$mY=AqT9$h6@fI8Ku|IFD2lH8&lAV zk}yAKDVRG>fm@t@D5yFJm9YWI-Xk}39t0zDkPGKTV!V9IobO-$kT~b_bR~i~(R=fP z-?I>^R&g;FZo7Ok_-~(I^{FPP(0SD2YM}wskQdk95H}#0zNfe>Bz`h_qmV_EV5{dN z)=VMu-%Y_X(PfYdVaVe^udz&P${Aq*x*L~~9OzpQBZVNQSSr$Yt7>0Gws;2K4)i}$ zVG*Dgg$?suntTu5ziqOLz&h7`hVmZ$8r!dl=TFNfQ_r;``7Z@6Bfm~_GC(UYqh6~E z$LT*Ne4T$0D?6GR39a&T8!wC;F&73Ie`GoH_X)vq7YX;RM{&I*;RH;q7G+@Let>9aWq6u4YwVcLtX`lj*FW=ZfpOXBN8Jj2JpW z@;_B<09rA~U&$i#9rcNn^IOm%_Y?25iMY2EQfVN%<?nfVOz zg(UVIR>nnK$orNHE|u&)rBIOEGruf)-MM+HIQ4+jq#%0+MZ~oGmY7zerQ}{ha9Tb% zO>h3RJ;yyIW|IWBZi;e(+rPoeb2IOGteg|Y#Yg9D#32X3^C~!GPrK+i`)`l)bs7Z6 z;lQ@*Lm($nGFzqD3AyjzB7c{uod8*{D}waVZy}wUgiTdqbo{W#(?*ob`22T8@E;&q zHb8*jU+;rmvPaje8Ssw31aiYvl%LU?v-%hEl;@7SA1b;g?AvR`I??9R61k$^%`bch z2m!w)-G=!qWZ@$KUXZI9<@ z`9ZXCoIX#>^^Xgd2C^I=duA_L@$Dr;JbNu|;nI*pG9vAOnb!{2@(brQW}8ULoh%l9 zH?O3ZUI-xd&PiROmPpDO^S2!E^g33muoQA&zbHkP^p#t>9CGKrF3_0drI5r~3;OA2 zDjIW-dn|gK-?xwZDZYO$1ep|(GHQX8um=}wr`;fX0z@3BUby5Z4iv011Cb&cEc{j# zInQtBWn8KK>@fo6;13rjfzTwGXIBwRp%Oga}jZ^O%A_61QvSwMc|4vNvINH5uek zArY7w&Ju`usz~*R%_}*`1zV6HuYJ2nveOGBJJNtL3y?O0Y_$fA#bd6EDh-6k1k%B> z-yXa8A$9zU<eK<C2Ct@4UnUosmmac^{R;C*FpaYNj?~@hyU(# z(?Nl2ngh8oalfgmc7tE}?0xQo;&AY&@*rH>-wHghZ5S_y7=4Rh@8xspJ(2qzI@h%8 z!FD%Vy$`78zpX0^{_BG#8}r*kAJkx!Uw{Dog_*Ta;#SYxeG(@>v-a;eW#ZbU9}YpU z_TJKV6nz-uPHFoaXE-Hc$^j06yqca_`!vpkxOUm=0%pp$Fe|4m66;;$aqxo(#KE`t zHOZ2XO=1mirRh-?o}toR8@Yw7F3|8vV7_`aPPKPrJ@Q8%n}}h5oNQ*(2At{Hpe6=^ zOs)LHzpI0--eG^Y#F@<8{UhecD~%wlZ&B#fc}_MURX$YEWl4|#;C7mEXo%A#I{Y|Z z7?>tIvDpH!p~g0yNRP6N#bGH5(B7t)v}5Y3HI8bR04c74_s`t@J5H5&cT;PeCGqaE z7(42&Z?SU91hid^c3Cu$eFy5m?{Zq=bZ5F6;`S1|etccXOkFr>B^RSvILHas_qrxg zKkejqXEEhNCcxs?2P5rg+DDjj5w#j*`() zSqP-k2)caS{a4NXdo^fv!%=n3wgCO58iFeh?>C{PppiL-8jMP+_()@?32Cs5Cl4pXne4z{AoX~zeBp1hN#zx|Lv?`W)k&UFEQ29CJ- z2LCg2OW!Xyglq>t`D zL(zxaV-28MHtuUQtkPe6!sAmU8>8ooJFvID0LY?KywF0%%Vz+;!)wQ7%R*-Ix1ij9 z2j`riC)V$4Dp9uKGOCIdVtJcc3CLtt4BOcl@KLIb`31F|OA8)5c^@i%^of6pMx3$cGZ}gy(GEHNu^AgxwLWL=& z7&#JJe*pM+yH~sO5Q7_rxH*8|?zLj6Wjg-Tg9e^4@xB3d2xS!V!9(fa0cyQJ{&tS$ zC(SJb>j2)okjcg@PN?_p^~C+5|9?B%ZhwliGQ-Mo{oS%-T>#j+PxDO{b$V* z*iPlOsTy*qshu}v$Zy0YTJL?2LuP=a-M#}ge7tLbm>C3eXa!vO#sB$)=G)Y`zIS&+fFv5~M3$#r1`7C$gj8+W zHN^xCx%{cmNd3+-Nz`prI}*{7Xc^ls!sW!3pJy-Slhw2Xu_L=l=RbvkN#8+QWA0_& z?swwz00Z33xm?|@C7{8&|P*5g;ef2Rd zbb%)?`VB=<0RZ=w zGO_kEYxl>%KVFHVzI?HOr!h&L`qK%6Y0wYLxdT5!AY1d6CHVE8>>a@G^16BXPy=&l z=v$R${)(HjEmf|F({?F!s7^)n^r^DzG8I#0H)RIb=MWqh1TcVu`Av)O{D+iv<_=gI zpk?Db{8kCP>Ky$w4Pb8H(P=+!IaiBFa5ogxZP%rWcyF88NQhPwJE z4ge3q5=!BVl%4cu(KY@Tw0WXPki|5Y9hH*ZLUj?*ra8_=Cf0ezVSkMOFJ9-dbqhj0 zWF?;;W&^;P_=9G!=x8GM`L}Muoi`WSy|r{Zxe30vm?feoRssu{dh`Fo9}L;zGhKE^ zL^G=M{|^^DHtQu6qtviR2-PB4OHj<#kgJ|!7V%@mBGl6LtXNqh5G?Z=%@PR+^Uik! zG%96DS?xC4#f(m>b_q@%vz+TQ<>YkTC2 zQoMow`24B!u0?O*;9>Ox5a7bzZ&Cw7a7z#Nt!G!VvcUaFporg}J$2rmChG^K(Ut*o z^ErjS@l}uf5LE0{y86qf&Ks_P{8+pC^m50UIG)qvb{K&}$a?Y4=q_Y%humC&;F#X@D_T%q#E*^KRlGc)0P#&)$S^{f1F;p9U4_yu1# zadtn!9O`bnSnPOq{nB0s|3?Q_4(%9YnZq~1y>RAs(Ms6@DWAZP;g5qBlEca2^^0v1 z+h^M(w$J6K*UJE=95hteGcv)D4Hwjt^E&;egk!j5mL~YT&%#Anv#ogZy6?72uZgMW zi6>Sg-ZZK1=_&60RfQz44VJ10eM)u2Wo2b+nz)hOE;Cr)&_(JNeV@8np=}7ME*$O+ zmyVv~sb5nYv(#}obH#QLmS9mqJ#xSV+0|j(pwwHlOHK zA~`V(r#P}RSYqIwk&n^2#MZoD*4RHF7LYtQl(43c@QxDQN1~b7U>5{=2H|VC_{{3r z2Ap~A*t|OYb`JhhdgpIcYKxF~e^q1s=|Xsp3EkRrlw)fl87tdxzjkPmM={^$(T#V# z77shd5q`6bwyL)h_l#kQ`}ap5jpGamus)Iq>uSkG0U~`tk^waxuWcpcVlw+^Gulsu z&+aD{l6*GM#yzj?u7UfUeTcqqqbn{HVw0R+{&>IqX|1m&yZU^JF&%ts_i>&=yHm|Ddd1=^zHU-=np2v$ z)56VGyIQE9{9BsHRsX|KVXGF)Ni|2ALb=IQs0^#KFE)vtzBT#O=PzqNJWI z%jsDI(a0%Pyi$PIHcfj1w~*VSapmI`M~9##$L2x`9edIu*iHZLdb+p>?z0|Kl2D7bSTkDR{ zQG><&NFlAfEc@pS7e30lk}S-y73daFU9W{jtQ15P&{b>*u844fOOa5)w1IE>0Dbv0 z8uWbKU_D8-Lsu>vjyC}8*6otT}t)m@!h7qPih7-J<=kz65mN}$Kjd; zwK>NIRBGt7=` zM|96VGuB#*)j+~>#$~K5(nt=0h)z}^T*coF<#|2OXo490zZ8cXN*}_v_)|h`$&d?C zL{|1K4N65UQ=upUfSR<#f_3bh^zpRXgCbS}zDsLL-ChKX}iu#2$r8AJ$dLk`b2lYZk zsJKN!eKQR+SFk-msL$DGHjVY^ASk8x+t z1k31S^Vl6-+;O4N*tm9HVPJb=4z;AreX>o5&=mW^k75@W{je5Fjk+-&c*ab?EucDG z>)F53i)ICl|JMx)>fk;Bew4hQ+*F#mAz^LDfN^L5DZ*SBC^EJu2lYzxb zi#qxr491NdT}EmFeRZ1*BWC|k7bzsvOK58LC1cf{I~Rg}hfCqq?P2>ZlA z-)7D_S+6l{R!08tOx<7|Ue2CKx*Ap-Kgn0GQdRyCdNWKiwFg1W43TRIh3BXe7*{p= zM{b50MF+(W%+y(YpgMRiqI>_B0?fR6Fn)=g*1Y%PU9i}i%R^>%D(;~1#BJi@ z_Q5S|;#dP|)jCi{4fT+O!;D}6>n0N(>yKVdbr>`{db!6Gp`jDy8DermPe)V z`fv5**`9&gP_|f1Gh>-XF19xP6q=J|E_g`ibwPdop1-U;WIQtv`(uDAdx?;T~J#<*Vqp?-F8mFX&SuVaR-gNx{WxJw^NmyVs1>NkVCy> z4_sYTt7pEnKWa1nf(vF7S1WsBS9;uPTiIIs3bKp9oZ#64NhU&nlqCmSq+ya|&KJSA z#u!?&_~vPuzziSA=t*kpB^@VH^A*Tzkr_<#QDJ?Ti~`J^IGCD+&Mv^CD|K)vW|ly( zKjME$fwAzAP#emNi2>}El*!1N9GP&0e@1jz#47*(L7O;EmQiaB6!<$(&EzlzRv~_? zKP<%Gt=-Kco7B6?i$~{w!SqJ%(!4?wO#zNa zZv?At0po2l1MOPvuTS@OxEr$SMLzF-xUo%KOZ4swH^Kx&ci9{IhbFU(NKJ0~{%#c7 z_>C~1RW_|2MAESbgRv7+O3*l4X;xjRf2p>c45~3{6*Xw=$UrMDaZ&Ko|9jZ5JwAsa zT>xLbl3Jgj!(fR*bT;F|(@09rtMK#w=-;S1BhioJfbC?}HF85uBmH)?LZLwld|Vta z5MC2AO>|Gx!P5%i8k@)1sXRA#BI$baq{r)zyy$IXBi8UuHwsXetgfV0tk9vxNW+bv zg{fZ#LjyAxdu#{XDm%1RvAXj9FjVw9_A7PV?E!qNO*NTVdvo{wYZEhBM(V;y+PI|+ z*{h(;U9EF`yG`rGu;MI*5tDTqM+Ww6BeZTF8#nC>PNTGYUT4a*sx&cQu3U6b?Y^H! z5k)R0@$dLqTN%b3`DmMKKSWT!s}| zh~NJ!i&dzQ3NCqlf_Xsi_wg)jXNSA2myfe5fA_XiG##K$K``cK69@OCg=%aj1c@~- z0;A-gfKkA34HZN-)G!jNo`yg0^pKUft8(;)0D!fS6P@X)mJP|r+ECB>Q$J&pBq zZ{1DPGGLuoCX9DVQ&}Ynr~>9X7FRh`>UX43f&n_Wk?W{U?MjVrs=Xd4e_e&Z*sGNP zDd=K-9dx<0tqEFx6h@?y)1WJeY#Fc7+9Jl>OMq88621Fbbbw^Xpp5xMAi-jgDo@oc zRe-?)#eVlnWtl{likz(ag4Qx!XtM2UDz&{!w^*IVxK}{70EQwsqerrBF_TmE^?2du z-gZhD>KV>-Ksglpw;g#0Fb@xu7;sHaWQGo2fm5GVK=$>*c;d9zvO+c40-p^wnMGkf z8Pf=5TI(j-H~@3R{z!?ime7d!Ow(=Af&B(mF-tt`?A^C&Z4YJ^90~Rh7n^0Kii;Ax3JjT?XYU z!XhGRJN?0YMElyCjiLNdOT+wtmDH|mNxAm(66im5P$PBb(~ilz7UjbJzstpH*EEh& z7FK!?XKjex=H)D*$;RFmJuloEsCRQP9zo+p+hk^nq>!Ab(I27$q|z%~LX*RiXBj>* zk5D`$@3Fx@4hGMDP=m|AQvelione}V$^XGzgrUUjWY>Hrgi`XbN$oa4R$nhPts{6f ztE(VOvY6;zU$Atw`g6q-HksCm@WxPIOMY<{HaLYT76c$dGy;04DOMLyd~*!7n|N#o zp|u8^hv*;LC+`e=B8fsg@QGBFn_FviaZ)+Tu7H-uN>4)BCLm9M- zV`UjZtqt9-Gu`3rW5 z?9H&^l|zOF1q^wUP+yuqu)~K%NN2fZD+Jo1QVUQq`%=O(Y$(+}w9tq6RV#<6E>^^y zYgV!hompK`ZF+GnkufH(Qy51)?ksvmUS9}(iNDQJVxXsaS%&upn(nZP{KxY#5@A^i zf$ro2bpq3iCS}LEd%}}8)vn{qs%LQ-Kfl` zO||qXXPc*?f|e)Ye}A5A*7!2nB7G|8I7_J3e8sR?@p%bL?7(4~ZYKb~23yQW=JM0& zffbKar!OBi%yW>@zZwcS&~wxdRX;PL`<$yf>C(Z`TE*d55=qvp5YCqMn+NK!SzaI! zY6nG4|I9bYQm}Y{&sS}3-PK%c{8>(w|KDxk!y|u$G77eNVv{>uV(O_0O+e|{RIg?M z3{YPoGs7a>Eb3ZGa)435H!8q~+r(-=uZ53NSFt>@6`WydgYak2s|6@!Wje{E*L#02 zzX(oD?rc?(1m8gWh10uHZ&3w*l{*985Ya@FrnoiKU--K(K~bQ(*APEFU;tyCw~hoU zgt{X9cK_wSTJq~U+;1p9_ufWpHHH@zl7jj80%89lK0LeysL1O)Uir3|dbL?f(1IMU z^(5K8=U5Xn&Dy|{`uNP_-UjMt79G`w{~jfH9cc-*-bm+^rDPX!-U79=k6}1uVmMp? z1dp8(=7vDh{2Q|}+=1mWe~{^IhaXXt9^(JqL@1gTM5RpEzAMBxtqy_#%N%LaL1gS! zCIp_Labm&_8k-U90zq<+8`$2<>?4kUo=o?45p+=@KU zbGM5-LJS2Sn+^E7NMNQNxB$2Z%-BD+{;hjzm@A&^hI(5YGhIv?b z6a*3|A60-`QtzNI?K~4D#-h+vfQT|u_w(x$hIx-H>(|ht!!ojOR50pv@cmufo6|CJ z)4M&l3C?F5*_*(B`#ceN@5@|tv^VjJHm&T+Q?BtNUF0u0+X#v`Kt#}Unz+QY0k654 z0Ld!M{6Fk{XyoTdTORR7I&EP!xd#2-8wUWK-E>38;XPT>=CHgml^} zRS*OOWK9(T!xCgkAcUnAF%lpV0z?uZvIYntLP&swz&#->V)f2_-rmo1Ui3xfoZt5S zt^Xxe5mtTz&bJfX4m;M_y@7TpTa>WDfWoaDpczqU;O3vo9_XPJu)N5L$V@Zd!P|yp z;0N!|p)rryRX(>PJ_iB~e{1Di1Ap%imj@@$kJYI!FPZz*0rOeAqt5RwyMLT!L=}G2zELFvtXx60RwXFH-h3fc7^_y~K4t|>{kvdyh9IkJPVejR zFTUi!=$$~9dg@4?>94s<#uKI-U=iaD?28d4gQz%h&dm3{tY=-G`^^P@+RZK&uw7H}2UnT2oO^3M+}!LGCp9dD$CHy_q=y6!SckMEowH(!TQvc6%Y(6hEWox^qC)j0E7f7lJ!t5$&O$k7b-p znmf9W-nsPYErIyxtOV$`%q%b-mKI`xBkKqF1~}6;b(DG2F}EJ7c%1Guy*jw~_uo}K zJ`2tXkqY(+IS>EocE>^-b?@;<536ZqhnkRe9924Sx?{+W%{aRhnF}cpvutCR_mDyA zpJM}3lOC70-(-9vG&v(BFD~<>pWHv%17#yU@6Vygua4@gv|j$_zFz`9Mg?u#um6eK zA3yx_Kb>K&CHU{PKKW!n_Ry~)=;@z+?SzJrx_o)Q0aq1YwI=rwyK3F-SsHx9)it}n9zDh|F{{)!Ai9S?Jl%fh3kg3RYEF|?Y(>Zu5*a#$a&5w{^ZW{7<-zB6A$3#$? z@L@<49?cM#(uTx!eyZWZ>WG^A6w7>Yjfg%O&ArID4xV^e*3@E6w4?+Z>ITLvEe z!>%ba&=>aU>FrfCvl9Lty_DcAsOn80_At0_Rx&#sika;x7JSv?h1oh*3z=l%_LGTc z=(>jsc(xd86I=jPjNB`LepQZRkdIAfe3M#$bThLiXJ--exF?SB#k((K>M=F9hhAUK z?>#hPsFW!#YSNbr|HST5aJgO%;q!b#ewpL;svGv|lGXuXdgGW~Ak&m+6LNlhKB}@sDfhLq%NiZqEzo10vCR?H6`7sDoGILcqmdROEhG|qWQwYkJF_C8Q z#NF1c)|#>Dl8(??b!SuxW#GFHnOyMAc;;Op>WhM7S9lr2C3Ncs+rhIe1g~m;z+~i6 zY#bIvIRg+yiNpH|>W9wwNSsN52E%STm%%rOeN-7=kou43bISdqTDvPALlXp#1-NrL zcaDTxfVPwKmwcyzlk2i)@@t>0qjw4de!-Ub6sI%>?C)w$9wlDkfzikCT%}Wtg~YIo zQ0ybkEj18rVbNv`Fh>Yeu4h2o;y;~G_dR;@LKo-V1V z%d=x9k6RWr8AK_HHeQ2iu|7VceS|#ls?$SDJ0|&4tDQkL*z6HXzn1$QGs4r% z3Mb*Y?v6>}%zRHPN7c{(&q7#^*^ZduV=Z87iPBtcNbvdAd>#xsLUezIo4cXSm~>|0 zwY&o`OAu~3tB=zM9I($Sz}mA!DR3LsMwD&K1D84=v`BvG zTs`%!@G0yh*c9|A8!EZh4;nunlOZzIlstb~D+uoeoo1&i@BHXl;N$zC|sDCH&9X?Z~rpM;ffWmGHJm zZs-u7r65qs_y$`X}?}gctGv2LCd~3yh9i!&ZxaM&;7Y1T``bQJshORsXdUSJG zM$yDRGXcR=<}O467d&U+x&#CwV>`P&TrY+IYBM(upQ>?Fh53L-le79GtCc&w zDfe^c>YLw>*X{9U4UG=gv|_sTp#)7k<)U(~Zi>%xSUJtaGeVmVtH8Nhsp4r1nZ7ni z;)Yf?TaI%RJbNmG0&cc{evWh)W~Ye!-66BZ$281BB_Tvqy-PQR-Q;Vjq0=)GHFu=6 zc!_&{D%ghBr1#I8sa3kHU4ZDDCw}E`D8rNuMQz}`sr2lV1!MH}8V?oh*{3sXYxdVc|_#kC^=-)eDmZ$2@>4hbN9{U)L_6VcG zwSp!Nklj(HCaMmO3{NYO$<%P|g$vI?e3nO;72X3Mbk)OF<}gw;Gk-|cR|EZz7q2}cr&hAPDaYKrLv zw+nMQsE>cT*aZ6D2w_R}6i= zpsh$;f7CECRzns{9##_E|A2>8Fdl0wvY*L@o34eBW>vdBG2zMuj9||DY)}hJs3=K}T z4LmXW!gF06Ah?FrV}p$B_7n^p2yw*L;t(ZOerWZWEH1r9d)J-Ditq5brP$sFIo(9C zji9b*BeI=(*F=?Q!0pIAbKghpRmcWbgRo|bZP|!EpJkk*Nvzs$Q0VOUro(QZffTS= z$b5KB_3ir7nO!_?m@d9Fw&?1zvpU7xnoNGH&H#=G$e#Dq0E*d>lSZm3l>1>lA#PB& zO4ak}D7Vn#MHCRWGK*f;`rY}K_u_k6E$I=!qj&~^76=~rEG+@6u z%vnh7FxjIEQ(O&kZi}vCowtT(4jpVWF$Z@c9}}bZpghRst@Z9(i#+nBSxjR;JbDjj z$dIH$*61UIM0Ibt_n{BU>KA{2$L^{ZoP}}@+C)xywMSo=y>6F|nsiydi|AHz*6>Tw zKBmJkIGm6{F``zY6=Ny`67|EvQbHaSg|^STiIm2j-E>Csa~j_AFUPk8ZXPP$>gsI4 zuKl+SJ98kfzHV# zH4~zMVRm-e_dm=T#UGo^xPsmYPKdX&*RsPzGtNyr|=&A*`j*sd`l^qcz> zLn_@Rt19l~WwQi<9@l*~YR-gV=Jxzy>3+^bK5}=5V(jd)hB&(Y`US-pQQNW0JVPfG zzpy+ln13;RgN5+_DGN4=V@g%03zq6zx14wAuHEu|?9mCVG_~%ue-G*`VmtN!8DWnLzcL;v?BujeEPKyN%Ow%YlXRUHV^x9MXdPECA=4>i zJ(iK*iYaOqN!*JR==u1@GHk=)ap007+4JQ1+l}8b5Hz&Y`nb+F{CA@I4g%fPQV)H;MOf)X0lOVSR|cP|68?Ml9~nXmo>>V z9@RFUUXS0K`Fg*ulhm5SzFUhk$@ju{Zwx**^YvZ`pV_S3ne7L6-q+|4wKKB&_d9%% z|Eq&-R_-RFlZN~- zVP=a)&bJj1=d!M){5Q`@4o@E%P-^-Y=g6s7jCDWRWXh>{k9i_y$Ffc8;4jdx)!p(o z(E46fG!L7MUNvb|9{W70b87 z3cQv;=X>aW&LCAtVYau$P_`&IDU=sLbo+M z9(o&(Ag+nW%*mM8%!&W{d1<}GJjX$?O}&IoNAy~+2ylJxnzDmTKATa8^XCtlW@!F zKecb~$VpvOk}m_ebW8SZu8-go*;Vzk3VB_%&U1{y?Q2970vM*e439HBAE%KTseggK$XSytRMP2K!NM2 zI`c=hYIq}f`G|J@ZH3gSgT~eyqP%zX;I(jzfBoNXoqEC@qt_AcD?Xv^yPiKc<(euw zdVWV$*NtNgrZ$hpbBuZtJNRosz}g=(PXL!r$)3$q$HjTu-d&1%Z7!@lDN%a+OR8Pm zwLNP<^hq{EF+%v_c8iWE0YKfzKGLMjpD3|?}aE@Rasys2EDACcZWQ%J&G?=%~{ zobT*Msx+?7ObD+Vp1CR5F(0Lm@@ zjk2DE1wL7a1kd3@ncIe1b=7q_G~%UsP=Rgx0YJH9YqVxfGj+2;p>+ zd$6Zlb!QjTG^K@iF76r-np)Odu+60rYY5da*fKG=+n+^C*^-`|v`>${%A_ti4gPStM6S5D zb;W&v(jOkt?5ne6Qxkw%M#*BKdN)nzwBq>%8%^paw=@NdS$uI>`*BqXbl-@Sk1DPC z!ZENbdBOw6jmj4o+fN7A4h|&^4qtEjgtQ)I5pD9;?QcdW-)cL(O-+|@s{2rbWo-Pp zeiT1)WQ&C*VZuUKS`h}JBi(=0h1Xv8D&7rp*DYs#K>o?nS_wXvs}kDqsdnJwu%!v( zd;#gP!_>ig-cp%+;LW1A1nktoOuo;vPZA8)sv0fe_cvtE=Ba;qgDQH?&ph08EIRl< z3I>f}beIZ#0ebVYmjI|VIfb5HifcKZq4sv0Iy|z}VdQSYl%$pxeKU0P0;h=T3dXcJ zM)~Iy>`}FAY92-I3Vy1&_OeqfOm6ngX+&u%n}1z6HWjo`Q+@p=*C)Fa95fz<`$4xH z=mV&NN>8t}je*MB1I~|(hqt`p)rO7ht{*$ z;9fcT&8SnHqgE}w8JpS~W$k9wD11V-%`OOCvN6*wUy_k82Bd#K`QhfJ*-UqkSR`Q= z{2cFhIXeNYGai*q81$6770ln)u<53S#qBtgB*_34x((!{HK zv7(G>o}Ct2PX#2821tkWUG~ac)7@spdl{f7KG#u|-q2Zk`Y9wQQ9&#&QlOcEO<^VT zhLM_(r5Kj`E*h=oTN`SBSbf6Tq+SNF0!)3M@!<>oNLI_Kxu|Ky=S}*?%KRm9k$E_7 zTFF*9b2rG90XfxcPVEg#NDNrda+`f#y2uYiHIHTTBOSKvw{4!_Pt1ivx~->!#g6D@ zysNtsja;K)W5c$tNti5Os||doS0ljY1W+WPy1Ez;^PrM@{37#qny zvS8`)l_LK;SE++ERUEQ!=Rv7V`M9PRhu4=YHdn7*^CG8SLor21I{2xhil@+4XgARi z;1tywo;KuJXqaAV*rUBcdvzA~4(&*OMAMzz$okpXq01`qB}>@oGt#PUw-Xm`f2n(n z22Dt`Ua!12!>&l!rh{UR=}3vDcHg($>Qe2}!8RZO{O1i0ooQ-Lxt+{g_Z*&B4!UrO<4H6Ja!$^(?2h(sa(sR-zv7h43gH9T!w4LJIu)BNKcL8kf9gyJqfs?j-P~(cI z=q-Gz3$HZ#w6ubITVcBWWbgqSLf=LrTrKK02R{al_GxEe&F|)_yZ1IAv{5g>K(I1Q(;bL zgYI*#UHG@`J zM~N%&FqOB0yO>{d=ZlolSL0y|1jx>V#^>g~#9=?+%c{!>9t(erjo2b*$0|&R5*+RZ z#f8;|^@c5Cp>FXjyF z-#svB{CQx&^bLDA_qv@?^o%(X_1$uT(1wv(L#eSW-rVl_phs#`J?VSXo ziV;1IvA-5n`~j35d@stZOs5~mJUeNYd083ltC%Jq`7zeZ|99kwkx!Q<3YHQ-ow|dM zZ(t@e$xUCYPChgo$X{QAM<}oQ^ro79WOQK~Q}3xK?lk>M@$-N<9^g(`l$$Is0#fCE zKq>`~nMdaK7lUfM;yWSQ`77Ot!A4N_d%yG2Y>mgiME?2mrZev}Z)i~72q}Lc7WYUz zqIqVyrVGxPbN{KEb~Gw9!DcJTNxc;(K0sD(u)!Hk(Z4?t1^8cJQzoM z%>9y3`g{N0AYl3f{JE7JbL(qSzKqpo^&QNmhD%=$4gK)ITBsdjY`F^S4~#Tv{ie$& zF;3stop6ji|34rx7uC*bI{8DU?@G{=NBElcZftYaP$3M^X(;%K$KPDAW)4Ah8g#Y4a3O@7z5!aOx;IB;D-$gQW(u=K z);|DIDEmW(6L-ytJ3bvtS-2atIML3I5NPTY31Iy*g^j>I*EDZD_v*SG-rTo zmWx8E*H|fg>GlPFr??gv`sy_LdK9Mh$(==iJ#-`b@e0HVY8p+WTO|jRHyHBx`GxmLUI%O z#2pvd!QCRfheptsMJOx;v1*<5n8*x!zjX%sJi;N`pSnCFHH`L-QFB6!@hDyz#q}8= zOG4i&Jk}lm9e7N`*fWhgUswtmJYf2z>We?#)s(z*C2$ZnbkX)+ zU5Fz&uF;0t|JpmzH@9%RJ$oB@0V{^~mbW_)N~oW(UMItaSLgXFd4!;uiDp{>olZ zjz%fPS}4Rb(rR_avnX~olgIr)(W3XOfyknbs*+}dg_ZOwa37xq9xeU&;nKbVd z58)%RML49ET>zCP!Gz7)zyFfmANILeP|3)#-K5YCk`Gbx;D zZLL#|#tPq_qEhb=6Dnt#UIc!?R`fXw2BiAu>vuhbBk1VaDOIH7(QpSxltU0WN82&d zj@T++)!et{icTbVxr#PVLVzDqdxlqtkNv6UIxTH^p*_36m~1C>s2jgtBNJ}sy1Ym| zvX;U(iqr;CdzE`Tl`XxYK>mGre1T;u?Edt1e~zTw)Tm}Cg%;a!yV5V!iu?u z9n)c(qPGc@hWnBETo1~AW>2`(UtfB$)+6>!o5 zL1>^LvgM4fhs@Uuh`3vmDJC_%8895B2nD!(4MsYPpg|Ll`2zFsfLFi>BnYU}Xy=Ng zK4|yzala)3HHU7wg=74&%qn~BWZ>{hHm?-EzGkO7_cR=^I=TOPloby+;xeIGqqNfYwST(RkvVUJ^J(0^m`BbD$G!wL5CE6LD(O$IFz!nbWY zY3hwaFmBSOJEv)Wp3H=_uYr~^=kkwn{z;o*owdF@MR5{6d>fJ4AgEA{`?I;rrL;C+ zApFL18)mV%UAF(A7iL!?S~NfpR$Pu`NV^(fdGot9AvTU#aXZynu#v+@U|6@qD`!WF zp{o_)di6a2TD@WXHesn%(!YZYCAxR}C7NTt+O@dYOxxRs_YN`c%8@OHHHmN%?gQF+ z-D8e%N~OE^Xr9t1Q_fuxq*A1AC4gJ>)-QBa*?n%P1^Hm zudPdLM)d3+_Hl8;@|5|YC^HaW#B|~rGp8nEDy6s$t!f&iw>fFVSP(zg0kL4-q*HRc zuZNs#{O48B_HKjznRvq~if|@JM^cfdH3M?%oQ>n}kq)F%r|n6Fs4i*?Ga#OGq3O5X z=>@yLJeYZ9GxF6}=hOjj+G%w|w+Yd7VO9LnYp1H&!^Hf0{s|iPcavG$=uuTw=}SA+ z<6>Gv%|7;+Gp`tEybDlgrHO<-hqt^+EhV#cR5PXVQSZ&PEM7%9R8`LQhMubx>m|1d z5Ycni&?`*_*~mkXWw|X1B1lM>ikR>`g|0pb=$h&qgpXuy?NSY9DLbwq_Le&vey^uC z51tMj`ojE02P8(!^f)?nwu2wpIc6%Ikqqjy z0`VlKhg8OM&nBLaIAN6zkLcHWO;r7Lvz5-P{|^y1d-#Qfbo-=aA%)7>Gf(7@8x*3kl{pB~38GKEE@ zCcCAh5%Fn z3+XXOQ!io5$V5VMbdSI0R;-(sz9}{<57@aze*=d}a9NMu*{VaBALH}~pNwo67rUCH z-xR6d@+>oUH4W50Sm>dR)hQQ`B#93cd2}-<78;xIKeD!TU=5k*aXQ>GVuzW_-~9yG zt5!7YLqmtOT5G^8h@X|}3E6+yHrd3f7nwO#RvA~1^Ma42s#(nFm1L=nXvk4&CDmWs zx|dv9BULwNFV1T=MAQ}ENNSjCsXr&%k=9QEY0RAlSpe@;JaW6~lp((Mmo0(AH(<5A z+xi|TUQ>He^rSS{ZG}>?+aq{VkMX)rO4hX+oX8;;$2mH zLBoX9sS-8O#ZZaU?q3wOt1eQ^FMpS;X0d%*AzFAa#-qrcc#85pg>vrv)YGB&5fq%G z)eqf9qzMmW^lU7Fy?bX8A~P?J96gdYkdnWYxf-6Uy2uUlCQg;hkJ+KplHBlK5w)Ga zIJwpHEI{kJ3fPi| z{M3;e-20Daio=?T2iCyQ)?IZ%d2WK2>y94c<;p@~JFR}XE>~8uBt*|Cs?h;spV2gb z?DhbDlcJ16pHj&(@*O!)|2@1KYx@_+xdU0wS5P(LxR#O@+Y;(b<2iuwwHqlXIy4C_ zlye$R7NE~6u&V1Jv+5ty)yt&cR?N2jz!CS>&eG&}_Z&KM(LdUv&bp&`e~!J*uSx(y zD=-4kzbn`O$iduLg9hn$jMJdg<1KJ;rZ{*l4{zaWT66pOPBD=EzpWE#4^O^V@y5%}V=YIK zhGVxwj@7)012TyY@V71H@H>jCQF@~XCW9u1{t^pGw&*TNJW41!SK2TwYTp-xB#*)R z=W^O*ncciWvQ0HU_8S0?fv%t*Mm|kBZa;NRdVI8@q#B+`u{e)=x{wdaUPViIJK&U! zx#p&G@$DZ4z1bTGz^XdM&OY!Q=oH1ou(y-_7fo)#py@GycU~9N0#=S{TIw)emhWmNXGnBQ8SD@U;|@cCHz;)P7T=O-UsmC zD2L9+|2cA>+W4+U5lv2gOUb$BCX#p{_IaF6QcDXMQA&xl-<|~DfsmO%zLIDq3l2p` z3bNmgbGU_vza+$I8xI9I&3+DSI4v)B8v@++4)I@-O!4(q6CajsWbL{~w|RGeuoG&^ z&(t`gHG~a{c>TG1qji#+Vio+hOj)U(QQwZLC}-tQ@g}{cuJ&yr z(_=LQ@6`r(r_?-2&3VmEXF)%Xe07BU8g9k8^k2BUBO?g2b2nOznt*pFaeXO1?|RcI znQkhMl&)DE5VkqO14c0DN5LeD+M8YY3y4Ff|L=DX-il+p30_^lfIGVpmg)EdNl z*LHxhnt=%geziEz1wK#2a|3mZ>m)jfLAKXPAW;n|?mpql=%015Ww?Uswss=0N=Z^{PynfaQ%9wjlnHAy zv*z4qjzr%W%{iUK@}l^$1c9=nF}lr{RMH-af=?fFl{_j>@bmgiVD%soiO3FjACA3#I>}AZ&YhJLvT~Zm0ue4oPgh#{V@1wCdMBz8T@<0a z(~BiH(bY+ffFen_zwktQW;7lP0@%3-uoJS%P9W}b@NhA7*Ewe($Ud)G>Ikc};zE)p z4kK%i6a+AK$iK2W2Ra20_M<)?_40G)RD%Y`Vfem>9`_O{wk|+%L#|03g$~)Jg=Ht# z%H$R5%g(^uA8yWh{uG-_jjZne$dH$2B?&?1glgwEpJyg;Ikm&#>5kCx-KVbrM!~Hf z@mgIFzcc(@ny$KA$iK~YozE*z8dYh&)QztwUU;q1^0Nh87tjQ*bmguHwS+l6MH}T4 z$}OGAyGGkGL>%~Jgvz3)x0DxRt!26GQ1euN7U7-6p}oz~xXU?@`InpRPj%ED?4dc} z=EfpdSr0q0!n&8Jc-QGvbrUOz+Jp~Un~a+tP}St}ouS;ME9IFo?W}I(T+>c@<_TN6 zc1em$$XV=^GritI5btGRJ0#BWrKU{=$#kM8*GWpw$UmBA9(~L>&@6m;pqZzeT}6Mw zHH~YA8jMu(_O4ElR+pEu1SZc=o^&bv)B-!mgaWRc9`rO)Q zi0<*9o9!Ku9Usj;l3uZvgn(Ssh$i_0L?<{VtSz-d(hW1?@AGCyZY9Rb%xfx_Yt5sZ zHBE|6+5zV!Pn-zA=<%z;l4qxx819g>JKqPAKAe(YOMsfm9VwnyG5_XqtnrvHyYi2T zM;}+wfD0MqA|)85p>VvDyUyg#oOVA3n1;D|nfXCcT?81rsDk#irqYx_ zMsmtW!?-SILjDw1LAbhC!tNyXSs?>RH*d=BzDQOw@=CPBoU~U5Nq&cn5A<*zWC5V9 zJPnV`GD$Nl}#KT;?hd(;~sT`#u+97300BeSk+T5DSNE2I=@eByn zemj6jKq=93rA#hgGiv#`a>9mu&q|`MdBhN_u**V;;Z_80$Hs*BIo$i6U zJqqH!a_SM`R20*~{Jk-#_?qG&BdwUMSm2YEGB6mu0|rZiWkD6W%`Z$?y--@o1K5Jo zN89w9GfJ&dIY3CND;C%@W)2A6cw?&XXHfPdiAPEU7&O_5^26L=>Jwz53W%d=Oe)}yfB=}|1$)$5$c?nlq_!0e7aRirpSC9mJ ztr>6$zjs^$gm|IEKE}zr!g<=EZIqy9}ho)VlPLjJ2j znhf0C^YT$1Z~>5YzgP{6$~L)KZ$Ueo(2dM0@}KJjJ7K4iCp*mP(Y(mI&p|mAp{Hvf z!Qbz3+cWJfN3v4KuP0Rq?gK~5vmDwQw7^@PMpO;>23qLEEx~4^z4_Fl$gU7v>}-HC zOM}(lrIAaSup?VYl)?}VVO9Kv)82P$du2ffwi4QiTtZ7;MXZ`$OF7NVkcY979F93` z4^4Vd&P)|yPVh)aVF{NK)hZLrxyIB=w#MW>4pjV@u%{!9d#z1a=2vW+ea9dPIB*IW z>m7fU(D+)O=Xwt0d!9jLJ54Z`9tf^=d0=b3n9FLCdnU9YCLx zEG*3vRD;3@2rq)OZ=m?<@4}!5qQHeK@6r>lEB4nszaJQRXs7H-s)`x=5e$HV6|SbQ zLUc>0jnbQ)_iM1{)MPNPC85OWy$-TWyE;l;%d}q03mNv$eKO*Y^n|%%xr?JOk=2lp zNuT8*x2CA8zMjm$9#Q&7lcXyT63jSrcFq+xRlLYWvxM>D$sK8#`c>9BDKdX~d5y~B zu|ck{HQ8^H%iwS)UdA^z*4|kJR6J-^sYjtZVziqtGj*K9@@EP^R~HUULQvw!FB7^! ze(Yy{r%B0B+MI^uYVudj6Q!Lula(T@6G@jxa1{&wBh|c(O-?<*<;U(|6G?&Wg$%t? z1aL#d8H7nUE+pKHGa#M~W}5cEAeF!_=DdK^07eL&g0NLSrkYMk#}imL7TP;+4!Olh zJ|TW0&jO(Ef!C(3yXHGw(FYuDrV3HX@qjj1C$L)drRK^Y_|>N!cns&z=Pfp4MJ5&A z`JNZT;NVgZI;2^fhpVX}6!@8nZZ+2bp=}q4;PDP#Mu`D8%P*LVaahIS-C=?3>CJ-4WP7 zFNRiShAHESH~>JH48YIwR+a7XYFV*jOa1Fn1u~@>3S6NS00m%Sk=pS#*RjwR3rUU_ z-p>UgHYOJ9w^jI=01M#DhIV_JwK-8KpDaFBt&uhREDx<bq~fH;I>BS-sXo~S-guuJY^!rjCDCuNZ)9Hb z%nPoMD+Iz~;F+Y}Dg_%&n`Y~(6$>nIdEoVGDnwDVJT!TsrzLRVs*=;e@E!y)6w!{X z^-ho1>JeTiQC&aJE}a0=F62uSn6UT~f^|rYa4vZ?5LRW&I?76>I$)MGV+fJm5RMVA z;i$o-r7GKoRI}1UaK)0vF#UoO<0`0SBdDrsIi%u7G;Le|PKBR(9awoCJ;`6C(<|c- zDtjDT_k-l$kf`3gEc%&$r*`EhPe$Nkx)UiGcuO<@rCi`L2^TzSdB0uY4>Pv7U)X}5 z!#z@4Qd5Qq8{6!teC5TKp0Z;>U#jU;@w3_v2(Gv2DZ7?)rxDqtePAa>0fxS0ZY@y* zOy$6Wae`r0dyZ8BwTYAo3&dNJ(xC3?B4{c!VL-3M^Gq4NE6jff5$CuB&ZIl$uyA<; zmH+5BaUC0SzX~)yK&ulj#J=Fnq&N1HNIg()$+ayPw)N%r`-eFyk!BHTmIyTOaLW2 z29fkY5QY+ic&2ziklAQ=KI&x{TV{6D#P|+O3xwmt_D-L0#X~uLF|RnriN>SWKWotf z81!o{`)S!)vl;u?d)Jh|znP&ineqDGO4bC;{+)ed{62#WZP}CYaxU080Jz<$zc{uM z?{~D#h>yBlZhS`&;vx!&OsDKm?;}kIq>YsCz`V^kU|#{ic>OmqwY`vSFDYTpz4rhB zh4q(l7rS19NDyh32*bo@(G6`|eCft%CrmRZ!RTyJ_6Z)NtEaku{kF($u@&sSlO4RA~%@01g%J|jaU3Jd- zqGS4ZIz`sx9(;*eiw>z!RnN*-lCKT&u5BpRaq!{?6&CsI-#iG1&Dig;rQ!FZmLs@46 ze@4BO!4b{X-!S* zuijm;YNS;3UaDWC#$|XPFEaY((!u0jZ%tTso2O<~0OM;puTMg<*evvm$?H8;#!DhY z!YLq`j22UX>mu*V5#ne1S$4xj%`4!>KwF+2wNNontH@ftM_7BbqZ@!Blp4w8t2v>f zca`x`4-&Y&fpF-0Xt9e84&=Z7rDBWepfmrst=~^Q%&i4jTHK>u3SDxhmeUBP?a}f+|bS$$d68T$O^SIJ)|H!kM$iM(mWkWoo3{`BB{VAR6K0eGwA1N=8TWW{>3=IXz*4|zHqr5Tjn z=B?QP)GvcsERr;&^yuQn*~LtVlETg26OzMFAm|}>ZtLi%Zw_-g`)aN?+bgB3Dxl~p z{hXaMY@}Ad_`qRC`+i;ZJ%?i!YEz5yl?ZSaB491=H!p0HsAel4j~rI$|C0Rb1++(s`uM4}oYcrh80GZI=z`nFFcITg9x)8V{iq;`dMGh_sFb9{FK&@XEjEjtB( zMGjX3?;H0}Twx(SlpvO{LW#=fEf4x{Nm%^8wD)jJeQeJ8Jp}=nA9dw92GZ$-|NYvF zhxp`4dc6bQ$5(?}oW;25@NBpSq}HwW=kPVvg~_|$Q~~Xhs~o+5P~lTgo7AcCnJD_) z_1JGf_`k5a46s^3`?;ZVx!9+netSSmr+1Z4_XewoJI!HLR9k1BbLi(kq$E##;~ex= zRHHZ2-+3W`gh*{9ZjvTH%)G+emszB*JS2kxPM7H8XnYh-J)b^| zb6xPIc*>yA2Ct@t_sj&QB|#IaxV?Q~t#2nIr$&3`dT|e}&3VT$5qtAJjeVKwL~X{)LZ!WFtbU#0F95TJ zyQMayrZ5P8SBP1pC(Eflw^gn;@}_@(0foOX(@F^eK=b=b6<99T{mpWRK(n9cC-LLlwiDRoxy7~$TZv;=J{C19dU7o?c)?#Ra&9Wr|x1a7^vsdm>8!u-4%(7LJEo zxtGs$-fM6KW+iDOA<;q0uq&D*wx@@xLVWKb3FIXe>^!yFd4p!Z?>~_Ub@-}F9h(g< zEh!45JI^_*I(?Z8^I_DR7OJJW#iwLaM~hM#3EWkc{_(3IQWAtwHkXW&BV)gD1`?{p zTKOzv(!nE7%ISeko#3&X3O!$BLCY;0N@}bDy*l|YcAt?PCZ$%5i*6-gM-!SY9dp2$ zBu5Afm(q1DkU!c0KY62#4B^LL|7jRk?W0h+q0_mVqx0P4(=?-0hbk-=sUl8D^OosU zGiK4#@@GqJGR49U=K}I4b<6SjH>+x2{%EPCr;prD5~{E`BDy?QR^}&!bqV57qYU~I zig#QQnT74m#no76>SHqEHHq69G@o=PI*HV2Fl^x%oiWo*fwA!AU@A{J>KdUV(sXff zY>NTfP~}@9umYZ2h0UU0+XFwv%!wG{k zf7(Xd8J~qaO=f(n64vuc$e~m8tz3(3rD7#N8Bk5AmCM06&xbGZ%cIJc4EW9G7MLFw z9$g204%i<}BzJ#Hy$SIS1lUc99IMg?cK8mGXy$jNxWaCI% zsQV?e(nStErz+NlCfgzg*j6A(;F~_-N}D#v=kSMUxo6BoV5DlFsIwk~7?* zL2U$b>=1BaacV)kVo~S|VM81@XBfC7dainS{&ha0Zq>C6Zvuc-O;>i36mkRi)LmOX z7<@Focc+AE*DV5!cTCw1sJ(n;g*tEQcvY&%BNfG&KflM$HrErle6v3sV{he#4=L;Zp9>D`MR*=;)^J^)L&$s=sZu0i@U!(FUIVJ21^u zKlBI%G>w=^mI!&P=kAuIk(bevo(uxjC>4UzuY{r|3|r8(LJ(jL=gfG48e>59@G#*5 z5k|4lbSlZhLqk?^5TJqS>86`fzL#K*N5{lr7f>VKoDIxuQF$?#?yoKVx z!mnE9;&zVqr0OGG{nEZs97{W)Upun?O26Cd&YmGds)i2GXk!*iao(w2rr5Y-tC6ED z`=DdI$j(Yz_|>e??1CIc9pG9~oEK)k$BUFh0&aqR0iCX9_&fD8aS6a>M*U4rc^`>W z$!ti(2#@&XFu_Qz%=L!)AqXmoS&4>2nI<~7&ambzevqi?_$=pYSI}(0Y_qR$aCdem z3~!BeQ&v}IoGXf}L>~iW+0J7-+>cM_b<|`OXB{7Tn37BITP@@My>)5Oj4j&3nG>On z?%+)XUHu1M^Z_@7671bo>Bg6tffJJia*yzpxYZMpt)K)p$e_3ou>D@lxyL@3;g`F| zj~CR$8M-|SwQ8<_P={w<*<@()hwesNFu1pSPbEYqUmPh)sRH2UZy0`QnM3~%iaghF zNdMQ~EfO{LqM1iAS1?1C5>*&;xiUNL$2?SicYVs(Bd-DoH@P;{+NGo347f$kE@JT*ZxVx4K*;zKRpxxAR{1Gw(}*vJ&|;g zWc*L(5HGV`7rG$iHj5m4cP0cBq%C|5jMlOY)T>`ad20mqOfQu<6s5e_sjLZGsgojm5tbc}$m zlr;WY6d2{|{{?Z|cGN4amgKM6XD^z{;+1w%FQUv6L$%!10V(>fE8Fc#)}xsM5s1?= z`^eCEVokQ35!28Y^h$W^y~rG(OsPuAB>bQDuJoa)YumruR;iC$xouxTtHRZ5wUq*D z)i5S)tx_sOs{+atl_CZV0WnM=NyTevi^A2&pdhIesDwl$0}zr36i^USLIh_hpft$%B?0+yVA)4t3p*npyq)j@?4s$G!)-oVW0h|i zP!15*8ei=+T!Z|2w~?TVh;>yq@`zclVD3fkvf@9jxwx(7fLrj5fWq;ZmrD^EZf)mL zW@*3pnvyQZQ>Fc^dhHeAF>Sfp@&_jFazV*tRHdAKsebZ7zK%W)Kr!sjc`LixvOh@A zF2F81&)@m+gzCHvbT}g>yu4=QyHB4d1_s<$Cdd{v)?XP>* z)KwP4n_eUBy7~)4aZ+3I*(585d**qD#~U@1NGKU%!+X8V_aciJFh~x zr#FL!gCbf7d9Sip?Ug593oYUmb$?fKXt>_|{p!EUPie0d zAF7I}Z_Y7H%*x+vt^Hre{=&|gYQg&hUx51Vj)r7_j((?w{=Fk_8dJdOO{$~}g64$z zc3iJ!Ne@W|)D}K?k?qJ%gw^#O`D%3lXgb??8MyX?bpDQ|*{k!1$g~?Hd)%eL=0xq? zVkUn!UgRwh(Bpka7sN#k39m`jy6;ki#h+H3ytUJ7$F%WS7~m5#VF~ZepNej}>gV-q zRKmgfZ(FZ)%_eWdWMBU^R)#%F4rV{Ew+2388QIh`hicb9I`i~ud`a zJZ<*v$5tPgnV)b;k4T`;`hUAWzDwO5yZBGcDphRu@>no{77QzJP&LDCUWJu&W0{>R z8N$&Aj7f6Tzxb!3Ydq>#eObD5@{=FJJ03C2XIa!(uEesXsAekvM;XoMQdvxK;*C$; zc5iTNoW^QCG594cOdnKsHnMz%vL|O)W5CZul4FK3TOxHw<0fN~k`ZTjroTBP&qaoY}SMVx{gLx55ukno|f*VG8dQ3JCe+_WV z@%4YzAq@ZN3J?IwkTyL#QsK+RCovgvJd=}r!uw^jha#bKQ}&?v)nJk9#Kq4_{m0(( zu)jeE)=v!8rlsEvREFT)Gh1sLAJmt*T*tVTK|!mpg+=Y7RHxHoyD9@JZ%0s`W4pNq z>yJcc*K~4eI*-+48RFrVxpn!te*xA5=~&x19pQS^{A1S_-IsD0qXQI-8$x#}R?n5m zio|F>%|GLv#shp^{CP=#e0hq5F#qit5p~90omu8ywf@Y^ek*hD|H;tfpVRHFJz-(v=~@xUr4G=v z8-9(?eaVkD#_K-t`6xNmb3@w?VB36OJr_hFdp2Z{`5v)d0Zr7My86%ad(V0P1KU(t z5jjV8|v;22zOXEqj93gg3fzin?>zLn?y6Mr5(%e7l zRjz*j1Su=RMD5R?CHjw%!V7(ZF#qn@GNTdn>|NxI9ghlO6s$Jmr(g~o8|Z**3kl(U&5kkTa0O}e;Z|LQw*3ixpfO2SHoP>&Z{>s zEtJw{8XX--?_RRp0U64Qz}G&o)tf*T8x-AmO#9EzA}gYN60{|Z&);N!JpL@rq_Vg5 zmnmOxD9>v1NxuZe(_eZXaTJyCet*z4Bg}fh#}O{aY*6fUzWv@&?~H?#&}|&Urbh`C z=QmR6SHdH}yBrGEx9tS=+h{wekkRfpKPs%lh9PW*QQp@XLinQbzum)FxjVF%56#vN z=%R*mdv!|dy>tp9VSemmT876d2fXp-5Xe%wGJKtUeoV4mg%qcxb2o@ zav1jpiNBi#{axrbr(m<@eL831X;4z5GKMEW!zZEDv`@~qel+O!)Xi4mw_loFyZJiE zRP1N1XP;^8nm)+JBx`>ub`O32X5GktB|j0dg*xN1K`{4fNySo|UI;FV4O+vc`26MJ zSuHVWakMh{Zf3d;&}W(X=`Nnwy4`kW@7azz6>3rJFUlHRmb}YOf2`gLByTvn zF-m*7Sa`QWzXl~}e`f6ZcdJ&WrjXE%?Sq*YOg~@SDtdMTH03Gr(Sndo`heA(cI9im z=cLz&brArygyB2oZ9wit2RhLf%J&y#>-_@5Pz_uZmoXbS{`Bv%;n&Hyks&nk0wnRZ z+{9R44k>777Jr-Q#-krH7ZU_~Nam@jySm8(p0u-zS0J@ldi|oBFfpl#(A$4 z9DYgPHy*ddo)88uhmJ0Zkrf8lSTrxQP>h%jAIGBB%tBYBxsI%Jvs8dXB@HZv$h_zZ z97`AJ_EEq6MP(Ki_&?76Sm7jsu94hgKV{DI+n5xLn0U75kNf~zEpzc^7-e=!c-@Jb zgbZ7I4fO7*joL#he@h``R7eaxw@LJ_rAd7DLb<|rFSmcHMK^(W`iX?7Fqd3kSi8e= z$A`8yr!Cg%CYOEXeFCfo>ca?_PR+-y_0|$d&?c?SLNt7y^n9ha=WW)`npbVUQL*8Z zeew6=AH-9hz(qy8!p4Z{BE?+EKZxrDJUR80sqHwvS@)2{4tQ0aMvs+Qi1Pe_>LP24 z-)cm_B33ZTd<}mmBV4TU!>Z;q-j=t~ln7U&q_BQX+CA zDPb_v}tOISy_HgYp>ba3!?&4W_4U(oT-xheFQBmhQ@e>9|z2f`^@P@Xt zRYH^1?C2ml+dhP!OLisCYW(+J>ZzW8oii;c?hMC9X2*8Hd6L8CvCZUf(w68SgTTlB zDORWo6_Wg7zB%m3OiBy&sJee)J8$%^Gyla~B~e)!4cm#2ytSz}yEM1B^JB%4!OnL= z=wg;n+^d~&)}3)>P+!{x0?&KmLViKNTxy#iyXL}W@_wH%V~Sj4SEbZgpQFD+^66U@ zb>r=X@fCi!?+dcxsDvF-%_gRs`SDhlssLNK!Ece!1Iqc($)(EnC(_~Af4zU-KHlT0 z)Z=o^t=o0HckerYQXJ^AZJ_n;{HCh;ysEZ#0%Kn&Z*)3U)315!-hAG@s5QG|H}8(! z74CSiE9(1jwQIN|3lyYEYnYpnCyK`z1^r3K2hZLd+#SYgCy>?!EE2R?U&%CM?1{`0 zMmxdOYgtG=KBQ^oVnUp&W24IU#0EpZM_q+P;9?Jn+PZFQ!Jn$7r|pE-VC~?a@(M!@ ztG8J@3@Ln+&HN|~7TWoJm>AU3XLCg?eVAPm`;K@C+srExo96OxR`ZkS)h-uC1T^fz zU|%F|tYo6SkIEK$TOKTE;tk4Xi~MAP%yMn4>}5)hiFc?I{*BY5FKI(MsB+sK{U@e+ zy*s(cr&-3OgeoaBuDf_~iTaLRXF@G!%K)}O6-p)F-qf%3L|nF3`lEA73hn} zQWj-W)2e(b+T}4)OG#7aqg^jmx!P9cJT67!N^*b zP;lcJO^R`D?Tg9-Iw01&s05Av><^~pg3@a%N#hrUV=yYpSUu-VIwmB5Rm8alVEtd6*M3TNa6YTqE_4V^f>cG7cwE($^w=(bBvu9H-^v~Eo~ zcZU882VEtyPa1Z86~-3k<6rDD@62oSO1|!Gd+WUAR+Z)J^Oh@BVd9|Fg?(UiVaNB` zxs1`J?hiXiADnij)wRBSkHGMX^R}$y0qwBV?Nk{~dfy-1I*wUYD7iPy0P4&uDG4{* zeSOC9?^ts}2<|Z6c?YdKntHTfQ~u6Y?ERyp{B@F`xqwpD<|Fw1Kj4G=SY8hrH2Vy{ zkv^!PHMV!DHZNTi6oEMDf|srgRK(Kmx>X$X(1+Hh*5ZfGPJOtQ^kGHr@V6q5hvM8H zEh}55qu&wcu6@caOu_HpnO!iNfZzWpPw+0ESt|Y}xi|Wq@_PLKgCs^UsHyeqE0&#C zN^dMWw}9^hbw}#5Ka1$BL!KU~`~x~(AvtMXk&s`9Gq*m<(!=574BzoZn>XNdjGU+aq(kj9NdrjwF-H= z@w37D`wMkVUWaKB__3=Cf^nAr!-bZhq+dK=$9eGNf6JP^rXw79&Gcu$RZcX`hMyzc^`XZo%3`MmO{tz5|X=mLR(aVX9Q zk9;2L7gLt8_9s5_SGj`FB%g6DD3;J7IxliA!p6QMUn3rgUs<_*A za~^p;Z35f|!FJ+_xUxNQQ{=H$ZkyoUIQy0sQu*iG$~||j^9rVn15_g2_&j)>WMAs5 z*NczJT{9I^e=%)o0Wnbq>2J-bCR&iLm-6)0r*jU$$*;n3`gC` zZTJaJ)#&dgc4TSvet%v)M=Xs({}4PK+QiAe#`odps#W$78U%U$J1V;oDj#9(8ZqouF* zdWnJ_R%*fY@jlvs+#VOKiWDDo%&acRbaK&$I&;HZ<_5op=Ud@3)cy% z2U>n7Z%r3fY^=hOX@-eTrrK}m+u0%1U(pAECuSXL+?*)9d8Q%A_f2Z(F$W*IuH|hh zeuyr6wg&LA&4SjFNnS4Liskrl1%0SxU=m&}HjG@bWiAIl0?nKMm23#`C^TS!7l-ZK znaN+*>9*{+lSId$crZCoRA)p-Xedr@p#RB7)QSxBCJ{!6Lb@7@s^x0BfNCd!lgWsg zGwYrVjvm>pF5vPYAF0;C^brUN)EVhRe54kF(MGDhoQSGVF;NjTj1F0ax_U{!-Cjc! zN|bhcx6X(^EkSb;+oU`Z*H5$8@|Eh-bab80$V}2&oph6(*Xgx38)pywmJJS6@ZD0F z)O$qu+yo-0qge_#KAkTH=3$~RkteZz={~-8(kmu62eA=|LIW*()522*(nI`LBJ!-h#|-JVA?X zAi}@bc)mil5vbtB`O%r=6x!o(nVvr0n4Hpt4#-ps@8Pq1T4YQiGzx|Q4xsn+;18j( z^crbzZ2G*NN7OC*Ce$$rhmVGzERH1~jtx&7pbn^7_Rb!eDVrGrZ8SIe4*E{iumiHr z@-$YS3x~5OT#;0l&Ev^qh$P!{{cQ)ex{IDvW0OsCxM74MDmF0>=5mqE zCIV*!YI?m6fuog@0UGK6#PtOh>Fl9Dwl1k9G{2 zo{!l!cvMO_XShL4E8iFk%$Fc;h=oT~_Djo}=H$nTU~a!WdBi;8&7mkKLk|X2IOB~& z(OBb*C|WA`@P~p6w1>84p1E#9O|TDwfS?*nKaRplm%l?qaY$CCM<_uxY)8g>(8Nbs zcd@7x)H2Z{<^DcT-cUmGl=gR>1T`e@2u5t`LYK~0Xs8h{u+fBOFNGS-R_K|0ojpi_ zF1O=0Ux_;GWiUzK1Bp@O3H6;(?`875WkgF3`a?2rgGla*`$J5rtv54tA?u@%<#^DSqJm z4ev}`AD?nmVYo{ay2x6AJUI3mlvXQ*q$}$Dg;e`_qR{Q$ib$exUGiTB!zmb?X@uv( zd~UZ>CpiIG&oyJ*WT(pS9R=*B3^c9wq5(_S^~ms4%~*WWxk&f%%e>JVB0LAk1_fh? zrS{m{l8z5GCVSEn{n+_2o$_m(rU+xXO3g2vt~y@uF;&o!GG?7ql5Zz zL2r~opJ_;5Efx=;WZtK>fG?1HJYb(pod?WrUk{6+u|{BC8pCVZoCeEHj!+gE*D&pM zLpWig-`vnyN^RdQu$l*d_97^3!xVg$(uLmMmSVY81H~B=^|n7{^p@9-74d@5(>`6} zM(d^@QeP!=^3~R@oSY{$=pR%zr6r>#Jx-%fymS8Y`x@Kjutxn}W(538Q)KS^@(nW4_{jMnkp9wisur-UfeXHBsKaz zolokI8h!f#$4ub5Da%mPPAg{>#)Lp!9yP@fOm0UTC~4s8 zMCl9Ty<%h`2g{St*tQ;^$gbu^_3#QM>SA1$C^{&R=YZxiS9=X>eE`Q zR9CM~QrOxcGq6uCJZ)7v0vZIgLi_@Mcnl#4Wo20r>gmQ zk7g7X?Zf*U9TfVt$4G$pBT(Z?NYGUr6?(YwkoNohg8b;v@PhH=9f-YNE=^uW^IYvQ z#^2u{%uk^&j=+ucN9G;M(N4|s8tu)p=UuYZMt+GT+!Yfv{RW7ZzrJ*tq0hLX(nS-l z&dml!ITi< zvIReQ_BfiU09XtOu3=WWT8(_XC}@t!J!4Np4D;phC``|>2%>}405EcpR|X!{-U%Tu zL-HIX&-sXgD8whxk@g8GhR!BpfGN6E9*gE6iZC7#MIaay0H?T0f>I5m;Dy#s0v&b; z>$k9_#RgYMQs-lrwT41;+ZOQp4wKHX9Mtd>SJ|k$Ngp&%Ol%z}J^NQG;~3TFNgX(m zYQ*4yLEGM~+B~G~{`Nt|bm`?_^(JrI&MOZjh5w+f4KfJc9W1@ff;{qcBmrMNOT0`v z)k`TD1!DMr|1k4V0#tm;0)1V^gB*2P!TYskE3W36U7mQxYIbGHV4Rf();V_8&@?x zEi~2%25om2xItb2);P-pUGkNvPM4~Axf^i;c-0;?dWhs;N$!!PdlMv?b6#Nh5Mw^6 z=e}S(*tseM{KBhNF5m$zbbpL+JCXG0H3e}f743USg`}%p2EbiBoBgW|dp*PiiJNC| zG7h+8ctjegai*Ns=@kt4a=)%V-nnWZx&T_fCc%qAA;du5_FC1ZtdZBn)8bw;p{TM5 z()$&$%bz3kY{Yig1Xb0HzrMp>pVSiWj`Iqx0jxr>{HS;0ae%5D$1e~t_{Qxz`FV%9 zsIrXy1jl}P23#vh;8`RyBRq^!sLDG$pPphEiyTz&^wge~QoL#IykrPM9AVm058m>T ztQnFrDND6r?sS<%U!`W?L^U`kD4FKo9| zn~sn+)dKOd-BEoX1(z<7mhpd(FAO5`RET0zD?5WK6ZR#wl@!r|H``T&)s zPbA}OkZ6S=jx4D`;s9b|kHBqBUW2weJwu0u$&q+UKZZ*&*=Ddg%P<-_YtI?rxkyYi zqH@<4`cicStOgtYEJ00fz-AoF9E^@r*r4O4L0JY}0?mLFBR9_xQR>{2CXOun#4NFn z%;5++T~!EBlwS)war4&Q4q-{s%Z8f7Z$xBh?poSX&II(=VZWtw5GyOf-8gKz=#x7b zQ;+7gB}#WYc%^8Y?>HZj)!)~_drhn)#lRP{fir6(3RnsQhsadEXl^9RD#^v}>qnxb zk0E58Ts?bjsS2Wq2m;`cfFkUEQR~z`v=Y6f+GbZlFwB#x)qLcbdnU-*=cNi%i_ySe zG!v3X2bLrTJbje*8c+U|f04}?wR3>+!Xv`TIs@P + +## Install ClusterLink CLI + +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} + +## Initialize clusters + +In this tutorial we set up a local environment using [kind][]. + +To setup three kind clusters: + +1. Install kind using the [kind installation guide][]. +2. Create a directory for all the tutorial files: + + ```sh + mkdir bookinfo-tutorial && cd bookinfo-tutorial + ``` + +3. Create three kind clusters: + + ```sh + kind create cluster --name=client + kind create cluster --name=server1 + kind create cluster --name=server2 + ``` + + {{< notice note >}} + kind uses the prefix `kind`, so the name of created clusters will be **kind-client**, **kind-server1**, and **kind-server2**. + {{< /notice >}} + +## Deploy BookInfo application + +Install the BookInfo application on the clusters: + +```sh +export BOOKINFO_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/bookinfo/manifests +kubectl config use-context kind-client +kubectl apply -f $BOOKINFO_FILES/product/product.yaml +kubectl apply -f $BOOKINFO_FILES/product/product2.yaml +kubectl apply -f $BOOKINFO_FILES/product/details.yaml + +kubectl config use-context kind-server1 +kubectl apply -f $BOOKINFO_FILES/review/review-v2.yaml +kubectl apply -f $BOOKINFO_FILES/review/rating.yaml + +kubectl config use-context kind-server2 +kubectl apply -f $BOOKINFO_FILES/review/review-v3.yaml +kubectl apply -f $BOOKINFO_FILES/review/rating.yaml +``` + +## Deploy ClusterLink + +1. Create the fabric and peer certificates and deploy ClusterLink to the clusters: + + *Client cluster*: + + ```sh + clusterlink create fabric + + kubectl config use-context kind-client + clusterlink create peer-cert --name client + clusterlink deploy peer --name client --ingress=NodePort --ingress-port=30443 + + kubectl config use-context kind-server1 + clusterlink create peer-cert --name server1 + clusterlink deploy peer --name server1 --ingress=NodePort --ingress-port=30443 + + kubectl config use-context kind-server2 + clusterlink create peer-cert --name server2 + clusterlink deploy peer --name server2 --ingress=NodePort --ingress-port=30443 + ``` + + {{< notice note >}} + This tutorial uses NodePort to create an external access point for the kind clusters. + By default `deploy peer` creates an ingress of type LoadBalancer, + which is more suitable for Kubernetes clusters running in the cloud. + {{< /notice >}} + +2. Verify that the ClusterLink control and data plane components are running. + + It may take a few seconds for the deployments to be successfully created. + + ```sh + kubectl rollout status deployment cl-controlplane -n clusterlink-system + kubectl rollout status deployment cl-dataplane -n clusterlink-system + ``` + + {{% expand summary="Sample output" %}} + + ```sh + deployment "cl-controlplane" successfully rolled out + deployment "cl-dataplane" successfully rolled out + ``` + + {{% /expand %}} + +## Enable cross-cluster access + +In this step, we enable connectivity access for the BookInfo application + by connecting the productpage service (client) to the reviews-v2 service (server1) + and reviews-v3 (server2). We establish connections between the peers, export the reviews service on the server side, + import the reviews service on the client side, and create a policy to allow the connection. + +{{% readfile file="/static/files/tutorials/envsubst.md" %}} + + ```sh + kubectl config use-context kind-client + export SERVER1_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server1-control-plane` + curl -s $BOOKINFO_FILES/clusterlink/peer-server1.yaml | envsubst | kubectl apply -f - + export SERVER2_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' server2-control-plane` + curl -s $BOOKINFO_FILES/clusterlink/peer-server2.yaml | envsubst | kubectl apply -f - + kubectl apply -f $BOOKINFO_FILES/clusterlink/import-reviews.yaml + kubectl apply -f $BOOKINFO_FILES/clusterlink/allow-policy.yaml + + kubectl config use-context kind-server1 + export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` + curl -s $BOOKINFO_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - + kubectl apply -f $BOOKINFO_FILES/clusterlink/export-reviews.yaml + kubectl apply -f $BOOKINFO_FILES/clusterlink/allow-policy.yaml + + kubectl config use-context kind-server2 + export CLIENT_IP=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' client-control-plane` + curl -s $BOOKINFO_FILES/clusterlink/peer-client.yaml | envsubst | kubectl apply -f - + kubectl apply -f $BOOKINFO_FILES/clusterlink/export-reviews.yaml + kubectl apply -f $BOOKINFO_FILES/clusterlink/allow-policy.yaml + ``` + +## BookInfo test + +To run the BookInfo application use a Firefox web browser to connect the productpage microservice: + + ```sh + kubectl config use-context kind-client + firefox http://$CLIENT_IP:30001/productpage + firefox http://$CLIENT_IP:30002/productpage + ``` + +{{< notice note >}} +By default, a round robin policy is set. +{{< /notice >}} + +## Apply privileged access policy + +In the previous steps, an unprivileged access policy was set to allow connectivity. +To enforce high-priority policy use the `PrivilegedAccessPolicy` CR. +In this example, we enforce that the productpage service can access only reviews-v3 from server2, +and deny all services from server1: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl config use-context kind-client +kubectl apply -f $BOOKINFO_FILES/clusterlink/deny-server1-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: PrivilegedAccessPolicy +metadata: + name: deny-from-server1 +spec: + action: deny + from: + - workloadSelector: {} + to: + - workloadSelector: { + matchLabels: { + clusterlink/metadata.gatewayName: server1 + } + } +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +To remove the privileged access policy use the following command: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl delete -f $BOOKINFO_FILES/clusterlink/deny-server1-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: PrivilegedAccessPolicy +metadata: + name: deny-from-server1 +spec: + action: deny + from: + - workloadSelector: {} + to: + - workloadSelector: { + matchLabels: { + clusterlink/metadata.gatewayName: server1 + } + } +" | kubectl delete -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies documentation][]. + +## Apply random load-balancing policy + +To apply a random load-balancing policy on connection to reviews import: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl config use-context kind-client +kubectl apply -f $BOOKINFO_FILES/clusterlink/import-reviews-lb-random.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: reviews + namespace: default +spec: + port: 9080 + sources: + - exportName: reviews + exportNamespace: default + peer: server1 + - exportName: reviews + exportNamespace: default + peer: server2 + lbScheme: random + +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +## Apply static load balancing policy + +To apply a static policy that selects the first peer in the sources array and uses the other peer for failover cases, + use the following: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl config use-context kind-client +kubectl apply -f $BOOKINFO_FILES/clusterlink/import-reviews-lb-static.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: reviews + namespace: default +spec: + port: 9080 + sources: + - exportName: reviews + exportNamespace: default + peer: server1 + - exportName: reviews + exportNamespace: default + peer: server2 + lbScheme: static + +" | kubectl apply -f - + +{{% /tab %}} +{{< /tabpane >}} + +## Apply round robin load-balancing policy + +To apply a round robin load-balancing policy (which is used by default) to the connection to reviews import: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl config use-context kind-client +kubectl apply -f $BOOKINFO_FILES/clusterlink/import-reviews.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: reviews + namespace: default +spec: + port: 9080 + sources: + - exportName: reviews + exportNamespace: default + peer: server1 + - exportName: reviews + exportNamespace: default + peer: server2 + lbScheme: round-robin + +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +## Cleanup + +1. Delete the `kind` clusters: + + ```sh + kind delete cluster --name=client + kind delete cluster --name=server1 + kind delete cluster --name=server2 + ``` + +2. Remove the tutorial directory: + + ```sh + cd .. && rm -rf bookinfo-tutorial + ``` + +3. Unset the environment variables: + + ```sh + unset BOOKINFO_FILES CLIENT_IP SERVER1_IP SERVER2_IP + ``` + + +[Istio BookInfo application]: https://istio.io/latest/docs/examples/bookinfo/ +[policies documentation]: {{< relref "../../concepts/policies/_index.md" >}} +[kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start +[kind]: https://kind.sigs.k8s.io/ diff --git a/website/content/en/docs/v0.3/tutorials/iperf/index.md b/website/content/en/docs/v0.3/tutorials/iperf/index.md new file mode 100644 index 000000000..5a6afbfb6 --- /dev/null +++ b/website/content/en/docs/v0.3/tutorials/iperf/index.md @@ -0,0 +1,251 @@ +--- +title: iPerf3 +description: Running basic connectivity between iPerf3 applications across two sites using ClusterLink +--- + +In this tutorial we'll establish iPerf3 connectivity between two kind clusters using ClusterLink. +The tutorial uses two kind clusters: + +1) Client cluster - runs ClusterLink along with an iPerf3 client. +2) Server cluster - runs ClusterLink along with an iPerf3 server. + +## Install ClusterLink CLI + +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} + +## Initialize clusters + +In this tutorial we set up a local environment using [kind][]. + You can skip this step if you already have access to existing clusters, just be sure to + set KUBECONFIG accordingly. + +To setup two kind clusters: + +1. Install kind using [kind installation guide][]. +1. Create a directory for all the tutorial files: + + ```sh + mkdir iperf3-tutorial + ``` + +1. Open two terminals in the tutorial directory and create a kind cluster in each terminal: + + *Client cluster*: + + ```sh + cd iperf3-tutorial + kind create cluster --name=client + ``` + + *Server cluster*: + + ```sh + cd iperf3-tutorial + kind create cluster --name=server + ``` + + {{< notice note >}} + kind uses the prefix `kind`, so the name of created clusters will be **kind-client** and **kind-server**. + {{< /notice >}} + +1. Setup `KUBECONFIG` on each terminal to access the cluster: + + *Client cluster*: + + ```sh + kubectl config use-context kind-client + cp ~/.kube/config $PWD/config-client + export KUBECONFIG=$PWD/config-client + ``` + + *Server cluster*: + + ```sh + kubectl config use-context kind-server + cp ~/.kube/config $PWD/config-server + export KUBECONFIG=$PWD/config-server + ``` + +{{< notice tip >}} +You can run the tutorial in a single terminal and switch access between the clusters +using `kubectl config use-context kind-client` and `kubectl config use-context kind-server`. +{{< /notice >}} + +## Deploy iPerf3 client and server + +Install iPerf3 (client and server) on the clusters: + +*Client cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests +kubectl apply -f $TEST_FILES/iperf3-client/iperf3-client.yaml +``` + +*Server cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/iperf3/testdata/manifests +kubectl apply -f $TEST_FILES/iperf3-server/iperf3.yaml +``` + +## Deploy ClusterLink + +{{% readfile file="/static/files/tutorials/deploy-clusterlink.md" %}} + +## Enable cross-cluster access + +In this step, we enable connectivity access between the iPerf3 client and server. + For each step, you have an example demonstrating how to apply the command from a + file or providing the complete custom resource (CR) associated with the command. + +{{% readfile file="/static/files/tutorials/envsubst.md" %}} + +### Set-up peers + +{{% readfile file="/static/files/tutorials/peer.md" %}} + +### Export the iPerf server endpoint + +In the server cluster, export the iperf3-server service: + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/export-iperf3.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Export +metadata: + name: iperf3-server + namespace: default +spec: + port: 5000 +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up import + +In the client cluster, import the iperf3-server service from the server cluster: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/import-iperf3.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: iperf3-server + namespace: default +spec: + port: 5000 + sources: + - exportName: iperf3-server + exportNamespace: default + peer: server +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up access policies + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +## Test service connectivity + +Test the iperf3 connectivity between the clusters: + +*Client cluster*: + +```sh +export IPERF3CLIENT=`kubectl get pods -l app=iperf3-client -o custom-columns=:metadata.name --no-headers` +kubectl exec -i $IPERF3CLIENT -- iperf3 -c iperf3-server --port 5000 +``` + +{{% expand summary="Sample output" %}} + +```sh +Connecting to host iperf3-server, port 5000 +[ 5] local 10.244.0.5 port 51666 connected to 10.96.46.198 port 5000 +[ ID] Interval Transfer Bitrate Retr Cwnd +[ 5] 0.00-1.00 sec 639 MBytes 5.36 Gbits/sec 0 938 KBytes +[ 5] 1.00-2.00 sec 627 MBytes 5.26 Gbits/sec 0 938 KBytes +[ 5] 2.00-3.00 sec 628 MBytes 5.26 Gbits/sec 0 938 KBytes +[ 5] 3.00-4.00 sec 635 MBytes 5.33 Gbits/sec 0 938 KBytes +[ 5] 4.00-5.00 sec 630 MBytes 5.29 Gbits/sec 0 938 KBytes +[ 5] 5.00-6.00 sec 636 MBytes 5.33 Gbits/sec 0 938 KBytes +[ 5] 6.00-7.00 sec 639 MBytes 5.36 Gbits/sec 0 938 KBytes +[ 5] 7.00-8.00 sec 634 MBytes 5.32 Gbits/sec 0 938 KBytes +[ 5] 8.00-9.00 sec 641 MBytes 5.39 Gbits/sec 0 938 KBytes +[ 5] 9.00-10.00 sec 633 MBytes 5.30 Gbits/sec 0 938 KBytes +- - - - - - - - - - - - - - - - - - - - - - - - - +[ ID] Interval Transfer Bitrate Retr +[ 5] 0.00-10.00 sec 6.19 GBytes 5.32 Gbits/sec 0 sender +[ 5] 0.00-10.00 sec 6.18 GBytes 5.31 Gbits/sec receiver + +iperf Done. +``` + +{{% /expand %}} + +## Cleanup + +1. Delete the kind clusters: + *Client cluster*: + + ```sh + kind delete cluster --name=client + ``` + + *Server cluster*: + + ```sh + kind delete cluster --name=server + ``` + +1. Remove the tutorial directory: + + ```sh + cd .. && rm -rf iperf3-tutorial + ``` + +1. Unset the environment variables: + + *Client cluster*: + + ```sh + unset KUBECONFIG TEST_FILES IPERF3CLIENT + ``` + + *Server cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + +[kind]: https://kind.sigs.k8s.io/ +[kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start diff --git a/website/content/en/docs/v0.3/tutorials/nginx/index.md b/website/content/en/docs/v0.3/tutorials/nginx/index.md new file mode 100644 index 000000000..bda0d5e80 --- /dev/null +++ b/website/content/en/docs/v0.3/tutorials/nginx/index.md @@ -0,0 +1,231 @@ +--- +title: nginx +description: Running basic connectivity between nginx server and client across two clusters using ClusterLink. +--- + +In this tutorial, we'll establish connectivity across clusters using ClusterLink to access a remote nginx server. +The tutorial uses two kind clusters: + +1) Client cluster - runs ClusterLink along with a client. +2) Server cluster - runs ClusterLink along with a nginx server. + +## Install ClusterLink CLI + +{{% readfile file="/static/files/tutorials/cli-installation.md" %}} + +## Initialize clusters + +This tutorial uses [kind][] as a local Kubernetes environment. + You can skip this step if you already have access to existing clusters, just be sure to + set KUBECONFIG accordingly. + +To setup two kind clusters: + +1. Install kind using [kind installation guide][]. +1. Create a directory for all the tutorial files: + + ```sh + mkdir nginx-tutorial + ``` + +1. Open two terminals in the tutorial directory and create a kind cluster in each terminal: + + *Client cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=client + ``` + + *Server cluster*: + + ```sh + cd nginx-tutorial + kind create cluster --name=server + ``` + + {{< notice note >}} + kind uses the prefix `kind`, so the name of created clusters will be **kind-client** and **kind-server**. + {{< /notice >}} + +1. Setup `KUBECONFIG` on each terminal to access the cluster: + + *Client cluster*: + + ```sh + kubectl config use-context kind-client + cp ~/.kube/config $PWD/config-client + export KUBECONFIG=$PWD/config-client + ``` + + *Server cluster*: + + ```sh + kubectl config use-context kind-server + cp ~/.kube/config $PWD/config-server + export KUBECONFIG=$PWD/config-server + ``` + +{{< notice tip >}} +You can run the tutorial in a single terminal and switch access between the clusters +using `kubectl config use-context kind-client` and `kubectl config use-context kind-server`. +{{< /notice >}} + +## Deploy nginx client and server + +Setup the ```TEST_FILES``` variable, and install nginx on the server cluster. + +*Client cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata +``` + +*Server cluster*: + +```sh +export TEST_FILES=https://raw.githubusercontent.com/clusterlink-net/clusterlink/main/demos/nginx/testdata +kubectl apply -f $TEST_FILES/nginx-server.yaml +``` + +## Deploy ClusterLink + +{{% readfile file="/static/files/tutorials/deploy-clusterlink.md" %}} + +## Enable cross-cluster access + +In this step, we enable access between the client and server. + For each step, you have an example demonstrating how to apply the command from a + file or providing the complete custom resource (CR) associated with the command. + +{{% readfile file="/static/files/tutorials/envsubst.md" %}} + +### Set-up peers + +{{% readfile file="/static/files/tutorials/peer.md" %}} + + +### Export the nginx server endpoint + +In the server cluster, export the nginx server service: + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/export-nginx.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Export +metadata: + name: nginx + namespace: default +spec: + port: 80 +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up import + +In the client cluster, import the nginx service from the server cluster: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/import-nginx.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +```sh +echo " +apiVersion: clusterlink.net/v1alpha1 +kind: Import +metadata: + name: nginx + namespace: default +spec: + port: 80 + sources: + - exportName: nginx + exportNamespace: default + peer: server +" | kubectl apply -f - +``` + +{{% /tab %}} +{{< /tabpane >}} + +### Set-up access policies + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +## Test service connectivity + +Test the connectivity between the clusters with a batch job of the ```curl``` command: + +*Client cluster*: + +```sh +kubectl apply -f $TEST_FILES/nginx-job.yaml +``` + +Verify the job succeeded: + +```sh +kubectl logs jobs/curl-nginx-homepage +``` + +{{% readfile file="/static/files/tutorials/nginx/nginx-output.md" %}} + +## Cleanup + +1. Delete the kind clusters: + *Client cluster*: + + ```sh + kind delete cluster --name=client + ``` + + *Server cluster*: + + ```sh + kind delete cluster --name=server + ``` + +1. Remove the tutorial directory: + + ```sh + cd .. && rm -rf nginx-tutorial + ``` + +1. Unset the environment variables: + *Client cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + + *Server cluster*: + + ```sh + unset KUBECONFIG TEST_FILES + ``` + +[kind]: https://kind.sigs.k8s.io/ +[kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start From 9283022c06c4c5cdd035c85a2167a1ff066005ec Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Thu, 13 Jun 2024 15:33:42 +0300 Subject: [PATCH 49/53] controlplane: Add error message when hearbeat fail (#645) Add logs to controlplane when a heartbeat fails. Signed-off-by: Kfir Toledo --- pkg/controlplane/control/peer.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/pkg/controlplane/control/peer.go b/pkg/controlplane/control/peer.go index e5189250a..643bf15ea 100644 --- a/pkg/controlplane/control/peer.go +++ b/pkg/controlplane/control/peer.go @@ -121,7 +121,8 @@ func (m *peerMonitor) Start() { break } - heartbeatOK := m.getClient().GetHeartbeat() == nil + heartbeatErr := m.getClient().GetHeartbeat() + heartbeatOK := heartbeatErr == nil if healthy == heartbeatOK { if !healthy { ticker.Reset(unhealthyInterval) @@ -138,6 +139,8 @@ func (m *peerMonitor) Start() { if strikeCount < threshold { <-ticker.C continue + } else if heartbeatErr != nil { + m.logger.Errorf("Failed to send heartbeat to %s : %v", m.pr.Name, heartbeatErr) } m.logger.Infof("Peer reachable status changed to: %v", heartbeatOK) From abcd6d95923b1b4afeb8524e92f4b7200c751152 Mon Sep 17 00:00:00 2001 From: Kfir Toledo Date: Mon, 17 Jun 2024 11:08:25 +0300 Subject: [PATCH 50/53] website: Fix allow-all-policy.md (#646) Remove the tabpane function from the included file. Signed-off-by: Kfir Toledo --- .../en/docs/main/tutorials/iperf/index.md | 37 ++++++++++++++ .../en/docs/main/tutorials/nginx/index.md | 37 ++++++++++++++ .../en/docs/v0.3/tutorials/iperf/index.md | 37 ++++++++++++++ .../en/docs/v0.3/tutorials/nginx/index.md | 37 ++++++++++++++ .../files/tutorials/allow-all-policy.md | 50 ------------------- 5 files changed, 148 insertions(+), 50 deletions(-) diff --git a/website/content/en/docs/main/tutorials/iperf/index.md b/website/content/en/docs/main/tutorials/iperf/index.md index 5a6afbfb6..af835993d 100644 --- a/website/content/en/docs/main/tutorials/iperf/index.md +++ b/website/content/en/docs/main/tutorials/iperf/index.md @@ -173,8 +173,44 @@ spec: ### Set-up access policies +Create access policies on both clusters to allow connectivity: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + {{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies][] documentation. + ## Test service connectivity Test the iperf3 connectivity between the clusters: @@ -249,3 +285,4 @@ iperf Done. [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start +[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/content/en/docs/main/tutorials/nginx/index.md b/website/content/en/docs/main/tutorials/nginx/index.md index bda0d5e80..06da58408 100644 --- a/website/content/en/docs/main/tutorials/nginx/index.md +++ b/website/content/en/docs/main/tutorials/nginx/index.md @@ -173,8 +173,44 @@ spec: ### Set-up access policies +Create access policies on both clusters to allow connectivity: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + {{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies][] documentation. + ## Test service connectivity Test the connectivity between the clusters with a batch job of the ```curl``` command: @@ -229,3 +265,4 @@ kubectl logs jobs/curl-nginx-homepage [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start +[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/content/en/docs/v0.3/tutorials/iperf/index.md b/website/content/en/docs/v0.3/tutorials/iperf/index.md index 5a6afbfb6..af835993d 100644 --- a/website/content/en/docs/v0.3/tutorials/iperf/index.md +++ b/website/content/en/docs/v0.3/tutorials/iperf/index.md @@ -173,8 +173,44 @@ spec: ### Set-up access policies +Create access policies on both clusters to allow connectivity: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + {{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies][] documentation. + ## Test service connectivity Test the iperf3 connectivity between the clusters: @@ -249,3 +285,4 @@ iperf Done. [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start +[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/content/en/docs/v0.3/tutorials/nginx/index.md b/website/content/en/docs/v0.3/tutorials/nginx/index.md index bda0d5e80..06da58408 100644 --- a/website/content/en/docs/v0.3/tutorials/nginx/index.md +++ b/website/content/en/docs/v0.3/tutorials/nginx/index.md @@ -173,8 +173,44 @@ spec: ### Set-up access policies +Create access policies on both clusters to allow connectivity: + +*Client cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + +{{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} + +{{% /tab %}} +{{< /tabpane >}} + +*Server cluster*: + +{{< tabpane text=true >}} +{{% tab header="File" %}} + +```sh +kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml +``` + +{{% /tab %}} +{{% tab header="Full CR" %}} + {{% readfile file="/static/files/tutorials/allow-all-policy.md" %}} +{{% /tab %}} +{{< /tabpane >}} + +For more details regarding policy configuration, see [policies][] documentation. + ## Test service connectivity Test the connectivity between the clusters with a batch job of the ```curl``` command: @@ -229,3 +265,4 @@ kubectl logs jobs/curl-nginx-homepage [kind]: https://kind.sigs.k8s.io/ [kind installation guide]: https://kind.sigs.k8s.io/docs/user/quick-start +[policies]: {{< relref "../../concepts/policies/_index.md" >}} diff --git a/website/static/files/tutorials/allow-all-policy.md b/website/static/files/tutorials/allow-all-policy.md index e11d37d48..3f8b06bc7 100644 --- a/website/static/files/tutorials/allow-all-policy.md +++ b/website/static/files/tutorials/allow-all-policy.md @@ -1,49 +1,4 @@ -Create access policies on both clusters to allow connectivity: - -*Client cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - -```sh -echo " -apiVersion: clusterlink.net/v1alpha1 -kind: AccessPolicy -metadata: - name: allow-policy - namespace: default -spec: - action: allow - from: - - workloadSelector: {} - to: - - workloadSelector: {} -" | kubectl apply -f - -``` - -{{% /tab %}} -{{< /tabpane >}} - -*Server cluster*: - -{{< tabpane text=true >}} -{{% tab header="File" %}} - -```sh -kubectl apply -f $TEST_FILES/clusterlink/allow-policy.yaml -``` - -{{% /tab %}} -{{% tab header="Full CR" %}} - ```sh echo " apiVersion: clusterlink.net/v1alpha1 @@ -59,8 +14,3 @@ spec: - workloadSelector: {} " | kubectl apply -f - ``` - -{{% /tab %}} -{{< /tabpane >}} - -For more details regarding policy configuration, see [policies](../../concepts/policies) documentation. From 819d320d0146cce49bc048ad290676c6d5549b04 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 17 Jun 2024 12:06:55 +0300 Subject: [PATCH 51/53] build(deps): bump google.golang.org/protobuf in the grpc-go group (#650) Bumps the grpc-go group with 1 update: google.golang.org/protobuf. Updates `google.golang.org/protobuf` from 1.34.1 to 1.34.2 --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-patch dependency-group: grpc-go ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kfir Toledo --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index ce1674225..fe77eaceb 100644 --- a/go.mod +++ b/go.mod @@ -16,7 +16,7 @@ require ( golang.org/x/net v0.26.0 google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 google.golang.org/grpc v1.64.0 - google.golang.org/protobuf v1.34.1 + google.golang.org/protobuf v1.34.2 k8s.io/api v0.30.1 k8s.io/apimachinery v0.30.1 k8s.io/client-go v0.30.1 diff --git a/go.sum b/go.sum index 44065f672..ad793e13a 100644 --- a/go.sum +++ b/go.sum @@ -255,8 +255,8 @@ google.golang.org/grpc v1.64.0 h1:KH3VH9y/MgNQg1dE7b3XfVK0GsPSIzJwdF617gUSbvY= google.golang.org/grpc v1.64.0/go.mod h1:oxjF8E3FBnjp+/gVFYdWacaLDx9na1aqy9oovLpxQYg= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg= -google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= +google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= From 0a5de24d0c521df3b413a7280be1843099cbec0a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 17 Jun 2024 14:11:57 +0300 Subject: [PATCH 52/53] build(deps): bump github.com/spf13/cobra in the cli group (#649) Bumps the cli group with 1 update: [github.com/spf13/cobra](https://github.com/spf13/cobra). Updates `github.com/spf13/cobra` from 1.8.0 to 1.8.1 - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](https://github.com/spf13/cobra/compare/v1.8.0...v1.8.1) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-patch dependency-group: cli ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 2 +- go.sum | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/go.mod b/go.mod index fe77eaceb..3dcdb9265 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/google/uuid v1.6.0 github.com/lestrrat-go/jwx v1.2.29 github.com/sirupsen/logrus v1.9.3 - github.com/spf13/cobra v1.8.0 + github.com/spf13/cobra v1.8.1 github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.9.0 golang.org/x/net v0.26.0 diff --git a/go.sum b/go.sum index ad793e13a..0471a1b54 100644 --- a/go.sum +++ b/go.sum @@ -12,7 +12,7 @@ github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50 h1:DBmgJDC9dTfkVyGgipamEh2BpGYxScCH1TOF1LL1cXc= github.com/cncf/xds/go v0.0.0-20240318125728-8a4994d93e50/go.mod h1:5e1+Vvlzido69INQaVO6d87Qn543Xr6nooe9Kz7oBFM= -github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= @@ -138,8 +138,8 @@ github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncj github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= -github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= -github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= +github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= +github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= From f034f988934e8880d91edd468d0cdc4dd501c154 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 17 Jun 2024 15:50:21 +0300 Subject: [PATCH 53/53] build(deps): bump the k8s group with 3 updates (#648) Bumps the k8s group with 3 updates: [k8s.io/api](https://github.com/kubernetes/api), [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.30.1 to 0.30.2 - [Commits](https://github.com/kubernetes/api/compare/v0.30.1...v0.30.2) Updates `k8s.io/apimachinery` from 0.30.1 to 0.30.2 - [Commits](https://github.com/kubernetes/apimachinery/compare/v0.30.1...v0.30.2) Updates `k8s.io/client-go` from 0.30.1 to 0.30.2 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/kubernetes/client-go/compare/v0.30.1...v0.30.2) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s - dependency-name: k8s.io/apimachinery dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-patch dependency-group: k8s ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 6 +++--- go.sum | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 3dcdb9265..09dee52b2 100644 --- a/go.mod +++ b/go.mod @@ -17,9 +17,9 @@ require ( google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 google.golang.org/grpc v1.64.0 google.golang.org/protobuf v1.34.2 - k8s.io/api v0.30.1 - k8s.io/apimachinery v0.30.1 - k8s.io/client-go v0.30.1 + k8s.io/api v0.30.2 + k8s.io/apimachinery v0.30.2 + k8s.io/client-go v0.30.2 k8s.io/utils v0.0.0-20230726121419-3b25d923346b sigs.k8s.io/controller-runtime v0.18.4 sigs.k8s.io/e2e-framework v0.4.0 diff --git a/go.sum b/go.sum index 0471a1b54..062242770 100644 --- a/go.sum +++ b/go.sum @@ -268,14 +268,14 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/api v0.30.1 h1:kCm/6mADMdbAxmIh0LBjS54nQBE+U4KmbCfIkF5CpJY= -k8s.io/api v0.30.1/go.mod h1:ddbN2C0+0DIiPntan/bye3SW3PdwLa11/0yqwvuRrJM= +k8s.io/api v0.30.2 h1:+ZhRj+28QT4UOH+BKznu4CBgPWgkXO7XAvMcMl0qKvI= +k8s.io/api v0.30.2/go.mod h1:ULg5g9JvOev2dG0u2hig4Z7tQ2hHIuS+m8MNZ+X6EmI= k8s.io/apiextensions-apiserver v0.30.1 h1:4fAJZ9985BmpJG6PkoxVRpXv9vmPUOVzl614xarePws= k8s.io/apiextensions-apiserver v0.30.1/go.mod h1:R4GuSrlhgq43oRY9sF2IToFh7PVlF1JjfWdoG3pixk4= -k8s.io/apimachinery v0.30.1 h1:ZQStsEfo4n65yAdlGTfP/uSHMQSoYzU/oeEbkmF7P2U= -k8s.io/apimachinery v0.30.1/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= -k8s.io/client-go v0.30.1 h1:uC/Ir6A3R46wdkgCV3vbLyNOYyCJ8oZnjtJGKfytl/Q= -k8s.io/client-go v0.30.1/go.mod h1:wrAqLNs2trwiCH/wxxmT/x3hKVH9PuV0GGW0oDoHVqc= +k8s.io/apimachinery v0.30.2 h1:fEMcnBj6qkzzPGSVsAZtQThU62SmQ4ZymlXRC5yFSCg= +k8s.io/apimachinery v0.30.2/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= +k8s.io/client-go v0.30.2 h1:sBIVJdojUNPDU/jObC+18tXWcTJVcwyqS9diGdWHk50= +k8s.io/client-go v0.30.2/go.mod h1:JglKSWULm9xlJLx4KCkfLLQ7XwtlbflV6uFFSHTMgVs= k8s.io/component-base v0.30.1 h1:bvAtlPh1UrdaZL20D9+sWxsJljMi0QZ3Lmw+kmZAaxQ= k8s.io/component-base v0.30.1/go.mod h1:e/X9kDiOebwlI41AvBHuWdqFriSRrX50CdwA9TFaHLI= k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw=