Skip to content

Commit

Permalink
Refactor ClusterMap
Browse files Browse the repository at this point in the history
  • Loading branch information
@XAMPPRocky
XAMPPRocky committed Sep 10, 2023
1 parent d653025 commit fb10779
Show file tree
Hide file tree
Showing 57 changed files with 697 additions and 1,181 deletions.
6 changes: 2 additions & 4 deletions agones/src/sidecar.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,8 @@ filters:
on_write: DO_NOTHING
bytes: c2lkZWNhcg== # sidecar
clusters:
default:
localities:
- endpoints:
- address: 127.0.0.1:7654
- endpoints:
- address: 127.0.0.1:7654
"#;

let config_map = config_maps
Expand Down
2 changes: 1 addition & 1 deletion benches/throughput.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ fn run_quilkin(port: u16, endpoint: SocketAddr) {
let runtime = tokio::runtime::Runtime::new().unwrap();
let config = Arc::new(quilkin::Config::default());
config.clusters.modify(|clusters| {
clusters.insert_default(vec![quilkin::endpoint::Endpoint::new(endpoint.into())])
clusters.insert_default([quilkin::endpoint::Endpoint::new(endpoint.into())].into())
});

let proxy = quilkin::cli::Proxy {
Expand Down
1 change: 1 addition & 0 deletions build.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
"proto/data-plane-api/envoy/type/metadata/v3/metadata.proto",
"proto/data-plane-api/envoy/type/tracing/v3/custom_tag.proto",
"proto/quilkin/relay/v1alpha1/relay.proto",
"proto/quilkin/config/v1alpha1/config.proto",
"proto/quilkin/filters/capture/v1alpha1/capture.proto",
"proto/quilkin/filters/compress/v1alpha1/compress.proto",
"proto/quilkin/filters/concatenate_bytes/v1alpha1/concatenate_bytes.proto",
Expand Down
68 changes: 34 additions & 34 deletions docs/src/deployment/quickstarts/agones-xonotic-xds.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@

## 1. Overview

In this quickstart, we'll be setting up an example [Xonotic](https://xonotic.org/) [Agones](https://agones.dev/)
In this quickstart, we'll be setting up an example [Xonotic](https://xonotic.org/) [Agones](https://agones.dev/)
Fleet, that will only be accessible through Quilkin, via utilising the [TokenRouter]
Filter to provide routing and access control to the Allocated `GameServer` instances.
Filter to provide routing and access control to the Allocated `GameServer` instances.

To do this, we'll take advantage of the Quilkin [Agones xDS Provider](../../services/xds/providers/agones.md) to provide
an out-of-the-box control plane for integration between Agones and [Quilkin's xDS configuration API](../../services/xds.md) with
To do this, we'll take advantage of the Quilkin [Agones xDS Provider](../../services/xds/providers/agones.md) to provide
an out-of-the-box control plane for integration between Agones and [Quilkin's xDS configuration API](../../services/xds.md) with
minimal effort.

## 2. Install Quilkin Agones xDS Provider
Expand All @@ -26,11 +26,11 @@ kubectl apply -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITH

This applies several resources to your cluster:

1. A [ConfigMap] with a [Capture] and [TokenRouter] Filter set up to route packets to Endpoints, to be the base
1. A [ConfigMap] with a [Capture] and [TokenRouter] Filter set up to route packets to Endpoints, to be the base
configuration for all the Quilkin proxies.
2. Appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions for the
2. Appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions for the
`quilkin manage agones` process to inspect Agones resources.
3. A matching [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) that runs the
3. A matching [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) that runs the
`quilkin manage process` xDS control plane and a
[Service](https://kubernetes.io/docs/concepts/services-networking/service/) that the Quilkin proxies can connect to,
to get their Filter and Endpoint configuration from.
Expand All @@ -44,7 +44,7 @@ quilkin-manage-agones-54b787654-9dbvp 1/1 Running 0 76s
```

We can now run `kubectl get service quilkin-manage-agones` and see the
service that is generated in front of the above Deployment for our Quilkin proxies to connect to and receive their
service that is generated in front of the above Deployment for our Quilkin proxies to connect to and receive their
configuration information from.

```shell
Expand All @@ -55,55 +55,55 @@ quilkin-manage-agones ClusterIP 10.104.2.72 <none> 80/TCP 1m23s

## 3. Install Quilkin Proxy Pool

To install the Quilkin Proxy pool which connects to the above xDS provider, we can create a Deployment of Quilkin
To install the Quilkin Proxy pool which connects to the above xDS provider, we can create a Deployment of Quilkin
proxy instances that point to the aforementioned Service, like so:

```shell
kubectl apply -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/proxy-pool.yaml
```

This will set up three instances of Quilkin running as `quilkin proxy --management-server http://quilkin-manage-agones:80`
This will set up three instances of Quilkin running as `quilkin proxy --management-server http://quilkin-manage-agones:80`
all connected to the `quilkin-manage-agones` service.

Now we can run `kubectl get pods` until we see that the Pods for the proxy Deployment is up and running.

```shell
$ kubectl get pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
quilkin-manage-agones-54b787654-9dbvp 1/1 Running 0 5m7s
quilkin-proxies-78965c446d-dqvjg 1/1 Running 0 6s
quilkin-proxies-78965c446d-fr6zs 1/1 Running 0 6s
quilkin-proxies-78965c446d-m4rr7 1/1 Running 0 6s
```

Let's take this one step further, and check the configuration of the proxies that should have come from the `quilkin
Let's take this one step further, and check the configuration of the proxies that should have come from the `quilkin
manage agones` instance.

In another terminal, run: `kubectl port-forward deployments/quilkin-proxies 8000`, to port forward the
[admin endpoint](../admin.md) locally, which we can then query.

Go back to your original terminal and run `curl -s http://localhost:8000/config`
Go back to your original terminal and run `curl -s http://localhost:8000/config`

> If you have [jq](https://stedolan.github.io/jq/) installed, run `curl -s http://localhost:8000/config | jq` for a
> nicely formatted JSON output.
```shell
$ curl -s http://localhost:8000/config
{"admin":{"address":"0.0.0.0:8000"},"clusters":{},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-dqvjg","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
{"admin":{"address":"0.0.0.0:8000"},"clusters":{},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-dqvjg","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
```

This shows us the current configuration of the proxies coming from the xDS server created via `quilkin manage
agones`. The most interesting part that we see here, is that we have a matching set of
[Filters](../../services/proxy/filters.md) that are found in the `ConfigMap` in the
This shows us the current configuration of the proxies coming from the xDS server created via `quilkin manage
agones`. The most interesting part that we see here, is that we have a matching set of
[Filters](../../services/proxy/filters.md) that are found in the `ConfigMap` in the
[xds-control-plane.yaml](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/xds-control-plane.yaml)
we installed earlier.

## 4. Create the Agones Fleet

Now we will create an [Agones Fleet](https://agones.dev/site/docs/reference/fleet/) to spin up all our Xonotic
Now we will create an [Agones Fleet](https://agones.dev/site/docs/reference/fleet/) to spin up all our Xonotic
game servers.

Thankfully, Agones Fleets require no specific configuration to work with Quilkin proxies, so this yaml is a
Thankfully, Agones Fleets require no specific configuration to work with Quilkin proxies, so this yaml is a
[standard Agones Fleet configuration](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/fleet.yaml)

```shell
Expand All @@ -126,18 +126,18 @@ To let the Quilkin xDS provider know what token will route to which `GameServer`
`quilkin.dev/tokens` annotation to an allocated `GameServer`, with the token content as its value.

> This token would normally get generated by some kind of player authentication service and passed to the client
> via the matchmaking service - but for demonstrative purposes, we've hardcoded it into the example
> via the matchmaking service - but for demonstrative purposes, we've hardcoded it into the example
> `GameServerAllocation`.
Since you can add annotations to `GameServers` at
[allocation time](https://agones.dev/site/docs/reference/gameserverallocation/), we can both allocate a `GameServer`
Since you can add annotations to `GameServers` at
[allocation time](https://agones.dev/site/docs/reference/gameserverallocation/), we can both allocate a `GameServer`
and apply the annotation at the same time!

```shell
kubectl create -f https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/gameserverallocation.yaml
```

If we check our `GameServers` now, we should see that one of them has moved to the `Allocated` state, marking it as
If we check our `GameServers` now, we should see that one of them has moved to the `Allocated` state, marking it as
having players playing on it, and therefore it is protected by Agones until the game session ends.

```shell
Expand All @@ -150,7 +150,7 @@ xonotic-d7rfx-sn5d6 Ready 34.168.170.51 7036 gke-agones-default-534a

> Don't do this more than once, as then multiple allocated `GameServers` will have the same routing token!
If we `kubectl describe gameserver <allocated-gameserver>` and have a look at the annotations section, we
If we `kubectl describe gameserver <allocated-gameserver>` and have a look at the annotations section, we
should see something similar to this:

```shell
Expand All @@ -168,7 +168,7 @@ Kind: GameServer
...
```

Where we can see that there is now an annotation of `quilkin.dev/tokens` with the base64 encoded version of `456` as
Where we can see that there is now an annotation of `quilkin.dev/tokens` with the base64 encoded version of `456` as
our authentication and routing token ("NDU2").

> You should use something more cryptographically random than `456` in your application.
Expand All @@ -177,22 +177,22 @@ Let's run `curl -s http://localhost:8000/config` again, so we can see what has c

```shell
$ curl -s http://localhost:8000/config
{"admin":{"address":"0.0.0.0:8000"},"clusters":{"default":{"localities":[{"locality":null,"endpoints":[{"address":"34.168.170.51:7226","metadata":{"quilkin.dev":{"tokens":["NDU2"]}}}]}]}},"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-tfgsj","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
{"admin":{"address":"0.0.0.0:8000"},"clusters": [{"locality":null,"endpoints":[{"address":"34.168.170.51:7226","metadata":{"quilkin.dev":{"tokens":["NDU2"]}}}]}],"filters":[{"name":"quilkin.filters.capture.v1alpha1.Capture","config":{"metadataKey":"quilkin.dev/capture","suffix":{"size":3,"remove":true}}},{"name":"quilkin.filters.token_router.v1alpha1.TokenRouter","config":null}],"id":"quilkin-proxies-78965c446d-tfgsj","management_servers":[{"address":"http://quilkin-manage-agones:80"}],"port":7000,"version":"v1alpha1","maxmind_db":null}%
```

Looking under `clusters` > `localities` > `endpoints` we can see an address and token that matches up with the
Looking under `clusters` > `endpoints` we can see an address and token that matches up with the
`GameServer` record we created above!

The xDS process saw that allocated `GameServer`, turned it into a Quilkin `Endpoint` and applied the set routing
token appropriately -- without you having to write a line of xDS compliant code!
The xDS process saw that allocated `GameServer`, turned it into a Quilkin `Endpoint` and applied the set routing
token appropriately -- without you having to write a line of xDS compliant code!

## Connecting Client Side

Instead of connecting to Xonotic or an Agones `GameServer` directly, we'll want to grab the IP and exposed port of
the `Service` that fronts all our Quilkin proxies and connect to that instead -- but we'll have to append our
the `Service` that fronts all our Quilkin proxies and connect to that instead -- but we'll have to append our
routing token `456` from before, to ensure our traffic gets routed to the correct Xonotic `GameServer` address.

Run `kubectl get service quilkin-proxies` to get the `EXTERNAL-IP` of the Service you created.
Run `kubectl get service quilkin-proxies` to get the `EXTERNAL-IP` of the Service you created.

```shell
$ kubectl get service quilkin-proxies
Expand All @@ -201,14 +201,14 @@ quilkin-proxies LoadBalancer 10.109.0.12 35.246.94.14 7000:30174/UDP
```

We have a [Quilkin config yaml](https://github.com/googleforgames/quilkin/blob/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml)
file all ready for you, that is configured to append the routing token `456` to each
packet that passes through it, via the power of a
file all ready for you, that is configured to append the routing token `456` to each
packet that passes through it, via the power of a
[ConcatenateBytes](../../services/proxy/filters/concatenate_bytes.md) Filter.

Download `client-token.yaml` locally, so you can edit it:

```shell
curl https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml --output client-token.yaml
curl https://raw.githubusercontent.com/googleforgames/quilkin/{{GITHUB_REF_NAME}}/examples/agones-xonotic-xds/client-token.yaml --output client-token.yaml
```

We then take the EXTERNAL-IP and port from the `quilkin-proxies` service, and replace the`${LOADBALANCER_IP}`
Expand Down
46 changes: 22 additions & 24 deletions docs/src/services/proxy.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"Proxy" is the primary Quilkin service, which acts as a non-transparent UDP
proxy.

To view all the options for the `proxy` subcommand, run:
To view all the options for the `proxy` subcommand, run:

```shell
$ quilkin proxy --help
Expand All @@ -17,51 +17,49 @@ $ quilkin proxy --help

## Endpoints

An Endpoint represents an address that Quilkin forwards packets to that it has received from the
An Endpoint represents an address that Quilkin forwards packets to that it has received from the
source port.

It is represented by an IP address and port. An Endpoint can optionally be associated with an arbitrary set of
It is represented by an IP address and port. An Endpoint can optionally be associated with an arbitrary set of
[metadata](#endpoint-metadata) as well.

## Proxy Filters

Filters are the way for a Quilkin proxy to intercept UDP packet traffic from the
source and [Endpoints][Endpoint] in either direction, and be able to inspect,
manipulate, and route the packets as desired.
manipulate, and route the packets as desired.

See [Filters] for a deeper dive into Filters, as well as the list of build in Filters that come with
See [Filters] for a deeper dive into Filters, as well as the list of build in Filters that come with
Quilkin.

## Endpoint Metadata

Endpoint metadata is an arbitrary set of key value pairs that are associated with an Endpoint.

These are visible to Filters when processing packets and can be used to provide more context about endpoints (e.g
These are visible to Filters when processing packets and can be used to provide more context about endpoints (e.g
whether or not to route a packet to an endpoint). Keys must be of type string otherwise the configuration is rejected.

Metadata associated with an endpoint contain arbitrary key value pairs which [Filters] can consult when processing packets (e.g they can contain information that determine whether or not to route a particular packet to an endpoint).

### Specialist Endpoint Metadata

Access tokens that can be associated with an endpoint are simply a special piece of metadata well known to Quilkin
Access tokens that can be associated with an endpoint are simply a special piece of metadata well known to Quilkin
and utilised by the built-in [TokenRouter] filter to route packets.

Such well known values are placed within an object in the endpoint metadata, under the special key `quilkin.dev`.
Such well known values are placed within an object in the endpoint metadata, under the special key `quilkin.dev`.
Currently, only the `tokens` key is in use.

As an example, the following shows the configuration for an endpoint with its metadata:
```yaml
clusters:
default:
localities:
- endpoints:
- address: 127.0.0.1:26000
metadata:
canary: false
quilkin.dev: # This object is extracted by Quilkin and is usually reserved for built-in features
tokens:
- MXg3aWp5Ng== # base64 for 1x7ijy6
- OGdqM3YyaQ== # base64 for 8gj3v2i
- endpoints:
- address: 127.0.0.1:26000
metadata:
canary: false
quilkin.dev: # This object is extracted by Quilkin and is usually reserved for built-in features
tokens:
- MXg3aWp5Ng== # base64 for 1x7ijy6
- OGdqM3YyaQ== # base64 for 8gj3v2i
```
An endpoint's metadata can be specified alongside the endpoint in [static configuration][file-configuration] or using the [xDS endpoint metadata][xds-endpoint-metadata] field when using [dynamic configuration][dynamic-configuration-doc] via xDS.
Expand All @@ -74,17 +72,17 @@ Quilkin uses the "Session" concept to track traffic flowing through the proxy be
Session serves the same purpose, and can be thought of as a lightweight version of a `TCP` session in that, while a
TCP session requires a protocol to establish and teardown:

- A Quilkin session is automatically created upon receiving the first packet from a client via the [Local Port] to be
- A Quilkin session is automatically created upon receiving the first packet from a client via the [Local Port] to be
sent to an upstream [Endpoint].
- The session is automatically deleted after a period of inactivity (where no packet was sent between either
- The session is automatically deleted after a period of inactivity (where no packet was sent between either
party) - currently 60 seconds.

A session is identified by the 4-tuple `(client IP, client Port, server IP, server Port)` where the client is the
downstream endpoint which initiated the communication with Quilkin and the server is one of the upstream Endpoints
A session is identified by the 4-tuple `(client IP, client Port, server IP, server Port)` where the client is the
downstream endpoint which initiated the communication with Quilkin and the server is one of the upstream Endpoints
that Quilkin proxies traffic to.

Sessions are established *after* the filter chain completes. The destination Endpoint of a packet is determined by
the [filter chain][Filters], so a Session can only be created after filter chain completion. For example, if the
Sessions are established *after* the filter chain completes. The destination Endpoint of a packet is determined by
the [filter chain][Filters], so a Session can only be created after filter chain completion. For example, if the
filter chain drops all packets, then no session will ever be created.

[Endpoint]: #endpoints
Expand Down
Loading

0 comments on commit fb10779

Please sign in to comment.