Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: document TLS and TCP edges #313

Merged
merged 1 commit into from
Oct 16, 2023
Merged

docs: document TLS and TCP edges #313

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 0 additions & 38 deletions docs/developer-guide/internal-crds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,44 +2,6 @@

Kubernetes has the concept of [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs) which allow you to define your own custom resources. This document covers the CRDs created and managed by the controller internally to manage the state of the system across various controller components. It's generally unsafe to modify these directly and would likely result in strange effects as the controller fights you. They are useful however to inspect the state of the system and to debug issues.

## Domains

Domains are automatically created by the controller based on the ingress objects host values. Standard ngrok subdomains will automatically be created and reserved for you. Custom domains will also be created and reserved, but will be up to you to configure the DNS records for them. See the [custom domain](./user-guide/custom-domain.md) guide for more details.

If you delete all the ingress objects for a particular host, as a safety precaution, the ingress controller does *NOT* delete the domains and thus does not unregister them. This ensures you don't lose domains while modifying or recreating ingress objects. You can still manually delete a domain CRD via `kubectl delete domain <name>` if you want to unregister it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| apiVersion | string | Yes | The API version for this custom resource. |
| kind | string | Yes | The kind of the custom resource. |
| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) |
| spec | [DomainSpec](#domainspec) | Yes | Specification of the domain. |
| status | [DomainStatus](#domainstatus) | No | Observed status of the domain. |


## Tunnels

Tunnels are automatically created by the controller based on the ingress objects' rules' backends. A tunnel will be created for each backend service name and port combination. This results in tunnels being created with those labels which can be matched by various edge backends. Tunnels are useful to inspect but are fully managed by the controller and should not be edited directly.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| apiVersion | string | Yes | The API version for this custom resource. |
| kind | string | Yes | The kind of the custom resource. |
| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) |
| spec | [TunnelSpec](#tunnelspec) | Yes | Specification of the tunnel. |
| status | [TunnelStatus](#tunnelstatus) | No | Observed status of the tunnel. |

### TunnelSpec
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| forwardsTo | string | Yes | The name and port of the service to forward traffic to. |
| labels | map[string]string | No | Key/value pairs that are attached to the tunnel. |

### TunnelStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| No fields defined. | | | |

## HTTPS Edges

HTTPS Edges are the primary representation of all the ingress objects and various configuration's states that will be reflected to the ngrok API. While you could create https edge CRDs directly, it's not recommended because:
Expand Down
116 changes: 96 additions & 20 deletions docs/user-guide/crds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,27 +108,9 @@ It's optional to create IP Policies this way vs using the ngrok dashboard or [te
| CIDR | The CIDR block that the rule applies to | No | `string` | `"1.2.3.4/24"` |
| Action | The action to take for the rule, either "allow" or "deny" | No | `string` | `"allow"` |


### DomainSpec
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. |
| domain | string | Yes | The domain name to reserve. |
| region | string | Yes | The region in which to reserve the domain. |

### DomainStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| id | string | No | The unique identifier of the domain. |
| domain | string | No | The domain that was reserved. |
| region | string | No | The region in which the domain was created. |
| uri | string | No | The URI of the reserved domain API resource. |
| cnameTarget | string | No | The CNAME target for the domain. |


## TCP Edges

The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kubernetes Ingress Controller supports TCP traffic via the [TCP Edge](https://ngrok.com/docs/api#tcp-edge) resource. This is a first class CRD that you can manage to control these edges in your account. This is in progress and not yet fully supported. Check back soon for updates to the [TCP Edge Guide](./tcp-edge.md).
The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kubernetes Ingress Controller supports TCP traffic via the [TCP Edge](https://ngrok.com/docs/api/resources/edges-tcp/) resource. This is a first class CRD that you can manage to control these edges in your account. See the [TCP and TLS Edges guide](./tcp-tls-edges.md) for more details.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
Expand Down Expand Up @@ -162,4 +144,98 @@ The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kub
### TunnelGroupBackendStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| id | string | No | The unique identifier for this backend. |
| id | string | No | The unique identifier for this backend. |

## TLS Edges

ngrok's TLS Edges function similarly to TCP Edges in that they may contain arbitrary application data, not just HTTP. As such, the Kubernetes Ingress spec isn't a perfect fit for them either. The ngrok Kubernetes Ingress Controller supports arbitrary TLS endpoints via the [TLS Edge](https://ngrok.com/docs/api/resources/edges-tls/) resource. This is a first class CRD that you can manage to control these edges in your account. See the [TCP and TLS Edges guide](./tcp-tls-edges.md) for more details.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| apiVersion | string | Yes | The API version for this custom resource. |
| kind | string | Yes | The kind of the custom resource. |
| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) |
| spec | [TLSEdgeSpec](#tlsedgespec) | Yes | Specification of the TCP edge. |
| status | [TLSEdgeStatus](#tlsedgestatus) | No | Observed status of the TCP edge. |

### TLSEdgeSpec
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. |
| backend | [TunnelGroupBackend](#tunnelgroupbackend) | Yes | The definition for the tunnel group backend that serves traffic for this edge. |
| hostports | []string | Yes | A list of hostports served by this edge. |
| ipRestriction | [EndpointIPPolicy](https://ngrok.com/docs/api#type-EndpointIPPolicy) | No | An IPRestriction to apply to this edge. |
| tlsTermination | [TLSTermination](https://ngrok.com/docs/api/resources/tls-edge-tls-termination-module/) | No | TLS Termination behaviour for this edge. |
| mutualTls | [MutualTLS](https://ngrok.com/docs/api/resources/tls-edge-mutual-tls-module/) | No | Mutual TLS validation for this edge. |

### TLSEdgeStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| id | string | No | The unique identifier for this edge. |
| uri | string | No | The URI of the edge. |
| hostports | []string | No | Hostports served by this edge. |
| backend | [TunnelGroupBackendStatus](#tunnelgroupbackendstatus) | No | Stores the status of the tunnel group backend, mainly the ID of the backend. |
## Domains

Domains are automatically created by the controller based on the ingress objects host values. Standard ngrok subdomains will automatically be created and reserved for you. Custom domains will also be created and reserved, but will be up to you to configure the DNS records for them. See the [custom domain](./custom-domain.md) guide for more details.

If you delete all the ingress objects for a particular host, as a safety precaution, the ingress controller does *NOT* delete the domains and thus does not unregister them. This ensures you don't lose domains while modifying or recreating ingress objects. You can still manually delete a domain CRD via `kubectl delete domain <name>` if you want to unregister it.

If using a [TCP](#tcp-edges) or [TLS](#tls-edges) CRD directly, a Domain will not be created for you automatically, so you will need to create and manage it yourself. See the [TCP and TLS Edges](./tcp-tls-edges.md) guide for details.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| apiVersion | string | Yes | The API version for this custom resource. |
| kind | string | Yes | The kind of the custom resource. |
| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) |
| spec | [DomainSpec](#domainspec) | Yes | Specification of the domain. |
| status | [DomainStatus](#domainstatus) | No | Observed status of the domain. |

### DomainSpec
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. |
| domain | string | Yes | The domain name to reserve. |
| region | string | Yes | The region in which to reserve the domain. |

### DomainStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| id | string | No | The unique identifier of the domain. |
| domain | string | No | The domain that was reserved. |
| region | string | No | The region in which the domain was created. |
| uri | string | No | The URI of the reserved domain API resource. |
| cnameTarget | string | No | The CNAME target for the domain. |

## Tunnels

Tunnels are automatically created by the controller based on the ingress objects' rules' backends. A tunnel will be created for each backend service name and port combination. This results in tunnels being created with those labels which can be matched by various edge backends. Automatically-created are useful to inspect but are fully managed by the controller and should not be edited directly.

If using a [TCP](#tcp-edges) or [TLS](#tls-edges) CRD, you may need to create and manage a Tunnel yourself. See the [TCP and TLS Edges](./tcp-tls-edges.md) guide for details.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| apiVersion | string | Yes | The API version for this custom resource. |
| kind | string | Yes | The kind of the custom resource. |
| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) |
| spec | [TunnelSpec](#tunnelspec) | Yes | Specification of the tunnel. |
| status | [TunnelStatus](#tunnelstatus) | No | Observed status of the tunnel. |

### TunnelSpec
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| forwardsTo | string | Yes | The name and port of the service to forward traffic to. |
| backend | [TunnelBackend](#tunnelbackend) | Yes | The type of backend this tunnel forwards to. |
| labels | map[string]string | No | Key/value pairs that are attached to the tunnel. |

### TunnelBackend

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| protocol | string | Yes | The protocol understood by this backend. Either TCP or TLS.

### TunnelStatus
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| No fields defined. | | | |

6 changes: 0 additions & 6 deletions docs/user-guide/tcp-edge.md
View file
Open in desktop

This file was deleted.

190 changes: 190 additions & 0 deletions docs/user-guide/tcp-tls-edges.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
# TCP and TLS Edges

ngrok offers [TCP](https://ngrok.com/docs/cloud-edge/edges/tcp/) and
[TLS](https://ngrok.com/docs/cloud-edge/edges/tcp/) Edges which can be used to
provide ingress to TCP or TLS based services. Both are implemented as CRDs and
function similarly in broad strokes, albeit with slightly different
configuration options offered. [Their CRD reference](./crds.md#tcp-edges) is a
useful companion to this guide.

## (TLS Only) Get a Domain

At least one `hostports` must be specified when creating a TLSEdge resource,
which takes the form `<fqdn>:443`. The fully qualified domain name must first be
reserved either via the ngrok dashboard or the [Domain](./crds.md#domains) CRD.

Example:

```yaml
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: Domain
metadata:
name: tlsedgetest-ngrok-app
spec:
domain: tlsedgetest.ngrok.app
```

## Create the Edge

Create the edge CRD. These resources are fairly similar, and both require you to
specify a [TunnelGroupBackend](./crds.md#tunnelgroupbackend). This consists of a
list of labels that determine which specific [Tunnel](./crds.md#tunnels) should
receive traffic from the edge. Both may also specify [IP
Policies](https://ngrok.com/docs/api/resources/ip-policies/) for limiting access
to the edge. At the time of writing, these policies must be provided as a
reference in the form `ipp_<id>`.

On top of the options available to TCP Edges, TLS Edges support (and require) a few other options:

- (required) `hostports`: A list of `"<fqdn>:443"` strings declaring the list of
reserved domains for the edge to listen on.
- [`tlsTermination`](https://ngrok.com/docs/api/resources/tls-edge-tls-termination-module/): Configure the TLS Termination behavior. The `terminateAt` field may be set to `upstream` to pass the encrypted stream to the Tunnel backend, or `edge` to terminate the TLS stream at the ngrok edge, and pass plaintext bytes to the Tunnel.
- [`mutualTls`](https://ngrok.com/docs/api/resources/tls-edge-mutual-tls-module/): Configure client certificate validation at the edge. Requires a reference to a [Certificate Authority](https://ngrok.com/docs/api/resources/certificate-authorities/).

TCP Example:

```yaml
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: TCPEdge
metadata:
name: test-edge
spec:
backend:
labels:
app: tcptestedge
```

Because TCP Edges don't currently support providing a reserved TCP address. On edge creation, one will be allocated for them, and will be visible by checking the status of the resource:

```bash
$ kubectl get tcpedges test-edge
NAME ID HOSTPORTS BACKEND ID AGE
test-edge edgtcp_2Wg5AzVE878vQoNMP3Z8wONIr76 ["7.tcp.ngrok.io:27866"] bkdtg_2Wg5Amjb4GiQoV7SAnpEdM0Dg3n 2m35s
```

TLS Example:

```yaml
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: TLSEdge
metadata:
name: test-edge
spec:
hostports:
- tlstestedge.ngrok.app:443
backend:
labels:
app: tlstestedge
tlsTermination:
terminateAt: upstream
```

## Start the Tunnel

Finally, create a Tunnel to receive and forward traffic for your edge.

Important fields:

- `forwardsTo`: The `<hostname>:<port>` to forward traffic to. This can be any
hostname resolvable and accessible from the ingress controller pod.
- `labels`: a map of labels corresponding to the edge to receive traffic for.
These must match the labels specified when creating your edge.
- `backend.protocol`: The protocol understood by the backend service. `TCP` will
forward connections to the backend as-is, while `TLS` will create a TLS
connection to the backend _first_, and then forward the connection stream over
that.

Example:

```yaml
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: Tunnel
metadata:
name: test-tunnel
spec:
backend:
protocol: TCP
forwardsTo: kubernetes.default.svc:443
labels:
app: tlsedgetest
```

# Full Example

This is an example of using a TLS Edge to expose the kubernetes control plane via ngrok.

```yaml
---
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: Domain
metadata:
name: tlsedgetest-ngrok-app
spec:
# Reserve the tlsedgetest.ngrok.app domain.
domain: tlsedgetest.ngrok.app
---
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: TLSEdge
metadata:
name: test-edge
spec:
hostports:
# Listen for connections on the domain we reserved
- tlsedgetest.ngrok.app:443
backend:
labels:
app: tlsedgetest
# Pass the TLS stream on to the backend - let the application do its own TLS
# handshake.
tlsTermination:
terminateAt: upstream
---
apiVersion: ingress.k8s.ngrok.com/v1alpha1
kind: Tunnel
metadata:
name: test-tunnel
spec:
# Forward the raw TCP stream to our backend.
# It will technically contain TLS, and the backend speaks TLS, but we don't
# want the Tunnel to terminate TLS before forwarding incoming connections.
# We don't want a TLS turducken.
backend:
protocol: TCP
# Forward to the kubernetes control plane.
forwardsTo: kubernetes.default.svc:443
# Listen for connections using the labels from our edge.
labels:
app: tlsedgetest
```

Check the status of your resources:

```
$ kubectl get domain
NAME ID REGION DOMAIN CNAME TARGET AGE
tlsedgetest-ngrok-app rd_2Wg986lvMqsiB1J5WV5lOcmT21a tlsedgetest.ngrok.app 4s
$ kubectl get tlsedge
NAME ID HOSTPORTS BACKEND ID AGE
test-edge edgtls_2Wg989BMmZLWXixStL8BjAxMcxW ["tlsedgetest.ngrok.app:443"] bkdtg_2Wg981gcSnxaX5cTL28LWwVg4xD 12s
$ kubectl get tunnel
NAME FORWARDSTO AGE
test-tunnel kubernetes.default.svc:443 52m
```

Our domain and edge both have IDs allocated, so we know they've been created successfully!

Edit your kubeconfig and replace the `server` with
`https://tlsedgetest.ngrok.app`, comment out `certificate-authority-data` and
add `insecure-skip-tls-verify: true` to your `cluster` config. This is needed
because kubernetes is completing the TLS handshake with its own certificate,
which won't be valid for your ngrok domain.

Use `kubectl cluster-info` to verify that everything is still working:

```
$ kubectl cluster-info
Kubernetes control plane is running at https://tlsedgetest.ngrok.app
CoreDNS is running at https://tlsedgetest.ngrok.app/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```