From 01a69dd8304cfdd1390b359d844e0680c6a1ed02 Mon Sep 17 00:00:00 2001 From: Josh Robson Chase Date: Thu, 12 Oct 2023 16:45:46 -0400 Subject: [PATCH] docs: document TLS and TCP edges --- docs/developer-guide/internal-crds.md | 38 ------ docs/user-guide/crds.md | 116 +++++++++++++--- docs/user-guide/tcp-edge.md | 6 - docs/user-guide/tcp-tls-edges.md | 190 ++++++++++++++++++++++++++ 4 files changed, 286 insertions(+), 64 deletions(-) delete mode 100644 docs/user-guide/tcp-edge.md create mode 100644 docs/user-guide/tcp-tls-edges.md diff --git a/docs/developer-guide/internal-crds.md b/docs/developer-guide/internal-crds.md index ff437b01..d3acaf8d 100644 --- a/docs/developer-guide/internal-crds.md +++ b/docs/developer-guide/internal-crds.md @@ -2,44 +2,6 @@ Kubernetes has the concept of [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs) which allow you to define your own custom resources. This document covers the CRDs created and managed by the controller internally to manage the state of the system across various controller components. It's generally unsafe to modify these directly and would likely result in strange effects as the controller fights you. They are useful however to inspect the state of the system and to debug issues. -## Domains - -Domains are automatically created by the controller based on the ingress objects host values. Standard ngrok subdomains will automatically be created and reserved for you. Custom domains will also be created and reserved, but will be up to you to configure the DNS records for them. See the [custom domain](./user-guide/custom-domain.md) guide for more details. - -If you delete all the ingress objects for a particular host, as a safety precaution, the ingress controller does *NOT* delete the domains and thus does not unregister them. This ensures you don't lose domains while modifying or recreating ingress objects. You can still manually delete a domain CRD via `kubectl delete domain ` if you want to unregister it. - -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| apiVersion | string | Yes | The API version for this custom resource. | -| kind | string | Yes | The kind of the custom resource. | -| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) | -| spec | [DomainSpec](#domainspec) | Yes | Specification of the domain. | -| status | [DomainStatus](#domainstatus) | No | Observed status of the domain. | - - -## Tunnels - -Tunnels are automatically created by the controller based on the ingress objects' rules' backends. A tunnel will be created for each backend service name and port combination. This results in tunnels being created with those labels which can be matched by various edge backends. Tunnels are useful to inspect but are fully managed by the controller and should not be edited directly. - -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| apiVersion | string | Yes | The API version for this custom resource. | -| kind | string | Yes | The kind of the custom resource. | -| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) | -| spec | [TunnelSpec](#tunnelspec) | Yes | Specification of the tunnel. | -| status | [TunnelStatus](#tunnelstatus) | No | Observed status of the tunnel. | - -### TunnelSpec -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| forwardsTo | string | Yes | The name and port of the service to forward traffic to. | -| labels | map[string]string | No | Key/value pairs that are attached to the tunnel. | - -### TunnelStatus -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| No fields defined. | | | | - ## HTTPS Edges HTTPS Edges are the primary representation of all the ingress objects and various configuration's states that will be reflected to the ngrok API. While you could create https edge CRDs directly, it's not recommended because: diff --git a/docs/user-guide/crds.md b/docs/user-guide/crds.md index a2993813..199b29b6 100644 --- a/docs/user-guide/crds.md +++ b/docs/user-guide/crds.md @@ -108,27 +108,9 @@ It's optional to create IP Policies this way vs using the ngrok dashboard or [te | CIDR | The CIDR block that the rule applies to | No | `string` | `"1.2.3.4/24"` | | Action | The action to take for the rule, either "allow" or "deny" | No | `string` | `"allow"` | - -### DomainSpec -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. | -| domain | string | Yes | The domain name to reserve. | -| region | string | Yes | The region in which to reserve the domain. | - -### DomainStatus -| Field | Type | Required | Description | -| --- | --- | --- | --- | -| id | string | No | The unique identifier of the domain. | -| domain | string | No | The domain that was reserved. | -| region | string | No | The region in which the domain was created. | -| uri | string | No | The URI of the reserved domain API resource. | -| cnameTarget | string | No | The CNAME target for the domain. | - - ## TCP Edges -The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kubernetes Ingress Controller supports TCP traffic via the [TCP Edge](https://ngrok.com/docs/api#tcp-edge) resource. This is a first class CRD that you can manage to control these edges in your account. This is in progress and not yet fully supported. Check back soon for updates to the [TCP Edge Guide](./tcp-edge.md). +The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kubernetes Ingress Controller supports TCP traffic via the [TCP Edge](https://ngrok.com/docs/api/resources/edges-tcp/) resource. This is a first class CRD that you can manage to control these edges in your account. See the [TCP and TLS Edges guide](./tcp-tls-edges.md) for more details. | Field | Type | Required | Description | | --- | --- | --- | --- | @@ -162,4 +144,98 @@ The Kubernetes ingress spec does not directly support TCP traffic. The ngrok Kub ### TunnelGroupBackendStatus | Field | Type | Required | Description | | --- | --- | --- | --- | -| id | string | No | The unique identifier for this backend. | \ No newline at end of file +| id | string | No | The unique identifier for this backend. | + +## TLS Edges + +ngrok's TLS Edges function similarly to TCP Edges in that they may contain arbitrary application data, not just HTTP. As such, the Kubernetes Ingress spec isn't a perfect fit for them either. The ngrok Kubernetes Ingress Controller supports arbitrary TLS endpoints via the [TLS Edge](https://ngrok.com/docs/api/resources/edges-tls/) resource. This is a first class CRD that you can manage to control these edges in your account. See the [TCP and TLS Edges guide](./tcp-tls-edges.md) for more details. + +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| apiVersion | string | Yes | The API version for this custom resource. | +| kind | string | Yes | The kind of the custom resource. | +| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) | +| spec | [TLSEdgeSpec](#tlsedgespec) | Yes | Specification of the TCP edge. | +| status | [TLSEdgeStatus](#tlsedgestatus) | No | Observed status of the TCP edge. | + +### TLSEdgeSpec +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. | +| backend | [TunnelGroupBackend](#tunnelgroupbackend) | Yes | The definition for the tunnel group backend that serves traffic for this edge. | +| hostports | []string | Yes | A list of hostports served by this edge. | +| ipRestriction | [EndpointIPPolicy](https://ngrok.com/docs/api#type-EndpointIPPolicy) | No | An IPRestriction to apply to this edge. | +| tlsTermination | [TLSTermination](https://ngrok.com/docs/api/resources/tls-edge-tls-termination-module/) | No | TLS Termination behaviour for this edge. | +| mutualTls | [MutualTLS](https://ngrok.com/docs/api/resources/tls-edge-mutual-tls-module/) | No | Mutual TLS validation for this edge. | + +### TLSEdgeStatus +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| id | string | No | The unique identifier for this edge. | +| uri | string | No | The URI of the edge. | +| hostports | []string | No | Hostports served by this edge. | +| backend | [TunnelGroupBackendStatus](#tunnelgroupbackendstatus) | No | Stores the status of the tunnel group backend, mainly the ID of the backend. | +## Domains + +Domains are automatically created by the controller based on the ingress objects host values. Standard ngrok subdomains will automatically be created and reserved for you. Custom domains will also be created and reserved, but will be up to you to configure the DNS records for them. See the [custom domain](./custom-domain.md) guide for more details. + +If you delete all the ingress objects for a particular host, as a safety precaution, the ingress controller does *NOT* delete the domains and thus does not unregister them. This ensures you don't lose domains while modifying or recreating ingress objects. You can still manually delete a domain CRD via `kubectl delete domain ` if you want to unregister it. + +If using a [TCP](#tcp-edges) or [TLS](#tls-edges) CRD directly, a Domain will not be created for you automatically, so you will need to create and manage it yourself. See the [TCP and TLS Edges](./tcp-tls-edges.md) guide for details. + +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| apiVersion | string | Yes | The API version for this custom resource. | +| kind | string | Yes | The kind of the custom resource. | +| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) | +| spec | [DomainSpec](#domainspec) | Yes | Specification of the domain. | +| status | [DomainStatus](#domainstatus) | No | Observed status of the domain. | + +### DomainSpec +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| ngrokAPICommon | [ngrokAPICommon](#ngrokapicommon) | No | Common fields shared by all ngrok resources. | +| domain | string | Yes | The domain name to reserve. | +| region | string | Yes | The region in which to reserve the domain. | + +### DomainStatus +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| id | string | No | The unique identifier of the domain. | +| domain | string | No | The domain that was reserved. | +| region | string | No | The region in which the domain was created. | +| uri | string | No | The URI of the reserved domain API resource. | +| cnameTarget | string | No | The CNAME target for the domain. | + +## Tunnels + +Tunnels are automatically created by the controller based on the ingress objects' rules' backends. A tunnel will be created for each backend service name and port combination. This results in tunnels being created with those labels which can be matched by various edge backends. Automatically-created are useful to inspect but are fully managed by the controller and should not be edited directly. + +If using a [TCP](#tcp-edges) or [TLS](#tls-edges) CRD, you may need to create and manage a Tunnel yourself. See the [TCP and TLS Edges](./tcp-tls-edges.md) guide for details. + +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| apiVersion | string | Yes | The API version for this custom resource. | +| kind | string | Yes | The kind of the custom resource. | +| metadata | [metav1.ObjectMeta](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta) | No | Standard object's metadata. More info: [https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata) | +| spec | [TunnelSpec](#tunnelspec) | Yes | Specification of the tunnel. | +| status | [TunnelStatus](#tunnelstatus) | No | Observed status of the tunnel. | + +### TunnelSpec +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| forwardsTo | string | Yes | The name and port of the service to forward traffic to. | +| backend | [TunnelBackend](#tunnelbackend) | Yes | The type of backend this tunnel forwards to. | +| labels | map[string]string | No | Key/value pairs that are attached to the tunnel. | + +### TunnelBackend + +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| protocol | string | Yes | The protocol understood by this backend. Either TCP or TLS. + +### TunnelStatus +| Field | Type | Required | Description | +| --- | --- | --- | --- | +| No fields defined. | | | | + diff --git a/docs/user-guide/tcp-edge.md b/docs/user-guide/tcp-edge.md deleted file mode 100644 index 96dd1d98..00000000 --- a/docs/user-guide/tcp-edge.md +++ /dev/null @@ -1,6 +0,0 @@ -# TCP Edges - -ngrok offers [TCP Edges](https://ngrok.com/docs/cloud-edge/edges/tcp/) which can be used to provide ingress to tcp based services. -This will be available via a CRD soon! - -Under Construction \ No newline at end of file diff --git a/docs/user-guide/tcp-tls-edges.md b/docs/user-guide/tcp-tls-edges.md new file mode 100644 index 00000000..2ffcb8de --- /dev/null +++ b/docs/user-guide/tcp-tls-edges.md @@ -0,0 +1,190 @@ +# TCP and TLS Edges + +ngrok offers [TCP](https://ngrok.com/docs/cloud-edge/edges/tcp/) and +[TLS](https://ngrok.com/docs/cloud-edge/edges/tcp/) Edges which can be used to +provide ingress to TCP or TLS based services. Both are implemented as CRDs and +function similarly in broad strokes, albeit with slightly different +configuration options offered. [Their CRD reference](./crds.md#tcp-edges) is a +useful companion to this guide. + +## (TLS Only) Get a Domain + +At least one `hostports` must be specified when creating a TLSEdge resource, +which takes the form `:443`. The fully qualified domain name must first be +reserved either via the ngrok dashboard or the [Domain](./crds.md#domains) CRD. + +Example: + +```yaml +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: Domain +metadata: + name: tlsedgetest-ngrok-app +spec: + domain: tlsedgetest.ngrok.app +``` + +## Create the Edge + +Create the edge CRD. These resources are fairly similar, and both require you to +specify a [TunnelGroupBackend](./crds.md#tunnelgroupbackend). This consists of a +list of labels that determine which specific [Tunnel](./crds.md#tunnels) should +receive traffic from the edge. Both may also specify [IP +Policies](https://ngrok.com/docs/api/resources/ip-policies/) for limiting access +to the edge. At the time of writing, these policies must be provided as a +reference in the form `ipp_`. + +On top of the options available to TCP Edges, TLS Edges support (and require) a few other options: + +- (required) `hostports`: A list of `":443"` strings declaring the list of + reserved domains for the edge to listen on. +- [`tlsTermination`](https://ngrok.com/docs/api/resources/tls-edge-tls-termination-module/): Configure the TLS Termination behavior. The `terminateAt` field may be set to `upstream` to pass the encrypted stream to the Tunnel backend, or `edge` to terminate the TLS stream at the ngrok edge, and pass plaintext bytes to the Tunnel. +- [`mutualTls`](https://ngrok.com/docs/api/resources/tls-edge-mutual-tls-module/): Configure client certificate validation at the edge. Requires a reference to a [Certificate Authority](https://ngrok.com/docs/api/resources/certificate-authorities/). + +TCP Example: + +```yaml +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: TCPEdge +metadata: + name: test-edge +spec: + backend: + labels: + app: tcptestedge +``` + +Because TCP Edges don't currently support providing a reserved TCP address. On edge creation, one will be allocated for them, and will be visible by checking the status of the resource: + +```bash +$ kubectl get tcpedges test-edge +NAME ID HOSTPORTS BACKEND ID AGE +test-edge edgtcp_2Wg5AzVE878vQoNMP3Z8wONIr76 ["7.tcp.ngrok.io:27866"] bkdtg_2Wg5Amjb4GiQoV7SAnpEdM0Dg3n 2m35s +``` + +TLS Example: + +```yaml +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: TLSEdge +metadata: + name: test-edge +spec: + hostports: + - tlstestedge.ngrok.app:443 + backend: + labels: + app: tlstestedge + tlsTermination: + terminateAt: upstream +``` + +## Start the Tunnel + +Finally, create a Tunnel to receive and forward traffic for your edge. + +Important fields: + +- `forwardsTo`: The `:` to forward traffic to. This can be any + hostname resolvable and accessible from the ingress controller pod. +- `labels`: a map of labels corresponding to the edge to receive traffic for. + These must match the labels specified when creating your edge. +- `backend.protocol`: The protocol understood by the backend service. `TCP` will + forward connections to the backend as-is, while `TLS` will create a TLS + connection to the backend _first_, and then forward the connection stream over + that. + +Example: + +```yaml +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: Tunnel +metadata: + name: test-tunnel +spec: + backend: + protocol: TCP + forwardsTo: kubernetes.default.svc:443 + labels: + app: tlsedgetest +``` + +# Full Example + +This is an example of using a TLS Edge to expose the kubernetes control plane via ngrok. + +```yaml +--- +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: Domain +metadata: + name: tlsedgetest-ngrok-app +spec: + # Reserve the tlsedgetest.ngrok.app domain. + domain: tlsedgetest.ngrok.app +--- +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: TLSEdge +metadata: + name: test-edge +spec: + hostports: + # Listen for connections on the domain we reserved + - tlsedgetest.ngrok.app:443 + backend: + labels: + app: tlsedgetest + # Pass the TLS stream on to the backend - let the application do its own TLS + # handshake. + tlsTermination: + terminateAt: upstream +--- +apiVersion: ingress.k8s.ngrok.com/v1alpha1 +kind: Tunnel +metadata: + name: test-tunnel +spec: + # Forward the raw TCP stream to our backend. + # It will technically contain TLS, and the backend speaks TLS, but we don't + # want the Tunnel to terminate TLS before forwarding incoming connections. + # We don't want a TLS turducken. + backend: + protocol: TCP + # Forward to the kubernetes control plane. + forwardsTo: kubernetes.default.svc:443 + # Listen for connections using the labels from our edge. + labels: + app: tlsedgetest +``` + +Check the status of your resources: + +``` +$ kubectl get domain +NAME ID REGION DOMAIN CNAME TARGET AGE +tlsedgetest-ngrok-app rd_2Wg986lvMqsiB1J5WV5lOcmT21a tlsedgetest.ngrok.app 4s +$ kubectl get tlsedge +NAME ID HOSTPORTS BACKEND ID AGE +test-edge edgtls_2Wg989BMmZLWXixStL8BjAxMcxW ["tlsedgetest.ngrok.app:443"] bkdtg_2Wg981gcSnxaX5cTL28LWwVg4xD 12s +$ kubectl get tunnel +NAME FORWARDSTO AGE +test-tunnel kubernetes.default.svc:443 52m +``` + +Our domain and edge both have IDs allocated, so we know they've been created successfully! + +Edit your kubeconfig and replace the `server` with +`https://tlsedgetest.ngrok.app`, comment out `certificate-authority-data` and +add `insecure-skip-tls-verify: true` to your `cluster` config. This is needed +because kubernetes is completing the TLS handshake with its own certificate, +which won't be valid for your ngrok domain. + +Use `kubectl cluster-info` to verify that everything is still working: + +``` +$ kubectl cluster-info +Kubernetes control plane is running at https://tlsedgetest.ngrok.app +CoreDNS is running at https://tlsedgetest.ngrok.app/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + +To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. +```