Skip to content

Commit

Permalink
v0.2.0 Pre-release (#27)
Browse files Browse the repository at this point in the history 
* Sidecar/init containers, 23.1, core example update

* Clean up unused env vars

* Distinct overrides/ dir for ro/configmap mounts

* 2regions-hrr ds/lothlorien-a for sidecar/init

* 2regions-hrr ds/lothlorien-b for sidecar/init

* Updated image paths

* Don't restart meshrr sidecars pending crpd

* Mirkwood on LBs and multiple containers

* Image pull cleanup

* Restructuring

* Initial YAML-driven config; 2region-hrr core only

* bgpgroups to j2 as dict

* Mirkwood example and configmap compatibility

* pylint

* Lothlorien example w/ YAML

* Documentation updates

* Minor doc update

* 2regions-hrr docs update

* Timestamps

* GitHub Action to push container image

* Docs update

* Minor docs update

* Docs updates

* Container image tag normalization

* Add cRPD image import instructions

* Fix Junos config syntax error for evpnrs

* Update A side evpnrs use case

* b side evpnrs

* Remove regions from routeserver example

* 3 client groups (#24)

* Base 3clientgroups example

* Junos config cleanup

* Routeservers readme update

* Removed references to regions from RS example

* Update manifests to v0.2 image tag

---------

Co-authored-by: Jason R. Rokeach <[email protected]>
  • Loading branch information
@jrokeach
jrokeach and jrokeach authored Jan 5, 2024
1 parent d3afae6 commit 42ec518
Show file tree
Hide file tree
Showing 35 changed files with 2,417 additions and 1,447 deletions.
62 changes: 62 additions & 0 deletions .github/workflows/publish-image.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#
name: Publish container image

on:
push:
branches:
- 'main'
- 'next'
tags:
- 'v*'
pull_request:
branches:
- 'main'
- 'next'

# Defines two custom environment variables for the workflow. These are used for the Container registry domain, and a name for the Docker image that this workflow builds.
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

# There is a single job in this workflow. It's configured to run on the latest available version of Ubuntu.
jobs:
build-and-push-image:
runs-on: ubuntu-latest
# Sets the permissions granted to the `GITHUB_TOKEN` for the actions in this job.
permissions:
contents: read
packages: write
#
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Uses the `docker/login-action` action to log in to the Container registry registry using the account and password that will publish the packages. Once published, the packages are scoped to the account defined here.
- name: Log in to the Container registry
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# This step uses [docker/metadata-action](https://github.com/docker/metadata-action#about) to extract tags and labels that will be applied to the specified image. The `id` "meta" allows the output of this step to be referenced in a subsequent step. The `images` value provides the base name for the tags and labels.
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
# set latest tag for default branch
type=raw,value=latest,enable={{is_default_branch}}
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
# This step uses the `docker/build-push-action` action to build the image, based on your repository's `Dockerfile`. If the build succeeds, it pushes the image to GitHub Packages.
# It uses the `context` parameter to define the build's context as the set of files located in the specified path. For more information, see "[Usage](https://github.com/docker/build-push-action#usage)" in the README of the `docker/build-push-action` repository.
# It uses the `tags` and `labels` parameters to tag and label the image with the output from the "meta" step.
- name: Build and push Docker image
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with:
context: "{{defaultContext}}:meshrr"
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
2 changes: 1 addition & 1 deletion LICENSE
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) Juniper Networks, Inc. 2020
Copyright (c) Juniper Networks, Inc. 2023

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
137 changes: 67 additions & 70 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,106 +9,103 @@ At this time and in the project's raw form, *meshrr* should not be considered fo
- [Introduction](#introduction)
- [Instructions](#instructions)
- [Prerequisites](#prerequisites)
- [Quickstart](#quickstart)
- [Usage](#usage)
- [Environment Variables](#environment-variables)
- [Methodology](#methodology)
- [Containers](#containers)
- [BGP Group Types](#bgp-group-types)
- [Examples](#examples)
- [Example Commands](#example-commands)

## Instructions

### Prerequisites
1. An operational Kubernetes cluster with sufficient resources for the topology you wish to build.
2. A *private* container registry accessible to your Kubernetes cluster.
- You'll need to be logged in to your registry using `docker login` to push the image you'll build.
- You'll need to store your registry credentials in a secret in your cluster to pull from this registry. In this project, all examples use a secret named `regcred`. There are a few ways you can do this.
1. If you already have a simple means of generating the secret manifest (e.g. using `doctl`), you can do this in one line:
```
doctl registry kubernetes-manifest --name regcred | kubectl apply -f -
```
2. You can generate the secret manually with all the parameters:
```
kubectl create secret docker-registry regcred \
--docker-server=<server> --docker-username=<username> \
--docker-password=<password> --docker-email=<email>
```
3. Any number of [other reasonable approaches](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).

1. An operational Kubernetes cluster with sufficient resources for the topology you wish to build.
2. The cRPD software. The current tested version is **23.2R1.13**. The software must be available via a private repository or preloaded onto all nodes it may be run on. If using k3s, this can be accomplished with `k3s ctr images import junos-routing-crpd-docker-amd64-23.2R1.13.tar`.
- If the import fails, you may need to convert the tarfile into a format that can be imported. You can do so with
```sh
docker load -i junos-routing-crpd-docker-amd64-23.2R1.13.tgz \
&& docker image tag crpd:23.2R1.13 localhost/juniper/crpd:23.2R1.13 \
&& docker image save localhost/juniper/crpd:23.2R1.13 --output=junos-routing-crpd-docker-amd64-23.2R1.13.tar
```
3. A cRPD license for the number of nodes you wish to deploy. At the time of writing, Juniper offers [free trial licenses](https://www.juniper.net/us/en/dm/crpd-trial/). Standard licenses are limited to 16 BGP peers and 4M RIB entries.

### Quickstart
1. (If required) modify [`juniper.conf.j2`](meshrr/juniper.conf.j2)
2. Build and push your image. **Do not push to a public registry.**
### Usage

```bash
docker build -t <tag> meshrr
docker push <tag>
```
1. (If required) copy a configuration file template from [`the default templates`](meshrr/defaults/) and edit it to your liking.

e.g.
```bash
docker build meshrr -t registry.example.com/meshrr/meshrr:latest
docker push registry.example.com/meshrr/meshrr:latest
```
3. Either:
2. Either:
1. Pick an example topology from [`examples`](examples/) and modify the YAML files as required for your topology. Details for how to use examples and reasonable modifications are below in the [Examples](#Examples) section.
2. Create your own YAML files if you need a completely custom topology.
4. Populate the YAML files with the required information. You will need to, at a minimum, replace the following:
1. Names
1. Service
2. Labels (if following the 2regions-hrr example, your regions probably are not in Middle Earth)

3. Populate the Kubernetes manifest YAML files with the required information. You will need to, at a minimum, replace the following:
1. Names of elements
2. [Environment Variables](#Environment-Variables)
3. Licensing mechanism. Examples here currently use a secret mounted as a volume mapped to `/config/license/safenet/junos_sfnt.lic`. This may be appropriate for bundle licenses where it is appropriate to use the same license file for many similar devices in a deployment or daemonset. You can create this using:
3. Licensing mechanism. Examples here currently use a secret exposed as an environment variable in the meshrr-init container which will populate the license into the config. This may be appropriate for bundle licenses where it is appropriate to use the same license file for many similar devices in a deployment or daemonset. You can create this using:
```
kubectl create secret generic crpd-license --from-file=junos_sfnt.lic=<filepath>
kubectl create secret generic crpd-license --from-file=crpd-license=<filepath>
```
4. Custom configuration Jinja2 templates loaded into ConfigMaps and mapped as volumes. See [Examples](#Examples).
5. Port mapping IP addresses (`hostIP`). No `hostIP` must be specified for instances only accessible within the cluster. Detailed strategy information to be defined in [Examples](#Examples).
5. Apply appropriate labels to the nodes:
Note that `<filepath>` must point to a file that contains the singular license line and not an entire license file.
4. (If required) Custom configuration Jinja2 templates loaded into ConfigMaps and mapped as volumes. See [Examples](#Examples).
4. (If required - e.g., for [2regions-hrr](examples/2regions-hrr/) where only certain nodes should host certain clusters of RRs) Apply appropriate labels to the nodes:
```bash
kubectl label nodes <node> <label1>=<value> <label2>=<value>
```
6. Apply your configuration:
5. Apply your configuration:
```bash
kubectl [-n namespace] apply -f <file1>
kubectl [-n namespace] apply -f <file2>
```

### Environment Variables
| Variable | Required? | Description |
| --------------------- | --------- | ------------------------------------------------------------ |
| POD_IP | Yes | The pod's IP address. Should be set by Kubernetes (`valueFrom: fieldRef: fieldPath: status.podIP`) |
| MESH_SERVICE_NAME | Yes* | The name of the mesh service. Set to the name of the headless Kubernetes service used for mesh BGP neighbor discovery. *Usually, a `MESH_SERVICE_NAME` is desirable. However, it may be skipped if there is an `UPSTREAM_SERVICE_NAME` in cases such as unmeshed regions learning routes from upstream HRRs. |
| UPSTREAM_SERVICE_NAME | No | The name of the upstream service. Set to the name of the headless Kubernetes service used for upstream BGP neighbor discovery. Defaults to `None`. |
| KUBE_NAMESPACE | No | Optional name of the Kubernetes namespace. Defaults to `default`. |
| ENCRYPTED_ROOT_PW | Yes | Encrypted ($6) root password for cRPD |
| AUTONOMOUS_SYSTEM | Yes | ASN for the router. |
| MESHRR_CLIENTRANGE | Yes | Range to allow. Currently, this accepts only one CIDR block. Format: `network/mask-length` |
| MESHRR_MODE | No | `routereflector` or `routeserver`. Defaults to `routereflector`. |
| MESHRR_ASRANGE | No | Range of ASNs to allow for `routeserver` mode. Defaults to `65001-65500` |
| MESHRR_FAMILY_INET | No | `true` or `false`. Defaults to `true` |
| MESHRR_FAMILY_EVPN | No | `true` or `false`. Defaults to `false` |
| SERVICE_ROOT_DOMAIN | No | Defaults to `svc.cluster.local`. You probably don't need to change this. |

## Methodology
- Build container image based on crpd.
- Requires additional packages installed via `apt-get`:
- cron
- python3
- Builds crontab
- Sets up `runit-init.sh`
- `runit-init.sh` initializes the environment:
- Saves environment variables, including the necessary pod's IP address, to `/etc/envvars`
- Sets up the cRPD configuration based on the template `juniper.conf.template`
- Calls `render_config.py` to create configuration file from Jinja2 template.
- `update_peers.py` called every minute via cron.
- Uses a Kubernetes headless service DNS A records to detect peers.

| Variable | Required for | Optional for | Description |
| -------------- | ------------------- | ------------ | ------------------------------------------------------------ |
| LICENSE_KEY | meshrr-init | | License key to be used for the cRPD container; expected to be a single line. |
| POD_IP | meshrr-init, meshrr | | The pod's IP address. Must be set by Kubernetes manifest in all pod templates for all meshrr containers. This does not need to be set for the cRPD containers. (`valueFrom: fieldRef: fieldPath: status.podIP`) |
| UPDATE_SECONDS | | meshrr | Frequency in seconds that `meshrr` container will attempt to update `crpd` container with changes to peers. (Default: 30) |
## Containers
- Init Container - `meshrr-init`:
- `run.sh` with arg `init`
- Creates configuration from default template or mounted template or derives from existing `/config/juniper.conf` if pod uses persistent storage.
- Container - `crpd`:
- Unmodified cRPD image running Juniper cRPD.
- Container - `meshrr`:
- Conducts periodic BGP peer configuration changes on `crpd` container via Netconf.
## BGP Group Types
- `mesh`
- Discovers peers and connects to all or a limited number of them.
- Currently, the only BGP peer discovery mode is `dns`, which uses a Kubernetes headless service DNS A records to detect peers.
New peers are added to config, removed peers are removed from config.
- Only occurs once a minute, so, given BGP timers, assume pod readiness 100 seconds from initiation.
- Supports a `max_peers` setting, which limits the number of peers added in this group. This is suitable for connections to a higher tier in a hierarchical route reflector / route server topology.
- `subtractive`
- This can be seen as a "wildcard". This is suitable for an environment in which not all peers are strictly defined and uses Junos BGP group `allow` config to permit connections from a range.
- The `allow` config is dynamically generated based on the list of all prefixes in the meshrr configuration with all peers from any mesh groups removed.
## Examples
- [2regions-hrr](examples/2regions-hrr)
- Hierarchicial route reflectors broken into two regions with a single core region unifying them.
- Reachability via static routes and Kubernetes NodeIP Services referencing additional loopbacks on the Kubernetes nodes.
- [load-balanced-route-servers](examples/load-balanced-route-servers)
- EVPN route servers deployed in a full iBGP mesh with each other serving eBGP peers. Intended to scale DCI for multi-region deployment.
- Reachability for external devices achieved through use of MetalLB in BGP mode.
- Reachability for external devices achieved through use of MetalLB in BGP mode.
## Example Commands
| Command | Description |
| ------------------------------------------------------------ | ------------------------------------------------------------ |
| `kubectl [-n NAMESPACE] get pods -o wide` | List pods and the nodes on which they run |
| `kubectl [-n NAMESPACE] exec -it POD -c crpd -- cli` | Access the CLI of cRPD |
| `kubectl [-n NAMESPACE] exec POD -c crpd -- cli show bgp summary` | See the `show bgp summary` output of a pod |
| `kubectl [-n NAMESPACE] exec POD -c crpd -- cli show bgp group summary \|except \"Allow\|orlonger\|^Default\|^$\"` | See the status of the neighbor groups of a pod |
| `kubectl [-n NAMESPACE] logs [-f] POD -c meshrr` | View the logs from the meshrr sidecar container. `-f` will follow the logs. |
| `kubectl [-n NAMESPACE] delete pod POD` | Delete POD. Because pods should be created by DaemonSet, StatefulSet, or Deployment, a new pod should be recreated in its place; in this context, this may be considered functionally more similar to a "restart" than to a "delete". |
Loading

0 comments on commit 42ec518

Please sign in to comment.