Introduction

This guide explains how to set up an environment to develop and test the Vertica operator.

Repo Structure

docker-vertica/: has the necessary files to build a Vertica server container. The build process requires that you provide a Vertica version 10.1.1 or higher RPM package. The RPM is not included in this repo.
docker-operator/: has the necessary files to build the container that holds the operator
docker-vlogger/: has the necessary files to build the container that runs the vlogger sidecar. This sidecar will tail the vertica.log to stdout. This is used only for development purposes to aid in testing out the sidecar property in the custom resource (CR).
scripts/: contains scripts for the repository. Some scripts run Makefile targets, while others are used during e2e testing.
api/: defines the spec of the custom resource definition (CRD)
pkg/: includes all of the packages that for the operator
cmd/: contains source code for each of the executables
bin/: contains the compiled or downloaded binaries that this repository depends on
config/: generated files of all of the manifests that make up the operator. Of particular importance is config/crd/bases/vertica.com_verticadbs.yaml, which shows a sample spec for our CRD.
tests/: has the test files for e2e testing
changes/: stores the changelog for past releases and details about the changes for the next release
hack/: includes a boilerplate file of a copyright that is included on the generated files
helm-charts/: contains the Helm charts that this repository builds

Software Setup

Prior to developing, the following software needs to be installed manually. There is other software that needed, but it is downloaded through make targets in the repo's bin directory.

docker
go (version 1.19.6)
kubectl (version 1.20.1)
helm (version 3.5.0)
kubectx (version 0.9.1)
kubens (version 0.9.1)
krew (version 0.4.1) $HOME/.krew/bin must be in your path
kuttl (version 0.9.0)
changie (version 1.2.0)
jq (version 1.5+)

Kind

Setup

Kind is a way to set up a multi-node Kubernetes cluster using Docker. It mimics a multi-node setup by starting a separate container for each node. The machine requirements for running Kind are minimal - it is possible to set this up on your own laptop. This is the intended deployment to run the tests in an automated fashion.

We have a wrapper that you can use to set up Kind and create a cluster that is suitable for testing Vertica. The following command creates a cluster named cluster1 with one node. It completes after a few minutes:

$ scripts/kind.sh init cluster1

When the command completes, change the context to use the cluster. The cluster has its own kubectl context named kind-cluster1:

$ kubectx kind-cluster1

To test the container, check the status of the nodes:

$ kubectl get nodes

Cleanup

After you are done with the cluster, you can delete it with our helper script. Substitute cluster1 with the name of your cluster:

$ scripts/kind.sh term cluster1

If you forgot the cluster name, run Kind directly to get all of the installed clusters:

$ PATH=$PATH:$HOME/go/bin
$ kind get clusters
cluster1

Add a changelog entry

The changelog file is generated by Changie. It separates the changelog generation from commit history, so any PR that has a notable change should add new changie entries.

Developer Workflow

IMPORTANT: You must have a Vertica version 11.0.0 or higher RPM to build the docker-vertica container. This repo DOES NOT include an RPM.

Store the RPM in the docker-vertica/packages directory with the name vertica-x86_64.RHEL6.latest.rpm:

$ cp /dir/vertica-x86_64.RHEL6.latest.rpm docker-vertica/packages/

1. Building and Pushing Vertica Container

We currently use the following containers:

docker-vertica/Dockerfile: The long-running container that runs the vertica daemon.
docker-operator/Dockerfile: The container that runs the operator and webhook
docker-vlogger/Dockerfile: The container that runs the vertica logger. It will tail the output of vertica.log to stdout. This is used for testing purposes. Some e2e tests use this as a sidecar to the Vertica server container.
docker-bundle/Dockerfile: The container that contains the 'bundle' for the operator. This is used by OLM. The contents of the docker-bundle/ directory are generated with make docker-build-bundle.

To run Vertica in Kubernetes, we need to package Vertica inside a container. This container is later referenced in the YAML file when we install the Helm chart.

By default, we create containers that are stored in the local docker daemon. The tag is either latest or, if running in a Kind environment, it is kind. You can control the container names by setting the following environment variables prior to running the make target.

OPERATOR_IMG: Operator image name.
VERTICA_IMG: Vertica image name.
VLOGGER_IMG: Vertica logger sidecar image name.
BUNDLE_IMG: OLM bundle image name.

If necessary, these variables can include the url of the registry. For example, export OPERATOR_IMG=myrepo:5000/verticadb-operator:latest.

Vertica provides two container sizes: the default, full image, and the minimal image that does not include the 240MB Tensorflow package.

Run the docker-build make target to build the necessary containers. The following command uses the default image:
```
$ make docker-build
```
To build the minimal container, invoke the make target with MINIMAL_VERTICA_IMG=YES:
```
$ make docker-build MINIMAL_VERTICA_IMG=YES
```
Due to the size of the vertica image, this step can take in excess of 10 minutes when run on a laptop.
Make these containers available to the Kubernetes cluster. Push them to the Kubernetes cluster with the following make target:
```
$ make docker-push
```
This command honors the same environment variables used when creating the image.

If your image builds fail silently, confirm that there is enough disk space in your Docker repository to store the built images.

2. Generating controller files

Vertica uses the operator-sdk framework for the operator. It provides tools to generate code so that you do not have to manually write boilerplate code. Depending on what you changed you may need to periodically regenerate files with the following command:

$ make generate manifests

3. Running Linters

We run three different linters:

Helm lint: Uses the chart verification test that is built into Helm.
Go lint: Uses a few linters that you can run with golang.
Dockerfile lint: Uses hadolint to check the various Dockerfile's that we have in our repo

Both of these linters can be run with this command:

$ make lint

4. Running Unit Tests

We have unit tests for both the Helm chart and the Go operator.

Unit tests for the Helm chart are stored in helm-charts/verticadb-operator/tests. They use the unittest plugin for helm. Some samples that you can use to write your own tests can be found at unittest github page. For details about the test format, review the helm-unittest GitHub repo.

Unit tests for the Go operator use the Go testing infrastructure. Some of the tests stand up a mock Kubernetes control plane using envtest, and runs the operator against that. Per Go standards, the test files are included in package directories and end with _test.go.

The Helm chart testing and Go lang unit tests can be run like this:

$ make run-unit-tests

5. Running the Operator

There are two ways to run the operator:

Locally in your shell.
Packaged in a container and deployed in Kubernetes as a deployment object

Option 1: Locally

This method runs the operator synchronously in your shell. It is the fastest way get up and running, but it does not mimic the way that the operator runs in a real Kubernetes environment.

Enter the following command:

make install run

Press Ctrl+C to stop the operator.

NOTE: When you run the operator locally, you can run only ad-hoc tests, not the e2e tests

This disables the webhook from running too, as running the webhook requires TLS certs to be available.

This will monitor VerticaDB CR's in the current namespace only. You need to update your kubectl config and change the namespace if you want to monitor a different one.

Option 2: Kubernetes Deployment

When run in this mode, the operator will watch only a single namespace. It will automatically use the current namespace as defined in the current config context.

The default deployment model will install the operator with helm. You can also have it install with olm or randomly pick between olm and helm. You can control this by specifing the DEPLOY_WITH environment variable; valid values are: helm, olm or random.

The operator pod contains a webhook, which needs TLS certificates setup. When deploying with helm, the default behaviour is generate a self-signed TLS certificate internally. Although, there are helm chart parameters to provide a custom TLS certificate or have one created through cert-manager (see the webhook.certSource helm chart parameter for info).

When installing with olm, you need to have olm setup.

To deploy helm and all of its prereqs, use the following command:

DEPLOY_WITH=helm make config-transformer deploy

To deploy olm all of its prereqs, use the following command:

DEPLOY_WITH=olm make setup-olm deploy

To remove the operator, regardless of how it was deployed, run the undeploy make target:

make undeploy

6. Running e2e Tests

The end-to-end (e2e) tests are run through Kubernetes itself. We use kuttl as the testing framework. The operator must be running as a Kubernetes deployment, which means the operator container needs to be built and pushed to a repository before starting.

Ensure that your Kubernetes cluster has a default storageClass. Most of the e2e tests do not specify any storageClass and use the default. For details about setting your storageClass, refer to the Kubernetes documentation.

Push the operator with the following command:

make docker-build-operator docker-push-operator

Start the test with the following make target. There is a lot of output generated by the tests, so it is advisable to pipe the output to a file.
```
make run-int-tests | tee kuttl.out
```

Running Individual Tests

You can also call kubectl kuttl from the command line if you want more control, such as running a single test or preventing cleanup when the test ends. For example, you can run a single e2e test and persist namespace with this command:

kubectl kuttl test --test <name-of-your-test> --skip-delete

For a complete list of flags that you can use with kuttl, see the kuttl docs.

The communal endpoint must be setup prior to calling the tests in this manner. You can set that up with a call to make setup-minio.

Customizing Communal Endpoints

The default endpoint for e2e tests is minio with TLS, which is created through the make setup-minio target. The endpoint are set for the test when calling scripts/setup-kustomize.sh. That script can take a config file that you can use to override the endpoint and credentials.

Here are the steps on how to override them:

Make a copy of tests/kustomize-defaults.cfg

cp tests/kustomize-defaults.cfg my-defaults.cfg

Edit my-defaults.cfg by setting your own access point, bucket, region and credentials that you want to use.
Set the KUSTOMIZE_CFG environment vairable to point to my-defaults.cfg
```
export KUSTOMIZE_CFG=my-defaults.cfg
```

Setup the commmunal endpoint.

AWS S3 BUCKET

If you have an AWS account, you can configure your environment so that it uses AWS instead of minio.

Prerequisite: The s3 bucket must already be created prior to running the tests.

Here are the steps to set that up:

Edit my-defaults.cfg and fill in the details.

Fill in the ACCESSKEY and SECRETKEY with your unique IDs.
Use the chart below to know what to fill in depending on what AWS region you want to write to.

Env Name	Description	Sample value
ENDPOINT	Endpoint and credentials for s3 communal access in the tests.	https://s3.us-east-1.amazonaws.com
REGION	The AWS region.	us-east-1
S3_BUCKET	This is the name of the bucket	aws-s3-bucket-name
PATH_PREFIX	his is used to place the communal path in a subdirectory.	/<userID>
COMMUNAL_EP_CERT_SECRET		<leave blank>
COMMUNAL_EP_CERT_NAMESPACE		<leave blank>

Google Cloud Storage

Prerequisite: The Google Cloud bucket must already be created prior to running the tests.

Here are the steps:

You need to create a ‘User account HMAC’. This will give you an access key and secret that you can use later on.
Edit my-defaults.cfg and fill in the details. Use the chart below as a guide.

Env Name	Description	Sample value
ACCESSKEY	Use the access key that you got when you generated the HMAC
SECRETKEY	Use the secret that you got when you generated the HMAC
PATH_PROTOCOL	This tells the kustomize scripts to setup for Google cloud.	gs://
BUCKET_OR_CLUSTER	Name of the bucket to use.	gc-bucket-name
PATH_PREFIX	Include your user name here so that all dbs that we know who created the DBs. It must begin and end with a slash.	/johndoe/

Azure Blob Storage

You have to decide whether to connect with the accountKey or a shared access signature (SAS). If it is a SAS, you can generate one in a self-serve manner.
- In the WebUI (https://portal.azure.com) go to the storage container that you want access to
- On the left is a link called “Shared access tokens”. Click that.
- Fill in the form to create a SAS token.
Edit my-defaults.cfg and fill in the details. Use the chart below as a guide.

Env Name	Description	Sample value
CONTAINERNAME	Name of the azure container	container-name
ACCOUNTKEY	If authenticating with an account key, fill this in. Otherwise it can be left blank.
SHAREDACCESSSIGNATURE	If authenticating with a SAS, fill this in. This can be left blank if using an accountKey. Before to include it in quotes ("") because of the special characters that are typically used.
PATH_PROTOCOL	Set this to tell the e2e tests that Azure is being used.	azb://
BUCKET_OR_CLUSTER	Fill in the account name	account-name
PATH_PREFIX	Include your user name here so that all dbs that we know who created the DBs. It must begin and end with a slash.	/johndoe/

Run the integration tests.
```
kubectl kuttl test
```

Stern output

The e2e tests use stern to save off logs of some pods. This is done to aid in debugging any failures. If needed, the logs are stored in the int-tests-output directory by default. Cleanup of the stern process is only done if kuttl runs to completition. If you abort the kuttl run, then you will need to stop the stern process manually.

Help

To see the full list of make targets, run the following command:

$ make help

Problem Determination

The following sections provide troubleshooting tips for your deployment.

Kubernetes Events

The operator generates Kubernetes events for some key scenarios. This can be a useful tool when trying to understand what the operator is doing. Use the following command to view the events:

$ kubectl describe vdb mydb

...<snip>...
Events:
  Type    Reason                   Age    From                Message
  ----    ------                   ----   ----                -------
  Normal  Installing               2m10s  verticadb-operator  Calling update_vertica to add the following pods as new hosts: mydb-sc1-0
  Normal  InstallSucceeded         2m6s   verticadb-operator  Successfully called update_vertica to add new hosts and it took 3.5882135s
  Normal  CreateDBStart            2m5s   verticadb-operator  Calling 'admintools -t create_db'
  Normal  CreateDBSucceeded        92s    verticadb-operator  Successfully created database with subcluster 'sc1'. It took 32.5709857s
  Normal  ClusterRestartStarted    36s    verticadb-operator  Calling 'admintools -t start_db' to restart the cluster
  Normal  ClusterRestartSucceeded  28s    verticadb-operator  Successfully called 'admintools -t start_db' and it took 8.8401312s

Retrieving Logs with vertica.log

You might need to inspect the contents of the vertica.log to diagnose a problem with the Vertica server. There are a few ways this can be done:

Drop into the container and navigate to the directory where is is stored. The exact location depends on your CR. You can refer to the Vertica documentation to find the location.
Deploy a sidecar to capture the vertica.log and print it to stdout. If this sidecar is enabled you can use kubectl logs to inspect it. This sidecar can be used by adding the following into your CR:
```
spec:
  ...
  sidecars:
  - name: vlogger
    image: vertica/vertica-logger:latest
```
The sidecars[i].image shown here is a container that Vertica publishes on its docker repository. After the sidecar container is running, inspect the logs with the following command:
```
$ kubectl logs <vertica-pod-name> -c vlogger
```

Memory Profiling

The memory profiler lets you view where the big allocations are occurring and to help detect any memory leaks. The toolset is Google's pprof.

By default, the memory profiler is disabled. To enable it, add a parameter when you start the operator. The following steps enable the memory profiler for a deployed operator.

Use kubectl edit to open the running deployment for editing:

$ kubectl edit deployment verticadb-operator-controller-manager

Locate where the arguments are passed to the manager, and add --enable-profiler:

      ...
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=127.0.0.1:8080
        - --leader-elect
        - --enable-profiler
        command:
        - /manager
      ...

Wait until the operator is redeployed.
Port forward 6060 to access the webUI for the profiler. The name of the pod differs for each deployment, so be sure to find the one specific to your cluster:
```
$ kubectl port-forward pod/verticadb-operator-controller-manager-5dd5b54df4-2krcr 6060:6060
```
Use a web browser or the standalone tool to connect to http://localhost:6060/debug/pprof. If you use a web browser, replace localhost with the host that you used in the previous kubectl port-forward command. Invoke the standalone tool with the following command:
```
$ go tool pprof http://localhost:6060/debug/pprof
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEVELOPER.md

DEVELOPER.md

Introduction

Repo Structure

Software Setup

Kind

Setup

Cleanup

Add a changelog entry

Developer Workflow

1. Building and Pushing Vertica Container

2. Generating controller files

3. Running Linters

4. Running Unit Tests

5. Running the Operator

Option 1: Locally

Option 2: Kubernetes Deployment

6. Running e2e Tests

Running Individual Tests

Customizing Communal Endpoints

Stern output

Help

Problem Determination

Kubernetes Events

Retrieving Logs with vertica.log

Memory Profiling

Files

DEVELOPER.md

Latest commit

History

DEVELOPER.md

File metadata and controls

Introduction

Repo Structure

Software Setup

Kind

Setup

Cleanup

Add a changelog entry

Developer Workflow

1. Building and Pushing Vertica Container

2. Generating controller files

3. Running Linters

4. Running Unit Tests

5. Running the Operator

Option 1: Locally

Option 2: Kubernetes Deployment

6. Running e2e Tests

Running Individual Tests

Customizing Communal Endpoints

Stern output

Help

Problem Determination

Kubernetes Events

Retrieving Logs with vertica.log

Memory Profiling