wazuh-pipeline

A CI pipeline built for Wazuh.

Link to original repository.

Table of Contents

1. Key Features
2. Quickstart
3. Release Configuration
4. Pipeline Organization
   • Decoders folder
   • Rules folder
   • Tests folder
5. Decoder Resolution
6. Building Containers
7. Running Locally
8. Debugging
9. Troubleshooting
10. Related
11. License
12. Deprecated Sections
    • Public Setup
    • Private Setup

Key Features

• Test Decoders

  • Verify that your decoders are extracting info from logs.

• Test Rules

  • Ensure that the correct rules alert on logs.

• Maintain Rulesets Easily

• Catch Errors Early

• Prevent Regression

• Decoder Conflict Resolution

  • Automatically disables default decoders that overlap with custom decoders.

Quickstart

1. Click Use this tempalte in the top right corner.

2. Click Create a new repository.

3. Configure the repository name, description, and visibility (Public or Private).

4. This will create a new repository in your GitHub account.

   • Note: The pipeline will fail with the intial commit. This is expected

5. Configure the three following Action secrets:

   • DOCKER_IMAGE - The docker image used for testing. Publicly supported docker image: alexchristy/wazuh-test-pipeline
   • DOCKER_USERNAME - Username for the docker account pulling the docker image. Ex: [email protected]
   • DOCKER_PASSWORD - Password for the docker account pulling the docker image.

   

6. [Private Repos Only] Create a fine-grained GitHub token for the next step. The token should only have access to the new private repository we just created.

   Required Repository permissions:

   • Pull requests (Read-only)
   • Metadata (Read-only)
   • Contents (Read-only)

7. [Private Repos Only] Configure an additional secret called TOKEN with the value of the token created in the previous step.

8. This pipeline is configured to use auto to create releases automatically.

   • ✅ Keep it: Jump to the Release Configuration section to set it up and return to this step when finished.

   • ❌ Remove it: Delete two files: .autorc and .github/workflows/release.yml. Commit changes. Continue to the next step.

9. Navigate to Actions

10. Click the correct workflow run:

    • If you configured auto release, click the Update auto configuration workflow.
    • If you removed auto release, click the Initial commit workflow. (Should be the only workflow run available)

11. In the top right, click Re-run jobs, then click Re-run all jobs

12. On the pop up click the green button Re-run jobs

13. The pipeline should completed successfully.

    • If auto release was configured, there should be a new release, v0.0.1, on the repo homepage in GitHub

Release Configuration

This section details how to setup and configure automatic releases for your repository.

1. Clone your pipeline repository.

   • It should have files in it from when we created the template.

2. Download the appropriate auto binary from the Auto Releases page.

3. Unzip the auto binary. On Ubuntu:

   gunzip auto-linux.gz

4. Make the auto binary executable. On Ubuntu:

   chmod +x ./auto-linux

5. Move the executable into the cloned repo and enter the directory. On Ubuntu:

   mv ./auto-linux ./wazuh-pipeline/
   cd ./wazuh-pipeline/

6. Initialize auto. On Ubuntu:

   ./auto-linux init

7. Options to select:

   • Package Manager Plugin --> Git Tag
   • Plugins --> Optionally: First Time Contributor and Released
   • GitHub Project --> Fill in your repository details here.
   • Git User --> Generally, put the username and email of account that owns the pipeline repo.
   • Release only on PR --> false
   • Enterprise GitHub --> Generally set to false.
   • Create .env file --> true
     • This should be a GitHub personal access token of type Tokens (classic) with all repo permissions.
   • Customize default labels --> false
   • Add more labels --> false

   You may see an error at the end. This is expected.

8. Create version labels. On Ubuntu:

   ./auto-linux create-labels

9. Delete auto binary. On Ubuntu:

   rm ./auto-linux

10. Commit .autorc changes. On Ubuntu:

    git add ./.autorc
    git commit -m "Update auto configuration"
    git push origin main

11. Back on the repo's page on GitHub, navigate to Settings > Actions > General.

    • Find the section called Workflow permissions
    • Select Read and write permissions
    • Click Save

12. The auto release is now configured. Continue with step 9 in the Quickstart section.

Pipeline Organization

The pipeline repository has three main folders: decoders, rules, and tests. The .sh scripts in the root of repository should not be modified unless explicitly trying to change the behavior of the pipeline.

Decoders folder

Located at decoders/, this folder is where you should put all the custom decoder files (.xml) that will get automatically installed when the container is run.

Note: If there is a decoder name in this folder that conflicts/overlaps with a default Wazuh decoder, the default decoder file will be disabled. (See the Decoder Resolution section for more information.)

Rules folder

Located at rules/, this folder is where you should put all custom rule files (.xml) that will get automatically installed when the container is run.

Tests folder

Located at tests/, this folder is where you should put all of your WazuhTest files (.json) and associated raw log files. See the WazuhTest repository for information on test syntax and organization.

Note: If a rule's ID conflicts with an existing/default rule only the first rule definition will be used.

Decoder Resolution

If a custom decoder name overlaps with a default Wazuh decoder's name, the Wazuh manager will fail to startup. To address this, this pipeline will automatically disable default decoder names that overlap/conflict with custom decoders names.

This can be useful, but it can also break detection logic as the pipeline will exclude entire default decoder files when any conflict with a custom decoder file is detected.

Example:

You add the custom auditd decoder below to the decoders/ folder in this pipeline.

custom_auditd_decoder.xml contents:

<decoder name="auditd">
  <prematch>My Special Custom Pattern</prematch>
</decoder>

The pipeline scripts will find an overlapping default decoder 0040-auditd_decoders.xml and disable the entire file. This is because both files contain a decoder with the name auditd. As a result of being in the same file, the decoder auditd-syscall will also be disabled.

0040-auditd_decoders.xml contents:

<html><body><decoder name="auditd">
  <prematch>^type=</prematch>
</decoder>

<decoder name="auditd-syscall">
  <parent>auditd</parent>
  <prematch offset="after_parent">^SYSCALL </prematch>
  <regex offset="after_parent">^(SYSCALL) msg=audit\(\d\d\d\d\d\d\d\d\d\d.\d\d\d:(\d+)\): </regex>
  <order>audit.type,audit.id</order>
</decoder>

(...)

Because of this behavior, it is recommended that when you are modifying default decoders copy the entire original decoder file and make the modifications inside of the copy.

Building Containers

This project maintains a public docker image for ease of use here.

If you wish to build your own docker images for the pipeline you can build them using the two Dockerfiles.

Dockerfiles:

• Dockerfile.auto - This is the image used for the pipeline or other automations.
• Dockerfile.live - This is an interactive image that will run indefinitely after running the pipeline logic.
  • Mainly used for debugging or local testing.

Build image:

docker build --no-cache -f Dockerfile.{auto or live} -t local-wazuh-pipeline-image .

For the pipeline to work correctly in GitHub you will need to upload your docker image to Docker Hub and then set the value of the DOCKER_IMAGE GitHub Action secret to your new image name.

Example Image Name:

Docker Hub image link: https://hub.docker.com/r/alexchristy/wazuh-test-pipeline

DOCKER_IMAGE secret value: alexchristy/wazuh-test-pipeline

Running Locally

1. Clone the repository.

   git clone https://github.com/alexchristy/wazuh-pipeline

2. Enter repository directory.

   cd wazuh-pipeline

3. Build docker image.

   docker build --no-cache -f Dockerfile.{auto or live} -t local-wazuh-pipeline-image .

   Choose the .live image if you are trying to debug the container or interact with the configured Wazuh manager.

4. Run docker container.

   docker run -d --name wazuh-pipeline-container \
   -e REPO_URL={URL_TO_YOUR_REPO} \
   -e BRANCH_NAME=main \
   -e TOKEN={GITHUB_TOKEN_IF_REPO_PRIVATE} \
   local-wazuh-pipeline-image

Debugging

The pipeline scripts generate three logs during runtime inside of the /root/wazuh_pipeline/ directory.

Logs:

• wazuh_pipeline_script.log - Human friendly and easily readable log.
• wazuh_pipeline_shell.log - Debug shell logging with done with set -x.
• wazuh_pipeline_wazuh_test.log - WazuhTest tool log.

The easiest way to debug the container is to build the interactive image (Dockerfile.live) and run the image locally. The interactive image will execute the pipeline scripts initially and then you can connect and inspect the logs.

Troubleshooting

Push Repository not Found

When creating a copy of this pipeline in your own private repository you might run into an error like the one below when trying to push the files to your newly created repo:

me@hostname:/tmp/wazuh-pipeline.git$ git push private main
remote: Repository not found.
fatal: repository 'https://github.com/myusername/wazuh-pipeline/' not found

Fix Steps:

1. Create a new SSH key for authenticating to GitHub (or use an exiting one).

2. Add the SSH key to your GitHub account (GitHub Documentation).

3. Add the SSH key to your SSH client. On Ubuntu:

   ssh-add /path/to/ssh_key

4. Add configuration for the new SSH key. On Ubuntu, edit or create ~/.ssh/config:

   Host github.com
     HostName github.com
     User git
     IdentityFile /path/to/ssh_key

5. Change to the remote URL for the repo clone:

   git remote remove private
   git remote add [email protected]:$MY_USERNAME/$REPO_NAME.git
   git push private main

Pipeline Could not Read Username

This error causes the pipeline step named Run container with branch and repo info to fail on the first run. Usually this happens after pushing the copied files to your private repository. The error in the pipeline log will say something like:

Cloning repository without token
Cloning into '/root/wazuh_pipeline'...
fatal: could not read Username for 'https://github.com': No such device or address
Could not cd in repo directory. The repo was likely not cloned properly!
If this is a private repo, make sure to include a GitHub token.
/root/init.sh: line 17: cd: /root/wazuh_pipeline: No such file or directory
Error: Process completed with exit code 1.

This happens because the pipeline does not have the TOKEN secret configured, so it assumes that your Wazuh pipeline repository that you created is public.

Fix Steps:

1. Create a fine-grained GitHub token for accessing this repository following Step 2 in the Private Setup section. Save this value for the next step.

2. Navigate to Settings > Secrets and variables > Actions.

3. Create a new repository secret called TOKEN with the value of the GitHub fine-grained token you created earlier.

4. Navigate to Actions and click the failed workflow run. (Usually there is only one run)

5. In the top right, click Re-run jobs, then click Re-run all jobs

6. On the pop up click the green button Re-run job

7. The pipeline should completed successfully.

If this did not work, look at other sections in the Troubleshooting section for other solutions.

Write Access to Repository not Granted

This error causes the pipeline step named Run container with branch and repo info and usually occurs on the first or second run of the pipeline. The error in the pipeline/action log will say something like:

Cloning repository with token
Cloning into '/root/wazuh_pipeline'...
remote: Write access to repository not granted.
fatal: unable to access 'https://github.com/myusername/wazuh-pipeline/': The requested URL returned error: 403
/root/init.sh: line 17: cd: /root/wazuh_pipeline: No such file or directory
Could not cd in repo directory. The repo was likely not cloned properly!
If this is a private repo, make sure to include a GitHub token.
Error: Process completed with exit code 1.

This happens because the repository TOKEN secret is configured with a GitHub access token that does not have the correct permissions for the repository.

Fix Steps:

1. Starting at step 6, follow all the steps in the Quickstart Section.

Related

wazuh-pipeline - Wazuh CI pipeline that leverages this tool

License

GNU General Public License v3.0

Deprecated Sections

[Deprecated] Public Setup

1. Fork this repository and only copy the main branch

   

   

2. Configure the three following Action secrets:

   • DOCKER_IMAGE - The docker image used for testing. Publicly supported docker image: alexchristy/wazuh-test-pipeline
   • DOCKER_USERNAME - Username for the docker account pulling the docker image. Ex: [email protected]
   • DOCKER_PASSWORD - Password for the docker account pulling the docker image.

   

3. Enable GitHub Actions

   

4. Done!

   You can now start creating pull requests or committing directly to main and see the tests run automatically. However, due to the nature of forked repositories, you repository will always be public. If this is an issue, follow the steps for a private repoistory setup.

[Deprecated] Private Setup

This section is for the people people who need to run this pipeline in a private repository.

1. Create a new private repository in GitHub.

   Note: Ensure that the repository is NOT initialized with a README.md or any other files.

2. Create a fine-grained GitHub token for the next step.

   The token should only have access to the new private repository we just created.

   Required Repository permissions:

   • Pull requests (Read-only)
   • Metadata (Read-only)
   • Contents (Read-only)

3. Configure the GitHub Action secrets, from step 2 in the Quickstart and an additional secret called TOKEN with the value of the token created in the previous step.

4. Clone this repository.

   git clone --bare https://github.com/alexchristy/wazuh-pipeline

5. Enter directory.

   cd wazuh-pipeline.git

6. Add a new remote

   git remote add private {NEW_REPO_URL}

7. Push main branch to new repository

   git push private main

   Note: If this step is failing ensure that the account you are using has proper access to the new repository.

8. Finished!

   Pushing the main branch will kick off the CI pipeline which should run the default tests. If it passes then the repository is ready for use. If it fails then the repository is not functional and an issue should be filed with the GitHub Action log.