Secret Detectors have these two major functions:
- Given some bytes, extract possible secrets, typically using a regex.
- Validate the secrets against the target API, typically using a HTTP client.
The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
We are interested in detectors for services that meet at least one of these criteria
- host data (they store any sort of data provided)
- have paid services (having a free or trial tier is okay though)
If you think that something should be included outside of these guidelines, please let us know.
- When reasonable, favor using the
net/http
library to make requests instead of bringing in another library. - Use the
common.SaneHttpClient
for thehttp.Client
whenever possible.
- Go 1.17+
- Make
-
Identify the Secret Detector name from the /proto/detectors.proto
DetectorType
enum. If necessary, runmake protos
when adding new ones. -
Generate the Secret Detector
go run hack/generate/generate.go detector <DetectorType enum name>
-
Add Secret Scanner
Add the secret scanner to the
pkg/engine/defaults.go
file likegithub.com/trufflesecurity/trufflehog/v3/pkg/detectors/<detector_name>
and<detector_name>.Scanner{},
-
Complete the secret detector.
The previous step templated a boilerplate + some example code as a package in the
pkg/detectors
folder for you to work on. The secret detector can be completed with these general steps:- Create a test secrets file, and export the variable
- Update the pattern regex and keywords. Try iterating with regex101.com.
- Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
- Update the tests with these test cases at minimum:
- Found and verified (using a credential loaded from GCP Secrets)
- Found and unverified
- Not found
- Any false positive cases that you come across
- Create a pull request for review.
-
Create a file called
.env
with this env file format:SECRET_TYPE_ONE=value SECRET_TYPE_ONE_INACTIVE=v@lue
-
Export the
TEST_SECRET_FILE
variable, pointing to the env file:export TEST_SECRET_FILE=".env"
Now, the detector test should attempt to load the given env key from that file.
- Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
- Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
- Open Ubuntu cli and install
dos2unix
.sudo apt install dos2unix
- Identify the
trufflehog
local directory and convertscripts/gen_proto.sh
file in Unix format.dos2unix ./scripts/gen_proto.sh
- Open /proto/detectors.proto file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
make protos
go test ./pkg/detectors/<detector> -tags=detectors