Develop (#96)

* clean up flags * update * update * update flag strings * formating issues * webserver enable/disable flaf * enhanced debug and header output * csv and json output * bring in ability to update signatures to a specific version * Update sample configuration and documentation (#91) Update the yaml configuration file example, as well as the description in step 1 and the yaml example in the README.md file. According to implementations found in scanLocalGitRepo.go:69 and scanLocalPath.go:85, the sample.yml and the quickstart instructions weren't referring to the updated parameter names. * cleanup * fix the progress bar Still not quite right due to threads but better * remove dead functions * Updated the changelog * implement persistent flags (#95) move the flags that are common to all the commands into the root command. The updateRules command does not adhere to all the flags but for know we will deal with it. Signed-off-by: Matty <[email protected]> Co-authored-by: Santiago Botta <[email protected]> Co-authored-by: Matty <[email protected]>
N0MoreSecr3ts · Apr 4, 2021 · 471568b · 471568b
1 parent 8933ca8
commit 471568b
Show file tree

Hide file tree

Showing 23 changed files with 706 additions and 442 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,27 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Changed
+- Adjust flag descriptions
+- Enhanced debug output and header
+- Update sample.yml and quickstart documentation
+- Condensed the increment stats to provide a more streamlined approach
+
+### Fixed
+- flag names were pointing to the wrong variables
+- change yml -> yaml in the documentation accross the board
+- the web interface now has the right stats and the progress bar works again
+
+### Added
+- flag to enable/disable the webserver, default is disable
+- add csv output format
+- add json output format
+- ability to update signatures to a given release
+- add consistent flags to streamline the adding of additional functionality
+
+### Removed
+- several dead functions
+
 
 ## [0.0.6] - 2020-01-22
 ### Changed
@@ -25,7 +46,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [0.0.4] - 2020-08-10
 ### Changed
-- change internal name from gitrob to wraith
+- change the internal name from gitrob to wraith
 - condense number of packages to remove cyclic dependencys. Better code organization is still needed.
 - rules are known as signatures
 - all signatures are in a single yaml file

diff --git a/README.md b/README.md
@@ -66,9 +66,9 @@ Wraith uncovers forgotten secrets and brings them back to life, haunting securit
 
 ## Quickstart
 
-1. Download the latest [release][3] and either build it yourself with `make build` or you can use a prebuilt binary, currently they only exist for OSX. This project uses a branching git flow. Details are in the developer doc, surfice it to say **Master** is stable **develop** shoud be considered beta.
+1. Download the latest [release][3] and either build it yourself with `make build` or you can use a prebuilt binary, currently they only exist for OSX. This project uses a branching git flow. Details can be found in the developer doc, suffice it to say **Master** is stable **develop** shoud be considered beta.
 2. Download or clone the latest set of [signatures][4] and either copy *signatures/default.yml* to *~/.wraith/signatures/* or adjust the location in the configuration file below.
-3. Copy the below configuration to *~/.wraith/config.yml*. This will allow you to get up and running for basic scans without having to figure out the flags. Any of these values can be overwritten on the commnd line as well. You will need to generate your own api tokens for github and gitlab if you are scanning against them.
+3. Copy *config/sample.yml* or the below configuration to *~/.wraith/config.yml*. This will allow you to get up and running for basic scans without having to figure out the flags. Any of these values can be overwritten on the commnd line as well. You will need to generate your own api tokens for github and gitlab if you are scanning against them.
 4. Once you have this done, just run a scan command.
 - `wraith scanGithub`
 - `wraith scanGitlab`
@@ -79,11 +79,11 @@ Wraith uncovers forgotten secrets and brings them back to life, haunting securit
 ---
 commit-depth: 0
 debug: false
-github-api-token: <token>>
+github-api-token: <token>
 github-targets:
   - mattyjones
   - N0MoreSecr3ts
-gitlab-api-token: <token>>
+gitlab-api-token: <token>
 gitlab-targets:
   - 5034914
   - mattyjones
@@ -95,11 +95,11 @@ ignore-path:
   - static/
   - docs/
 in-mem-clone: false
-repo-paths:
+local-repos:
   - ../wraith-test
 match-level: 3
 num-threads: 0
-repo-dirs:
+local-paths:
   - relative/path/to/repo
   - absolute/path/to/repo
 signature-file: ../wraith-signatures/signatures/default.yml

diff --git a/cmd/root.go b/cmd/root.go
@@ -5,16 +5,24 @@ package cmd
 import (
 	"fmt"
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 	"os"
+	"wraith/core"
 	"wraith/version"
 )
 
-// rootCmd represents the base command when called without any subcommands
-var rootCmd = &cobra.Command{
-	Use:   "wraith",
-	Short: "A tool to scan for secrets in various digital hiding spots",
-	Long:  "A tool to scan for secrets in various digital hiding spots - v" + version.AppVersion(), // TODO write a better long description
-}
+var (
+
+	// wraithConfig holds the configuration data the commands
+	wraithConfig *viper.Viper
+
+	// rootCmd represents the base command when called without any subcommands
+	rootCmd = &cobra.Command{
+		Use:   "wraith",
+		Short: "A tool to scan for secrets in various digital hiding spots",
+		Long:  "A tool to scan for secrets in various digital hiding spots - v" + version.AppVersion(), // TODO write a better long description
+	}
+)
 
 // Execute adds all child commands to the root command and sets flags appropriately.
 // This is called by main.main(). It only needs to happen once to the rootCmd.
@@ -25,4 +33,45 @@ func Execute() {
 	}
 }
 
-func init() {}
+func init() {
+	wraithConfig = core.SetConfig()
+
+	rootCmd.PersistentFlags().String("bind-address", "127.0.0.1", "The IP address for the webserver")
+	rootCmd.PersistentFlags().Int("bind-port", 9393, "The port for the webserver")
+	rootCmd.PersistentFlags().Int("confidence-level", 3, "The confidence level level of the expressions used to find matches")
+	rootCmd.PersistentFlags().Bool("csv", false, "output csv format")
+	rootCmd.PersistentFlags().Bool("debug", false, "Print available debugging information to stdout")
+	rootCmd.PersistentFlags().Bool("hide-secrets", false, "Do not print secrets to any supported output")
+	rootCmd.PersistentFlags().StringSlice("ignore-extension", nil, "List of file extensions to ignore")
+	rootCmd.PersistentFlags().StringSlice("ignore-path", nil, "List of file paths to ignore")
+	rootCmd.PersistentFlags().Bool("json", false, "output json format")
+	rootCmd.PersistentFlags().Int("max-file-size", 10, "Max file size to scan (in MB)")
+	rootCmd.PersistentFlags().Int("num-threads", -1, "Number of execution threads")
+	rootCmd.PersistentFlags().Bool("scan-tests", false, "Scan suspected test files")
+	rootCmd.PersistentFlags().String("signature-file", "$HOME/.wraith/signatures/default.yaml", "file(s) containing detection signatures.")
+	rootCmd.PersistentFlags().String("signature-path", "$HOME/.wraith/signatures", "path containing detection signatures.")
+	rootCmd.PersistentFlags().Bool("silent", false, "Suppress all output. An alternative output will need to be configured")
+	rootCmd.PersistentFlags().Bool("web-server", false, "Enable the web interface for scan output")
+
+	err := wraithConfig.BindPFlag("bind-address", rootCmd.PersistentFlags().Lookup("bind-address"))
+	err = wraithConfig.BindPFlag("bind-port", rootCmd.PersistentFlags().Lookup("bind-port"))
+	err = wraithConfig.BindPFlag("debug", rootCmd.PersistentFlags().Lookup("debug"))
+	err = wraithConfig.BindPFlag("confidence-level", rootCmd.PersistentFlags().Lookup("confidence-level"))
+	err = wraithConfig.BindPFlag("csv", rootCmd.PersistentFlags().Lookup("csv"))
+	err = wraithConfig.BindPFlag("hide-secrets", rootCmd.PersistentFlags().Lookup("hide-secrets"))
+	err = wraithConfig.BindPFlag("ignore-extension", rootCmd.PersistentFlags().Lookup("ignore-extension"))
+	err = wraithConfig.BindPFlag("ignore-path", rootCmd.PersistentFlags().Lookup("ignore-path"))
+	err = wraithConfig.BindPFlag("json", rootCmd.PersistentFlags().Lookup("json"))
+	err = wraithConfig.BindPFlag("max-file-size", rootCmd.PersistentFlags().Lookup("max-file-size"))
+	err = wraithConfig.BindPFlag("num-threads", rootCmd.PersistentFlags().Lookup("num-threads"))
+	err = wraithConfig.BindPFlag("scan-tests", rootCmd.PersistentFlags().Lookup("scan-tests"))
+	err = wraithConfig.BindPFlag("signature-file", rootCmd.PersistentFlags().Lookup("signature-file"))
+	err = wraithConfig.BindPFlag("signature-path", rootCmd.PersistentFlags().Lookup("signature-path"))
+	err = wraithConfig.BindPFlag("silent", rootCmd.PersistentFlags().Lookup("silent"))
+	err = wraithConfig.BindPFlag("web-server", rootCmd.PersistentFlags().Lookup("web-server"))
+
+	if err != nil {
+		fmt.Printf("There was an error binding a flag: %s\n", err.Error())
+	}
+
+}
diff --git a/cmd/scanGithub.go b/cmd/scanGithub.go
@@ -5,14 +5,12 @@ package cmd
 import (
 	"fmt"
 	"github.com/spf13/cobra"
-	"github.com/spf13/viper"
 	"os"
 	"time"
 	"wraith/core"
-	"wraith/version"
 )
 
-var viperScanGithub *viper.Viper
+//var viperScanGithub *viper.Viper
 
 // scanGithubCmd represents the scanGithub command that will enumerate and scan github.com
 var scanGithubCmd = &cobra.Command{
@@ -23,32 +21,40 @@ var scanGithubCmd = &cobra.Command{
 
 		// Set the scan type and start a new session
 		scanType := "github"
-		sess := core.NewSession(viperScanGithub, scanType)
+		sess := core.NewSession(wraithConfig, scanType)
 
 		// Ensure user input exists and validate it
-		sess.ValidateUserInput(viperScanGithub)
+		sess.ValidateUserInput(wraithConfig)
 
 		// Check for a token. If no token is present we should default to scan but give a message
 		// that no token is available so only public repos will be scanned
 		// TODO do not exit out if no token but drop a message saying only public repos will be scanned
-		sess.GithubAccessToken = core.CheckGithubAPIToken(viperScanGithub.GetString("github-api-token"), sess)
-
-		sess.Out.Warn("%s\n\n", core.ASCIIBanner)
-		sess.Out.Important("%s v%s started at %s\n", core.Name, version.AppVersion(), sess.Stats.StartedAt.Format(time.RFC3339))
-		sess.Out.Important("Loaded %d signatures.\n", len(core.Signatures))
-		sess.Out.Important("Web interface available at http://%s:%d\n", "127.0.0.1", 9393)
+		sess.GithubAccessToken = core.CheckGithubAPIToken(wraithConfig.GetString("github-api-token"), sess)
+
+		// By default we display a header to the user giving basic info about application. This will not be displayed
+		// during a silent run which is the default when using this in an automated fashion.
+		if !sess.JSONOutput && !sess.CSVOutput {
+			sess.Out.Warn("%s\n\n", core.ASCIIBanner)
+			sess.Out.Important("%s v%s started at %s\n", core.Name, sess.WraithVersion, sess.Stats.StartedAt.Format(time.RFC3339))
+			sess.Out.Important("Loaded %d signatures.\n", len(core.Signatures))
+			if sess.WebServer {
+				sess.Out.Important("Web interface available at http://%s:%d\n", sess.BindAddress, sess.BindPort)
+			}
+		}
 
-		sess.Out.Debug("We have these orgs: %s\n", sess.UserOrgs)
-		sess.Out.Debug("We have these users: %s\n", sess.UserLogins)
-		sess.Out.Debug("We have these repos: %s\n", sess.UserRepos)
+		if sess.Debug {
+			sess.Out.Debug("We have these orgs: %s\n", sess.UserOrgs)
+			sess.Out.Debug("We have these users: %s\n", sess.UserLogins)
+			sess.Out.Debug("We have these repos: %s\n", sess.UserRepos)
+		}
 
 		//Create a github client to be used for the session
 		sess.InitGitClient()
 
-		// If we have github users and no orgs or repos then we default to scan
-		// the visible repos of that user.
 		if sess.UserLogins != nil && sess.UserOrgs == nil && sess.UserRepos == nil {
 
+			// If we have github users and no orgs or repos then we default to scan
+			// the visible repos of that user.
 			core.GatherUsers(sess)
 			core.GatherGithubRepositoriesFromOwner(sess)
 
@@ -79,9 +85,9 @@ var scanGithubCmd = &cobra.Command{
 		core.AnalyzeRepositories(sess)
 		sess.Finish()
 
-		core.PrintSessionStats(sess)
+		core.SummaryOutput(sess)
 
-		if !sess.Silent {
+		if !sess.Silent && sess.WebServer {
 			sess.Out.Important("Press Ctrl+C to stop web server and exit.\n")
 			select {}
 		}
@@ -91,51 +97,17 @@ var scanGithubCmd = &cobra.Command{
 func init() {
 	rootCmd.AddCommand(scanGithubCmd)
 
-	viperScanGithub = core.SetConfig()
-
-	scanGithubCmd.Flags().String("bind-address", "127.0.0.1", "The IP address for the webserver")
-	scanGithubCmd.Flags().Int("bind-port", 9393, "The port for the webserver")
-	scanGithubCmd.Flags().Int("confidence-level", 3, "The confidence level level of the expressions used to find matches")
-	scanGithubCmd.Flags().Float64("commit-depth", -1, "Set the commit depth to scan")
-	scanGithubCmd.Flags().Bool("debug", false, "Print debugging information")
-	scanGithubCmd.Flags().Bool("gather-org-members", false, "Add members to targets when processing organizations")
-	scanGithubCmd.Flags().String("github-api-token", "", "API token for access to github, see doc for necessary scope")
+	scanGithubCmd.Flags().Bool("add-org-members", false, "Add members to targets when processing organizations")
+	scanGithubCmd.Flags().String("github-api-token", "", "API token for github access, see documentation for necessary scope")
 	scanGithubCmd.Flags().StringSlice("github-orgs", nil, "List of github orgs to scan")
 	scanGithubCmd.Flags().StringSlice("github-repos", nil, "List of github repositories to scan")
 	scanGithubCmd.Flags().StringSlice("github-users", nil, "List of github.com users to scan")
-	scanGithubCmd.Flags().Bool("hide-secrets", false, "Hide secrets from any supported output")
-	scanGithubCmd.Flags().StringSlice("ignore-extension", nil, "List of extensions to ignore")
-	scanGithubCmd.Flags().StringSlice("ignore-path", nil, "List of paths to ignore")
-	//scanGithubCmd.Flags().Bool("in-mem-clone", false, "Clone repos in memory")
-	scanGithubCmd.Flags().Int("max-file-size", 10, "Max file size to scan in MB")
-	scanGithubCmd.Flags().Int("num-threads", -1, "Number of threads to execute with")
-	scanGithubCmd.Flags().Bool("scan-forks", false, "Scan repositories forked by users or orgs")
-	scanGithubCmd.Flags().Bool("scan-tests", false, "Scan suspected test files")
-	scanGithubCmd.Flags().String("signature-file", "$HOME/.wraith/signatures/default.yml", "file(s) containing detection signatures.")
-	scanGithubCmd.Flags().String("signature-path", "$HOME/.wraith/signatures", "path containing detection signatures.")
-	scanGithubCmd.Flags().Bool("silent", false, "Suppress all output except for errors")
-
-	err := viperScanGithub.BindPFlag("bind-address", scanGithubCmd.Flags().Lookup("bind-address"))
-	err = viperScanGithub.BindPFlag("bind-port", scanGithubCmd.Flags().Lookup("bind-port"))
-	err = viperScanGithub.BindPFlag("commit-depth", scanGithubCmd.Flags().Lookup("commit-depth"))
-	err = viperScanGithub.BindPFlag("debug", scanGithubCmd.Flags().Lookup("debug"))
-	err = viperScanGithub.BindPFlag("gather-org-members", scanGithubCmd.Flags().Lookup("gather-org-members"))
-	err = viperScanGithub.BindPFlag("github-api-token", scanGithubCmd.Flags().Lookup("github-api-token"))
-	err = viperScanGithub.BindPFlag("hide-secrets", scanGithubCmd.Flags().Lookup("hide-secrets"))
-	err = viperScanGithub.BindPFlag("ignore-extension", scanGithubCmd.Flags().Lookup("ignore-extension"))
-	err = viperScanGithub.BindPFlag("ignore-path", scanGithubCmd.Flags().Lookup("ignore-extension"))
-	//err = viperScanGithub.BindPFlag("in-mem-clone", scanGithubCmd.Flags().Lookup("in-mem-clone"))
-	err = viperScanGithub.BindPFlag("confidence-level", scanGithubCmd.Flags().Lookup("confidence-level"))
-	err = viperScanGithub.BindPFlag("max-file-size", scanGithubCmd.Flags().Lookup("max-file-size"))
-	err = viperScanGithub.BindPFlag("num-threads", scanGithubCmd.Flags().Lookup("num-threads"))
-	err = viperScanGithub.BindPFlag("scan-forks", scanGithubCmd.Flags().Lookup("scan-forks"))
-	err = viperScanGithub.BindPFlag("scan-tests", scanGithubCmd.Flags().Lookup("scan-tests"))
-	err = viperScanGithub.BindPFlag("signature-file", scanGithubCmd.Flags().Lookup("signature-file"))
-	err = viperScanGithub.BindPFlag("signature-path", scanGithubCmd.Flags().Lookup("signature-path"))
-	err = viperScanGithub.BindPFlag("silent", scanGithubCmd.Flags().Lookup("silent"))
-	err = viperScanGithub.BindPFlag("github-orgs", scanGithubCmd.Flags().Lookup("github-orgs"))
-	err = viperScanGithub.BindPFlag("github-repos", scanGithubCmd.Flags().Lookup("github-repos"))
-	err = viperScanGithub.BindPFlag("github-users", scanGithubCmd.Flags().Lookup("github-users"))
+
+	err := wraithConfig.BindPFlag("add-org-members", scanGithubCmd.Flags().Lookup("add-org-members"))
+	err = wraithConfig.BindPFlag("github-api-token", scanGithubCmd.Flags().Lookup("github-api-token"))
+	err = wraithConfig.BindPFlag("github-orgs", scanGithubCmd.Flags().Lookup("github-orgs"))
+	err = wraithConfig.BindPFlag("github-repos", scanGithubCmd.Flags().Lookup("github-repos"))
+	err = wraithConfig.BindPFlag("github-users", scanGithubCmd.Flags().Lookup("github-users"))
 
 	if err != nil {
 		fmt.Printf("There was an error binding a flag: %s\n", err.Error())