diff --git a/README.md b/README.md index e633da5..504220a 100644 --- a/README.md +++ b/README.md @@ -26,15 +26,17 @@ Usage of ./logveil: -d value Path to directory with anonymizing data -i value - Path to input file containing logs to be anonymized + Path to input file containing logs to be anonymized (mandatory - if you don't specify input, code will fail) -o value Path to output file (default: Stdout) + -c value + Path to input file with custom anonymization mapping -v Enable verbose logging -e Change input file type to LM export (default: LM Backup) -p - Disable proof wrtier (default: Enabled) + Disable proof writer (default: Enabled) -h Help for logveil ``` @@ -61,72 +63,102 @@ Usage of ./logveil: `./logveil -d example_anon_data/ -e -i lm_export.csv -p -v` -### How it works +6. Read log data from LM Export file (CSV), output anonymization result to standard output (STDOUT) and load custom mapping from custom_mapping.txt -**This is only a simplified example and does not match 1:1 with how anonymization is actually implemented** +`./logveil -d example_anon_data/ -e -i lm_export.csv -c custom_mapping.txt` -Consider below log line. It is formatted in a common `key:value` format. + +## Anonymization functionality + +There are three ways LogVeil anonymizes data: + +### Custom anonymization mappings + +You can provide custom anonymization mappings for LogVeil to use. They will take precedence over any other anonymization functionality. + +Custom mappings can be enabled by using flag `-c ` and must have the following format: + +`:` + +Each custom mapping must be separated by new line. For example: + +`test_custom_replacement:test_custom_replacement123`\ +`replace_this:with_that`\ +`test123:test1234` + +### Anonymization data + +You can also provide sets of fake data to use when anonymizing. + +Consider below log line: ``` -{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip":"89.239.31.49", "username":"test.user@test.cz", "organization":"TESTuser.test.com", "mac": "71:e5:41:18:cb:3e"} +{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip":"89.239.31.49", "username":"test.user@test.cz", "organization":"TESTuser.test.com", "mac": "71:e5:41:18:cb:3e", "replacement_test":"replace_this"} ``` -First, LogVeil will load anonymization data from supplied directory (`-d example_anon_data/`). Each file in that folder should be named according to the values it will be masking. For example, lets assume we have following directory structure: +If you want to anonymize values in `organization` and `username` keys, you need to have two files of the same name in anonymization data folder and enable them by using `-d ` flag. 1. `username.txt` 2. `organization.txt` -Next, LogVeil will go over each log line in supplied input and extract `key:value` pairs from it. When applied to above log line it would look like this: +Both files should contain appropriate fake data for the values they will be masking. + +### Regexp scanning and dynamic fake data generation -1. `"@timestamp": "2024-06-05T14:59:27.000+00:00"` -2. `"src_ip":"89.239.31.49"` -3. `"username":"test.user@test.cz"` -4. `"organization":"TESTuser.test.com"` -5. `"mac": "71:e5:41:18:cb:3e"` +LogVeil implements regular expressions to look for common patterns: IP (v4, v6), Emails, MAC and URL. Once such pattern is found it is replaced with fake data generated on the fly. -Then, LogVeil will try to match extracted pairs to anonymization data it loaded in previous step. Two paris should be matched: +## Output -1. `"src_ip":"89.239.31.49"` with `src_ip.txt` -2. `"username":"test.user@test.cz"` with `username.txt` -3. `"organization":"TESTuser.test.com"` with `organization.txt` +Anonymized data will be written to provided file path in txt format. Alternatively, if you don't provide output file path it will be written to the console (stdout). -And one pair should be matched by regular expression scanning: +Additionally LogVeil will write anonymization proof to `proof.json`, to show which values were anonymized. Proof has a following format: -1. `"mac": "71:e5:41:18:cb:3e"` +``` +{"original":"", "new":"} +``` -Now LogVeil will grab values (randomly) from files which filenames matched with keys, generate new value for `mac` key and create a replacement map in format `"original_value":"new_value"`: +## How it works -1. `"89.239.31.49":"10.20.0.53"` -1. `"test.user@test.cz":"ladislav.dosek"` -2. `"TESTuser.test.com":"Apple"` -3. `"71:e5:41:18:cb:3e": "0f:da:68:92:7f:2b"` +**This is only a simplified example and does not match 1:1 with how anonymization is actually implemented** -Now each element from the above list will be iterated over and compared against log line. Whenever `original_value` is found it will be replaced with `new_value`. Outcome should look like this: +Consider below log line. It is formatted in a common `key:value` format. ``` -{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip":"10.20.0.53", "username":"ladislav.dosek", "organization":"Apple", "mac": "0f:da:68:92:7f:2b"} +{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip":"89.239.31.49", "username":"test.user@test.cz", "organization":"TESTuser.test.com", "mac": "71:e5:41:18:cb:3e", "replacement_test":"replace_this"} ``` -``` -{"original": "27.221.126.209", "new": "10.20.0.53"}, -"{"original":"test.user@test.cz","new":"ladislav.dosek"}" -"{"original":"TESTuser.test.com","new":"Apple"}" -{"original": "71:e5:41:18:cb:3e", "new": "0f:da:68:92:7f:2b"}, -``` +First, LogVeil will load anonymization data from supplied directory (`-d example_anon_data/`). Each file in that folder should be named according to the values it will be masking. For example, lets assume we have following directory structure: -### Anonymization data +1. `username.txt` +2. `organization.txt` + +Second, if available, LogVeil will load the custom anonymization mapping from user supplied path. For example, assume we have following file `custom_mapping.txt` with below content: + +1. `test_custom_replacement:test_custom_replacement123` +2. `replace_this:with_that` +3. `test123:test1234` -Each `key:value` pair which you want to anonymize data must have its equivalent in anonymization data folder. +Now anonymization process can start. LogVeil will grab log lines from supplied input, one by one, and apply anonymization to it three steps: -If anonymization data does not exist for any given `key:value` pair then LogVeil will attempt to use regular expressions to match and replace common values such as: IPv4, IPv6, MAC, Emails and URLs. +1. Replace values based on custom anonymization mapping +2. Replace values based on loaded anonymization data +3. Replace values based on regular expression matching and fake data generation -For example, if you want to anonymize values in `organization` and `username` keys, you need to have two files of the same name in anonymization folder containing some random data. +Final output should look like this: -### Output +``` +{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip":"10.20.0.53", "username":"ladislav.dosek", "organization":"Apple", "mac": "0f:da:68:92:7f:2b", "replacement_test":"with_that"} +``` -Anonymized data will be outputted to provided file path in txt format. +And anonymization proof: -Alternatively, if you don't provide file path, output will be written to the console. +``` +{"original":"replace_this", "new":"with_that"} +{"original": "27.221.126.209", "new": "10.20.0.53"}, +{"original":"test.user@test.cz","new":"ladislav.dosek"}, +{"original":"TESTuser.test.com","new":"Apple"}, +{"original": "71:e5:41:18:cb:3e", "new": "0f:da:68:92:7f:2b"}, +``` ## Release diff --git a/internal/anonymizer/anonymizer.go b/internal/anonymizer/anonymizer.go index 98622bf..2d4ff46 100644 --- a/internal/anonymizer/anonymizer.go +++ b/internal/anonymizer/anonymizer.go @@ -4,7 +4,6 @@ import ( "fmt" "log/slog" "regexp" - "strings" "github.com/logmanager-oss/logveil/internal/config" "github.com/logmanager-oss/logveil/internal/generator" @@ -16,31 +15,38 @@ import ( // Anonymizer represents an object responsible for anonymizing indivisual log lines feed to it. It contains anonymization data which will be used to anonymize input and a random number generator funtion used to select values from anonymization data. type Anonymizer struct { - anonData map[string][]string - randFunc func(int) int - proofWriter *proof.ProofWriter - lookup *lookup.Lookup - generator *generator.Generator - replacementMap map[string]string + anonymizationData map[string][]string + customAnonymizationMapping map[string]string + randFunc func(int) int + proofWriter *proof.ProofWriter + lookup *lookup.Lookup + generator *generator.Generator + replacementMap map[string]string } func CreateAnonymizer(config *config.Config, proofWriter *proof.ProofWriter) (*Anonymizer, error) { - anonymizingData, err := loader.Load(config.AnonymizationDataPath) + customAnonymizationMapping, err := loader.LoadCustomAnonymizationMapping(config.CustomAnonymizationMappingPath) + if err != nil { + return nil, fmt.Errorf("loading custom anonymization mappings from path %s: %v", config.CustomAnonymizationMappingPath, err) + } + + anonymizationData, err := loader.LoadAnonymizationData(config.AnonymizationDataPath) if err != nil { return nil, fmt.Errorf("loading anonymizing data from dir %s: %v", config.AnonymizationDataPath, err) } return &Anonymizer{ - anonData: anonymizingData, - randFunc: rand.Intn, - proofWriter: proofWriter, - lookup: lookup.New(), - generator: &generator.Generator{}, + anonymizationData: anonymizationData, + customAnonymizationMapping: customAnonymizationMapping, + randFunc: rand.Intn, + proofWriter: proofWriter, + lookup: lookup.New(), + generator: &generator.Generator{}, }, nil } func (an *Anonymizer) Anonymize(logLine map[string]string) string { - an.replacementMap = make(map[string]string) + an.replacementMap = an.customAnonymizationMapping an.loadAndReplace(logLine) @@ -51,7 +57,6 @@ func (an *Anonymizer) Anonymize(logLine map[string]string) string { an.generateAndReplace(logLineRaw, an.lookup.ValidEmail, an.generator.GenerateRandomEmail()) an.generateAndReplace(logLineRaw, an.lookup.ValidUrl, an.generator.GenerateRandomUrl()) - an.proofWriter.Write(an.replacementMap) an.proofWriter.Flush() return an.replace(logLineRaw) @@ -76,7 +81,7 @@ func (an *Anonymizer) loadAndReplace(logLine map[string]string) { continue } - if anonValues, exists := an.anonData[field]; exists { + if anonValues, exists := an.anonymizationData[field]; exists { newAnonValue := anonValues[an.randFunc(len(anonValues))] an.replacementMap[value] = newAnonValue @@ -98,8 +103,19 @@ func (an *Anonymizer) generateAndReplace(rawLog string, regexp *regexp.Regexp, g } func (an *Anonymizer) replace(rawLog string) string { - for oldValue, newValue := range an.replacementMap { - rawLog = strings.ReplaceAll(rawLog, oldValue, newValue) + for originalValue, newValue := range an.replacementMap { + // Added word boundary to avoid matching words withing word. For example "test" in "testing". + r := regexp.MustCompile(fmt.Sprintf(`\b%s\b`, originalValue)) + + var found bool + rawLog = r.ReplaceAllStringFunc(rawLog, func(originalValue string) string { + found = true + return newValue + }) + + if found { + an.proofWriter.Write(originalValue, newValue) + } } return rawLog diff --git a/internal/anonymizer/anonymizer_test.go b/internal/anonymizer/anonymizer_test.go index 30efd6d..1f63d35 100644 --- a/internal/anonymizer/anonymizer_test.go +++ b/internal/anonymizer/anonymizer_test.go @@ -12,32 +12,35 @@ import ( func TestAnonimizer_AnonymizeData(t *testing.T) { tests := []struct { - name string - anonymizingDataDir string - input map[string]string - expectedOutput string + name string + anonymizationDataDir string + customAnonymizationMappingPath string + input map[string]string + expectedOutput string }{ { - name: "Test AnonymizeData", - anonymizingDataDir: "../../tests/data/anonymization_data", + name: "Test AnonymizeData", + anonymizationDataDir: "../../tests/data/anonymization_data", + customAnonymizationMappingPath: "../../tests/data/custom_mappings.txt", input: map[string]string{ "@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip": "10.10.10.1", "src_ipv6": "7f1d:64ed:536a:1fd7:fe8e:cc29:9df4:7911", "mac": "71:e5:41:18:cb:3e", - "email": "test@test.com", + "email": "atest@atest.com", "url": "https://www.testurl.com", "username": "miloslav.illes", "organization": "Microsoft", - "raw": "2024-06-05T14:59:27.000+00:00, 10.10.10.1, 7f1d:64ed:536a:1fd7:fe8e:cc29:9df4:7911, miloslav.illes, Microsoft, 71:e5:41:18:cb:3e, test@test.com, https://www.testurl.com", + "custom:": "replacement_test", + "raw": "2024-06-05T14:59:27.000+00:00, 10.10.10.1, 7f1d:64ed:536a:1fd7:fe8e:cc29:9df4:7911, miloslav.illes, Microsoft, 71:e5:41:18:cb:3e, test@test.com, https://www.testurl.com, replace_this", }, - expectedOutput: "2024-06-05T14:59:27.000+00:00, 10.20.0.53, 8186:39ac:48a4:c6af:a2f1:581a:8b95:25e2, ladislav.dosek, Apple, 0f:da:68:92:7f:2b, QHtPwsw@RJSkoHl.top, http://soqovkq.com/NfkcUjG.php", + expectedOutput: "2024-06-05T14:59:27.000+00:00, 10.20.0.53, 8186:39ac:48a4:c6af:a2f1:581a:8b95:25e2, ladislav.dosek, Apple, 0f:da:68:92:7f:2b, QHtPwsw@RJSkoHl.top, http://soqovkq.com/NfkcUjG.php, with_that", }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - anonymizer, err := CreateAnonymizer(&config.Config{AnonymizationDataPath: tt.anonymizingDataDir}, &proof.ProofWriter{IsEnabled: false}) + anonymizer, err := CreateAnonymizer(&config.Config{AnonymizationDataPath: tt.anonymizationDataDir, CustomAnonymizationMappingPath: tt.customAnonymizationMappingPath}, &proof.ProofWriter{IsEnabled: false}) if err != nil { t.Fatal(err) } diff --git a/internal/config/config.go b/internal/config/config.go index faee97a..a6258d1 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -2,16 +2,19 @@ package config import ( "flag" + "fmt" + "os" ) // Config represents user supplied program input type Config struct { - AnonymizationDataPath string - InputPath string - OutputPath string - IsVerbose bool - IsLmExport bool - IsProofWriter bool + AnonymizationDataPath string + InputPath string + OutputPath string + CustomAnonymizationMappingPath string + IsVerbose bool + IsLmExport bool + IsProofWriter bool } // LoadAndValidate loads values from user supplied input into Config struct and validates them @@ -20,6 +23,8 @@ func (c *Config) LoadAndValidate() { flag.Func("i", "Path to input file containing logs to be anonymized", validateInput(c.InputPath)) + flag.Func("c", "Path to input file containing custom anonymization mappings", validateInput(c.CustomAnonymizationMappingPath)) + flag.Func("o", "Path to output file (default: Stdout)", validateOutput(c.OutputPath)) flag.BoolVar(&c.IsVerbose, "v", false, "Enable verbose logging (default: Disabled)") @@ -27,4 +32,11 @@ func (c *Config) LoadAndValidate() { flag.BoolVar(&c.IsProofWriter, "p", true, "Disable proof wrtier (default: Enabled)") flag.Parse() + + // Check if mandatory flags are set + if c.InputPath == "" { + fmt.Println("Error: -i flag is mandatory") + flag.Usage() + os.Exit(1) + } } diff --git a/internal/loader/loader.go b/internal/loader/loader.go index 0e410bf..ab5a261 100644 --- a/internal/loader/loader.go +++ b/internal/loader/loader.go @@ -7,13 +7,44 @@ import ( "log/slog" "os" "path/filepath" + "strings" ) -// Load() loads anonymization data from given directory and returns it in a map format of: [filename][]values. Anonymization data is needed for the purposes of masking original values. -func Load(anonDataDir string) (map[string][]string, error) { - var anonData = make(map[string][]string) +func LoadCustomAnonymizationMapping(path string) (map[string]string, error) { + customMapping := make(map[string]string) - files, err := os.ReadDir(anonDataDir) + file, err := os.OpenFile(path, os.O_RDONLY, os.ModePerm) + if err != nil { + return nil, err + } + defer file.Close() + + scanner := bufio.NewScanner(file) + for scanner.Scan() { + line := scanner.Text() + values := strings.Split(line, ":") + if len(values) == 1 { + slog.Error("wrong custom mapping: %s", "error", line) + } + + originalValue := values[0] + newValue := values[1] + + customMapping[originalValue] = newValue + } + + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("error reading custom anonymization mapping: %w", err) + } + + return customMapping, nil +} + +// LoadAnonymizationData() loads anonymization data from given directory and returns it in a map format of: [filename][]values. Anonymization data is needed for the purposes of masking original values. +func LoadAnonymizationData(path string) (map[string][]string, error) { + anonymizationData := make(map[string][]string) + + files, err := os.ReadDir(path) if err != nil { log.Fatal(err) } @@ -23,19 +54,19 @@ func Load(anonDataDir string) (map[string][]string, error) { continue } - data, err := loadAnonymizingData(filepath.Join(anonDataDir, file.Name())) + data, err := loadFromFile(filepath.Join(path, file.Name())) if err != nil { return nil, fmt.Errorf("loading anonymizing data from file %s: %v", file.Name(), err) } - anonData[file.Name()] = data + anonymizationData[file.Name()] = data slog.Debug(fmt.Sprintf("Loaded anonymizing data for field: %s; values loaded: %d\n", file.Name(), len(data))) } - return anonData, nil + return anonymizationData, nil } -func loadAnonymizingData(filepath string) ([]string, error) { +func loadFromFile(filepath string) ([]string, error) { anonDataFile, err := os.OpenFile(filepath, os.O_RDONLY, os.ModePerm) if err != nil { return nil, err diff --git a/internal/loader/loader_test.go b/internal/loader/loader_test.go index 64aa4dc..9bafd1b 100644 --- a/internal/loader/loader_test.go +++ b/internal/loader/loader_test.go @@ -9,21 +9,45 @@ import ( "github.com/stretchr/testify/assert" ) -func TestAnonimizer_Anondataloader(t *testing.T) { +func TestAnonimizer_LoadCustomAnonymizationMapping(t *testing.T) { + tests := []struct { + name string + customAnonymizationMappingPath string + expectedMapping map[string]string + }{ + { + name: "Test Loading Custom Anonymization Mapping", + customAnonymizationMappingPath: "../../tests/data/custom_mappings.txt", + expectedMapping: map[string]string{"replace_this": "with_that", "test123": "test1234", "test_custom_replacement": "test_custom_replacement123"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mapping, err := LoadCustomAnonymizationMapping(tt.customAnonymizationMappingPath) + if err != nil { + t.Fatal(err) + } + + assert.Equal(t, tt.expectedMapping, mapping) + }) + } +} + +func TestAnonimizer_LoadAnonymizationData(t *testing.T) { tests := []struct { name string anonDataDir string expectedFields []string }{ { - name: "Test Anondataloader", + name: "Test Anonymization Data Loading", anonDataDir: "../../tests/data/anonymization_data", expectedFields: []string{"dst_iface", "dst_ip", "ip", "name", "organization", "src_iface", "src_ip", "username"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - anonData, err := Load(tt.anonDataDir) + anonData, err := LoadAnonymizationData(tt.anonDataDir) if err != nil { t.Fatal(err) } diff --git a/internal/proof/proof.go b/internal/proof/proof.go index 461d15c..b188d30 100644 --- a/internal/proof/proof.go +++ b/internal/proof/proof.go @@ -35,30 +35,29 @@ func CreateProofWriter(config *config.Config, openFiles *files.FilesHandler) (*P return &ProofWriter{IsEnabled: false}, nil } -func (p *ProofWriter) Write(replacementMap map[string]string) { +func (p *ProofWriter) Write(originalValue string, newValue string) { if !p.IsEnabled { return } - for originalValue, newValue := range replacementMap { - proof := struct { - OriginalValue string `json:"original"` - NewValue string `json:"new"` - }{ - OriginalValue: originalValue, - NewValue: newValue, - } + proof := struct { + OriginalValue string `json:"original"` + NewValue string `json:"new"` + }{ + OriginalValue: originalValue, + NewValue: newValue, + } - bytes, err := json.Marshal(proof) - if err != nil { - slog.Error("marshalling anonymisation proof", "error", err) - } + bytes, err := json.Marshal(proof) + if err != nil { + slog.Error("marshalling anonymisation proof", "error", err) + } - _, err = fmt.Fprintf(p.writer, "%s\n", bytes) - if err != nil { - slog.Error("writing anonymisation proof", "error", err) - } + _, err = fmt.Fprintf(p.writer, "%s\n", bytes) + if err != nil { + slog.Error("writing anonymisation proof", "error", err) } + } func (p *ProofWriter) Flush() { diff --git a/internal/proof/proof_test.go b/internal/proof/proof_test.go index c2aaf42..87072dd 100644 --- a/internal/proof/proof_test.go +++ b/internal/proof/proof_test.go @@ -15,23 +15,22 @@ func TestProof_Write(t *testing.T) { tests := []struct { name string isProofWriter bool - replacementMap map[string]string + originalValue string + newValue string expectedOutput string }{ { - name: "Test case 1: write proof", - isProofWriter: true, - replacementMap: map[string]string{ - "test": "masked", - }, + name: "Test case 1: write proof", + isProofWriter: true, + originalValue: "test", + newValue: "masked", expectedOutput: "{\"original\":\"test\",\"new\":\"masked\"}\n", }, { - name: "Test case 2: proof writer disabled", - isProofWriter: false, - replacementMap: map[string]string{ - "test": "masked", - }, + name: "Test case 2: proof writer disabled", + isProofWriter: false, + originalValue: "test", + newValue: "masked", expectedOutput: "", }, } @@ -45,7 +44,7 @@ func TestProof_Write(t *testing.T) { t.Fatal(err) } - p.Write(tt.replacementMap) + p.Write(tt.originalValue, tt.newValue) p.Flush() file, err := os.OpenFile("proof.json", os.O_RDWR|os.O_CREATE, 0644) diff --git a/internal/reader/export_test.go b/internal/reader/export_test.go index 91098b6..e0ba689 100644 --- a/internal/reader/export_test.go +++ b/internal/reader/export_test.go @@ -22,7 +22,21 @@ func TestLmExport(t *testing.T) { { name: "Test LM Export Anonymizer", inputFilename: "../../tests/data/lm_export_test_input.csv", - expectedOutput: map[string]string{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip": "89.239.31.49", "username": "test.user@test.cz", "organization": "TESTuser.test.com", "raw": "{\"@timestamp\": \"2024-06-05T14:59:27.000+00:00\", \"msg.src_ip\":\"89.239.31.49\", \"username\":\"test.user@test.cz\", \"organization\":\"TESTuser.test.com\"}"}, + expectedOutput: map[string]string{"@timestamp": "2024-06-05T14:59:27.000+00:00", "src_ip": "89.239.31.49", "username": "test.user@test.cz", "organization": "TESTuser.test.com", "raw": "{\"@timestamp\": \"2024-06-05T14:59:27.000+00:00\", \"msg.src_ip\":\"89.239.31.49\", \"username\":\"test.user@test.cz\", \"organization\":\"TESTuser.test.com\", \"replacement_test\":\"replace_this\"}"}, + }, + { + name: "Test LM Export Anonymizer - RAW missing", + inputFilename: "../../tests/data/lm_export_test_input_raw_missing.csv", + expectedOutput: map[string]string{}, + wantErr: true, + expectedErr: fmt.Errorf("Malformed lm export file - RAW field is missing"), + }, + { + name: "Test LM Export Anonymizer - RAW empty", + inputFilename: "../../tests/data/lm_export_test_input_raw_empty.csv", + expectedOutput: map[string]string{}, + wantErr: true, + expectedErr: fmt.Errorf("Malformed lm export file - RAW field cannot be empty"), }, { name: "Test LM Export Anonymizer - RAW missing", diff --git a/tests/data/custom_mappings.txt b/tests/data/custom_mappings.txt new file mode 100644 index 0000000..eed4cd0 --- /dev/null +++ b/tests/data/custom_mappings.txt @@ -0,0 +1,3 @@ +test_custom_replacement:test_custom_replacement123 +replace_this:with_that +test123:test1234 diff --git a/tests/data/lm_export_test_input.csv b/tests/data/lm_export_test_input.csv index 8d4fadf..cb906cf 100644 --- a/tests/data/lm_export_test_input.csv +++ b/tests/data/lm_export_test_input.csv @@ -1,2 +1,2 @@ @timestamp,raw,msg.src_ip,msg.username,msg.organization -2024-06-05T14:59:27.000+00:00,"{""@timestamp"": ""2024-06-05T14:59:27.000+00:00"", ""msg.src_ip"":""89.239.31.49"", ""username"":""test.user@test.cz"", ""organization"":""TESTuser.test.com""}",89.239.31.49,test.user@test.cz,TESTuser.test.com +2024-06-05T14:59:27.000+00:00,"{""@timestamp"": ""2024-06-05T14:59:27.000+00:00"", ""msg.src_ip"":""89.239.31.49"", ""username"":""test.user@test.cz"", ""organization"":""TESTuser.test.com"", ""replacement_test"":""replace_this""}",89.239.31.49,test.user@test.cz,TESTuser.test.com diff --git a/tests/integration_test.go b/tests/integration_test.go index c41d424..90c0cab 100644 --- a/tests/integration_test.go +++ b/tests/integration_test.go @@ -28,34 +28,37 @@ func TestLogVeil_IntegrationTest(t *testing.T) { { name: "Test LM Backup Anonymizer", config: &config.Config{ - AnonymizationDataPath: "data/anonymization_data", - InputPath: "data/lm_backup_test_input.gz", - IsLmExport: false, - IsProofWriter: true, + AnonymizationDataPath: "data/anonymization_data", + CustomAnonymizationMappingPath: "data/custom_mappings.txt", + InputPath: "data/lm_backup_test_input.gz", + IsLmExport: false, + IsProofWriter: true, }, expectedOutput: "<189>date=2024-11-06 time=12:29:25 devname=\"LM-FW-70F-Praha\" devid=\"FGT70FTK22012016\" eventtime=1730892565525108329 tz=\"+0100\" logid=\"0000000013\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.20.0.53 srcport=57158 srcintf=\"lan1\" srcintfrole=\"wan\" dstip=227.51.221.89 dstport=80 dstintf=\"lan1\" dstintfrole=\"lan\" srccountry=\"China\" dstcountry=\"Czech Republic\" sessionid=179455916 proto=6 action=\"client-rst\" policyid=9 policytype=\"policy\" poluuid=\"d8ccb3e4-74d4-51ef-69a3-73b41f46df74\" policyname=\"Gitlab web from all\" service=\"HTTP\" trandisp=\"noop\" duration=6 sentbyte=80 rcvdbyte=44 sentpkt=2 rcvdpkt=1 appcat=\"unscanned\" srchwvendor=\"H3C\" devtype=\"Router\" mastersrcmac=\"0f:da:68:92:7f:2b\" srcmac=\"0f:da:68:92:7f:2b\" srcserver=0 dsthwvendor=\"H3C\" dstdevtype=\"Router\" masterdstmac=\"0f:da:68:92:7f:2b\" dstmac=\"0f:da:68:92:7f:2b\" dstserver=0\n", expectedProof: []map[string]interface{}{ {"original": "dev-uplink", "new": "lan1"}, - {"original": "wan1-lm", "new": "lan1"}, - {"original": "00:23:89:39:a4:ef", "new": "0f:da:68:92:7f:2b"}, {"original": "00:23:89:39:a4:fa", "new": "0f:da:68:92:7f:2b"}, {"original": "27.221.126.209", "new": "10.20.0.53"}, + {"original": "wan1-lm", "new": "lan1"}, + {"original": "00:23:89:39:a4:ef", "new": "0f:da:68:92:7f:2b"}, {"original": "95.80.197.108", "new": "227.51.221.89"}, }, }, { name: "Test LM Export Anonymizer", config: &config.Config{ - AnonymizationDataPath: "data/anonymization_data", - InputPath: "data/lm_export_test_input.csv", - IsLmExport: true, - IsProofWriter: true, + AnonymizationDataPath: "data/anonymization_data", + CustomAnonymizationMappingPath: "data/custom_mappings.txt", + InputPath: "data/lm_export_test_input.csv", + IsLmExport: true, + IsProofWriter: true, }, - expectedOutput: "{\"@timestamp\": \"2024-06-05T14:59:27.000+00:00\", \"msg.src_ip\":\"10.20.0.53\", \"username\":\"ladislav.dosek\", \"organization\":\"Apple\"}\n", + expectedOutput: "{\"@timestamp\": \"2024-06-05T14:59:27.000+00:00\", \"msg.src_ip\":\"10.20.0.53\", \"username\":\"ladislav.dosek\", \"organization\":\"Apple\", \"replacement_test\":\"with_that\"}\n", expectedProof: []map[string]interface{}{ + {"original": "replace_this", "new": "with_that"}, + {"original": "89.239.31.49", "new": "10.20.0.53"}, {"original": "test.user@test.cz", "new": "ladislav.dosek"}, {"original": "TESTuser.test.com", "new": "Apple"}, - {"original": "89.239.31.49", "new": "10.20.0.53"}, }, }, } @@ -93,6 +96,7 @@ func TestLogVeil_IntegrationTest(t *testing.T) { assert.Equal(t, tt.expectedOutput, output.String()) + proofWriter.Flush() actualProof, err := unpackProofOutput() if err != nil { t.Fatal(err)