Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization of SGs #191

Merged
merged 65 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
8ae3193
updated subcmds
YairSlobodin1 Aug 5, 2024
2187816
readme
YairSlobodin1 Aug 5, 2024
13ae9a2
Merge branch 'main' into first_step_optimization
YairSlobodin1 Aug 6, 2024
851e02c
generic
YairSlobodin1 Aug 6, 2024
210ddcc
merge
YairSlobodin1 Aug 6, 2024
20ffe15
Merge branch 'main' into use_models
YairSlobodin1 Aug 12, 2024
4aeaef5
updated
YairSlobodin1 Aug 18, 2024
9a4f06d
Merge branch 'main' into use_models
YairSlobodin1 Aug 21, 2024
cec430e
use ipblock size
YairSlobodin1 Aug 26, 2024
b8ee61b
minor changes
YairSlobodin1 Aug 26, 2024
c117941
template
YairSlobodin1 Aug 26, 2024
067edfe
another change
YairSlobodin1 Aug 26, 2024
19c5483
rename folder name
YairSlobodin1 Aug 26, 2024
74737e7
renaming
YairSlobodin1 Aug 26, 2024
4fb97b4
updated template
YairSlobodin1 Aug 26, 2024
02231d3
use ToIPAddressString
YairSlobodin1 Sep 2, 2024
150f697
Merge branch 'use_models' into optimize
YairSlobodin1 Sep 2, 2024
820a023
check protocol is not nil
YairSlobodin1 Sep 8, 2024
2916159
Merge branch 'use_models' into optimize
YairSlobodin1 Sep 8, 2024
9286689
inbound udp
YairSlobodin1 Sep 9, 2024
d25bf9c
make mod
YairSlobodin1 Sep 11, 2024
87180ef
merge wip
YairSlobodin1 Sep 11, 2024
dc400bf
update
YairSlobodin1 Sep 11, 2024
40c3a52
fixed
YairSlobodin1 Sep 12, 2024
4af3f7a
merged
YairSlobodin1 Sep 12, 2024
bd2acb8
merge main
YairSlobodin1 Sep 25, 2024
67f320b
merge
YairSlobodin1 Sep 25, 2024
3d120cc
Merge branch 'main' into optimize
YairSlobodin1 Sep 25, 2024
09528ae
Merge branch 'main' into optimize
YairSlobodin1 Sep 29, 2024
4919d00
Read sgs (#192)
YairSlobodin1 Oct 1, 2024
0fa5920
Merge branch 'main' into optimize
YairSlobodin1 Oct 1, 2024
02cae44
Optimize output and flags (#193)
YairSlobodin1 Oct 2, 2024
6fcb85f
Merge branch 'main' into optimize
YairSlobodin1 Oct 7, 2024
5f3d0d9
fixed
YairSlobodin1 Oct 9, 2024
af1c12c
merge
YairSlobodin1 Oct 9, 2024
68f23ed
wip
YairSlobodin1 Oct 10, 2024
38f9fa5
lookup functions
YairSlobodin1 Oct 13, 2024
c565d25
better lookup
YairSlobodin1 Oct 14, 2024
4ba19c2
lookup, blocked
YairSlobodin1 Oct 14, 2024
cd94ff0
fix bugs, linter, update old tests
YairSlobodin1 Oct 14, 2024
e763b72
readme
YairSlobodin1 Oct 14, 2024
d6fa8c3
rename acl_segment test
YairSlobodin1 Oct 14, 2024
4894fa8
generic synth acl func
YairSlobodin1 Oct 14, 2024
68503df
merged
YairSlobodin1 Oct 14, 2024
dc92129
added instance nif segment test (acl)
YairSlobodin1 Oct 14, 2024
e40861d
added vpe acl test
YairSlobodin1 Oct 14, 2024
6fa8417
merge test configs
YairSlobodin1 Oct 14, 2024
e34acb0
added sg segment tests
YairSlobodin1 Oct 14, 2024
737b803
small fix
YairSlobodin1 Oct 28, 2024
799cd73
small fix
YairSlobodin1 Oct 28, 2024
6407b27
rename1
YairSlobodin1 Oct 28, 2024
a9cb5ac
rename2
YairSlobodin1 Oct 28, 2024
f4bca3b
rename3
YairSlobodin1 Oct 28, 2024
c3fbb1a
Merge branch 'main' into update_conn
YairSlobodin1 Oct 28, 2024
9fbc33f
renaming
YairSlobodin1 Oct 29, 2024
7a62bc3
review comments
YairSlobodin1 Oct 29, 2024
61c623d
merge
YairSlobodin1 Oct 30, 2024
7b37e1c
merge
YairSlobodin1 Dec 9, 2024
ecbb4aa
small fixed
YairSlobodin1 Dec 9, 2024
aa83ea8
errors.Join
YairSlobodin1 Dec 10, 2024
e6dbbf4
Merge branch 'main' into optimize
YairSlobodin1 Dec 10, 2024
f1e93a4
-d optimization
YairSlobodin1 Dec 17, 2024
bc8133f
readme
YairSlobodin1 Dec 17, 2024
b4b86f1
Optimize algorithm (#195)
YairSlobodin1 Dec 18, 2024
1cc3ad6
update readme
YairSlobodin1 Dec 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 44 additions & 18 deletions README.md
zivnevo marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,45 +1,69 @@
# vpc-network-config-synthesis

## About vpc-network-config-synthesis
Tool for automatic synthesis of VPC network configurations, namely Network ACLs and Security Groups.
Tool for automatic synthesis and optimization of VPC network configurations, namely Network ACLs and Security Groups.


## Usage
Use the `vpcgen` CLI tool with one of the following commands to specify the type of network resources to generate.
Use the `vpcgen` CLI tool with one of the following commands:
* `vpcgen synth sg` - generate Security Groups.
* `vpcgen synth acl` - generate an nACL for each subnet separately.
* `vpcgen synth acl --single` - generate a single nACL for all subnets in the same VPC.
* `vpcgen optimize sg` - optimize SGs.
* `vpcgen optimize acl` - optimize nACLs (In progress).

### nACLs Generation
Specifying the `--single` flag results in generating a single nACL for all subnets in the same VPC. Otherwise, an nACL is generated for each subnet separately.
**Note**: A required connection between NIFs/VSIs/VPEs implies connectivity will be allowed between the subnets they are contained in.
## Synthesis
#### nACLs Generation
A required connection between NIFs/VSIs/VPEs implies connectivity will be allowed between the subnets they are contained in.

### SGs Generation
**Note**: A Security Group, generated for a specific VSI (or for one of its NIFs), will be applied to all the NIFs of the VSI. The same goes for Reserved IPs of a VPE.
#### SGs Generation
A Security Group, generated for a specific VSI (or for one of its NIFs), will be applied to all the NIFs of the VSI. The same goes for Reserved IPs of a VPE.

### Supported types
#### Supported types
The input supports subnets, subnet segments, CIDR segments, NIFs, NIF segments, instances (VSIs), instance segments, VPEs, VPE segments and externals.
**Note**: Segments should be defined in the spec file.

### Output
1. If the `output-dir` flag is used, the specified folder will be created, containing one file per VPC. Each generated file will contain the network resources (Security Groups or Network ACLs) relevant to its VPC. File names are set as `prefix_vpc`, where prefix is ​​the value received in the `prefix` flag. If the `prefix` flag is omitted, file names will match VPC names.
2. If the `output-file` flag is used, all generated resources will be written to the specified file.
3. if both `output-file` and `output-dir` flags are not used, the collection will be written to stdout.
#### Options
```commandline
Flags:
-s, --spec string JSON file containing spec file
```

## Optimization
#### SG optimization
SG optimizatin attempts to reduce the number of security group rules in a SG without changing the semantic.
Specifying the `-n` flag results in optimizing only one given SG. Otherwise, all SGs will be optimized.
```
Flags:
-n, --sg-name string which security group to optimize
```

### Global options
#### nACL optimization (in progress)
nACL optimizatin attempts to reduce the number of nACL rules in an nACL without changing the semantic.
Specifying the `-n` flag results in optimizing only one given nACL. Otherwise, all nACLs will be optimized.
```
Flags:
-n, --acl-name string which nACL to optimize
```


## Global options
```commandline
Flags:
-c, --config string JSON file containing a configuration object of existing resources
-f, --format string Output format; must be one of [tf, csv, md, json]
-h, --help help for vpc-synthesis
-h, --help help for vpcgen
-l, --locals whether to generate a locals.tf file (only possible when the output format is tf)
-d, --output-dir string Write generated resources to files in the specified directory, one file per VPC.
-o, --output-file string Write all generated resources to the specified file.
-o, --output-file string Write all generated resources to the specified file
-p, --prefix string The prefix of the files that will be created.
-s, --spec string JSON file containing spec file
```
**Note 1**: The infrastructure configuration must always be provided using the `--config` flag.
**Note 2**: Multi-vpc input is supported.
**Note**: The infrastructure configuration must always be provided using the `--config` flag.

## Output
1. If the `output-dir` flag is used, the specified folder will be created, containing one file per VPC. Each generated file will contain the network resources (Security Groups or Network ACLs) relevant to its VPC. File names are set as `prefix_vpc`, where prefix is ​​the value received in the `prefix` flag. If the `prefix` flag is omitted, file names will match VPC names.
2. If the `output-file` flag is used, all generated resources will be written to the specified file.
3. if both `output-file` and `output-dir` flags are not used, the collection will be written to stdout.

## Build the project
Make sure you have golang 1.23+ on your platform.
Expand All @@ -60,6 +84,8 @@ make build
bin/vpcgen synth acl -c test/data/acl_testing5/config_object.json -s test/data/acl_testing5/conn_spec.json

bin/vpcgen synth sg -c test/data/sg_testing3/config_object.json -s test/data/sg_testing3/conn_spec.json

bin/vpcgen optimize sg -c test/data/optimize_sg_redundant/config_object.json
```

**Note**: Windows environment users should replace all `/` with `\`.
31 changes: 26 additions & 5 deletions cmd/subcmds/optimize.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,38 @@ SPDX-License-Identifier: Apache-2.0

package subcmds

import "github.com/spf13/cobra"
import (
"fmt"

func NewOptimizeCommand(args *inArgs) *cobra.Command {
"github.com/spf13/cobra"

"github.com/np-guard/vpc-network-config-synthesis/pkg/ir"
"github.com/np-guard/vpc-network-config-synthesis/pkg/optimize"
)

func newOptimizeCommand(args *inArgs) *cobra.Command {
cmd := &cobra.Command{
Use: "optimize",
Short: "Optimize is not supported yet",
Long: `Optimize is not supported yet`,
Short: "optimization of existing SG (nACLS are not supported yet)",
Long: `optimization of existing SG (nACLS are not supported yet)`,
}

cmd.AddCommand(newOptimizeACLCommand(args))
// sub cmds
cmd.AddCommand(newOptimizeSGCommand(args))

return cmd
}

func optimization(cmd *cobra.Command, args *inArgs, newOptimizer func(ir.Collection, string) optimize.Optimizer, isSG bool) error {
cmd.SilenceUsage = true // if we got this far, flags are syntactically correct, so no need to print usage
collection, err := parseCollection(args, isSG)
if err != nil {
return fmt.Errorf("could not parse config file %v: %w", args.configFile, err)
}
optimizer := newOptimizer(collection, args.firewallName)
optimizedCollection, err := optimizer.Optimize()
if err != nil {
return err
}
return writeOutput(args, optimizedCollection, collection.VpcNames(), false)
}
4 changes: 2 additions & 2 deletions cmd/subcmds/optimizeACL.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ package subcmds

import "github.com/spf13/cobra"

func newOptimizeACLCommand(_ *inArgs) *cobra.Command {
// temporarily exported and currently unused
func NewOptimizeACLCommand(_ *inArgs) *cobra.Command {
cmd := &cobra.Command{
Use: "acl",
Short: "OptimizeACL is not supported yet",
Expand All @@ -17,6 +18,5 @@ func newOptimizeACLCommand(_ *inArgs) *cobra.Command {
return nil
},
}

return cmd
}
19 changes: 14 additions & 5 deletions cmd/subcmds/optimizeSG.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,27 @@ SPDX-License-Identifier: Apache-2.0

package subcmds

import "github.com/spf13/cobra"
import (
"github.com/spf13/cobra"

func newOptimizeSGCommand(_ *inArgs) *cobra.Command {
sgoptimizer "github.com/np-guard/vpc-network-config-synthesis/pkg/optimize/sg"
)

const sgNameFlag = "sg-name"

func newOptimizeSGCommand(args *inArgs) *cobra.Command {
cmd := &cobra.Command{
Use: "sg",
Short: "OptimizeSG is not supported yet",
Long: `OptimizeSG is not supported yet`,
Short: "OptimizeSG attempts to reduce the number of security group rules in a SG without changing the semantic.",
Long: `OptimizeSG attempts to reduce the number of security group rules in a SG without changing the semantic.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return nil
return optimization(cmd, args, sgoptimizer.NewSGOptimizer, true)
},
}

// flags
cmd.PersistentFlags().StringVarP(&args.firewallName, sgNameFlag, "n", "", "which security group to optimize")

return cmd
}
63 changes: 28 additions & 35 deletions cmd/subcmds/output.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,33 +16,27 @@ import (
"github.com/np-guard/vpc-network-config-synthesis/pkg/io/csvio"
"github.com/np-guard/vpc-network-config-synthesis/pkg/io/mdio"
"github.com/np-guard/vpc-network-config-synthesis/pkg/io/tfio"

"github.com/np-guard/vpc-network-config-synthesis/pkg/ir"
)

const defaultFilePermission = 0o644
const defaultDirectoryPermission = 0o755

func writeOutput(args *inArgs, collection ir.Collection, vpcNames []ir.ID) error {
if err := updateOutputFormat(args); err != nil {
return err
}
if args.outputDir != "" && args.outputFmt == apiOutputFormat {
return fmt.Errorf("-d cannot be used with format json")
}
func writeOutput(args *inArgs, collection ir.Collection, vpcNames []string, isSynth bool) error {
if args.outputDir != "" { // create the directory if needed
if err := os.MkdirAll(args.outputDir, defaultDirectoryPermission); err != nil {
return err
}
}
if err := writeLocals(args, collection, vpcNames); err != nil {

if err := writeLocals(args, vpcNames, collection); err != nil {
return err
}

var data *bytes.Buffer
var err error
if args.outputDir == "" {
if data, err = writeCollection(args, collection, ""); err != nil {
if data, err = writeCollection(args, collection, "", isSynth); err != nil {
return err
}
return writeToFile(args.outputFile, data)
Expand All @@ -55,7 +49,7 @@ func writeOutput(args *inArgs, collection ir.Collection, vpcNames []ir.ID) error
if args.prefix != "" {
args.outputFile = args.outputDir + "/" + args.prefix + "_" + suffix
}
if data, err = writeCollection(args, collection, vpc); err != nil {
if data, err = writeCollection(args, collection, vpc, isSynth); err != nil {
return err
}
if err := writeToFile(args.outputFile, data); err != nil {
Expand All @@ -65,9 +59,9 @@ func writeOutput(args *inArgs, collection ir.Collection, vpcNames []ir.ID) error
return nil
}

func writeCollection(args *inArgs, collection ir.Collection, vpc string) (*bytes.Buffer, error) {
func writeCollection(args *inArgs, collection ir.Collection, vpc string, isSynth bool) (*bytes.Buffer, error) {
var data bytes.Buffer
writer, err := pickWriter(args, &data)
writer, err := pickWriter(args, &data, isSynth)
if err != nil {
return nil, err
}
Expand All @@ -77,15 +71,7 @@ func writeCollection(args *inArgs, collection ir.Collection, vpc string) (*bytes
return &data, nil
}

func writeToFile(outputFile string, data *bytes.Buffer) error {
if outputFile == "" {
fmt.Println(data.String())
return nil
}
return os.WriteFile(outputFile, data.Bytes(), defaultFilePermission)
}

func pickWriter(args *inArgs, data *bytes.Buffer) (ir.Writer, error) {
func pickWriter(args *inArgs, data *bytes.Buffer, isSynth bool) (ir.Writer, error) {
w := bufio.NewWriter(data)
switch args.outputFmt {
case tfOutputFormat:
Expand All @@ -94,34 +80,41 @@ func pickWriter(args *inArgs, data *bytes.Buffer) (ir.Writer, error) {
return csvio.NewWriter(w), nil
case mdOutputFormat:
return mdio.NewWriter(w), nil
case apiOutputFormat:
return confio.NewWriter(w, args.configFile)
default:
return nil, fmt.Errorf("bad output format: %q", args.outputFmt)
case jsonOutputFormat:
if isSynth {
return confio.NewWriter(w, args.configFile)
}
}
return nil, fmt.Errorf("bad output format: %q", args.outputFmt)
}

func writeLocals(args *inArgs, collection ir.Collection, vpcNames []ir.ID) error {
if !args.locals {
func writeToFile(outputFile string, data *bytes.Buffer) error {
if outputFile == "" {
fmt.Println(data.String())
return nil
}
if args.outputFmt != tfOutputFormat {
return fmt.Errorf("--locals flag requires setting the output format to tf")
}
return os.WriteFile(outputFile, data.Bytes(), defaultFilePermission)
}

_, isACLCollection := collection.(*ir.ACLCollection)
func writeLocals(args *inArgs, vpcNames []ir.ID, collection ir.Collection) error {
if !args.locals {
return nil
}
var data *bytes.Buffer
var err error

_, isACLCollection := collection.(*ir.ACLCollection)

if data, err = tfio.WriteLocals(vpcNames, isACLCollection); err != nil {
return err
}

outputFile := ""
suffix := "/locals.tf"
pathSuffix := "/locals.tf"
if args.outputDir != "" {
outputFile = args.outputDir + suffix
outputFile = args.outputDir + pathSuffix
} else if args.outputFile != "" {
outputFile = filepath.Dir(args.outputFile) + suffix
outputFile = filepath.Dir(args.outputFile) + pathSuffix
}
return writeToFile(outputFile, data)
}
6 changes: 3 additions & 3 deletions cmd/subcmds/outputFormat.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ const (
tfOutputFormat = "tf"
csvOutputFormat = "csv"
mdOutputFormat = "md"
apiOutputFormat = "json"
jsonOutputFormat = "json"
defaultOutputFormat = csvOutputFormat
)

var outputFormats = []string{tfOutputFormat, csvOutputFormat, mdOutputFormat, apiOutputFormat}
var outputFormats = []string{tfOutputFormat, csvOutputFormat, mdOutputFormat, jsonOutputFormat}

func updateOutputFormat(args *inArgs) error {
var err error
Expand All @@ -40,7 +40,7 @@ func inferFormatUsingFilename(filename string) (string, error) {
case strings.HasSuffix(filename, ".md"):
return mdOutputFormat, nil
case strings.HasSuffix(filename, ".json"):
return apiOutputFormat, nil
return jsonOutputFormat, nil
default:
return "", fmt.Errorf("bad output format")
}
Expand Down
Loading
Loading