synthesis README

README.md

@@ -118,7 +118,7 @@ spec:
         - Egress
 ```
 
-
+More details [here](README_Synthesis.md)
 
 
 ## NSX Supported API versions and resources

README_Synthesis.md

@@ -0,0 +1,124 @@
+# Synthesize k8s network policy resources from NSX DFW config
+
+Synthesize a given NSX DFW configuration into an equivalent k8s network policy. 
+
+```
+Flags:
+  -- synthesis-dump-dir         flag to run synthesis; specify directory path to store k8s synthesis results
+  -- synth                      flag to run synthesis, even if synthesis-dump-dir is not specified
+  -- synth-create-dns-policy    flag to create a policy allowing access to target env dns pod
+  -- synthesize-admin-policies  flag to synthesize category environment into admin network policies (which included deny, pass and priority) (default false)
+  -- disjoint-hint              comma separated list of NSX groups/tags that are always disjoint in their VM members, 
+  needed for an effective and sound synthesis process, can specify more than one hint 
+  (example: --disjoint-hint frontend,backend --disjoint-hint app,web,db) 
+```
+
+## Overview
+Synthesize a given `NSX DFW` policy into k8s network policy.
+The result may not be totally equivalent, due to limitations of the target policy; more details regarding the k8s synthesis [here](#limitation).
+There are two main challenges here: 
+* *The flattening challenge*: translating prioritized set of rules with actions `allow/deny/jump-to-app` into a flat set of  `allow` rules (which is what k8s network policies support).
+* *The intent preserving challenge*: maintain the original semantic intent of the rules
+and not just generate a set of rules that preserves the connectivity between VMs given the current state of the configuration.
+
+
+### The flattening challenge
+There are two modes of policies synthesis, depending on the value of `synthesize-admin-policies`; when
+it is not active then priortized `allow, deny, jump-to-app` rules from all `NSX categories` should be synthesized to 
+`k8s network policy`, namely, to `flat allow rules`; when it is activated then rules from `NSX category environment` should be synthesized to  
+`admin network policy` which rules have `allow, deny, pass` and priority; other categories should be synthesized as
+before to `k8s network policy`. 
+
+For example, for [this example](pkg/data/exampleHogwarts.go) there are two related files: 
+[no admin policies](pkg/synthesis/tests_expected_output/abstract_models/ExampleHogwarts.txt), which is the result of execution
+without `-- synthesize-admin-policies` and contains translation of all rules to flat allow rules; and 
+[with admin policies](pkg/synthesis/tests_expected_output/abstract_models/ExampleHogwartsAdmin_AdminPoliciesEnabled.txt),
+result of execution with `-- synthesize-admin-policies`, contains the translation when `NSX category env` 
+is synthesized to admin polices that can use `deny/pass/allow` and priorities. Full synthesis results for this example can be found 
+[here](pkg/synthesis/tests_expected_output/k8s_resources/ExampleHogwarts)
+for non-admin polices and [here](pkg/synthesis/tests_expected_output/k8s_resources/ExampleHogwartsAdmin_AdminPoliciesEnabled) 
+for admin policies.
+
+The translation of priortized `allow,deny,jump-to-app` rules into flat `allow` rules is exponential in the size of the
+original rules (to be accurate, the number of allow rules generated for each original allow rule is
+exponential in the number of term in this allow rule and in higher priority deny and pass rules). To tackle this we:
+1. Ask the user to provide the tool with `hints` -  lists of disjoint tags/groups.
+E.g., tags `{frontend, backend}` are disjoint.
+In the future it is planned to "guess" these disjoint sets, and ask the user to approve them. E.g., 
+for [this example](pkg/data/exampleHint.go) there are two related files: [flat allow rules without hints](pkg/synthesis/tests_expected_output/abstract_models/ExampleHintsDisjoint_NoHint_NoHint.txt)
+contains the flat allow rules when executed without hints; and [flat allow rules with hints](pkg/synthesis/tests_expected_output/abstract_models/ExampleHintsDisjoint.txt) 
+contains the flat allow when executed with `--disjoint-hint sly, huf, gry, dum1, dum2` 
+2. Apply various optimization to simplify the resulting rules and to delete redundant rules; the more accurate hints the
+tool is provided, the more concise and readable rules it will synthesize.  
+
+### The policy preserving challenge
+The synthesis maintains the original semantic intent of the rules
+and not just generates a set of rules that preserves the connectivity between VMs given the current state of the configuration.
+For example:
+* When a `frontend` `VM` is added it should be granted the policies defined for the `frontend`
+* A DFW rule that uses an NSX group with no VMs at the moment of analysis and synthesis,
+will still be relevant to maintain in the conversion to network policies.
+
+To preserve the original intent of the policy, the synthesized policy prioritizes referencing non-ephemeral features.
+E.g., it prefers referencing`frontend` label instead of referencing `VMs'` names. `VM`s may be deleted and added, while
+the `frontend` label is always granted to any relevant `VM`.  
+Specifically,  given a rule with `src`defined as group `aaa` which is defined as `tag = backend and tag != DB`,
+the synthesized policy will reference the labels corresponding to the `backend` and `DB`  values.
+
+`todo: add tag -> labeling machanism` 
+
+## Currently supported
+Currently, the tool supports groups defined by expressions over tags; `nested NSX expression` are not yet supported.
+If a group is defined by an expression that we do not yet support, then the synthesized policy will refer just to the group, 
+and the relevant *VM*s will be granted labels of this group.
+
+For example, the expression `tag = backend and tag != DB` is supported, while the nested expression
+`(tag = backend and tag != DB) or (tag = research) ` is not supported. For a group defined over the former expression,
+the synthesis will reference labels corresponding to the above tags' values, while for a group defined over the latter 
+expression, the synthesis will reference a label corresponding to the group.
+
+## Output
+### Synthesized k8s resource
+`k8s_resources` folder under the folder specified in `synthesis-dump-dir` contains the following files:
+* **pods.yaml** the list pods (as placeholder for VMs resources for now) with the relevant labels of each pod.
+The labels are added based on original VMs' tags and groups in NSX env. 
+* **policies.yaml** the k8s policies
+
+The combination of the policies and the pods' labels:
+1. Satisfies the snapshot of the connectivity at the time of the synthesis
+2. Preserve the policy's intent, as expressed e.g. by *tags*, for future changes 
+(e.g. adding a `VM` or changing its functionality and thus its labeling)
+
+
+<a id="limitation"></a>
+#### limitations
+There are differences in the expression power between `NSX DFW` to `Kubernetes Network Policies`; e.g. `ICMP protocols`
+support depends on the network plugin and is not supported by our synthesis. 
+
+## Debugging
+The synthesize process is a complex one. Along it, in order to have the intent preserving synthesis as explained above,
+we use a *symbolic* representation of the *rules*: each *symbolic rule* is a `priority`, an `action`, a
+`src`, a `dst` and a `connection`; The priority in a natural number; the action is `allow/deny/pass`; The `src` and the `dst` are `Conjunctions` of simple expressions 
+(equal/not equal) over e.g. `tags`;  the `connection` is a protocol and potentially `src/dst` min and max ports.
+
+The synthesis dump directory (specified in `synthesis-dump-dir`) contains (among others) the following files:
+* under subdirectory **debug_dir**
+  * **config.txt** Contains the NSX config as being read by the tool; this includes `VMs`, `groups` and `firewall rules`.
+  * **pre_processing.txt** Contains the translation of the firewall rule into `symbolic rules` ; e.g., if a specific 
+src is a group which is an expression over tags, then this file will have this rule's `src` defined over tags.
+  * **abstract_model.txt** The tool translates the *allow/deny/pass* rules from `pre_processing.txt` into an abstract model that
+    contains the rules to be syntactically translated to `k8s policies`. If `synthesize-admin-policies`  is off then all rules must
+    be `flat allow rules`, and so all the rules in the abstract model are non-prioritized with action `allow`;
+    otherwise rules originating from the `environment` category are translated to admin policies with
+    a priority and an allow/deny/pass action; rules that originate from the other categories are flat allow rules.
+  `abstract_model.txt` contains these rules and a list of the groups, each groups with the expression that defined
+  it and the snapshot of the *VMs* in the group. 
+
+ The following log files contain warning messages and various debug printing of the different stages
+ of the synthesis, as following:
+
+  * **runPreprocessing.log** Log of the stage in which the NSX rules are translated to symbolic rules.
+  * **runConvertToAbstract.log** Log of the stage in which the symbolic rules from the preprocessing stage 
+are translated to the abstract model's rules.  
+  * **runK8SSynthesis.log** Log of the stage in which the k8s yaml pods and polices files are synthesized from 
+the abstract model.

cmd/commands.go

@@ -50,15 +50,16 @@ const (
 	outputFileHelp              = "file path to store analysis results"
 	explainHelp                 = "flag to explain connectivity output with rules explanations per allowed/denied connections (default false)"
 	synthesisDumpDirHelp        = "run synthesis; specify directory path to store k8s synthesis results"
-	synthesizeAdminPoliciesHelp = "include admin network policies in policy synthesis (default false)"
-	outputFormatHelp            = "output format; must be one of "
-	outputFilterFlagHelp        = "filter the analysis results by vm names, can specify more than one (example: \"vm1,vm2\")"
-	quietHelp                   = "flag to run quietly, report only severe errors and result (default false)"
-	verboseHelp                 = "flag to run with more informative messages printed to log (default false)"
-	colorHelp                   = "flag to enable color output (default false)"
-	createDNSPolicyHelp         = "flag to create a policy allowing access to target env dns pod"
-	synthHelp                   = "flag to run synthesis, even if synthesis-dump-dir is not specified"
-	disjointHintsHelp           = "comma separated list of NSX groups/tags that are always disjoint in their VM members," +
+	synthesizeAdminPoliciesHelp = "synthesize category environment into admin network policies" +
+		" (which included deny, pass and priority) (default false)"
+	outputFormatHelp     = "output format; must be one of "
+	outputFilterFlagHelp = "filter the analysis results by vm names, can specify more than one (example: \"vm1,vm2\")"
+	quietHelp            = "flag to run quietly, report only severe errors and result (default false)"
+	verboseHelp          = "flag to run with more informative messages printed to log (default false)"
+	colorHelp            = "flag to enable color output (default false)"
+	createDNSPolicyHelp  = "flag to create a policy allowing access to target env dns pod"
+	synthHelp            = "flag to run synthesis, even if synthesis-dump-dir is not specified"
+	disjointHintsHelp    = "comma separated list of NSX groups/tags that are always disjoint in their VM members," +
 		" needed for an effective and sound synthesis process, can specify more than one hint" +
 		" (example: \"--" + disjointHintsFlag + " frontend,backend --" + disjointHintsFlag + " app,web,db\")"
 )

pkg/data/exampleHint.go

@@ -0,0 +1,64 @@
+package data
+
+// ExampleHintsDisjoint for testing the hint of disjoint groups/tags and relevant optimization
+// Dumbledore1 can talk to all but Slytherin
+// Dumbledore2 can talk to all but Gryffindor
+var ExampleHintsDisjoint = Example{
+	Name: "ExampleHintsDisjoint",
+	VMs:  []string{sly, huf, gry, dum1, dum2},
+	GroupsByVMs: map[string][]string{
+		sly:  {sly},
+		huf:  {huf},
+		gry:  {gry},
+		dum1: {dum1},
+		dum2: {dum2},
+	},
+	Policies: []Category{
+		{
+			Name:         "From-Dumbledore-connection",
+			CategoryType: application,
+			Rules: []Rule{
+				{
+					Name:     "Dumb1-Not-Sly",
+					ID:       newRuleID,
+					Source:   dum1,
+					Dest:     sly,
+					Services: []string{anyStr},
+					Action:   Drop,
+				},
+				{
+					Name:     "Dumb2-Not-Gryf",
+					ID:       newRuleID + 1,
+					Source:   dum2,
+					Dest:     gry,
+					Services: []string{anyStr},
+					Action:   Drop,
+				},
+				{
+					Name:     "Dumb1-To-All",
+					ID:       newRuleID + 2,
+					Source:   dum1,
+					Dest:     anyStr,
+					Services: []string{anyStr},
+					Action:   Allow,
+				},
+				{
+					Name:     "Dumb2-To-All",
+					ID:       newRuleID + 3,
+					Source:   dum2,
+					Dest:     anyStr,
+					Services: []string{anyStr},
+					Action:   Allow,
+				},
+			},
+		},
+		{
+			Name:         defaultL3,
+			CategoryType: application,
+			Rules: []Rule{
+				DefaultDenyRule(denyRuleIDEnv),
+			},
+		},
+	},
+	DisjointGroupsTags: disjointHouses2Dum,
+}

pkg/data/exampleHogwarts.go

@@ -0,0 +1,164 @@
+package data
+
+import (
+	"github.com/np-guard/models/pkg/netp"
+	"github.com/np-guard/models/pkg/netset"
+	nsx "github.com/np-guard/vmware-analyzer/pkg/analyzer/generated"
+)
+
+/*
+ExampleHogwarts with macro and micro segmentation
+
+Slytherin House {Vms : S1, S2, S3}
+Hufflepuff House {VMs: H1, H2, H3}
+Gryffindor House {VMs: G1, G2, G3}
+Dumbledore {VMs: D1, D2}
+	 Web {S1, H1, G1}
+	 APP {S2, H2, G2}
+	 DB  {S3, H3, G3}
+
+
+Macro Segmentation
+- Houses (tenants / apps) must not communicate with each other
+- each house must be able to communicate to Dumbledore (shared services)
+- [Dumbledore must be able to communicate only to Gryffindor house (ML server / other special use case server, etc )] removed
+- Within each house (tenants/apps) tiers must be able to communicate with each other
+
+Macro Segmentation - the starting point to the land of zero trust
+
+micro segmentation
+- within each house (tenants/apps) tiers must have granular firewall policies
+	- anyone can access WEB servers
+	- only Web server can access App server over a whitelisted port
+	- only App Server can access DB Server over a whitelisted port
+
+
+*/
+
+var ExampleHogwarts = Example{
+	Name: "ExampleHogwarts",
+	VMs: []string{slyWeb, slyApp, slyDB, hufWeb, hufApp, hufDB,
+		gryWeb, gryApp, gryDB, dum1, dum2},
+	GroupsByVMs: map[string][]string{
+		sly: {slyWeb, slyApp, slyDB},
+		huf: {hufWeb, hufApp, hufDB},
+		gry: {gryWeb, gryApp, gryDB},
+		dum: {dum1, dum2},
+		web: {slyWeb, gryWeb, hufWeb},
+		app: {slyApp, gryApp, hufApp},
+		db:  {slyDB, gryDB, hufDB},
+	},
+	Policies: []Category{
+		{
+			Name:         "Gryffindor-to-Gryffindor-allow",
+			CategoryType: environment,
+			Rules: []Rule{
+				{
+					Name:   "allow-Gryffindor-to-Gryffindor",
+					ID:     10218,
+					Source: gry,
+					Dest:   gry,
+					Action: JumpToApp,
+					Conn:   netset.AllTCPTransport(),
+				},
+			},
+		},
+		{
+			Name:         "Hufflepuff-to-Hufflepuff-allow",
+			CategoryType: environment,
+			Rules: []Rule{
+				{
+					Name:   "allow-Hufflepuff-to-Hufflepuff",
+					ID:     10219,
+					Source: huf,
+					Dest:   huf,
+					Action: JumpToApp,
+					//nolint:mnd // these are the port numbers for the test
+					Conn:      netset.NewUDPTransport(netp.MinPort, netp.MinPort, 300, 320),
+					Direction: string(nsx.RuleDirectionIN),
+				},
+			},
+		},
+		{
+			Name:         "Slytherin-to-Slytherin-allow",
+			CategoryType: environment,
+			Rules: []Rule{
+				{
+					Name:     "allow-Slytherin-to-Slytherin",
+					ID:       10220,
+					Source:   sly,
+					Dest:     sly,
+					Services: []string{anyStr},
+					Action:   JumpToApp,
+				},
+			},
+		},
+		{
+			Name:         "Dumbledore-connection",
+			CategoryType: environment,
+			Rules: []Rule{
+				{
+					Name:     "allow-Dumbledore-to-all",
+					ID:       10221,
+					Source:   dum,
+					Dest:     gry,
+					Services: []string{anyStr},
+					Action:   JumpToApp,
+				},
+				{
+					Name:     "default-deny-env",
+					ID:       10300,
+					Source:   anyStr,
+					Dest:     anyStr,
+					Services: []string{anyStr},
+					Action:   Drop,
+				},
+			},
+		},
+
+		{
+			Name:         "Intra-App-Policy",
+			CategoryType: application,
+			Rules: []Rule{
+				{
+					Name:     "Client-Access",
+					ID:       10400,
+					Source:   anyStr,
+					Dest:     web,
+					Services: []string{anyStr},
+					Action:   Allow,
+				},
+				{
+					Name:     "Web-To-App-Access",
+					ID:       10401,
+					Source:   web,
+					Dest:     app,
+					Services: []string{anyStr},
+					Action:   Allow,
+				},
+				{
+					Name:     "App-To-DB-Access",
+					ID:       10405,
+					Source:   app,
+					Dest:     db,
+					Services: []string{anyStr},
+					Action:   Allow,
+				},
+			},
+		},
+		{
+			Name:         defaultL3,
+			CategoryType: application,
+			Rules: []Rule{
+				DefaultDenyRule(denyRuleIDEnv),
+			},
+		},
+	},
+	DisjointGroupsTags: [][]string{
+		{sly, huf, gry, dum},
+		{web, app, db},
+		{web, dum},
+		{app, dum},
+		{db, dum},
+	},
+}