Create additional NEGs based on subnets in Node Topology CR. #2702

sawsa307 · 2024-10-18T20:18:05Z

Query Node Topology CR for the current set of NEGs in the cluster.
When ensureNetworkEndpointGroups(), ensure NEGs are properly provisioned in the non-default subnets as well.

sawsa307 · 2024-10-22T15:55:32Z

/assign @gauravkghildiyal

pkg/neg/types/types_test.go

pkg/utils/zonegetter/zone_getter.go

pkg/neg/syncers/transaction.go

pkg/neg/syncers/utils.go

pkg/neg/syncers/transaction.go

sawsa307 · 2024-10-31T22:38:00Z

/label tide/merge-method-squash

sawsa307

Just addressed all the comments, @gauravkghildiyal could you take another look when you have time? Thank you!

pkg/neg/types/types_test.go

pkg/utils/zonegetter/zone_getter.go

pkg/neg/syncers/transaction.go

pkg/neg/types/types_test.go

pkg/neg/syncers/transaction.go

pkg/utils/zonegetter/zone_getter.go

gauravkghildiyal · 2024-11-05T01:14:59Z

Also, is this TODO something that we plan on fixing independently? I guess as long as the MSC flags are disabled, this shouldn't cause any problems but please do confirm. Thanks!

sawsa307 · 2024-11-05T21:01:39Z

Also, is this TODO something that we plan on fixing independently? I guess as long as the MSC flags are disabled, this shouldn't cause any problems but please do confirm. Thanks!

Yes I'm planning to tackle those TODO in follow up PRs(like #2728), but if you would like, I can also include some of them in this PR as well.

pkg/utils/zonegetter/zone_getter.go

gauravkghildiyal · 2024-11-08T02:39:07Z

pkg/utils/zonegetter/zone_getter.go

+	nodeTopologyNotSynced := z.nodeTopologyInformer != nil && !z.nodeTopologyHasSynced()
+	if z.onlyIncludeDefaultSubnetNodes || z.nodeTopologyInformer == nil || nodeTopologyNotSynced {
+		if z.nodeTopologyInformer == nil || nodeTopologyNotSynced {
+			logger.Info("NodeTopologyInformer is not ready, fall back to use default subnet only")


I'd recommend

logger.Info("Falling back to only using default subnet", "z.onlyIncludeDefaultSubnetNodes", z.onlyIncludeDefaultSubnetNodes, <Rest of the variables influencing this decision>)

Updated, thanks!

It may be useful to know what was the reason why we are using "default subnet only", which means it's worth logging ALL the variables which went into that decision. So maybe we don't have an if !z.nodeTopologyHasSynced() { condition for the logs, and instead log:

logger.Info("Falling back to only using default subnet when listing nodes", "z.onlyIncludeDefaultSubnetNodes", z.onlyIncludeDefaultSubnetNodes, "z.nodeTopologyHasSynced()", nodeTopologyHasSynced)

On a related note, it plan on using or logging z.nodeTopologyHasSynced(), it may be useful to store it in a variable before. Reason being that since it's a function, it may return a different value between two independent invocations -- if that ever happens for some reason, our logs would simply end up confusing us.

* Remove NodeTopologyInformer from controller context hasSynced so it won't block controllers from starting up in case of the CRD or CR does not exist.

pkg/utils/zonegetter/zone_getter.go

pkg/neg/syncers/transaction.go

pkg/utils/zonegetter/zone_getter.go

sawsa307 · 2024-11-19T01:04:57Z

@gauravkghildiyal @swetharepakula This PR is ready for another round of review, please take a look, thank you!

gauravkghildiyal · 2024-11-19T04:04:12Z

Github is just the worst at showing newly added comments so sharing the links:

sawsa307 · 2024-11-19T21:21:26Z

Github is just the worst at showing newly added comments so sharing the links:

Create additional NEGs based on subnets in Node Topology CR. #2702 (comment)

Create additional NEGs based on subnets in Node Topology CR. #2702 (comment)

Addressed both comments, please take a look! @gauravkghildiyal

pkg/neg/syncers/transaction.go

gauravkghildiyal · 2024-11-25T21:58:02Z

pkg/utils/namer/interfaces.go

@@ -60,6 +62,9 @@ type BackendNamer interface {
 	// RXLBBackendName returns the Regional External Ingress backend name,
 	// based on the service namespace, name and target port.
 	RXLBBackendName(namespace, name string, port int32) string
+	// NonDefaultSubnetCustomNEG returns the gce neg name for custom NEGs created
+	// in non-default subnets.
+	NonDefaultSubnetCustomNEG(customNEGName, subnetName string) (string, error)


I believe this needs to be cleaned up (because the embedded NonDefaultSubnetNEGNamer already defines the same)

Updated, thanks!

gauravkghildiyal · 2024-11-25T22:06:19Z

pkg/neg/manager.go

@@ -228,6 +232,11 @@ func (manager *syncerManager) EnsureSyncers(namespace, name string, newPorts neg
 				&portInfo.NetworkInfo,
 				portInfo.L4LBType,
 			)
+			negNamer := manager.namer


nit: Maybe rename to nonDefaultSubnetNEGNamer?

Updated, thanks!

gauravkghildiyal · 2024-11-25T22:10:45Z

pkg/utils/namer/l4_namer.go

@@ -79,6 +80,15 @@ func (namer *L4Namer) L4NonDefaultSubnetNEG(namespace, name, subnetName string)
 	}, "-")
 }

+// NonDefaultSubnetCustomNEG returns the gce neg name in the non-default subnet
+// when the NEG name is a custom one.
+func (n *L4Namer) NonDefaultSubnetCustomNEG(customNEGName, subnetName string) (string, error) {


Let's add a note here that as of today, we don't support custom NEG names in L4 NEGs. Let's also Log a function call to this as an error.

Updated, thanks!

gauravkghildiyal · 2024-11-25T22:18:23Z

pkg/neg/syncers/transaction.go

@@ -883,6 +928,19 @@ func (s *transactionSyncer) computeEPSStaleness(endpointSlices []*discovery.Endp
 	}
 }

+// getNonDefaultSubnetName returns the name of the NEG based on the subnet name.
+func (s *transactionSyncer) getNonDefaultSubnetName(subnet string) (string, error) {
+	if s.customName {


Don't we still need to adjust the value stored in s.customName such that for L4 NEGs, this is always false?

* Query Node Topology CR for the current set of NEGs in the cluster. * When ensureNetworkEndpointGroups(), ensure NEGs are properly provisioned in the non-default subnets as well.

sawsa307

Addressed all the comments, thank you!

sawsa307 · 2024-11-26T21:54:32Z

pkg/neg/syncers/transaction.go

@@ -883,6 +928,19 @@ func (s *transactionSyncer) computeEPSStaleness(endpointSlices []*discovery.Endp
 	}
 }

+// getNonDefaultSubnetName returns the name of the NEG based on the subnet name.
+func (s *transactionSyncer) getNonDefaultSubnetName(subnet string) (string, error) {
+	if s.customName {


sawsa307 · 2024-11-26T22:08:53Z

pkg/utils/namer/interfaces.go

@@ -60,6 +62,9 @@ type BackendNamer interface {
 	// RXLBBackendName returns the Regional External Ingress backend name,
 	// based on the service namespace, name and target port.
 	RXLBBackendName(namespace, name string, port int32) string
+	// NonDefaultSubnetCustomNEG returns the gce neg name for custom NEGs created
+	// in non-default subnets.
+	NonDefaultSubnetCustomNEG(customNEGName, subnetName string) (string, error)


Updated, thanks!

sawsa307 · 2024-11-26T22:18:41Z

pkg/utils/namer/l4_namer.go

@@ -79,6 +80,15 @@ func (namer *L4Namer) L4NonDefaultSubnetNEG(namespace, name, subnetName string)
 	}, "-")
 }

+// NonDefaultSubnetCustomNEG returns the gce neg name in the non-default subnet
+// when the NEG name is a custom one.
+func (n *L4Namer) NonDefaultSubnetCustomNEG(customNEGName, subnetName string) (string, error) {


Updated, thanks!

sawsa307 · 2024-11-26T22:20:53Z

pkg/neg/manager.go

@@ -228,6 +232,11 @@ func (manager *syncerManager) EnsureSyncers(namespace, name string, newPorts neg
 				&portInfo.NetworkInfo,
 				portInfo.L4LBType,
 			)
+			negNamer := manager.namer


Updated, thanks!

sawsa307 · 2024-11-26T22:22:02Z

Did a squash over the previous fix-up to make it cleaner, thanks!

gauravkghildiyal · 2024-12-02T22:12:44Z

/lgtm

Thank you for the patience!

k8s-ci-robot · 2024-12-02T22:12:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gauravkghildiyal, sawsa307

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gauravkghildiyal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 18, 2024

k8s-ci-robot requested review from aojea and panslava October 18, 2024 20:18

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 18, 2024

sawsa307 marked this pull request as draft October 18, 2024 20:18

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 18, 2024

sawsa307 force-pushed the create-non-default-neg branch from 2f593bb to 29bc49a Compare October 22, 2024 06:19

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 22, 2024

sawsa307 marked this pull request as ready for review October 22, 2024 15:53

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 22, 2024

k8s-ci-robot assigned gauravkghildiyal Oct 22, 2024

sawsa307 force-pushed the create-non-default-neg branch 3 times, most recently from 644e229 to 264bd6c Compare October 23, 2024 20:44

gauravkghildiyal reviewed Oct 23, 2024

View reviewed changes

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 31, 2024

sawsa307 commented Oct 31, 2024

View reviewed changes

gauravkghildiyal reviewed Nov 5, 2024

View reviewed changes

pkg/neg/types/types_test.go Outdated Show resolved Hide resolved

pkg/neg/syncers/transaction.go Outdated Show resolved Hide resolved

pkg/neg/syncers/transaction.go Outdated Show resolved Hide resolved

pkg/utils/zonegetter/zone_getter.go Outdated Show resolved Hide resolved

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 5, 2024

sawsa307 force-pushed the create-non-default-neg branch 2 times, most recently from 4bc66aa to 6d44fe7 Compare November 7, 2024 03:50

gauravkghildiyal reviewed Nov 8, 2024

View reviewed changes

sawsa307 added 2 commits November 14, 2024 17:06

Update namer interface to include NonDefaultSubnetNEG

460f7d6

Exclude NodeTopologyInformer from hasSynced.

fd001e7

* Remove NodeTopologyInformer from controller context hasSynced so it won't block controllers from starting up in case of the CRD or CR does not exist.

Add NodeTopologyInformer to zoneGetter.

be3c372

sawsa307 force-pushed the create-non-default-neg branch from 6d44fe7 to 055d691 Compare November 14, 2024 18:10

swetharepakula reviewed Nov 16, 2024

View reviewed changes

pkg/utils/zonegetter/zone_getter.go Outdated Show resolved Hide resolved

sawsa307 requested review from gauravkghildiyal and swetharepakula November 19, 2024 01:04

swetharepakula reviewed Nov 22, 2024

View reviewed changes

pkg/neg/syncers/transaction.go Outdated Show resolved Hide resolved

gauravkghildiyal reviewed Nov 25, 2024

View reviewed changes

sawsa307 added 2 commits November 26, 2024 20:32

Fall back to only include default if nodeTopology CR isn't ready.

d4d4874

Create additional NEGs based on subnets in Node Topology CR.

013d611

* Query Node Topology CR for the current set of NEGs in the cluster. * When ensureNetworkEndpointGroups(), ensure NEGs are properly provisioned in the non-default subnets as well.

sawsa307 commented Nov 26, 2024

View reviewed changes

sawsa307 force-pushed the create-non-default-neg branch from 2121863 to e1410e0 Compare November 26, 2024 22:21

sawsa307 requested review from gauravkghildiyal and swetharepakula December 2, 2024 17:46

Use correct namer based on the NEG type.

157daa7

sawsa307 force-pushed the create-non-default-neg branch from 55732d4 to 157daa7 Compare December 2, 2024 22:11

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 2, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 2, 2024

k8s-ci-robot merged commit 3ba1a57 into kubernetes:master Dec 2, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create additional NEGs based on subnets in Node Topology CR. #2702

Create additional NEGs based on subnets in Node Topology CR. #2702

sawsa307 commented Oct 18, 2024

sawsa307 commented Oct 22, 2024

sawsa307 commented Oct 31, 2024

sawsa307 left a comment

gauravkghildiyal commented Nov 5, 2024

sawsa307 commented Nov 5, 2024

gauravkghildiyal Nov 8, 2024

sawsa307 Nov 19, 2024

gauravkghildiyal Nov 19, 2024

sawsa307 commented Nov 19, 2024

gauravkghildiyal commented Nov 19, 2024

sawsa307 commented Nov 19, 2024

gauravkghildiyal Nov 25, 2024

sawsa307 Nov 26, 2024

gauravkghildiyal Nov 25, 2024

sawsa307 Nov 26, 2024

gauravkghildiyal Nov 25, 2024

sawsa307 Nov 26, 2024

gauravkghildiyal Nov 25, 2024

sawsa307 Nov 26, 2024

sawsa307 left a comment

sawsa307 Nov 26, 2024

sawsa307 Nov 26, 2024

sawsa307 Nov 26, 2024

sawsa307 Nov 26, 2024

sawsa307 commented Nov 26, 2024

gauravkghildiyal commented Dec 2, 2024

k8s-ci-robot commented Dec 2, 2024

Create additional NEGs based on subnets in Node Topology CR. #2702

Create additional NEGs based on subnets in Node Topology CR. #2702

Conversation

sawsa307 commented Oct 18, 2024

sawsa307 commented Oct 22, 2024

sawsa307 commented Oct 31, 2024

sawsa307 left a comment

Choose a reason for hiding this comment

gauravkghildiyal commented Nov 5, 2024

sawsa307 commented Nov 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawsa307 commented Nov 19, 2024

gauravkghildiyal commented Nov 19, 2024

sawsa307 commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawsa307 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawsa307 commented Nov 26, 2024

gauravkghildiyal commented Dec 2, 2024

k8s-ci-robot commented Dec 2, 2024