Skip to content

Commit

Permalink
roachprod: handle nodeIPPorts map update atomicity
Browse files Browse the repository at this point in the history
when creating cluster config service discovery
for each node happens concurrently, and the
information is added to a common map. If the
map getting updated at the same time, roachprod
panics for concurrent update.
the fix is to have a mutex to protect the same.

Fixes: cockroachdb#124400
Epic: none
  • Loading branch information
nameisbhaskar committed May 20, 2024
1 parent 93ad913 commit 36ac171
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions pkg/roachprod/roachprod.go
Original file line number Diff line number Diff line change
Expand Up @@ -795,6 +795,7 @@ func UpdateTargets(
// updatePrometheusTargets updates the prometheus instance cluster config. Any error is logged and ignored.
func updatePrometheusTargets(ctx context.Context, l *logger.Logger, c *install.SyncedCluster) {
nodeIPPorts := make(map[int]string)
nodeIPPortsMutex := syncutil.RWMutex{}
var wg sync.WaitGroup
for _, node := range c.Nodes {
if c.VMs[node-1].Provider == gce.ProviderName {
Expand All @@ -808,7 +809,10 @@ func updatePrometheusTargets(ctx context.Context, l *logger.Logger, c *install.S
return
}
nodeInfo := fmt.Sprintf("%s:%d", v.PublicIP, desc.Port)
nodeIPPortsMutex.Lock()
// ensure atomicity in map update
nodeIPPorts[index] = nodeInfo
nodeIPPortsMutex.Unlock()
}(int(node), c.VMs[node-1])
}
}
Expand Down

0 comments on commit 36ac171

Please sign in to comment.