discovery+graph: track job set dependencies in vb #9241

Crypt-iQ · 2024-11-01T19:43:26Z

This PR changes the ValidationBarrier to track abstract job dependencies. This just means that every time a child job comes in (i.e. channel update or node announcement), we track the set of possible parent jobs that are related to it (channel announcement(s)) that have registered via InitJobDependencies. The goroutines containing the child jobs will then wait to be notified every time one of their parent jobs completes. From the child job's POV, this just works as ref-counting except that you're only counting the parent jobs you're interested in.

With this, we can now extend the ValidationBarrier to track any sort of abstract job that requires both concurrency and waiting for another job to finish. It also makes it possible in a future PR to very easily make node announcements depend on channel announcements. See the commit messages for more details.

TODO:

Introduce a test to simulate goroutines randomly calling into the ValidationBarrier and ensure that all child jobs finish after their related parent jobs.
If time permits, write a fuzz test for the above as well.
Continue testing on mainnet

coderabbitai · 2024-11-01T19:43:32Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR. (Experiment)
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

saubyk · 2024-11-07T21:13:55Z

cc: @gijswijs for review

graph/builder.go

ellemouton · 2024-11-11T12:46:52Z

graph/validation_barrier.go

+func (v *ValidationBarrier) FetchJobSlot() {
+	// We'll wait for either a new slot to become open, or for the quit


since the builder code is called by the gossiper code which itself also uses a semaphore - why isnt that inheritance enough?

It looks like it is, good catch

fn/set.go

ellemouton · 2024-11-11T12:50:06Z

graph/validation_barrier.go

-func (v *ValidationBarrier) SignalDependants(job interface{}, allow bool) {
+// SignalDependents signals to any child jobs that this parent job has
+// finished.
+func (v *ValidationBarrier) SignalDependents(job interface{}, id JobID) error {


haha sneaky change from British spelling to American 😝

this has been driving me crazy for years

graph/validation_barrier.go

ellemouton · 2024-11-11T12:53:54Z

discovery/gossiper.go

+			"JobID=%v", spew.Sdump(nMsg.msg), jobID)
+	}


should we not handle the error similarly to how it is handled for WaitForParents above? (including returning after handling?)

graph/validation_barrier.go

ellemouton · 2024-11-11T13:02:12Z

graph/validation_barrier.go

+		info.activeParentJobIDs.Add(annJobID)
+	} else {


style nit: return after info.activeParentJobIDs.Add(annJobID) and remove & unindent the else block

graph/validation_barrier.go

ellemouton · 2024-11-11T13:04:58Z

graph/validation_barrier.go

-		signals, ok = v.chanEdgeDependencies[msg.ShortChannelID]
+		annID = msg.ShortChannelID
+
+		// TODO: If ok is false, we have serious issues.


throw the error? (if it is really impossible then panic)

Despite the many legitimate uses of panics, they have been rejected every time I have tried to use them (even for the provably impossible scenario). I believe that a critical log is the next best thing.

returning error now

ProofOfKeags

I'm finding the distinction between parent and child jobs here both confusing and unnecessary. What we have here is a dependency graph of undifferentiated jobIDs. once all of the dependencies have run we can run. once we run we want to signal all of our dependents. We should be able to accomplish this with a single removeJob that does this index cleanup and dependent signaling.

The main difficulty I'm noticing in this PR is that we have multiple IDs that we want to be able to map to JobIDs from disjoint domains. My recommendation here is to make the core algebra of this component undifferentiated and then have auxilliary mappings that help recover the relevant JobID from the other unique protocol identifiers.

fn/set.go

ProofOfKeags · 2024-11-11T20:56:48Z

graph/validation_barrier.go

+	// length and entries therefore cannot hash to the same keys.
+	// NOTE: IF OTHER TYPES OF KEYS ARE STORED, CHECK THAT COLLISION WON'T
+	// OCCUR.
+	jobInfoMap map[any]*jobInfo


I think we ought to use an explicit closed union (via an interface) in the key here. any is a disaster waiting to happen.

I was trying to figure out how to do this, but couldn't figure out how? map won't accept an interface as a key because interfaces aren't comparable

ProofOfKeags · 2024-11-11T21:01:10Z

graph/validation_barrier.go

+// should complete after another) for the (childJobID, annID) tuple. This must
+// only be called from InitJobDependencies.


why not just define this as a local function inside that scope to enforce that it is only referenceable there?

changed, this was only introduced to not have to deal with line-length issues

ProofOfKeags · 2024-11-11T21:04:33Z

graph/validation_barrier.go

+		// Copy over the parent job IDs at this moment for this annID.
+		// This job must be processed AFTER these parent IDs.
+		parentJobs := info.activeParentJobIDs.Union(fn.NewSet[JobID]())


I think this reveals a need for a set copying method.

ProofOfKeags · 2024-11-11T21:15:38Z

graph/validation_barrier.go

-		signals, ok = v.chanEdgeDependencies[msg.ShortChannelID]
+		annID = msg.ShortChannelID
+
+		// TODO: If ok is false, we have serious issues.


Despite the many legitimate uses of panics, they have been rejected every time I have tried to use them (even for the provably impossible scenario). I believe that a critical log is the next best thing.

ProofOfKeags · 2024-11-11T21:19:22Z

graph/validation_barrier.go

+		annID = msg.ShortChannelID
+
+		// TODO: If ok is false, we have serious issues.
+		parentJobIDs, ok = v.jobDependencies[childJobID]


why does this read not need a mutex lock?

it's locked above no?

graph/validation_barrier.go

ProofOfKeags · 2024-11-11T21:29:22Z

graph/validation_barrier.go

+// and cleans up its job dependency mappings. This MUST be called from
+// SignalDependents.
+// NOTE: MUST be called with the mutex held.
+func (v *ValidationBarrier) removeChildJob(annID any, childJobID JobID) {


ProofOfKeags · 2024-11-11T21:32:27Z

graph/validation_barrier.go

+		// We don't want to block when sending out the signal.
+		select {
+		case notifyChan <- struct{}{}:
+		default:
+		}


Are we ok with swallowing the signal instead? Seems like this could case jobs to never be run, particularly if lastJob is true

I'm not sure what you mean here. 1. How would jobs not be run in the current scenario? 2. What does swallowing the signal look like here?

ProofOfKeags · 2024-11-11T21:39:21Z

graph/validation_barrier.go

 	case *lnwire.NodeAnnouncement:
-		delete(v.nodeAnnDependencies, route.Vertex(msg.NodeID))
+		// Remove child job info.
+		v.removeChildJob(route.Vertex(msg.NodeID), id)


I don't think we need two distinct removal functions for parent and child.

changed, using a bool now since they have different reuqirements

saubyk · 2024-11-19T16:24:19Z

cc: @gijswijs for review

lightninglabs-deploy · 2024-12-03T17:41:30Z

@Crypt-iQ, remember to re-request review from reviewers when ready

Crypt-iQ · 2024-12-06T23:42:37Z

Outstanding things to do:

change from any to a union for map keys
update fn package to have a set copy method

ellemouton · 2024-12-09T16:00:18Z

@Crypt-iQ - is this ready for re-review given the itest failures?

Crypt-iQ · 2024-12-09T17:47:18Z

@Crypt-iQ - is this ready for re-review given the itest failures?

Sorry no, it seems like I broke something

This omits calls to InitJobDependencies, SignalDependants, and WaitForDependants. These changes have been made here because the router / builder code does not actually need job dependency management. Calls to the builder code (i.e. AddNode, AddEdge, UpdateEdge) are all blocking in the gossiper. This, combined with the fact that child jobs are run after parent jobs in the gossiper, means that the calls to the router will happen in the proper dependency order.

Crypt-iQ · 2024-12-09T20:59:06Z

Ready for review now, had an issue with any being a key so I think the type change mentioned above is a blocker.

ellemouton · 2024-12-10T12:17:30Z

@Crypt-iQ - defs ready given all the failures?

This commit does two things: - removes the concept of allow / deny. Having this in place was a minor optimization and removing it makes the solution simpler. - changes the job dependency tracking to track sets of abstact parent jobs rather than individual parent jobs. As a note, the purpose of the ValidationBarrier is that it allows us to launch gossip validation jobs in goroutines while still ensuring that the validation order of these goroutines is adhered to when it comes to validating ChannelAnnouncement _before_ ChannelUpdate and _before_ NodeAnnouncement.

Crypt-iQ · 2024-12-10T15:16:46Z

@Crypt-iQ - defs ready given all the failures?

now it is, the fn change broke it

Crypt-iQ force-pushed the fix_vb branch 3 times, most recently from 7acf321 to fc00572 Compare November 4, 2024 18:56

Crypt-iQ changed the title ~~[DRAFT] graph: fix validation barrier~~ [DRAFT] discovery+graph: track job set dependencies in ValidationBarrier Nov 4, 2024

Crypt-iQ changed the title ~~[DRAFT] discovery+graph: track job set dependencies in ValidationBarrier~~ [DRAFT] discovery+graph: track job set dependencies in vb Nov 4, 2024

Crypt-iQ force-pushed the fix_vb branch 3 times, most recently from fc72083 to 7d95cd2 Compare November 6, 2024 18:19

Crypt-iQ marked this pull request as ready for review November 6, 2024 18:46

Crypt-iQ changed the title ~~[DRAFT] discovery+graph: track job set dependencies in vb~~ discovery+graph: track job set dependencies in vb Nov 6, 2024

Crypt-iQ force-pushed the fix_vb branch from 7d95cd2 to c5e584a Compare November 6, 2024 19:36

Crypt-iQ added graph refactoring labels Nov 6, 2024

saubyk assigned Crypt-iQ Nov 7, 2024

saubyk added this to the v0.19.0 milestone Nov 7, 2024

saubyk requested review from ellemouton and ProofOfKeags November 7, 2024 15:28

ellemouton reviewed Nov 11, 2024

View reviewed changes

ProofOfKeags requested changes Nov 11, 2024

View reviewed changes

gijswijs self-requested a review December 3, 2024 15:57

Crypt-iQ force-pushed the fix_vb branch from c5e584a to c7178ff Compare December 6, 2024 23:40

Crypt-iQ requested review from ProofOfKeags and ellemouton December 6, 2024 23:42

Crypt-iQ force-pushed the fix_vb branch from c7178ff to a7ce2e7 Compare December 9, 2024 20:58

Crypt-iQ added 2 commits December 10, 2024 10:16

release-notes: update for 0.19.0

2143b9c

Crypt-iQ force-pushed the fix_vb branch from a7ce2e7 to 2143b9c Compare December 10, 2024 15:16

		func (v *ValidationBarrier) FetchJobSlot() {
		// We'll wait for either a new slot to become open, or for the quit

		// should complete after another) for the (childJobID, annID) tuple. This must
		// only be called from InitJobDependencies.

discovery+graph: track job set dependencies in vb #9241

Are you sure you want to change the base?

discovery+graph: track job set dependencies in vb #9241

Conversation

Crypt-iQ commented Nov 1, 2024 • edited Loading

coderabbitai bot commented Nov 1, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

saubyk commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ProofOfKeags left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saubyk commented Nov 19, 2024

lightninglabs-deploy commented Dec 3, 2024

Crypt-iQ commented Dec 6, 2024

ellemouton commented Dec 9, 2024

Crypt-iQ commented Dec 9, 2024

Crypt-iQ commented Dec 9, 2024

ellemouton commented Dec 10, 2024

Crypt-iQ commented Dec 10, 2024

Crypt-iQ commented Nov 1, 2024 •

edited

Loading

coderabbitai bot commented Nov 1, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)