lightning-liquidity: Introduce EventQueue notifier and wake BP for message processing

[tnull]

As discussed/mentioned in #3436, this is a prefactor to upcoming persistence/LSPS1 service/LSPS5 work.

Here, we introduce a EventQueueNotifierGuard type reducing the potential of lock contention arising from calling EventQueue::enqueue while holding peer state locks.

We furthermore wake the background processor to trigger message processing after we enqueue new messages.

[tnull]

Rebased after #3533 landed, should be ready for review (cc @TheBlueMatt)

[TheBlueMatt]

Why end up going with a direct function callback instead of the lightning Future logic (and integrating into the BP to drive the calls for LDK users and letting other users drive via an async task)?

[tnull]

Why end up going with a direct function callback instead of the lightning Future logic (and integrating into the BP to drive the calls for LDK users and letting other users drive via an async task)?

Mhh, that might be an alternative approach, yes. I wonder if it would make sense to add this more generally to the CustomMessageHandler interface as every CMH will likely need to call back into PeerHandler to trigger message processing?

[tnull]

Now switched to having the BP woken to trigger message processing. Had to rebase on top of #3546 to avoid silent rebase conflicts while adding a lightning-liquidity dependency on lightning-background-processor.

Should be generally good for review, but I'm currently still fighting some issues from RefCell not being Sync + Send in --no-default-features --features futures.

[codecov]

Codecov Report

Attention: Patch coverage is 70.39474% with 90 lines in your changes missing coverage. Please review.

Project coverage is 88.48%. Comparing base (2c3f11d) to head (eb3ea83).

Files with missing lines	Patch %	Lines
lightning-liquidity/src/lsps2/service.rs	59.15%	44 Missing and 14 partials ⚠️
lightning-background-processor/src/lib.rs	84.09%	14 Missing ⚠️
lightning-liquidity/src/lsps1/client.rs	0.00%	12 Missing ⚠️
lightning-liquidity/src/events.rs	85.71%	2 Missing and 2 partials ⚠️
lightning-liquidity/src/lsps0/client.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3509      +/-   ##
==========================================
- Coverage   88.53%   88.48%   -0.05%     
==========================================
  Files         149      149              
  Lines      114985   115039      +54     
  Branches   114985   115039      +54     
==========================================
- Hits       101803   101795       -8     
- Misses      10693    10750      +57     
- Partials     2489     2494       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheBlueMatt]

Would be nice to get some additional context in some of the commit messages, especially "Relax Sync + Send bounds on BackgroundProcessor where possible" which really need an explainer of why we think its a good idea to do so.

[TheBlueMatt]

Not sure I understand the point of this logic here - we've pushed a new htlc onto payment_queue, but we're erroring here which causes the HTLC to be failed-back. The comment seems to imply we want to revert to the old queue state (which I think would be right?), but that isn't what we're doing.

[tnull]

Hmm, maybe comment could be more specific, but the idea here is to preserve the previous behavior, which keeps all HTLCs in the queue until we get HTLCHandlingFailed, at which point we need to clean up the queue, no?

[TheBlueMatt]

Right, so two things:
(a) I think there's an existing bug here - LSPS2ServiceHandler::htlc_intercepted calls through to here, and when it gets an Err it calls channelmanager....fail_intercepted_htlc, but the code here keeps the HTLC listed as pending.
(b) doing the take at the top looks pretty brittle, and isn't required. eg the following patch compiles:

$ git diff -U2
diff --git a/lightning-liquidity/src/lsps2/service.rs b/lightning-liquidity/src/lsps2/service.rs
index 69acc5ba6..f6ba6aec7 100644
--- a/lightning-liquidity/src/lsps2/service.rs
+++ b/lightning-liquidity/src/lsps2/service.rs
@@ -142,6 +142,5 @@ impl OutboundJITChannelState {
                let new_state;
                let res = match self {
-                       OutboundJITChannelState::PendingInitialPayment { payment_queue: old_payment_queue } => {
-                               let mut payment_queue = core::mem::take(old_payment_queue);
+                       OutboundJITChannelState::PendingInitialPayment { payment_queue } => {
                                let (total_expected_outbound_amount_msat, num_htlcs) = payment_queue.add_htlc(htlc);

@@ -152,6 +151,4 @@ impl OutboundJITChannelState {
                                                debug_assert_eq!(num_htlcs, 1);
                                                if num_htlcs != 1 {
-                                                       // Revert the queue before error'ing
-                                                       core::mem::swap(old_payment_queue, &mut payment_queue);
                                                        return Err(ChannelStateError(
                                                                "Paying via multiple HTLCs is disallowed in \"no-MPP+var-invoice\" mode.".to_string()
@@ -164,6 +161,4 @@ impl OutboundJITChannelState {
                                        || expected_payment_size_msat > opening_fee_params.max_payment_size_msat
                                {
-                                       // Revert the queue before error'ing
-                                       core::mem::swap(old_payment_queue, &mut payment_queue);
                                        return Err(ChannelStateError(
                                                        format!("Payment size violates our limits: expected_payment_size_msat = {}, min_payment_size_msat = {}, max_payment_size_msat = {}",
@@ -181,6 +176,4 @@ impl OutboundJITChannelState {
                                        opening_fee
                                } else {
-                                       // Revert the queue before error'ing
-                                       core::mem::swap(old_payment_queue, &mut payment_queue);
                                        return Err(ChannelStateError(
                                                        format!("Could not compute valid opening fee with min_fee_msat = {}, proportional = {}, and expected_payment_size_msat = {}",
@@ -199,5 +192,5 @@ impl OutboundJITChannelState {
                                {
                                        new_state = OutboundJITChannelState::PendingChannelOpen {
-                                               payment_queue,
+                                               payment_queue: core::mem::take(payment_queue),
                                                opening_fee_msat,
                                        };
@@ -210,9 +203,7 @@ impl OutboundJITChannelState {
                                        if mpp_mode {
                                                new_state =
-                                                       OutboundJITChannelState::PendingInitialPayment { payment_queue };
+                                                       OutboundJITChannelState::PendingInitialPayment { payment_queue: core::mem::take(payment_queue) };
                                                Ok(None)
                                        } else {
-                                               // Revert the queue before error'ing
-                                               core::mem::swap(old_payment_queue, &mut payment_queue);
                                                return Err(ChannelStateError(
                                                        "Intercepted HTLC is too small to pay opening fee".to_string(),

[tnull]

(a) I think there's an existing bug here - LSPS2ServiceHandler::htlc_intercepted calls through to here, and when it gets an Err it calls channelmanager....fail_intercepted_htlc, but the code here keeps the HTLC listed as pending.

Well, we def. need to keep the payment as pending, question is when we'd remove the HTLC. My thinking was that it would always happen as part of handling HTLCHandlingFailed, but IIUC, we need #3551 for this to work reliably in all cases.

(b) doing the take at the top looks pretty brittle, and isn't required. eg the following patch compiles:

I think I considered this but somehow arrived at the opinion this wouldn't work (maybe something to do with persistence, as this commit came from that draft branch). I however can't recall why I thought that and indeed it seems the above approach should work just as well as being cleaner. Now added in a fixup.

[TheBlueMatt]

Well, we def. need to keep the payment as pending, question is when we'd remove the HTLC. My thinking was that it would always happen as part of handling HTLCHandlingFailed, but IIUC, we need #3551 for this to work reliably in all cases.

Ah, okay, I wasn't considering the fail_intercepted_htlc generated an HTLCHandlingFailed and we handled it that way....that said I'm not really convinced that's the right approach - that leaves a race where our state doesn't match the HTLC state and we might think we've received enough HTLCs if another one comes in before we process the event.

[TheBlueMatt]

Might be easier to just push the HTLC before we move to the success block instead of pushing it then trying to revert before returning an Err...like this we'll end up adding a new error return in the future and forgetting to revert.

[tnull]

Yeah, although note that all following calculations are made based on the state with the HTLC already added. Pushing only in the success case would mean to refactor the logic to run the same calcualtions twice essentially.

[tnull]

Would be nice to get some additional context in some of the commit messages, especially "Relax Sync + Send bounds on BackgroundProcessor where possible" which really need an explainer of why we think its a good idea to do so.

Alright, now added some rationale there.