routing: fix stuck inflight payments #8174

yyforyongyu · 2023-11-14T06:09:10Z

Replaces #8150, this PR attempts to fix #8146.

Timed out HTLCs are properly handle

The original assumption was, when the HTLC timed out, the attempt result wasn't sent back the payment lifecycle, thus causing the payment to be in stuck. However, this assumption is not true, as shown in the newly added itest - when an outgoing HTLC has timed out, the payment will be marked as failed. Moreover, if the outgoing HTLC is swept by the remote via the preimage path, the payment will be marked as succeeded. All these are expected behavior.

HTLC Attempt result is not saved

If there's already a result in db, GetAttemptResult will return it, so the result must not be found in db. Also notice that, since there's no result in db, the only line we can hit is:

lnd/htlcswitch/switch.go

Line 466 in e6fbaaf

nChan, err = s.networkResults.subscribeResult(attemptID)

as otherwise hitting this line will return the error ErrPaymentIDNotFound:

lnd/htlcswitch/switch.go

Line 456 in e6fbaaf

res, err := s.networkResults.getResult(attemptID)

And the method GetAttemptResult is blocking at line:

lnd/htlcswitch/switch.go

Line 456 in e6fbaaf

res, err := s.networkResults.getResult(attemptID)

On the other hand, these network results will be cleaned when ChannelRouter starts here:

lnd/routing/router.go

Line 657 in e6fbaaf

if err := r.cfg.Payer.CleanStore(toKeep); err != nil {

But this cleanup only applied to terminated payments. For inflight payments, we always keep the network results. This rules out the possibility that the HTLC attempt once had a result but was later removed.

This seems to leave us only one possibility - that the network result was never written to disk, causing an HTLC attempt waiting for its result forever, hence the payment stayed inflight, which leads to suspicion that this line is hit, thus the line go s.handleLocalResponse(packet) is skipped so the result is not saved:

lnd/htlcswitch/switch.go

Lines 1289 to 1293 in e02fd39

    
           // closeCircuit returns a nil circuit when a settle packet returns an 
        
           // ErrUnknownCircuit error upon the inner call to CloseCircuit. 
        
           if circuit == nil { 
        
           	return nil 
        
           }

This PR refactors handlePacketForward and makes sure the network result is always saved for locally-initiated packet.

Fixes,

saubyk · 2023-11-28T17:54:19Z

cc: @Roasbeef for approach ack

bitromortac

First pass done, looks really good already 🔥. I'll let this run on a node that has stuck inflight payments.

bitromortac · 2024-05-02T08:34:27Z

itest/list_on_test.go

@@ -562,4 +562,8 @@ var allTestCases = []*lntest.TestCase{
 		Name:     "payment succeeded htlc remote swept",
 		TestFunc: testPaymentSucceededHTLCRemoteSwept,
 	},
+	{
+		Name:     "send to route failed htlc timeout",
+		TestFunc: testSendToRouteFailHTLCTimeout,


Is the repeated itest needed? What's the main difference for SendPaymentV2?

Not sure I follow, but it's testing a different RPC? Tho SendToRouteV2 looks similar to SendPaymentV2, it's nonetheless an independent RPC that can diverge someday.

bitromortac · 2024-05-02T08:46:01Z

htlcswitch/switch.go

@@ -978,7 +978,7 @@ func (s *Switch) handleLocalResponse(pkt *htlcPacket) {
 		s.cfg.HtlcNotifier.NotifySettleEvent(key, htlc.PaymentPreimage,
 			eventType)

-	case *lnwire.UpdateFailHTLC:
+	case *lnwire.UpdateFailHTLC, *lnwire.UpdateFailMalformedHTLC:


is that change needed? I think for the same reason mentioned below, we don't expect UpdateFailMalformedHTLC here because of the conversion, right?

Good catch! Yeah based on the callsites I don't think this is needed, now removed.

htlcswitch/switch.go

bitromortac · 2024-05-02T09:56:34Z

routing/router.go

+			htlcs[a.AttemptID] = hash
+		}
+
+		payHash := payment.Info.PaymentIdentifier


nit: payHash can be used more to compress some lines below

routing/router.go

yyforyongyu · 2024-06-14T06:11:45Z

lightninglabs-deploy mute 336h

yyforyongyu · 2024-06-26T13:59:55Z

!lightninglabs-deploy mute 336h30m

yyforyongyu · 2024-07-10T15:04:12Z

!lightninglabs-deploy mute 336h30m

kilrau · 2024-07-20T10:18:39Z

Why are you guys muting this? This happens in prod and imo serious

saubyk · 2024-07-20T16:24:05Z

Why are you guys muting this? This happens in prod and imo serious

Hi @kilrau the notifications for reviews were muted as YY focused on other prs, but as you can see the pr is still tagged for 18.3 and will be picked up again soon

kilrau · 2024-07-20T18:51:23Z

Why are you guys muting this? This happens in prod and imo serious

Hi @kilrau the notifications for reviews were muted as YY focused on other prs, but as you can see the pr is still tagged for 18.3 and will be picked up again soon

Getting this into 0.18.3 sounds good 🙏

coderabbitai · 2024-07-22T13:10:44Z

Important

Review skipped

Auto reviews are limited to specific labels.

Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

ziggie1984 · 2024-08-01T22:31:22Z

Exchanged with the guys from the BOLTZ team, and it seems like they are suffering from another issue in regards of Inflight Payments which I am not sure this PR will solve.

There seems to be an edge case where we fail a payment ( exitresumePayment) although we have still Inflight HTLCs which need to be resolved. This has some side effects:

When a service tries to use trackpayment => this call remains pending although we already canceled all resultCollector goroutines, so it will pending until the next restart.

Moreover this payment will be inflight until the next restart in general, where we try to recollect all the Inflight HTLCs of a payment.

I think I might have found the culprit based on a logs of a payment attempt which got stuck:

So basically we are in the process of adding a new HTLC Attempt to the payment here:

Code parts taken from this PR.

attempt, err := p.registerAttempt(rt, ps.RemainingAmt)
		if err != nil {
			return exitWithErr(err)
		}

This will however return an error if the payment is failed already in
func (p *controlTower) RegisterAttempt(....

// Check if registering a new attempt is allowed.
		if err := payment.Registrable(); err != nil {
			return err
		}

hence the Payment Lifecycle will exit with (as seen in the picture):


	// If the payment is already failed, we won't allow adding more HTLCs.
	if m.State.PaymentFailed {
		return ErrPaymentPendingFailed
	}

I think the problem is For MPP when we hit the MPP Timeout error we would mark a payment prematurely as failed although HTLCs are still pending in handleSwitchErr

Because we would fail a payment as soon as the final hop reported the failure which is the case for the MPP Timeout for example.

I think my analysis describes whats happening in the above log picture, although it's not a full log but just a grep of the payment hash. However I think we really need to be carefull to not prematurely kill the payment lifecycle until all HTLCs reported back, or when killing the payment flow, we should at least also kill the trackpayment stream so that the subscriber is not waiting forever.

ziggie1984 · 2024-08-01T22:34:03Z

here is another log which shows the MPP-Timeout of one of the HTLCAttempts:

ziggie1984

First pass done, mostly nits. Except from my other comment related to the other problem with Inflight HTLCs.

itest/lnd_payment_test.go

ziggie1984 · 2024-07-31T13:31:00Z

htlcswitch/switch.go

@@ -3041,3 +2949,162 @@ func (s *Switch) handlePacketAdd(packet *htlcPacket,

 	return destination.handleSwitchPacket(packet)
 }
+
+// handlePacketAdd handles forwarding a settle packet.


Nit: handlePacketAdd => handlePacketSettle

ziggie1984 · 2024-07-31T13:31:37Z

htlcswitch/switch.go

+	// ErrUnknownCircuit error upon the inner call to CloseCircuit.
+	//
+	// NOTE: We can only get a nil circuit when it has already been deleted
+	// when `UpdateFulfillHTLC` is received. After which `RevokeAndAck` is


Nit: ... and when UpdateFulfillHTLC is received.

ziggie1984 · 2024-07-31T13:32:47Z

htlcswitch/switch.go

+	return s.mailOrchestrator.Deliver(packet.incomingChanID, packet)
+}
+
+// handlePacketAdd handles forwarding a fail packet.


Nit: godocs

ziggie1984 · 2024-08-02T10:58:41Z

itest/lnd_payment_test.go

+	payStream := alice.RPC.TrackPaymentV2(payHash[:])
+	dustPayStream := alice.RPC.TrackPaymentV2(dustPayHash[:])
+
+	// Check that the dust payment is failed


Nit: Maybe add a comment why both checks, the stream and the normal findpayment command should be done here ?

htlcswitch/switch.go

ziggie1984 · 2024-08-02T11:20:30Z

htlcswitch/switch.go

+// handlePacketAdd handles forwarding a settle packet.
+func (s *Switch) handlePacketSettle(packet *htlcPacket) error {
+	// If the source of this packet has not been set, use the circuit map
+	// to lookup the origin.


Q: when can this happen, would be good to describe it in the comment.

Cool let me dig a bit...

ziggie1984 · 2024-08-02T11:28:11Z

docs/release-notes/release-notes-0.18.3.md

@@ -33,6 +33,9 @@
  created](https://github.com/lightningnetwork/lnd/pull/8921) when multiple RPC
  calls are made concurrently.

+* [Fixed](https://github.com/lightningnetwork/lnd/pull/8174) old payments that
+  are stuck inflight. Check more info [here](https://github.com/lightningnetwork/lnd/issues/8146).
+


wondering whether the three commits on top of this PR are related ?

you mean the last three? yeah they are related as they are required to get the itest pass.

maybe switch the order so we have the release at the end ?

yyforyongyu · 2024-08-02T23:28:34Z

Interesting case @ziggie1984! Cool could you open a new issue to track this instead - think I know what happened, but don't have a good fix, plus I prefer not growing the scope of this PR to stay focused. In short it's due to async fetching of the same payment, and it happens when an invoice times out exactly during the period while the node is still restarting. Would also be great to have raw logs instead of screenshots, as the timestamps were messed up and difficult to analyze.

bitromortac

Nice, looks very close 🚀, just a few nits and questions.

htlcswitch/switch.go

routing/router.go

ziggie1984

LGTM 🐝, still had some general questions about this code change but all non-blocking

ziggie1984 · 2024-08-05T17:53:07Z

routing/router.go

+	}
+
+	log.Debugf("Cleaning network result store.")
+	if err := r.cfg.Payer.CleanStore(toKeep); err != nil {


So we clean the store only once on startup ? Would it make sense to run this periodically:

from the comment of the clean function:

This should be called // preiodically to let the switch clean up payment results that we have // handled.

Good q - yeah I think it should be cleaned more frequently. I also remember we can fetch whatever is stored here from our open channel, which makes this store redundant. Will make an issue to track it - either clean it more often or just don't store it.

ziggie1984 · 2024-08-05T17:56:02Z

routing/router.go

+			a := a
+
+			// We check whether the individual attempts have their
+			// HTLC hash set, if not we'll fall back to the overall


Q: Is the hash set equal of AMP being used ?

yeah I think so

Nit: maybe add it to the comment then, I would have found it useful ?

ziggie1984 · 2024-08-05T18:06:54Z

routing/router.go

+		// We pass in a non-timeout context, to indicate we don't need
+		// it to timeout. It will stop immediately after the existing
+		// attempt has finished anyway. We also set a zero fee limit,
+		// as no more routes should be tried.


should be => can we be sure that INFLIGHT HTLCs will never have payment status (e.g. StatusInFlight) which could lead us to retry the payment and send more HTCL attempts ?

would the zero-fee route cause a problem in terms of not setting a timeout here ?

if there exists a zero-fee route (which seems impossible?), then whether to send another attempt or not is determined by the remaining payment amt, if it's not zero, then another attempt will be tried.

A question is whether we really would like to set a zero fee limit here (and starve the payment), as in principle we may want to have payments to continue after a restart, right? I think that's why resumePayment was added. Do we have an itest for a payment that continues its pathfinding after a restart?

Good question! I think we need to revisit the restart process a bit - atm, there's the existing issue re how we should go and handle the payment timeout from the sender side. The second issue now is the one you mentioned above. Say we have this case,

invoice has a timeout of 60s.

the sender tries to fulfill the invoice via MPP, say htlc1 and htlc2.

htlc1 is sent, but if the sender restarts before sending htlc2(or even before the creation of it, as long as there's a remaining amount), and,

the sender's restart finishes before the invoice times out.

the payment is now resumed, with only htlc1 sent, and some remaining amount to try.

In this case, this payment is def gonna fail since it won't try another route. I also don't think this is an urgent issue, also guess that's why we log errors during the shutdown process if we still have inflight payments. There are multiple ways to fix it and think it's worthy having a tracking issue.

ziggie1984 · 2024-08-05T18:54:19Z

docs/release-notes/release-notes-0.18.3.md

@@ -33,6 +33,9 @@
  created](https://github.com/lightningnetwork/lnd/pull/8921) when multiple RPC
  calls are made concurrently.

+* [Fixed](https://github.com/lightningnetwork/lnd/pull/8174) old payments that
+  are stuck inflight. Check more info [here](https://github.com/lightningnetwork/lnd/issues/8146).
+


maybe switch the order so we have the release at the end ?

ziggie1984 · 2024-08-05T18:55:45Z

docs/release-notes/release-notes-0.18.3.md

@@ -48,6 +48,9 @@
 bumping an anchor channel closing was not possible when no HTLCs were on the
 commitment when the channel was force closed.

+* [Fixed](https://github.com/lightningnetwork/lnd/pull/8174) old payments that
+  are stuck inflight. Check more info [here](https://github.com/lightningnetwork/lnd/issues/8146).


maybe add in the description that the culprit of this problem is still not detected ? (Meaning why those payments are still inflight and did not resolve their corresponding payment attempt)

cool updated.

ziggie1984 · 2024-08-05T18:57:58Z

server.go

+	// Double check whether the reported closed channel has indeed finished
+	// closing.
+	//
+	// NOTE: There are misalignments regarding when a channel's FC is


Could you find out which misalignments in particular ?

That's what the TODO is for...

ok sure, was just asking what misalignment means in this context, afaict it means somehow we mark channels prematurely as closed although they are still pending ?

routing/router.go

bitromortac

LGTM ⚡

This commit adds a new test case to validate that when an HTLC has timed out, the corresponding payment is marked as failed.

Simply moves the code into a new method so it's easier to follow the method `handlePacketForward`.

This commit adds two methods, `handlePacketFail` and `handlePacketSettle` to handle the settle and fail packets differently.

startup

This commit adds a check during router's startup and fails the inflight HTLCs if they are routing using channels unknown to us. The channels are unknown because they are already closed, usually long time ago.

The method `FetchClosedChannels` sometimes prematurely mark a pending force closing channel as finalized, therefore we need to furthur check `FetchPendingChannels` to make sure the channel is indeed finalized.

To reflect the new sweeping behavior, also makes it easier to be used as we need to method to quickly cleanup force closes without concerning the details when we are not testing the force close behavior.

yyforyongyu added payments Related to invoices/payments bug fix labels Nov 14, 2023

yyforyongyu force-pushed the fix-inflight-payments branch 4 times, most recently from 1435e5b to 2793c53 Compare November 15, 2023 17:16

yyforyongyu self-assigned this Nov 16, 2023

yyforyongyu added this to the v0.18.0 milestone Nov 16, 2023

yyforyongyu force-pushed the fix-inflight-payments branch 6 times, most recently from d044321 to a80524d Compare November 19, 2023 14:42

yyforyongyu mentioned this pull request Nov 21, 2023

[bug]: trackpayment shows payment IN_FLIGHT but logs shows failed: no_route #8178

Closed

saubyk modified the milestones: v0.18.0, v0.18.1 Mar 3, 2024

Roasbeef requested review from bitromortac and bhandras April 1, 2024 18:38

bitromortac reviewed May 2, 2024

View reviewed changes

saubyk added the P2 should be fixed if one has time label Jun 25, 2024

yyforyongyu force-pushed the fix-inflight-payments branch from a80524d to d4ff44e Compare July 22, 2024 13:10

ziggie1984 removed the request for review from bhandras July 31, 2024 12:08

ziggie1984 reviewed Aug 2, 2024

View reviewed changes

yyforyongyu force-pushed the fix-inflight-payments branch from ae6ed43 to 06146ce Compare August 2, 2024 23:34

ziggie1984 mentioned this pull request Aug 5, 2024

[bug]: Payment Flow: Don't abort payment until HTLC attempts are still in flight #8975

Open

bitromortac reviewed Aug 5, 2024

View reviewed changes

ziggie1984 approved these changes Aug 5, 2024

View reviewed changes

yyforyongyu force-pushed the fix-inflight-payments branch from 06146ce to ba3741c Compare August 6, 2024 12:49

bitromortac reviewed Aug 7, 2024

View reviewed changes

routing/router.go Show resolved Hide resolved

bitromortac approved these changes Aug 7, 2024

View reviewed changes

yyforyongyu and others added 15 commits August 7, 2024 22:17

routing: fix and enhance logging

0928ba0

lntest+itest: add new test testPaymentHTLCTimeout

941716d

This commit adds a new test case to validate that when an HTLC has timed out, the corresponding payment is marked as failed.

htlcswitch: rename paymentID to attemptID for clarity

21112cf

itest: add testSendToRouteFailHTLCTimeout to check SendToRouteV2

d28d5d7

htlcswitch: add new method handlePacketAdd

6cb374a

Simply moves the code into a new method so it's easier to follow the method `handlePacketForward`.

htlcswitch: handle forwarding settle and fail seperately

2fc79d8

This commit adds two methods, `handlePacketFail` and `handlePacketSettle` to handle the settle and fail packets differently.

channeldb+htlcswitch: make sure circuit is not nil in teardownCircuit

3b6e28d

routing: add new method resumePayments to handle payments during

bbf58ab

startup

routing+htlcswitch: add new interface method HasAttemptResult

b998ce1

routing: fail stale HTLC attempts during startup

c2f7e6a

This commit adds a check during router's startup and fails the inflight HTLCs if they are routing using channels unknown to us. The channels are unknown because they are already closed, usually long time ago.

docs: update release notes re stuck payment fix

e8f292e

routing+lnd: prepare closed channel SCIDs in server

188aa9a

The method `FetchClosedChannels` sometimes prematurely mark a pending force closing channel as finalized, therefore we need to furthur check `FetchPendingChannels` to make sure the channel is indeed finalized.

routing: fix linter complains

7aba5cb

lntest: re-define CleanupForceClose

677f2c3

To reflect the new sweeping behavior, also makes it easier to be used as we need to method to quickly cleanup force closes without concerning the details when we are not testing the force close behavior.

routing: fix doc string

bc31a8b

yyforyongyu force-pushed the fix-inflight-payments branch from ba3741c to bc31a8b Compare August 7, 2024 14:21

yyforyongyu merged commit a449a5d into lightningnetwork:master Aug 7, 2024
24 of 27 checks passed

yyforyongyu deleted the fix-inflight-payments branch August 7, 2024 15:23

	// closeCircuit returns a nil circuit when a settle packet returns an
	// ErrUnknownCircuit error upon the inner call to CloseCircuit.
	if circuit == nil {
	return nil
	}

	// Based on the above variables, we derive the status using the following
	// table,
	// \| inflight \| settled \| htlc failed \| payment failed \| status \|
	// \|:--------:\|:-------:\|:-----------:\|:--------------:\|:--------------------:\|
	// \| true \| true \| true \| true \| StatusInFlight \|
	// \| true \| true \| true \| false \| StatusInFlight \|
	// \| true \| true \| false \| true \| StatusInFlight \|
	// \| true \| true \| false \| false \| StatusInFlight \|
	// \| true \| false \| true \| true \| StatusInFlight \|
	// \| true \| false \| true \| false \| StatusInFlight \|
	// \| true \| false \| false \| true \| StatusInFlight \|
	// \| true \| false \| false \| false \| StatusInFlight \|

routing: fix stuck inflight payments #8174

routing: fix stuck inflight payments #8174

Conversation

yyforyongyu commented Nov 14, 2023 • edited Loading

Timed out HTLCs are properly handle

HTLC Attempt result is not saved

saubyk commented Nov 28, 2023

bitromortac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yyforyongyu commented Jun 14, 2024

yyforyongyu commented Jun 26, 2024

yyforyongyu commented Jul 10, 2024

kilrau commented Jul 20, 2024

saubyk commented Jul 20, 2024

kilrau commented Jul 20, 2024

coderabbitai bot commented Jul 22, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

ziggie1984 commented Aug 1, 2024

ziggie1984 commented Aug 1, 2024

ziggie1984 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yyforyongyu commented Aug 2, 2024

bitromortac left a comment

Choose a reason for hiding this comment

ziggie1984 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitromortac Aug 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitromortac left a comment

Choose a reason for hiding this comment

yyforyongyu commented Nov 14, 2023 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

bitromortac Aug 7, 2024 •

edited

Loading