paying an invoices puts my server at highload and makes it crash

[warioishere]

Describe the bug

I am running a quite big node optimized on routing.

Got Zeus connected with it.

Everytime I pay an invoice with it, the server goes to almost 100% load, consuming all available RAM and then crashes

A friend told me that it could be a command used by your app which lists all payments listpayments, invoices and forwards without pagination maybe causing this, and because I ve millions due to rebelancing, the server collapse when asking for this. I dont know if this is the proper explanation, but you might look into it

I guess it has no big impact on smaller nodes with just a few channels and less routing.

Cannot use Zeus with my node at all

running lnd 0.18.3 here

greetings

Reproduce

pay an invoice with zeus of any kind
crashes the server

ZEUS version

v0.9.2

Node interface

LND (REST)

Network

Clearnet via domain of my btcppay server 443/8080

Device

android 15 pixel 9Pro XL

Device operating system

GrapheneOS latest

Log output

I dont risk to do this again to get logs from my server as I dont wanna corrupt my db or put this thing on any kind of danger. It happend twice, first I thought it was some kind of general crash on my server, the next time it happend again, i just did a payment checked the Server load and realized that it comes when paying invoices via Zeus

Additional Info:
this does not happen with bitbanana paying invoices

[kaloudis]

Payments, invoices, and payments aren't queried when making a payment. Maybe your node is having issues making a multi-path payment that Zeus attempts by default? Or there's an issue with the REST interface.

You should take a look at the logs. It shouldn't corrupt your DB. Your node is always writing those logs even if you're not looking at them.

Differences to be cognizant of btwn BitBanana and Zeus:
-BB uses GRPC, whereas Zeus uses REST or LNC
-BB may have different defaults for send payment (no MPP?) - unsure of this

[warioishere]

Payments, invoices, and payments aren't queried when making a payment. Maybe your node is having issues making a multi-path payment that Zeus attempts by default? Or there's an issue with the REST interface.

You should take a look at the logs. It shouldn't corrupt your DB. Your node is always writing those logs even if you're not looking at them.

Differences to be cognizant of btwn BitBanana and Zeus: -BB uses GRPC, whereas Zeus uses REST or LNC -BB may have different defaults for send payment (no MPP?) - unsure of this

last time I did a payment, it completly locked up my machine. --> hard reset with bbolt. Not funny, can kill your db and with a node of about 6btc capacity... I dont wanna risk this again.

Okay got another one that has similar issues: he is going to get some logs later.

Well I do believe, that paying invoices do not require invoices, payments and forwards, but could it be, that zeus calls all payments after an invoice is done to have a fresh list of it?

Here are my forwards,payments,invoices:

Forwards 207,281
Payments 565,220
Invoices 192,878

My machine has currently 16GB of RAM allocated. If I do a payment, it fills up completly and CPU load --> 99%

This is the Data of a user that has no Problems but has about 48Gigs of RAM:

Payments: 1.571.483
invoices: 939.753
forwards 261.161

He didnt check his system graphs yet when paying an invoice but he will do later

@Filouman

Will add some charts later if similar high system loads when paying invoices

[warioishere]

https://github.com/ZeusLN/zeus/blob/master/backends/LND.ts#L327

might it be an issue it calls payments without pagination? Micheal from Boltz said it can bring down the biggest node if you call this at once

[kaloudis]

Ah, I believe it's how we fetch the payment path. We can fix our query there.

Thanks for bringing this to our attention.

[Filouman]

Can confirm a similar issue on my server– Similar setup, larger routing node with high number of records in the database:

In my case LND is connected to Zeus via LNC
LND 0.18.3 on Postgresql backend
Zeus 0.9.3-beta3 (recently upgraded, however issue existed on 0.9.2 as well)

Making a payment using Zeus sends LND cpu consumption to near or above 100% for nearly 15 minutes straight. In my case, it does not crash the server but it does lock up LND during that time period and it will not respond to other calls.

Making a payment using lncli or bos pay does not cause the same effect.

Logs (RPCS=debug) after entering invoice into Zeus and paying:

Dec 07 15:51:18 nodelou lnd[1684]: 2024-12-07 15:51:18.571 [DBG] RPCS: [/lnrpc.Lightning/DecodePayReq] requested
Dec 07 15:51:18 nodelou lnd[1684]: 2024-12-07 15:51:18.784 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:18 nodelou lnd[1684]: 2024-12-07 15:51:18.786 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:18 nodelou lnd[1684]: 2024-12-07 15:51:18.792 [DBG] RPCS: [/lnrpc.Lightning/QueryRoutes] requested
Dec 07 15:51:22 nodelou lnd[1684]: 2024-12-07 15:51:22.524 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:22 nodelou lnd[1684]: 2024-12-07 15:51:22.528 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:22 nodelou lnd[1684]: 2024-12-07 15:51:22.530 [DBG] RPCS: [/routerrpc.Router/SendPaymentV2] requested
Dec 07 15:51:22 nodelou lnd[1684]: 2024-12-07 15:51:22.756 [DBG] RPCS: [/lnrpc.Lightning/ForwardingHistory] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.011 [INF] SRVR: Adding preimage=ef94e610c1b38b653f019b6bcef289bef0b014c1c0d156dcdfb8d8242b634b84 to witness cache for 39ac4fe8a6d5f8049e8ad7f76500638b421f622cb2b49e122b1be4f1e8532efe
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.056 [DBG] RPCS: [/routerrpc.Router/TrackPaymentV2] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.112 [ERR] RRPC: TrackPayment got error for payment 33396163346665386136643566383034396538616437663736353030363338623432316636323263623262343965313232623162653466316538353332656665: <nil>
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.140 [ERR] RRPC: TrackPayment got error for payment 33396163346665386136643566383034396538616437663736353030363338623432316636323263623262343965313232623162653466316538353332656665: <nil>
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.146 [DBG] RPCS: [/lnrpc.Lightning/GetNodeInfo] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.146 [DBG] RPCS: [/lnrpc.Lightning/GetNodeInfo] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.266 [DBG] RPCS: [/verrpc.Versioner/GetVersion] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.266 [DBG] RPCS: [/verrpc.Versioner/GetVersion] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.472 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.474 [DBG] RPCS: [/lnrpc.Lightning/CheckMacaroonPermissions] requested
Dec 07 15:51:23 nodelou lnd[1684]: 2024-12-07 15:51:23.476 [DBG] RPCS: [/lnrpc.Lightning/ListPayments] requested
Dec 07 15:51:44 nodelou lnd[1684]: 2024-12-07 15:51:44.169 [DBG] RPCS: [/lnrpc.Lightning/PendingChannels] requested
Dec 07 15:51:52 nodelou lnd[1684]: 2024-12-07 15:51:52.761 [DBG] RPCS: [/lnrpc.Lightning/ForwardingHistory] requested

LND cpu use before/after initiating payment:

It seems Zeus makes some RPC calls here, very possible that ListPayments call creates the load as this corresponds with the CPU spike and is generally a very intensive call if not restricted in scope.

After successful payment, processor load remains elevated for approximately 14 minutes and total memory used by LND slowly climbs from around 450MB to 2.0GB.

At around 16:05:36 (14 minutes and 13 seconds after calling listpayments) CPU use returns again to normal levels:

[Filouman]

The behavior from Zeus seems to reflect running lncli listpayments in full as a standalone call.

time lncli listpayments
real	14m40.232s
user	0m11.340s
sys	0m16.413s

[blckbx]

Payments, invoices, and payments aren't queried when making a payment.

https://github.com/ZeusLN/zeus/blob/master/views/SendingLightning.tsx#L86-L112

componentDidUpdate -> on successful sending -> fetchPayments() -> PaymentsStore.getPayments() -> BackendUtils.getPayments() -> this.getRequest('/v1/payments?include_incomplete=true')

am I wrong here?

[Filouman]

Payments, invoices, and payments aren't queried when making a payment.

https://github.com/ZeusLN/zeus/blob/master/views/SendingLightning.tsx#L86-L112

componentDidUpdate -> on successful sending -> fetchPayments() -> PaymentsStore.getPayments() -> BackendUtils.getPayments() -> this.getRequest('/v1/payments?include_incomplete=true')

am I wrong here?

Yes, in this case, it seems Zeus does not limit the scope of payments call:
getPayments = () => this.getRequest('/v1/payments?include_incomplete=true');
Best would be to use --index_offset and paginate to limit the call which would be 1000x lighter to process. Like ms instead of minutes.

[kaloudis]

Payments, invoices, and payments aren't queried when making a payment.

https://github.com/ZeusLN/zeus/blob/master/views/SendingLightning.tsx#L86-L112

componentDidUpdate -> on successful sending -> fetchPayments() -> PaymentsStore.getPayments() -> BackendUtils.getPayments() -> this.getRequest('/v1/payments?include_incomplete=true')

am I wrong here?

#2605 (comment)

[kaloudis]

Payments, invoices, and payments aren't queried when making a payment.

https://github.com/ZeusLN/zeus/blob/master/views/SendingLightning.tsx#L86-L112
componentDidUpdate -> on successful sending -> fetchPayments() -> PaymentsStore.getPayments() -> BackendUtils.getPayments() -> this.getRequest('/v1/payments?include_incomplete=true')
am I wrong here?

Yes, in this case, it seems Zeus does not limit the scope of payments call: getPayments = () => this.getRequest('/v1/payments?include_incomplete=true'); Best would be to use --index_offset and paginate to limit the call which would be 1000x lighter to process. Like ms instead of minutes.

index_offset plus max_payments should do the trick for this case

[kaloudis]

Fix available for testing in ZEUS v0.9.4-beta1 https://github.com/ZeusLN/zeus/releases/tag/v0.9.4-beta1

[Filouman]

MASSIVE improvement in my tests! Zeus is slick and fast again TY for the fix!! ⚡️

[warioishere]

thanks guys, closing this as done!

[warioishere]

or not, just saw Evan reopened?

[kaloudis]

All good now. v0.9.4 now out in general release