RTL section for managing the autoclean-once RPC command #1353
Replies: 3 comments 2 replies
-
|
Do you recall exactly which autoclean subsystem you used? I think RTL should probably autoclean by default, especially for failed forwards, payments and invoices after say 1 week or 1 month. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @rustyrussell, Yes I do - the commands I ran were: bitcoin-lightning-cli.sh autoclean-once failedforwards 34186698 The numbers are in seconds. There is a handy calculator here to convert seconds to days: https://www.calculateme.com/time/days/to-seconds/ 34,186,698 Seconds ≈ 8,000,000 Seconds ≈ I recommend before you run the commands to figure out how far back you want to have autoclean remove the failedforwards. The command took quite a while to run for me due to the size of the underlying db. I exported the data first from RTL into the CSV file for the succeededforwards before I ran the commands but note I had to run the autoclean-once commands on the failedforwards first in order for me to successfully export the successful forwards as a CSV file due to the sheer size of the underlying sqlite dd which had grown to over 5GB over the years. Best Regards, Clarke |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I suspected as much: I think most wins will be from simply enabling deleting of failedforwards after a month. I discussed with Shahana the idea of making |
Beta Was this translation helpful? Give feedback.
-
|
I think that's an excellent idea. The RTL interface I think should have a section for interacting with the autoclean commands if only for reporting purposes. As time goes on and more and more LN operators who were ahead of the curve with getting their nodes operational in 2021 and 2022, as they build up larger and larger entries in the underlying lightning sqlite db they will face issues which they may think are RTL related but are in reality caused by the build up of the lightning db. |
Beta Was this translation helpful? Give feedback.
-
|
I set myself a cron job on each of my BTCPay Server hosts with a script that does a clean up every 3 months. I have it disabled for now pending some further work with grafana and fully automating my collection and presentation of the successful forwarding logs. Run a cron job on the 1st January, 1st April 1st July and 1st October to get rid of all failed forwards and failed payments from this BTCPay Server0 0 1 1,4,7,10 * /root/scripts/BTCPayClean.sh#/bin/bash clean up the failed forwards failedpays and expired invoices BTCPayimplemented 7th February 2024 as a result of too many failed forwards pays and invoices clogging up BTCPay commandsin particular lightning commands and RTL GUIrun this script once every 3 months via cron8000000 seconds is approx 93 days in totalClarke Towsonwe will only have 12 weeks of failures to go back on but were keeping the successes for logging purposes/root/BTCPayServer/btcpayserver-docker/bitcoin-lightning-cli.sh autoclean-once failedforwards 8000000 |
Beta Was this translation helpful? Give feedback.
-
Hi Everyone,
I wanted to share some ideas based on running a production BTCPay Server with RTL since early 2021.
My node CHILLYCALENDAR has around 120 payment channels opened to it. Over the years the forwarding and routing logs grew substantially until the underlying lightning postgres database was around 6GB in size. As the entries in the db grew RTL started becoming very slow and the UI would fail on things like sending bitcoin out on chain, sending bitcoin via lightning to/from the node, listing the forwarding and routing events etc. When I first started experiencing these issues I was stumped as to what was going on but a c-lightning developer advised me that running the bitcoin-lightning-cli autoclean-once command would help me to trim the entries in the db down and this would likely resolve the issues I was experiencing.
After upgrading BTCPay to the latest version this morning I did exactly that and I noticed a substantial improvement in the node including the underlying lightningd and related processes which prior to the pruning would take up 99% of the CPU, max out the RAM and then the machine would lock up. In the older version of BTCPay I had installed the docker processes would also restart but the newer version this doesn't happen which is great.
I was new to lightning back in 2021 and up until now I was not fully aware of what was happening on my node or how to resolve the issues until I saw the autoclean-once command. I now have cron jobs set to run this command so I keep approx 3 months worth of logs and this seems to be an appropriate amount I think.
I would like to suggest that the RTL UI include methods to interact with the underlying autoclean-once command and perhaps a piece of text that suggests to the node operator that the underlying lightning processes are greatly affected by the size of the lightningd.sqlite3 database. I expect more node operators will come across these slow down and performance issues in RTL which are caused by a bloated lightningd.sqlite3 db and not by RTL itself as far as I can see.
Perhaps RTL could implement some checks on the underlying lightning db or cache it in some manner to ensure that the most important tasks a node operator does like opening channels, sending bitcoin out on chain, to/from the node via lightning etc that these tasks are always really quick.
I would also like to suggest that RTL include some custom error message explanations as the underlying errors from lightning are quite confusing especially for new users. The NGINX errors for example are common if a node operators lightning db has grown to a large size.
Please keep up the excellent work on this software it's such a great tool for us node operators!
With Very Best Regards,
Clarke
Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science)
CEO
INTJ Billing
m: +61 432 359 166
a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA
w: https://intjbilling.com/
e: [email protected]
Bitcoin Lightning Network Node Name: CHILLYCALENDAR
Node Public Key: 025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693
Status Page: https://intjbilling.com/INTJBillingStatusPage.php
amboss.space: https://amboss.space/node/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693
1ml.com: https://1ml.com/node/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693
Lightning network+: https://lightningnetwork.plus/nodes/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693
Beta Was this translation helpful? Give feedback.
All reactions