Reduce peer message traffic for ledger data #5126

ximinez · 2024-09-11T13:10:20Z

High Level Overview of Change

Several changes to help reduce message traffic and improve logging and visibility.

Suppress duplicate TMGetLedger and TMLedgerData messages, reducing the overhead of processing those messages.
Reduce the number of those messages sent to peers.
Improve logging related to those messages, as well as a few other areas.
Introduces a new protocol version to gate a new feature on the TMLedgerData message, which allows multiple identical requests to be replied to with one message.

These changes are organized into several commits which are organized logically separating each functional operation. They can be merged as-is, or squashed.

Context of Change

Analysis of the issue that led to #5115 identified heavy TMGetLedger request and TMLedgerData response traffic between nodes leading up to the syncing incidents. It was later determined that those messages were more a symptom of the problem, and not the root cause. However, leading up to identification of the root cause, these changes were being implemented to cut down on those messages, detect duplicates, etc. That reduction in unnecessary traffic is still valuable, so it's being included here.

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance (increase or change in throughput and/or latency)

API Impact

None.

Test Plan

Verify that nodes are at least as reliable at staying in sync as they are now.
Verify that the indicated messages are sent less frequently by upgraded nodes.
Verify that peers that both have this change send fewer ledger messages to each other.

Future Tasks

I am still working on a follow-up to #4764 that makes use of these changes and other improvements to reduce the number of requests initiated by a given node in the first place.

* Also log as warning when the state lowers

codecov · 2024-09-11T14:13:59Z

Codecov Report

Attention: Patch coverage is 23.95349% with 327 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (a079bac) to head (39fe006).

Files with missing lines	Patch %	Lines
src/xrpld/overlay/detail/PeerImp.cpp	0.0%	215 Missing ⚠️
src/xrpld/overlay/detail/ProtocolMessage.h	0.0%	32 Missing ⚠️
src/xrpld/app/misc/NetworkOPs.cpp	21.4%	22 Missing ⚠️
src/xrpld/overlay/detail/PeerSet.cpp	20.8%	19 Missing ⚠️
src/xrpld/app/ledger/detail/InboundLedgers.cpp	67.9%	17 Missing ⚠️
src/xrpld/app/misc/HashRouter.cpp	50.0%	6 Missing ⚠️
src/xrpld/app/ledger/detail/InboundLedger.cpp	54.5%	5 Missing ⚠️
src/xrpld/app/ledger/InboundLedger.h	60.0%	4 Missing ⚠️
src/xrpld/app/ledger/detail/LedgerMaster.cpp	0.0%	3 Missing ⚠️
src/xrpld/app/misc/HashRouter.h	77.8%	2 Missing ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5126     +/-   ##
=========================================
- Coverage     78.2%   77.9%   -0.3%     
=========================================
  Files          790     791      +1     
  Lines        67639   67967    +328     
  Branches      8160    8324    +164     
=========================================
+ Hits         52869   52950     +81     
- Misses       14770   15017    +247

Files with missing lines	Coverage Δ
include/xrpl/basics/base_uint.h	`96.9% <100.0%> (+<0.1%)`	⬆️
include/xrpl/protocol/LedgerHeader.h	`100.0% <ø> (ø)`
src/xrpld/app/ledger/detail/TimeoutCounter.cpp	`88.4% <100.0%> (+1.3%)`	⬆️
src/xrpld/app/ledger/detail/TimeoutCounter.h	`100.0% <ø> (ø)`
src/xrpld/app/misc/NetworkOPs.h	`100.0% <ø> (ø)`
src/xrpld/overlay/Peer.h	`100.0% <ø> (ø)`
src/xrpld/overlay/detail/PeerImp.h	`12.8% <ø> (ø)`
src/xrpld/overlay/detail/ProtocolVersion.cpp	`86.4% <ø> (ø)`
include/xrpl/basics/CanProcess.h	`95.5% <95.5%> (ø)`
src/xrpld/app/consensus/RCLConsensus.cpp	`65.4% <0.0%> (ø)`
... and 10 more

... and 6 files with indirect coverage changes

* Allow a retry after 30s in case of peer or network congestion. * Addresses RIPD-1870 * (Changes levelization. That is not desirable, and will need to be fixed.)

* Allow a retry after 15s in case of peer or network congestion. * Collate duplicate TMGetLedger requests: * The requestCookie is ignored when computing the hash, thus increasing the chances of detecting duplicate messages. * With duplicate messages, keep track of the different requestCookies (or lack of cookie). When work is finally done for a given request, send the response to all the peers that are waiting on the request, sending a separate message for each requestCookie. * Addresses RIPD-1871

* Addresses RIPD-1869 --------- Co-authored-by: Valentin Balaschenko <[email protected]> Co-authored-by: Ed Hennis <[email protected]>

* When work is done for a given TMGetLedger request, send the response to all the peers that are waiting on the request, sending one message per peer, including all the cookies and a "directResponse" flag indicating the data is intended for the sender, too.

Bronek · 2024-09-13T11:22:46Z

include/xrpl/basics/CanProcess.h

+    insert()
+    {
+        std::unique_lock<Mutex> lock_(mtx_);
+        bool exists = collection_.contains(item_);


This would avoid extra lookup

auto [_, inserted] = collection_.insert(item_); return inserted;

Good catch. Fixed.

* Avoid an unnecessary lookup in CanProcess

* upstream/develop: Set version to 2.3.0-b4 feat(SQLite): allow configurable database pragma values (5135) refactor: re-order PRAGMA statements (5140) fix(book_changes): add "validated" field and reduce RPC latency (5096) chore: fix typos in comments (5094) Set version to 2.2.3 Update SQLite3 max_page_count to match current defaults (5114)

vlntb

Only minor comments. Happy to approve once addressed.

include/xrpl/basics/base_uint.h

vlntb · 2024-10-10T14:22:16Z

src/xrpld/overlay/detail/PeerImp.cpp

+                        return ledger;
+                    }
+
+                    JLOG(p_journal_.trace())


[nit]: Should this be a warn instead of trace? Not having a peer to relay the request may indicate some configuration or environment issues.

Not necessarily, though. It could mean that the node has already sent the request to all it's peers. Also not that the original message is trace.

But there's something odd here. It looks like this code block was somehow duplicated! I must have messed up resolving a conflict when I rebased from master to develop. I've removed the duplicate.

vlntb · 2024-10-10T14:23:53Z

src/xrpld/overlay/detail/PeerImp.cpp

@@ -2936,7 +3078,9 @@ getPeerWithLedger(
 void
 PeerImp::sendLedgerBase(
    std::shared_ptr<Ledger const> const& ledger,
-    protocol::TMLedgerData& ledgerData)
+    protocol::TMLedgerData& ledgerData,
+    std::map<std::shared_ptr<Peer>, std::set<std::optional<uint64_t>>> const&


std::map<std::shared_ptr<Peer>, std::set<std::optional<uint64_t>>> is mentioned in four places. It would be more readable to define an alias:
using PeerCookieMap = std::map<std::shared_ptr<Peer>, std::set<std::optional<uint64_t>>>;

Good suggestion! Fixed.

* Add unit tests for to_short_string(base_uint * Remove duplicated code * Use type aliases for cookie maps

* That's what I get for rushing to push

* upstream/develop: Set version to 2.3.0 refactor(AMMClawback): move tfClawTwoAssets check (5201) Add a new serialized type: STNumber (5121) fix: check for valid ammID field in amm_info RPC (5188)

Bronek

Several notes for now, also I noticed there's a bunch of Added line not covered by tests annotations from codecov, perhaps worth checking these.

include/xrpl/basics/CanProcess.h

Bronek · 2024-11-25T13:19:33Z

include/xrpl/basics/CanProcess.h

+        return canProcess_;
+    }
+
+    operator bool() const


Please use explicit here; it would be consistent with std::optional and most other operator bool inside the project.

Please use explicit here; it would be consistent with std::optional and most other operator bool inside the project.

Fixed

Bronek · 2024-12-02T13:12:56Z

include/xrpl/basics/CanProcess.h

+    }
+
+    bool
+    canProcess() const


given that it is not covered by tests, perhaps we do not need this function ?

given that it is not covered by tests, perhaps we do not need this function ?

It's used in the call path for network traffic, most of which is not covered by tests. But I changed that call to use the boolean conversion, and everything seems fine.

Bronek · 2024-12-02T14:04:00Z

include/xrpl/basics/CanProcess.h

+        if (canProcess_)
+        {
+            std::unique_lock<Mutex> lock_(mtx_);
+            collection_.erase(item_);


How about replacing the (likely) O(log N) element lookup with O(1) erase of an iterator ? The iterator which is returned from insert to be specific, and currently ignored. In this case the Item wouldn't have to be stored inside CanProcess object, so that's also one less template parameter and probably also smaller object size (hashes are larger than iterators I guess).

Could even go one step further and replace all data members with std::function<void()> cleanup_ which would capture all that it needs if insert succeeded, or is empty if insert failed. In this case no template parameters would be needed at all.

How about replacing the (likely) O(log N) element lookup with O(1) erase of an iterator ? The iterator which is returned from insert to be specific, and currently ignored. In this case the Item wouldn't have to be stored inside CanProcess object, so that's also one less template parameter and probably also smaller object size (hashes are larger than iterators I guess).

I didn't really consider using an iterator because I didn't want to consider the risks of it getting invalidated for any valid Collection type. For example, it looks like unordered_map would fit the template, but can invalidate iterators. One option is to force the collection to be std::set, which the current callers use.

Could even go one step further and replace all data members with std::function<void()> cleanup_ which would capture all that it needs if insert succeeded, or is empty if insert failed. In this case no template parameters would be needed at all.

So only the ctor and insert would have template parameters? That could work.

I was incorrect in that last comment. unordered_map doesn't fit the template, but unordered_set does, and it also says it can potentially invalidate operators. So, I wrote a generic insert() that doesn't use iterators, and a specialized one that does, and wrote tests for both. It may be overkill....

Bronek · 2024-12-02T14:15:40Z

src/xrpld/app/ledger/detail/InboundLedgers.cpp

+                    return true;
+                return false;
+            }();
+            assert(


This looks like unnecessary repetition of the condition used to initialise shouldAcquire but there is also a subtle bug here. We are reading getOPs().isNeedNetworkLedger() more than once, but this function returns an atomic (I mean its implementation in NetworkOPs.cpp) so in principle each time we call it, we might get a different response. I am not certain this condition can actually happen, but at the very least it is brittle code.

Also please note 3rd call to getOPs().isNeedNetworkLedger() inside logging below, the same applies as it could generate some confusing logs

Ugh. I should have removed that assert before opening the PR, but forgot. I was just double-checking that I had rewritten old code into new code. I've done that, and stored the result of isNeedNetworkLedger into a local variable, so that all the uses are guaranteed consistent.

src/xrpld/app/ledger/detail/InboundLedgers.cpp

Bronek · 2024-12-02T14:30:55Z

src/xrpld/app/ledger/detail/InboundLedgers.cpp

-                << ": " << e.what();
-        }
-        catch (...)
+        if (CanProcess check{acquiresMutex_, pendingAcquires_, hash})


Looking good 👍

* upstream/develop: test: Add more test cases for Base58 parser (5174) test: Check for some unlikely null dereferences in tests (5004) Add Antithesis intrumentation (5042)

* upstream/develop: refactor: clean up `LedgerEntry.cpp` (5199)

* upstream/develop: Enforce levelization in libxrpl with CMake (5111)

* upstream/develop: Set version to 2.4.0-b1 fix: Add header for set_difference (5197) fix: allow overlapping types in `Expected` (5218) Add MPTIssue to STIssue (5200) Antithesis instrumentation improvements (5213)

* upstream/develop: chore: add macos dependency installation (5233) prefix Uint384 and Uint512 with Hash in server_definitions (5231) refactor: add `rpcName` to `LEDGER_ENTRY` macro (5202)

Co-authored-by: Bronek Kozicki <[email protected]>

* Rewrite CanProcess: Remove class template parameters. Erase collection items using iterator if appropriate. Remove the redundant canProcess() function. * Store the result of isNeedNetworkLedger() to prevent possible inconsistencies. * Make all the uses of CanProcess const objects,

* Older clang versions can't handle structured binding variables passed to lambas

* upstream/develop: chore: update deprecated Github Actions (5241) XLS-46: DynamicNFT (5048)

* upstream/develop: chore: Update Visual Studio CI to VS 2022, and add VS Debug builds (5240) Add [validator_list_threshold] to validators.txt to improve UNL security (5112) Switch from assert to XRPL_ASSERT (5245) Add missing space character to a log message (5251) Cleanup API-CHANGELOG.md (5207) test: Unit tests to recreate invalid index logic error (5242) Update branch management and merge / release processes (5215) fix: Error consistency in LedgerEntry::parsePermissionedDomains() (5252) Set version to 2.4.0-b2 fix: Use consistent CMake settings for all modules (5228) Fix levelization script to ignore commented includes (5194) Fix the flag processing of NFTokenModify (5246) Fix failing assert in `connect` RPC (5235) Permissioned Domains (XLS-80d) (5161)

* upstream/develop: Fix CI unit tests (5196) Update secp256k1 library to 0.6.0 (5254)

* upstream/develop: Set version to 2.4.0-b3 Set version to 2.3.1 Update conan in the "nix" CI jobs Add RPC "simulate" to execute a dry run of a transaction (5069) Set version to 2.3.1-rc1 Reduce the peer charges for well-behaved peers:

* upstream/develop: Add deep freeze feature (XLS-77d) (5187)

* upstream/develop: Amendment `fixFrozenLPTokenTransfer` (5227) Improve git commit hash lookup (5225)

* upstream/develop: Updates Conan dependencies (5256)

* upstream/develop: fix: Do not allow creating Permissioned Domains if credentials are not enabled (5275) fix: issues in `simulate` RPC (5265)

bthomee · 2025-02-10T17:52:31Z

@Bronek Any outstanding issues for @ximinez to address?

* upstream/develop: fix: Amendment to add transaction flag checking functionality for Credentials (5250) fix: Omit superfluous setCurrentThreadName call in GRPCServer.cpp (5280)

Bronek · 2025-02-11T17:45:54Z

src/xrpld/app/ledger/InboundLedger.h

+        case CONSENSUS:
+            return "CONSENSUS";
+        default:
+            assert(false);


please replace with UNREACHABLE and a nice & short description why (see CONTRIBUTING.md , section " Contracts and instrumentation" for naming guideline)

Bronek · 2025-02-11T17:47:14Z

src/xrpld/app/ledger/detail/InboundLedger.cpp

+        {
+            // Done. Something else (probably consensus) built the ledger
+            // locally while waiting for data (or possibly before requesting)
+            assert(isDone());


please replace with XRPL_ASSERT and a short description why; see CONTRIBUTING.md for contracts and instrumentation naming guidelines

Bronek · 2025-02-11T17:56:31Z

src/xrpld/app/misc/HashRouter.cpp

+
+    for (auto const peerID : s.peekPeerSet())
+    {
+        if (!callback(peerID))


How fast are these callbacks - we are running them over a loop inside critical section. Might need to think of an alternative solution, to avoid hogging the mutex_. Not a major issue, because as indicated in first sentence - I do not know if these callbacks are fast or slow (or we don't know ?)

Bronek · 2025-02-11T18:10:28Z

src/xrpld/overlay/detail/PeerImp.cpp

There are 10 assert added here, can you please turn them to XRPL_ASSERT, following naming guideline in CONTRIBUTING.md ?

Bronek

Only minor remarks related to use of assert, once fixed feel free to merge

Note to self: add unit tests for PeerImp.cpp in a future PR

Bronek

Actually I realized I added a comment re. use of a loop and callback inside critical section. A sufficient explanation why this is safe might be sufficient to address this comment, but for now this is not approved.

Bronek · 2025-02-11T18:26:24Z

@Bronek Any outstanding issues for @ximinez to address?

Yes, just added some. The loop and callback inside critical section can be probably resolved with an explanation e.g. a comment in code. The assert ones are trivial to fix.

* upstream/develop: chore: Rename missing-commits job, and combine nix job files (5268) docs: Add a summary of the git commit message rules (5283)

ximinez added 2 commits September 11, 2024 12:34

Log the caller / reason for server state changes:

f0cf1fd

* Also log as warning when the state lowers

Class "CanProcess" to keep track of processing of distinct items

756cad9

ximinez added the Perf Attn Needed Attention needed from RippleX Performance Team label Sep 11, 2024

ximinez force-pushed the pr/getledger branch from c357abb to 9e5487c Compare September 11, 2024 13:55

ximinez force-pushed the pr/getledger branch 3 times, most recently from 09c4156 to c69b443 Compare September 11, 2024 14:48

ximinez and others added 5 commits September 11, 2024 11:09

Drop duplicate outgoing TMGetLedger messages per peer:

8a17f16

* Allow a retry after 30s in case of peer or network congestion. * Addresses RIPD-1870 * (Changes levelization. That is not desirable, and will need to be fixed.)

Drop duplicate incoming TMLedgerData messages:

d5ec2d3

* Addresses RIPD-1869 --------- Co-authored-by: Valentin Balaschenko <[email protected]> Co-authored-by: Ed Hennis <[email protected]>

Improve logging related to ledger acquisition

e490e57

ximinez force-pushed the pr/getledger branch from c69b443 to e490e57 Compare September 11, 2024 15:10

ximinez changed the title ~~Reduce ledger protocol message traffic~~ Reduce peer message traffic for ledger data Sep 11, 2024

ximinez requested a review from mtrippled September 11, 2024 20:37

ximinez assigned mtrippled Sep 11, 2024

ximinez added this to the 2.3.0 (August 2024) milestone Sep 11, 2024

ximinez marked this pull request as ready for review September 11, 2024 20:45

ximinez mentioned this pull request Sep 12, 2024

Fix deadlock in InboundLedgers and NetworkOPs #5124

Merged

1 task

Bronek reviewed Sep 13, 2024

View reviewed changes

ximinez added 2 commits September 24, 2024 20:14

Review feedback from @Bronek:

30eee9b

* Avoid an unnecessary lookup in CanProcess

vlntb assigned vlntb and unassigned vlntb Oct 1, 2024

vlntb self-requested a review October 1, 2024 10:57

vlntb reviewed Oct 10, 2024

View reviewed changes

ximinez added 2 commits October 11, 2024 19:10

[FOLD] Review feedback from @vlntb:

b8b7b31

* Add unit tests for to_short_string(base_uint * Remove duplicated code * Use type aliases for cookie maps

[FOLD] Fix typo in unit test:

43b6e3e

* That's what I get for rushing to push

vlntb approved these changes Oct 15, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into pr/getledger

e23be8e

* upstream/develop: Set version to 2.3.0 refactor(AMMClawback): move tfClawTwoAssets check (5201) Add a new serialized type: STNumber (5121) fix: check for valid ammID field in amm_info RPC (5188)

Bronek reviewed Dec 2, 2024

View reviewed changes

ximinez and others added 8 commits December 3, 2024 17:44

Merge remote-tracking branch 'upstream/develop' into pr/getledger

75fcca5

* upstream/develop: test: Add more test cases for Base58 parser (5174) test: Check for some unlikely null dereferences in tests (5004) Add Antithesis intrumentation (5042)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

20ad383

* upstream/develop: refactor: clean up `LedgerEntry.cpp` (5199)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

f372585

* upstream/develop: Enforce levelization in libxrpl with CMake (5111)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

d670c79

* upstream/develop: Set version to 2.4.0-b1 fix: Add header for set_difference (5197) fix: allow overlapping types in `Expected` (5218) Add MPTIssue to STIssue (5200) Antithesis instrumentation improvements (5213)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

35acb44

* upstream/develop: chore: add macos dependency installation (5233) prefix Uint384 and Uint512 with Hash in server_definitions (5231) refactor: add `rpcName` to `LEDGER_ENTRY` macro (5202)

Apply suggestions from code review

1dcc061

Co-authored-by: Bronek Kozicki <[email protected]>

Fix: Work around compiler bug:

cb6daa2

* Older clang versions can't handle structured binding variables passed to lambas

ximinez requested a review from Bronek January 8, 2025 22:04

ximinez added 8 commits January 9, 2025 16:53

Merge remote-tracking branch 'upstream/develop' into pr/getledger

7894d8f

* upstream/develop: chore: update deprecated Github Actions (5241) XLS-46: DynamicNFT (5048)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

2090083

* upstream/develop: Fix CI unit tests (5196) Update secp256k1 library to 0.6.0 (5254)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

79f9bb8

* upstream/develop: Add deep freeze feature (XLS-77d) (5187)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

fd94b55

* upstream/develop: Amendment `fixFrozenLPTokenTransfer` (5227) Improve git commit hash lookup (5225)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

7e52abd

* upstream/develop: Updates Conan dependencies (5256)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

829e4f1

* upstream/develop: fix: Do not allow creating Permissioned Domains if credentials are not enabled (5275) fix: issues in `simulate` RPC (5265)

Merge remote-tracking branch 'upstream/develop' into pr/getledger

ca07176

* upstream/develop: fix: Amendment to add transaction flag checking functionality for Credentials (5250) fix: Omit superfluous setCurrentThreadName call in GRPCServer.cpp (5280)

Bronek reviewed Feb 11, 2025

View reviewed changes

Bronek approved these changes Feb 11, 2025

View reviewed changes

Bronek self-requested a review February 11, 2025 18:19

Bronek reviewed Feb 11, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into pr/getledger

39fe006

* upstream/develop: chore: Rename missing-commits job, and combine nix job files (5268) docs: Add a summary of the git commit message rules (5283)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce peer message traffic for ledger data #5126

Reduce peer message traffic for ledger data #5126

ximinez commented Sep 11, 2024 •

edited

Loading

codecov bot commented Sep 11, 2024 •

edited

Loading

Bronek Sep 13, 2024 •

edited

Loading

ximinez Sep 25, 2024

vlntb left a comment

vlntb Oct 10, 2024

ximinez Oct 11, 2024

vlntb Oct 10, 2024

ximinez Oct 11, 2024

Bronek left a comment

Bronek Nov 25, 2024

ximinez Jan 7, 2025

Bronek Dec 2, 2024

ximinez Jan 7, 2025

Bronek Dec 2, 2024 •

edited

Loading

ximinez Jan 7, 2025

ximinez Jan 7, 2025

Bronek Dec 2, 2024

ximinez Jan 7, 2025

Bronek Dec 2, 2024

bthomee commented Feb 10, 2025

Bronek Feb 11, 2025

Bronek Feb 11, 2025

Bronek Feb 11, 2025

Bronek Feb 11, 2025

Bronek left a comment

Bronek left a comment

Bronek commented Feb 11, 2025

Reduce peer message traffic for ledger data #5126

Are you sure you want to change the base?

Reduce peer message traffic for ledger data #5126

Conversation

ximinez commented Sep 11, 2024 • edited Loading

High Level Overview of Change

Context of Change

Type of Change

API Impact

Test Plan

Future Tasks

codecov bot commented Sep 11, 2024 • edited Loading

Codecov Report

Bronek Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlntb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bronek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bronek Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bthomee commented Feb 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bronek left a comment

Choose a reason for hiding this comment

Bronek left a comment

Choose a reason for hiding this comment

Bronek commented Feb 11, 2025

ximinez commented Sep 11, 2024 •

edited

Loading

codecov bot commented Sep 11, 2024 •

edited

Loading

Bronek Sep 13, 2024 •

edited

Loading

Bronek Dec 2, 2024 •

edited

Loading