JBTM-3971 Set the proper state while recovering timed out LRAs #27

marcosgopen · 2025-01-09T15:15:13Z

https://issues.redhat.com/browse/JBTM-3971

When shortLRAs time out when the server is down the abortLRA method is not called. When the server restarts, the scheduler is not restored because already timed out and the LRA status keeps being cancelling indefinitely

Second commit is related to https://issues.redhat.com/browse/JBTM-3966

normal scheduler for cancelletion of short LRAs: https://github.com/jbosstm/lra/blob/main/coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java#L1105

the scheduler sets the status to cancelled when lists are empty: https://github.com/jbosstm/lra/blob/main/coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java#L1189

when a short LRA times out while the server is down that method is not called, and when it is restored the status goes to cancelling: https://github.com/jbosstm/lra/blob/main/coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java#L322

Eventually when the shortLRA is cancelled the status keeps being cancelling when it is considered finished : https://github.com/jbosstm/lra/blob/main/coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java#L502-L506

Please note that if a LRA is in cancelling status but all the lists are empty it is considered finished: https://github.com/jbosstm/lra/blob/main/coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java#L427-L441

jbosstm-bot · 2025-01-09T15:20:20Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/98/

jbosstm-bot · 2025-01-09T15:20:38Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/98/

jbosstm-bot · 2025-01-09T15:41:33Z

LRA profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/98/): LRA Test failed with failures in arq profile

jbosstm-bot · 2025-01-09T15:41:48Z

JACOCO profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/98/): LRA Test failed with failures in arq profile

jbosstm-bot · 2025-01-09T15:45:22Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/99/

jbosstm-bot · 2025-01-09T15:45:38Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/99/

jbosstm-bot · 2025-01-09T15:46:46Z

JACOCO profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/99/): narayana build failed

mmusgrov

Nice fix - the PR is still marked draft but I'm approving it anyway.

[Edit] I asked for a re-review. Please let me know when you are happy with your fix.

jbosstm-bot · 2025-01-09T15:59:04Z

LRA profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/99/): LRA Test failed with failures in arq profile

marcosgopen · 2025-01-09T17:14:30Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

-                .put(null);
-
-        Assert.assertEquals("LRA participant action", 200, response.getStatus());
+        Long SHORTER_TIMEOUT = 1000L;


Using here a shorter timeout (1 sec) so that the shortLRA times out when the server is down.

jbosstm-bot · 2025-01-09T17:15:17Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/100/

jbosstm-bot · 2025-01-09T17:15:32Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/100/

jbosstm-bot · 2025-01-09T17:35:53Z

JACOCO profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/100/

jbosstm-bot · 2025-01-09T17:35:58Z

LRA profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/100/

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

tomjenkinson · 2025-01-10T09:34:51Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java


        for (Map.Entry<String, String> entry : containerDeploymentMap.entrySet()) {
            stopContainer(entry.getKey(), entry.getValue());
        }
+        clearRecoveryLogFromFS();


I understand this is clean up after the server is stopped so it's fine with me for this to move. I would be tempted to see if it's worth (in a follow up) being able to assert a fail if there is something there but this is just a thought and maybe some tests would still expect some file system records

Good point Tom. I think we wanted this clean up to prevent a test failure from contaminating following tests and appear (e.g. in the CI) that multiple tests have failed.

Just to be clear I see this topic as different to #27 (comment) but I will resolve this one

Actually, I am not mandating the fail checks being added but to add readability of my review I will unresolve it but it does not need addressing IMO

tomjenkinson · 2025-01-10T09:36:05Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

        clearRecoveryLogFromObjectStore();
-        clearRecoveryLogFromFS();


I am OK with this being moved before the start of the container, it seems clean to me to do that. I would be tempted to fail the test if there was something in the objectstore after starting the new test (or make the assumption there should be nothing and remove the call to clearRecoveryLogFromObjectStore but these are remarks rather than requests for change

Thanks Tom, I think this is related to my previous comment. In the CI we have the --fail-at-end flag when running tests. So we want to make sure that a failure is not impacting the following tests.

I guess part of what I am saying though is that I can't imagine a way that clearRecoveryLogFromObjectStore should have found a transaction and if it did something in Narayana is broken and needs fixing so a "fail()" would seem appropriate

Anyway, it was a remark so I will resolve this

Actually, I am not mandating the fail checks being added but to add readability of my review I will unresolve it but it does not need addressing IMO

coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java

tomjenkinson · 2025-01-10T09:46:18Z

coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java

@@ -319,7 +319,12 @@ public boolean restore_state(InputObjectState os, int ot) {
                    }

                    if (status == LRAStatus.Active) {
-                        status = LRAStatus.Cancelling; // transition from Active to Cancelling
+                        if (getSize(pendingList) == 0 && getSize(preparedList) == 0 && getSize(heuristicList) == 0) {
+                            updateState(LRAStatus.Cancelled);


Please can you share the pre-crash condition for the LRA such that (getSize(pendingList) == 0 && getSize(preparedList) == 0 && getSize(heuristicList) == 0) but the status was Active? Is this basically an LRA that had been created but no participants registered so we are just moving it to Cancelled as the TTL is expired. I guess so but if so maybe it's worth a comment because it at least made my curious to know how this could happen.

That is the case tested here: https://github.com/jbosstm/lra/pull/27/files#diff-1a03090f8ce8ff7635f6bc880a133ff1d5f01a56955be2da86c33199963bf879R151
You are right, it is the case where no participants are registered and the LRA is active. For instance if we create a shortLRA and then the server crashes before the participants are registered, then when the server restarts the LRA is already expired.

Thank you, I think it might be worth a comment in the code just to mention that as it's not immediately obvious but I suppose it's not essential to do that. Thank you for the answer on here, though :)

@marcosgopen please can I ask you to confirm if this change is related in some way to JBTM-3966 or is it unrelated?

Actually I think the two commits in the pull request should be prefixed with their numbers and then this would be clearer - is that OK with you, please?

And would the second commit be OK in it's own pull request or does it require the first commit? If so I think some linkage between https://issues.redhat.com/browse/JBTM-3966 and https://issues.redhat.com/browse/JBTM-3971 should be made in the JIRAs? I am not sure which but maybe JBTM-3966 should be marked as blocked by JBTM-3971?

I think from the JIRA linkage now it can now be understood that JBTM-3971 really should be part of the same pull request (or merged first, so that JBTM-3966 will be more predictable) - thank you!

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

tomjenkinson · 2025-01-10T09:48:37Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java


+        // The LRA transaction could have been started via lraClient but it is useful to


This statement is a little confusing because on the new line 155 lraClient is being used

Actually, I see where it was moved from in the old diff and my thought is that the intention of the test before was to use the annotated client directly. I am not -1 to change to using lraClient (as long as we have tests for the filters elsewhere) but it seems like this comment is no longer appropriate?

Good catch, I'll remove that comment.
As I mentioned above I will restore (with little changes) the previous test testing he annotated client.

OK, thank you!

tomjenkinson · 2025-01-10T09:51:18Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

-        // Verifies that the resource was notified that the LRA finished
-        String listenerStatus = getStatusFromListener(lraListenerURI);
+        Assert.assertTrue(String.format("LRA %s should have cancelled", shortLRA.toString()),
+                status == null || status == LRAStatus.Cancelled);


Is it possible for status to be null? What is the condition for that to happen, please?

It might be null only in case the server has not recovered the LRA yet. Since we have the wait() call before that we could harden the test so that we test the recovery is finished and the LRA is not null.

I think a "status == null" should be considered a failure if it means the server should know something about the LRA but does not

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

tomjenkinson · 2025-01-10T09:54:01Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

@@ -247,7 +213,6 @@ public void lraCoordinatorRecoveryTwoLRAs(@ArquillianResource @OperateOnDeployme
        }

        lraClient.closeLRA(longLRA);
-        lraClient.closeLRA(shortLRA);


Please can you mention why removing this call is definitely fine now?

There was a mistake in the test: the waiting time (doWait) was not enough to trigger the shortLRA timeout and so it needed to be closed manually.
SHORT_TIMEOUT is 10.000
LRAListener.LRA_SHORT_TIMELIMIT is 5L
(LRAListener.LRA_SHORT_TIMELIMIT +1L ) * 1000 = 6000

Now the shortLRA is already closed at the end of the test. I think now it makes more sense.
BTW calling closeLRA to a cancelled LRA returns a 412 error response (precondition failed)

This makes sense to me now. So if the LRA was cancelled (rather than cancelling) before these changes, this line would have caused a test failure.

Yes, I believe so.

There have been a lot of changes since the version I approved and I'm not clear on why so many changes are needed.

jbosstm-bot · 2025-01-10T10:20:21Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/101/

jbosstm-bot · 2025-01-10T10:20:38Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/101/

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

jbosstm-bot · 2025-01-10T10:41:09Z

LRA profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/101/): LRA Test failed with failures in arq profile

jbosstm-bot · 2025-01-10T10:41:16Z

JACOCO profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/101/

tomjenkinson · 2025-01-10T11:20:50Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

+        if (shortStatus == LRAStatus.Cancelling) {
+            recover();
+            shortStatus = getStatus(shortLRA);
+        }


@marcosgopen please can you clarify why this block is a better approach than the previous block: https://github.com/jbosstm/lra/pull/27/files#diff-1a03090f8ce8ff7635f6bc880a133ff1d5f01a56955be2da86c33199963bf879L221-L227 they look to be somewhat similar to me.

After some recent changes (wildfly/wildfly#18499) the recovery is now triggered by the server startup, so it should not need to be triggered manually. (Edited)

OK, that's clear then. I think I would say "should not need to be triggered manually" but that is clearer now to why it's different. Thank you

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

When shortLRAs time out when the server is down the abortLRA method is not called. When the server restart the scheduler is not restored because already timed out and the LRA status keeps being cancelling or closing indefinitely

jbosstm-bot · 2025-01-10T11:40:22Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/102/

jbosstm-bot · 2025-01-10T11:40:38Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/102/

jbosstm-bot · 2025-01-10T11:59:46Z

JACOCO profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/102/

jbosstm-bot · 2025-01-10T12:02:31Z

LRA profile tests failed (https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/102/): LRA Test failed with failures in arq profile

tomjenkinson · 2025-01-10T12:13:21Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

+        // wait almost until the LRA is timed out)
+        doWait((LRAListener.LRA_SHORT_TIMELIMIT * 1000) - 500);


Please can I check if it is mandatory to wait here before restarting? If so perhaps the comment could be expanded to clarify why?

Good idea, I'll expand my comment

tomjenkinson · 2025-01-10T12:18:55Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

@@ -163,36 +186,19 @@ public void lraCoordinatorShortTimeoutLRA(@ArquillianResource @OperateOnDeployme

        Assert.assertEquals("LRA participant action", 200, response.getStatus());

+        // The LRA transaction could have been started via lraClient but it is useful to


Please can I ask why this was moved before the restart of the container? This might end up a duplicate discussion of https://github.com/jbosstm/lra/pull/27/files#r1910297173 but I am not sure

jbosstm-bot · 2025-01-10T12:20:20Z

Started testing this pull request with JACOCO profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/103/

jbosstm-bot · 2025-01-10T12:20:34Z

Started testing this pull request with LRA profile: https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/103/

tomjenkinson · 2025-01-10T12:21:59Z

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java

-
-        // LRA with short timeout should have timed out and cancelled
-        status = getStatus(new URI(lraId));
-


I understand why this is being removed - it's the same reason as https://github.com/jbosstm/lra/pull/27/files#r1910235210

But I am wondering if it might be useful to add the replacement you made there (or something similar) because the doWait((LRAListener.LRA_SHORT_TIMELIMIT * 1000) - 500); (on line 194) is not long enough to timeout the LRA and even though when combined with the time restartContainer(LRA_COORDINATOR_CONTAINER_QUALIFIER); (from 196) takes, it is probably long enough to timeout the LRA, it might not be and so the LRAStatus status = getStatus(new URI(lraId)); on (now) line 200 might find the LRA Cancelling still and so the assert on 202-203 might fail?

jbosstm-bot · 2025-01-10T12:39:18Z

JACOCO profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=JACOCO,jdk=openJDK17,label=jnlp-agent/103/

jbosstm-bot · 2025-01-10T12:41:06Z

LRA profile tests passed - Job complete https://ci-jenkins-csb-narayana.apps.ocp-c1.prod.psi.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/103/

tomjenkinson · 2025-01-13T12:12:51Z

coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java

+                        }
+                        else {
+                            updateState(LRAStatus.Cancelling);
+                        }


There is a question if we can move this to new code to use the scheduleCancellation function somehow. As I understand it abortLRA causes a locking issue but scheduleCancellation would hopefully be in a different thread.

mmusgrov · 2025-01-13T15:45:34Z

Please note that if a LRA is in cancelling status but all the lists are empty it is considered finished:

The spec [1] says that a participant can ask to be notified of the outcome of the LRA via the @AfterLRA annotation as long as the LRA hasn't reached one of the terminal states.

[1] https://download.eclipse.org/microprofile/microprofile-lra-2.0/microprofile-lra-spec-2.0.html#_quick_overview_of_annotations
[2] https://github.com/eclipse/microprofile-lra/blob/2.0/api/src/main/java/org/eclipse/microprofile/lra/annotation/AfterLRA.java#L37

marcosgopen · 2025-01-16T17:04:16Z

I put this on hold while I investigate further the issue and address your comments.

marcosgopen force-pushed the JBTM-3966_fix branch from 627193f to 37676a8 Compare January 9, 2025 15:43

mmusgrov previously approved these changes Jan 9, 2025

View reviewed changes

marcosgopen force-pushed the JBTM-3966_fix branch from 37676a8 to a4b7d66 Compare January 9, 2025 17:10

marcosgopen commented Jan 9, 2025

View reviewed changes

mmusgrov self-requested a review January 9, 2025 17:20

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java Outdated Show resolved Hide resolved

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

coordinator/src/main/java/io/narayana/lra/coordinator/domain/model/LongRunningAction.java Show resolved Hide resolved

marcosgopen marked this pull request as ready for review January 10, 2025 09:42

marcosgopen mentioned this pull request Jan 10, 2025

JBTM-3966 fix crash tests #26

Closed

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java Outdated Show resolved Hide resolved

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java Outdated Show resolved Hide resolved

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

marcosgopen changed the title ~~Extra check for ShortLRA~~ JBTM-3971 Set the proper state while recovering timed ont LRAs Jan 10, 2025

marcosgopen force-pushed the JBTM-3966_fix branch 2 times, most recently from e0009a4 to 28c8ded Compare January 10, 2025 10:17

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java Show resolved Hide resolved

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

test/crash/src/test/java/io/narayana/lra/arquillian/LRACoordinatorRecoveryIT.java Outdated Show resolved Hide resolved

marcosgopen force-pushed the JBTM-3966_fix branch from 28c8ded to 616d01f Compare January 10, 2025 11:38

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

JBTM-3966 fix LRA crash tests

b748699

marcosgopen force-pushed the JBTM-3966_fix branch from 616d01f to b748699 Compare January 10, 2025 12:15

tomjenkinson reviewed Jan 10, 2025

View reviewed changes

marcosgopen changed the title ~~JBTM-3971 Set the proper state while recovering timed ont LRAs~~ JBTM-3971 Set the proper state while recovering timed out LRAs Jan 10, 2025

tomjenkinson reviewed Jan 13, 2025

View reviewed changes

marcosgopen added the Hold label Jan 16, 2025


		// The LRA transaction could have been started via lraClient but it is useful to

		// wait almost until the LRA is timed out)
		doWait((LRAListener.LRA_SHORT_TIMELIMIT * 1000) - 500);

		@@ -163,36 +186,19 @@ public void lraCoordinatorShortTimeoutLRA(@ArquillianResource @OperateOnDeployme

		Assert.assertEquals("LRA participant action", 200, response.getStatus());

		// The LRA transaction could have been started via lraClient but it is useful to


		// LRA with short timeout should have timed out and cancelled
		status = getStatus(new URI(lraId));

JBTM-3971 Set the proper state while recovering timed out LRAs #27

Are you sure you want to change the base?

JBTM-3971 Set the proper state while recovering timed out LRAs #27

Conversation

marcosgopen commented Jan 9, 2025 • edited Loading

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

mmusgrov left a comment • edited Loading

Choose a reason for hiding this comment

jbosstm-bot commented Jan 9, 2025

Choose a reason for hiding this comment

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

jbosstm-bot commented Jan 9, 2025

Choose a reason for hiding this comment

marcosgopen Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosgopen Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosgopen Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

Choose a reason for hiding this comment

marcosgopen Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbosstm-bot commented Jan 10, 2025

jbosstm-bot commented Jan 10, 2025

Choose a reason for hiding this comment

mmusgrov commented Jan 13, 2025

marcosgopen commented Jan 16, 2025

marcosgopen commented Jan 9, 2025 •

edited

Loading

mmusgrov left a comment •

edited

Loading

marcosgopen Jan 10, 2025 •

edited

Loading

marcosgopen Jan 10, 2025 •

edited

Loading

marcosgopen Jan 10, 2025 •

edited

Loading

marcosgopen Jan 10, 2025 •

edited

Loading