SOL-112355: Integration tests to cover the processing of requests and responses

[oodigie]

Changes:

• SOL-112354: Start and termination of a RequestReplyMessageReceiver integration tests
• feat: updated the tests to fix potential race conditions with the use of time.Millisecond
• feat: added an assertion to check the error returned from replier.Reply in one of the tests
• feat: fixed the failing tests to check that error returns from replier.Reply after service termination
• SOL-112355: Integration tests to cover the requests and responses processing using the RequestReplyMessagePublisher and RequestReplyMessageReceiver
• SOL-112355: updated the integration tests to provide a more consistent test outcome

[cjwmorgan-sol]

There is a failing test due to a receiver configuration issue.
@oodigie please address it.

[cjwmorgan-sol]

With PR #43 merged this PR might need to sync with the base feature branch SOL-62455 now.

[oodigie]

I had hoped it wouldn't but I had to make an update from the #43 for this PR so I guess it caused the conflict. I will resolve the conflict now. And yes this PR (#44) has all the changes from #43

[cjwmorgan-sol]

Changes look good and git actions passed.
But jenkins show request reply failure so there must some issue in here.
@oodigie please take look.
Since these test pass on relatively clean nodes I suspect this is timing related on busy nodes.

[oodigie]

@cjwmorgan-sol,
I am currently publishing 1000 messages for these tests and I also configured the receiver back pressure to have a buffer of 1000 messages (to accommodate these messages). The two test failures are due to timeout waiting for the reply for the published messages - "Timed out waiting to receive reply message". Apparently I do not see this failures on my local machine so it may be due to stress of the test nodes and/or available compute resource.

Seeing as the average time to typically receive a reply per message is about < 1 second, probably < 100 milliseconds (changes per version of the broker), I will either have to increase the timeout to allow receiving all the replies or reduce the amount of messages being published. I will reduce the total published message to 500.

[cjwmorgan-sol]

All check passing in git actions, and the only jenkins failure were independent of these changes (all known intermittent).
This looks good to me.
And is ready for merge.