Implement Job Scheduler logic

[f-galland]

Description

This PR adds job scheduler logic to the command-manager plugin.

Issues Resolved

Resolves #87

[mcasas993]

LGTM

[mcasas993]

When I tested this branch with ./gradlew run or ./gradlew run --debug after a couple of minutes this error appears in the terminal:

[2024-11-07T09:48:36,565][ERROR][o.o.a.s.TransportCreatePitAction] [integTest-0] PIT creation failed while updating PIT ID for indices [[.commands]]
[2024-11-07T09:48:36,565][ERROR][c.w.c.j.PointInTime      ] [integTest-0] all shards failed

This is the complete log file:
integTest.log

[f-galland]

An issue was observed where the plugin entered an infinite loop whenever API calls were received whilst outgoing http requests where being performed.
The following logs would be output in quick succession:

[2024-11-07T12:03:52,564][ERROR][c.w.c.j.PointInTime      ] [integTest-0] all shards failed
[2024-11-07T12:03:52,566][ERROR][o.o.a.s.TransportCreatePitAction] [integTest-0] PIT creation failed while updating PIT ID for indices [[.commands]]

I haven't been able to pin point the issue with exactitude, but I took the following measures to try and fix it:

• Made all the job scheduler related code synchronous by avoiding ActionListeners wherever possible.
  In cases where this was not possible (ie. for methods like client.createPit() only have an async version), I forced synchronous execution by blocking until the return value is obtained.
• I also found that a PointInTimeBuilder was being instantiated per-page, instead of per job executuion.
• Lastly, the SearchJob class was Singleton, but it seems there is no need for this.

With these changes, the problem seems to not manifest itself anymore. However this brings new issues as we now need to handle the lifecycle of the SearchJob object. I will research how this is usually done in opensearch and report back.