Revise wait_for_integration ES implementation #12150
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Instead of polling Elasticsearch for the presence of index templates, attempt to create a data stream; this will fail if there is no index template with a matching index pattern. This decouples apm-server from the integration package, and will enable apm-server to alternatively work with the apm-data ES plugin. Instead of eagerly checking the preconditions (which includes making these attempts to create a data stream), we now delay the checks to the first attempt to index/publish events. This means that the "publish_ready" attribute of the server info API response will always be false until the first time events are sent to apm-server.
axw
force-pushed
the
check-integration-create-datastream
branch
from
c7a56e5 to
d9cfe5b
Compare
|
This pull request does not have a backport label. Could you fix it @axw? 🙏
NOTE: |
mergify
bot
added
the
backport-skip
axw
added
v8.12.0
and removed
backport-skip
mergify
bot
added
the
backport-skip
kruskall
approved these changes
Dec 4, 2023
kruskall
reviewed
Dec 4, 2023
| @@ -889,7 +899,13 @@ func (s *Runner) newLibbeatFinalBatchProcessor( | |||
| } | |||
| return acker.Wait(ctx) | |||
| } | |||
| return publisher, stop, nil | |||
| processor := modelprocessor.Chained{ | |||
| modelpb.ProcessBatchFunc(func(ctx context.Context, batch *modelpb.Batch) error { | |||
There was a problem hiding this comment.
question: what's the reason for this ? Isn't this handled by the es client ?
There was a problem hiding this comment.
This one is for libbeat outputs, which is anything other than Elasticsearch.
axw
added a commit
to axw/apm-server
that referenced
this pull request
Dec 19, 2023
This reverts commit 3e54384.
|
Removing the test label, as this has been reverted in #12240 |
simitt
pushed a commit
that referenced
this pull request
Dec 19, 2023
mergify bot
added a commit
that referenced
this pull request
Dec 19, 2023
…" (backport #12240) (#12249) * Revert "Revise wait_for_integration ES implementation (#12150)" (#12240) This reverts commit 3e54384. (cherry picked from commit 54e367c) # Conflicts: # internal/beater/checkintegration.go * Fix merge conflict --------- Co-authored-by: Andrew Wilkins <[email protected]>
v1v
added a commit
to v1v/apm-server
that referenced
this pull request
Dec 21, 2023
…lease-process * upstream/main: Remove Nassim Kammah from list of maintainers (elastic#12269) docs: Add tagged regions to changelog (elastic#12251) Update apm data (elastic#12242) Revert "Revise wait_for_integration ES implementation (elastic#12150)" (elastic#12240)
v1v
added a commit
to v1v/apm-server
that referenced
this pull request
Dec 21, 2023
…r-elastic * upstream/main: [updatecli] Bump elastic stack version to 8.13.0-yufkxnwm (elastic#12246) Remove Nassim Kammah from list of maintainers (elastic#12269) docs: Add tagged regions to changelog (elastic#12251) Update apm data (elastic#12242) Revert "Revise wait_for_integration ES implementation (elastic#12150)" (elastic#12240) ci(release): automate release steps (elastic#12234) build(deps): bump the github-actions group with 1 update (elastic#12232) [updatecli] Update to elastic/beats@a633696068ed (elastic#12164) [updatecli] Bump elastic stack version to 8.13.0-h30gube9 (elastic#12235) smoke-test: semver sort latest versions (elastic#12229) [updatecli] Bump Golang version to 1.21.5 (elastic#12191) fix: always try to remap both span and error stacktraces (elastic#12221) [updatecli] Bump elastic stack version to 8.13.0-6av99u5d (elastic#12169) docs: Update changelogs for 8.12 release (elastic#12172)
amannocci
pushed a commit
that referenced
this pull request
Dec 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
Instead of polling Elasticsearch for the presence of index templates, attempt to create a data stream; this will fail if there is no index template with a matching index pattern. This decouples apm-server from the integration package, and will enable apm-server to alternatively work with the apm-data ES plugin.
Instead of eagerly checking the preconditions (which includes making these attempts to create a data stream), we now delay the checks to the first attempt to index/publish events. This means that the "publish_ready" attribute of the server info API response will always be false until the first time events are sent to apm-server.
Checklist
- [ ] Update package changelog.yml (only if changes toapmpackagehave been made)- [ ] Documentation has been updatedHow to test these changes
Related issues
#11528