Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dev): Upgrade kafka and zookeeper to support Apple M1 #28574

Merged
merged 6 commits into from
Sep 15, 2021
Merged

feat(dev): Upgrade kafka and zookeeper to support Apple M1 #28574

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
71e8c82
feat(ci): Upgrade kafka and zookeeper
armenzg Sep 14, 2021
c60a0d9
Merge branch 'master' into armenzg/dev/upgrade-kafka-zookeeper
armenzg Sep 15, 2021
eb8c165
Match to production versions
armenzg Sep 15, 2021
9cdb788
Update version
armenzg Sep 15, 2021
8edf6e9
Fix comment location
armenzg Sep 15, 2021
25ffa5d
Update comment
armenzg Sep 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actions/setup-sentry/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,12 +158,16 @@ runs:

# TODO: Use devservices kafka. See https://github.com/getsentry/sentry/pull/20986#issuecomment-704510570
if [ "$NEED_KAFKA" = "true" ]; then
# This is *not* the production version. Unclear reason as to why this was chosen
# https://github.com/getsentry/ops/blob/c823e62f930ecc6c97bb08898c71e49edc7232f6/cookbooks/getsentry/attributes/default.rb#L631
docker run \
--name sentry_zookeeper \
-d --network host \
-e ZOOKEEPER_CLIENT_PORT=2181 \
confluentinc/cp-zookeeper:4.1.0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the CI was running an older version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What version do we run in production?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we run 4.1.2-3 and version 3.4.11 for zookeeper.

Should I make the CI test against those versions?

As curiosity, we run a range of versions across the org: https://github.com/search?q=org%3Agetsentry+cp-kafka&type=code

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think we should have CI match production, let's leave comments here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@billyvg I've discovered that version of zookeeper we use in production is no longer available on Docker hub, thus, we cannot match the versions between CI and production.

FYI, this part of the PR is completely unnecessary to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asked about the versions picked in #20543 (review)


# This is the production version; do not change w/o changing it there as well
# https://github.com/getsentry/ops/blob/c823e62f930ecc6c97bb08898c71e49edc7232f6/cookbooks/getsentry/attributes/default.rb#L643
docker run \
--name sentry_kafka \
-d --network host \
Expand Down
7 changes: 5 additions & 2 deletions src/sentry/conf/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1726,15 +1726,18 @@ def build_cdc_postgres_init_db_volume(settings):
),
"zookeeper": lambda settings, options: (
{
"image": "confluentinc/cp-zookeeper:5.1.2",
# Upgrading to version 6.x allows zookeeper to run properly on Apple's arm64
# See details https://github.com/confluentinc/kafka-images/issues/80#issuecomment-855511438
"image": "confluentinc/cp-zookeeper:6.2.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we leave comments here about the versions and why it differs from CI/prod

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

"environment": {"ZOOKEEPER_CLIENT_PORT": "2181"},
"volumes": {"zookeeper": {"bind": "/var/lib/zookeeper"}},
"only_if": "kafka" in settings.SENTRY_EVENTSTREAM or settings.SENTRY_USE_RELAY,
}
),
"kafka": lambda settings, options: (
{
"image": "confluentinc/cp-kafka:5.1.2",
# We upgrade to version 6.x to match zookeeper's version (I believe they both release together)
"image": "confluentinc/cp-kafka:6.2.0",
"ports": {"9092/tcp": 9092},
"environment": {
"KAFKA_ZOOKEEPER_CONNECT": "{containers[zookeeper][name]}:2181",
Expand Down