Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RHCLOUD-29056 Upgrade debezium to 2.5 #356

Merged
merged 8 commits into from
Mar 27, 2024
Merged

RHCLOUD-29056 Upgrade debezium to 2.5 #356

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6cb12e8
RHCLOUD-29056 Upgrade debezium to 2.5
josejulio Mar 15, 2024
10ad572
`database.` is still used for these fields
josejulio Mar 15, 2024
f312463
Updating more related files
josejulio Mar 18, 2024
3a23cd8
Removing test files
josejulio Mar 18, 2024
7984e0d
Remove template-ref for pd-connector
josejulio Mar 26, 2024
3aceac3
Set template ref for pd-connect to current commit
josejulio Mar 26, 2024
be9ada1
Upgrade to debezium 2.5.3.Final
josejulio Mar 26, 2024
ad05825
Update docker-compose.yml
josejulio Mar 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deploy/connect-msk.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@ objects:
slot.name: debezium
plugin.name: pgoutput
slot.max.retries: 999999999
database.server.name: playbook-dispatcher
topic.prefix: playbook-dispatcher
table.include.list: public.runs,public.run_hosts
tombstones.on.delete: false

Expand Down
2 changes: 1 addition & 1 deletion deploy/connect.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ objects:
slot.name: debezium
plugin.name: pgoutput
slot.max.retries: 999999999
database.server.name: playbook-dispatcher
topic.prefix: playbook-dispatcher
table.include.list: public.runs,public.run_hosts
tombstones.on.delete: false

Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ services:
- kafka

db:
image: quay.io/debezium/postgres:12
image: quay.io/debezium/postgres:14
josejulio marked this conversation as resolved.
Show resolved Hide resolved
restart: always
Copy link
Member

@tahmidefaz tahmidefaz Mar 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, in PROD, I think we are using Postgres:12. Not sure if that will cause an issue with this one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll revert back to '12' and let the tests run with it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tried locally and still works fine.

environment:
POSTGRES_PASSWORD: insights
Expand Down
6 changes: 3 additions & 3 deletions event-streams/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ USER root

ENV EXTRA_PLUGINS=/deps/plugins \
EXTRA_LIBS=/deps/libs \
DEBEZIUM_VERSION=1.9.7.Final \
DEBEZIUM_CHECKSUM=161e362568163639fcde39da27e29456 \
DEBEZIUM_VERSION=2.5.3.Final \
DEBEZIUM_CHECKSUM=0e119e8fac1a0bc0c35c009786110508 \
CONFIG_PROVIDERS_VERSION=0.1.0 \
CONFIG_PROVIDERS_CHECKSUM=108e0bf4148a37676bed866ff45e1199

Expand All @@ -16,7 +16,7 @@ ADD schema/run.host.event.yaml /src
RUN microdnf install gzip

# Taken from https://github.com/debezium/docker-images/blob/master/connect-base/1.9/docker-maven-download.sh
# Debezium PostgreSQL checksum from https://github.com/debezium/container-images/blob/main/connect/1.9/Dockerfile
# Debezium PostgreSQL checksum from https://github.com/debezium/container-images/blob/main/connect/2.5/Dockerfile
RUN mkdir -p $EXTRA_PLUGINS $EXTRA_LIBS && \
MAVEN_DEP_DESTINATION=$EXTRA_PLUGINS /src/docker-maven-download.sh debezium postgres ${DEBEZIUM_VERSION} ${DEBEZIUM_CHECKSUM} && \
MAVEN_DEP_DESTINATION=$EXTRA_LIBS /src/docker-maven-download.sh central com/redhat/insights/kafka config-providers ${CONFIG_PROVIDERS_VERSION} ${CONFIG_PROVIDERS_CHECKSUM}
Expand Down
2 changes: 1 addition & 1 deletion event-streams/connector.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"slot.name": "debezium",
"plugin.name": "pgoutput",
"slot.max.retries": 999999999,
"database.server.name": "playbook-dispatcher",
"topic.prefix": "playbook-dispatcher",
"table.include.list": "public.runs,public.run_hosts",
"tombstones.on.delete": false,

Expand Down
21 changes: 20 additions & 1 deletion event-streams/docker-maven-download.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,27 @@ maven_apicurio_converter() {
return
fi
APICURIO_CONVERTER_PACKAGE="apicurio-registry-distro-connect-converter"
maven_dep $MAVEN_REPO_CENTRAL "io/apicurio" "$APICURIO_CONVERTER_PACKAGE" "$1" "$APICURIO_CONVERTER_PACKAGE-$1-converter.tar.gz" "$2"
maven_dep $MAVEN_REPO_CENTRAL "io/apicurio" "$APICURIO_CONVERTER_PACKAGE" "$1" "$APICURIO_CONVERTER_PACKAGE-$1.tar.gz" "$2"
mkdir "$EXTERNAL_LIBS_DIR/apicurio"
tar -xzf "$DOWNLOAD_FILE" -C "$EXTERNAL_LIBS_DIR/apicurio" && rm "$DOWNLOAD_FILE"
}

maven_otel_libs() {
if [[ -z "$EXTERNAL_LIBS_DIR" ]] ; then
echo "WARNING: EXTERNAL_LIBS_DIR is not set. Skipping loading OTEL libraries ..."
return
fi
if [[ ! -d "$EXTERNAL_LIBS_DIR" ]] ; then
echo "WARNING: EXTERNAL_LIBS_DIR is not a directory. Skipping loading OTEL libraries ..."
return
fi
if [[ ! -d "$EXTERNAL_LIBS_DIR/otel" ]] ; then
mkdir "$EXTERNAL_LIBS_DIR/otel"
fi
maven_dep $MAVEN_REPO_CENTRAL $1 $2 $3 "$2-$3.jar" $4
mv "$DOWNLOAD_FILE" $EXTERNAL_LIBS_DIR/otel
}

case $1 in
"central" ) shift
maven_central_dep ${@}
Expand All @@ -111,4 +127,7 @@ case $1 in
"apicurio" ) shift
maven_apicurio_converter ${@}
;;
"otel" ) shift
maven_otel_libs ${@}
;;
esac
2 changes: 1 addition & 1 deletion examples/connector-local.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"slot.name": "debezium",
"plugin.name": "pgoutput",
"slot.max.retries": 999999999,
"database.server.name": "playbook-dispatcher",
"topic.prefix": "playbook-dispatcher",
"table.include.list": "public.runs,public.run_hosts",

"key.converter": "org.apache.kafka.connect.storage.StringConverter",
Expand Down
4 changes: 2 additions & 2 deletions pr_check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ source $CICD_ROOT/build.sh

# IMAGE is set to the Connect image, setting dispatcher image as an extra arg
# hardcode connect to use a ref that works in ephemeral
EXTRA_DEPLOY_ARGS="--set-image-tag ${IMAGE_DISPATCHER}=${IMAGE_TAG} --set-template-ref ${CONNECT_COMPONENT_NAME}=709963354a31225591dd612f15c670a14ddf3c99"
EXTRA_DEPLOY_ARGS="--set-image-tag ${IMAGE_DISPATCHER}=${IMAGE_TAG} --set-template-ref ${CONNECT_COMPONENT_NAME}=${GIT_COMMIT}"

# Deploy to an ephemeral environment
source $CICD_ROOT/deploy_ephemeral_env.sh
Expand All @@ -48,7 +48,7 @@ bonfire deploy playbook-dispatcher cloud-connector \
--set-template-ref ${COMPONENT_NAME}=${GIT_COMMIT} \
--set-image-tag ${IMAGE_DISPATCHER}=${IMAGE_TAG} \
--set-image-tag ${IMAGE_CONNECT}=${IMAGE_TAG} \
--set-template-ref ${CONNECT_COMPONENT_NAME}=047e256da507f29d0e0ae803a4b1d688eb74a2cb \
--set-template-ref ${CONNECT_COMPONENT_NAME}=${GIT_COMMIT} \
--namespace ${NAMESPACE} \
--timeout ${DEPLOY_TIMEOUT} \
--set-parameter playbook-dispatcher/CLOUD_CONNECTOR_IMPL=impl
Expand Down
Loading