Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Docker-based tests of client installation package #200

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion client/.dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,5 @@ __pycache__
.pytest_cache
.idea
.mypy_cache
examples
node_modules
.coverage
25 changes: 16 additions & 9 deletions client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,29 @@ the distribution-specific commands to install these are:
* Red Hat-based distros: `sudo yum install python38 python38-pip` (for e.g. Python
3.8)

### GCP

You need the Google Cloud CLI client `gcloud` installed. [Read installation
steps for the `gcloud` CLI package.](https://cloud.google.com/sdk/docs/install)

You need to have credentials configured locally for a GCP user with suitable
permissions to perform batch translation. [Read how `gcloud` searches for
application default credentials.](https://cloud.google.com/docs/authentication/application-default-credentials)

You need a GCP project and a Google Cloud Storage bucket to use for uploading
your input SQL files and downloading the translated output. [Learn how to
create a GCS bucket manually][creating buckets], or see the [instructions for
using `provision.sh`](#running-using-runsh) to automatically provision a
bucket for translation.

### Support for Encodings other than UTF-8

If all of the files you wish to translate are UTF-8 encoded
(this is commonly the case), you can skip this section.
Otherwise, you will need to install additional system dependencies:

* Debian-based distros: `sudo apt install pkg-config libicu-dev`
* RedHat-based distros: `sudo yum install gcc gcc-c++ libicu-devel
* Red Hat-based distros: `sudo yum install gcc gcc-c++ libicu-devel
python38-devel`

**You must also remember**, upon reaching the step to `pip install` further down
Expand All @@ -64,14 +79,6 @@ in the Quickstart section below, to use this command instead:
pip install ../dwh-migration-tools/client[icu]
```

### GCP

You need a GCP project and a Google Cloud Storage bucket to use for uploading
your input SQL files and downloading the translated output. [Learn how to
create a GCS bucket manually][creating buckets], or see the [instructions for
using `provision.sh`](#running-using-runsh) to automatically provision a
bucket for translation.

## Quickstart

1. Download the repo from [google/dwh-migration-tools] in your choice of
Expand Down
31 changes: 31 additions & 0 deletions client/tests/packaging/Dockerfile-RedHat
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
FROM redhat/ubi9

# See https://cloud.google.com/sdk/docs/install
COPY tests/packaging/redhat-google-cloud-sdk.repo /etc/yum.repos.d/google-cloud-sdk.repo
RUN dnf install -y \
google-cloud-cli \
python3 \
python3-pip

WORKDIR /workspace
COPY . /workspace/dwh-migration-tools/client
COPY examples/teradata/sql /workspace/project

ARG ICU_ENABLED=false
RUN if [ "$ICU_ENABLED" = "true" ]; then \
dnf install -y \
gcc \
gcc-c++ \
libicu-devel \
python3-devel \
&& python3 -m pip install dwh-migration-tools/client[icu] ; \
else \
python3 -m pip install dwh-migration-tools/client ; \
fi

RUN dnf clean all -y && rm -rf /var/cache
WORKDIR /workspace/project
ENV BQMS_VERBOSE="True"
ENV BQMS_MULTITHREADED="True"

ENTRYPOINT [ "/workspace/project/run.sh" ]
32 changes: 32 additions & 0 deletions client/tests/packaging/Dockerfile-Ubuntu
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
FROM ubuntu:22.04

RUN apt-get update -y && apt-get install -y \
curl \
python3-pip
# See https://cloud.google.com/sdk/docs/install
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" \
| tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \
| apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - \
&& apt-get update -y \
&& apt-get install -y google-cloud-cli

WORKDIR /workspace
COPY . /workspace/dwh-migration-tools/client
COPY examples/teradata/sql /workspace/project

ARG ICU_ENABLED=false
RUN if [ "$ICU_ENABLED" = "true" ]; then \
apt-get install -y \
libicu-dev \
pkg-config \
&& python3 -m pip install dwh-migration-tools/client[icu] ; \
else \
python3 -m pip install dwh-migration-tools/client ; \
fi

RUN rm -rf /var/lib/apt/lists/*
WORKDIR /workspace/project
ENV BQMS_VERBOSE="True"
ENV BQMS_MULTITHREADED="True"

ENTRYPOINT [ "/workspace/project/run.sh" ]
7 changes: 7 additions & 0 deletions client/tests/packaging/redhat-google-cloud-sdk.repo
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[google-cloud-cli]
name=Google Cloud CLI
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el9-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
16 changes: 16 additions & 0 deletions client/tests/packaging/run-packaging-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/bin/bash -e

for BASE_OS in RedHat Ubuntu; do
for ICU_ENABLED in false true; do
docker build \
-t dwh-migration-tools-test \
-f Dockerfile-$BASE_OS \
--build-arg ICU_ENABLED=$ICU_ENABLED \
../..
docker run -it --rm \
-v ~/.config/gcloud:/root/.config/gcloud \
-e BQMS_PROJECT=$BQMS_PROJECT \
-e BQMS_GCS_BUCKET=$BQMS_GCS_BUCKET \
dwh-migration-tools-test
done
done