Skip to content

Commit

Permalink
Add Docker-based tests of client installation package.
Browse files Browse the repository at this point in the history
  • Loading branch information
arotenberg-google committed Apr 17, 2023
1 parent 2511c40 commit bab8a7d
Show file tree
Hide file tree
Showing 4 changed files with 88 additions and 9 deletions.
25 changes: 16 additions & 9 deletions client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,29 @@ the distribution-specific commands to install these are:
* Red Hat-based distros: `sudo yum install python38 python38-pip` (for e.g. Python
3.8)

### GCP

You need the Google Cloud CLI client `gcloud` installed. [Read installation
steps for the `gcloud` CLI package.](https://cloud.google.com/sdk/docs/install)

You need to have credentials configured locally for a GCP user with suitable
permissions to perform batch translation. [Read how `gcloud` searches for
application default credentials.](https://cloud.google.com/docs/authentication/application-default-credentials)

You need a GCP project and a Google Cloud Storage bucket to use for uploading
your input SQL files and downloading the translated output. [Learn how to
create a GCS bucket manually][creating buckets], or see the [instructions for
using `provision.sh`](#running-using-runsh) to automatically provision a
bucket for translation.

### Support for Encodings other than UTF-8

If all of the files you wish to translate are UTF-8 encoded
(this is commonly the case), you can skip this section.
Otherwise, you will need to install additional system dependencies:

* Debian-based distros: `sudo apt install pkg-config libicu-dev`
* RedHat-based distros: `sudo yum install gcc gcc-c++ libicu-devel
* Red Hat-based distros: `sudo yum install gcc gcc-c++ libicu-devel
python38-devel`

**You must also remember**, upon reaching the step to `pip install` further down
Expand All @@ -64,14 +79,6 @@ in the Quickstart section below, to use this command instead:
pip install ../dwh-migration-tools/client[icu]
```

### GCP

You need a GCP project and a Google Cloud Storage bucket to use for uploading
your input SQL files and downloading the translated output. [Learn how to
create a GCS bucket manually][creating buckets], or see the [instructions for
using `provision.sh`](#running-using-runsh) to automatically provision a
bucket for translation.

## Quickstart

1. Download the repo from [google/dwh-migration-tools] in your choice of
Expand Down
32 changes: 32 additions & 0 deletions client/tests/packaging/Dockerfile-RedHat
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
ARG ICU_ENABLED=false

FROM redhat/ubi9 AS dwh-migration-tools-base
RUN dnf install -y \
git \
python3 \
python3-pip
# See https://cloud.google.com/sdk/docs/install
COPY redhat-google-cloud-sdk.repo /etc/yum.repos.d/google-cloud-sdk.repo
RUN dnf install -y google-cloud-cli
WORKDIR /workspace
RUN git clone -b main https://github.com/google/dwh-migration-tools \
&& cp -R dwh-migration-tools/client/examples/teradata/sql project

FROM dwh-migration-tools-base AS dwh-migration-tools-icu-false
RUN python3 -m pip install dwh-migration-tools/client

FROM dwh-migration-tools-base AS dwh-migration-tools-icu-true
RUN dnf install -y \
gcc \
gcc-c++ \
libicu-devel \
python3-devel
RUN python3 -m pip install dwh-migration-tools/client[icu]

FROM dwh-migration-tools-icu-${ICU_ENABLED}
RUN dnf clean all -y && rm -rf /var/cache
WORKDIR /workspace/project
ENV BQMS_VERBOSE="True"
ENV BQMS_MULTITHREADED="True"

ENTRYPOINT [ "/workspace/project/run.sh" ]
33 changes: 33 additions & 0 deletions client/tests/packaging/Dockerfile-Ubuntu
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
ARG ICU_ENABLED=false

FROM ubuntu:22.04 AS dwh-migration-tools-base
RUN apt-get update -y && apt-get install -y \
curl \
git \
python3-pip
# See https://cloud.google.com/sdk/docs/install
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" \
| tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \
| apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - \
&& apt-get update -y \
&& apt-get install -y google-cloud-cli
WORKDIR /workspace
RUN git clone -b main https://github.com/google/dwh-migration-tools \
&& cp -R dwh-migration-tools/client/examples/teradata/sql project

FROM dwh-migration-tools-base AS dwh-migration-tools-icu-false
RUN python3 -m pip install dwh-migration-tools/client

FROM dwh-migration-tools-base AS dwh-migration-tools-icu-true
RUN apt-get install -y \
libicu-dev \
pkg-config
RUN python3 -m pip install dwh-migration-tools/client[icu]

FROM dwh-migration-tools-icu-${ICU_ENABLED}
RUN rm -rf /var/lib/apt/lists/*
WORKDIR /workspace/project
ENV BQMS_VERBOSE="True"
ENV BQMS_MULTITHREADED="True"

ENTRYPOINT [ "/workspace/project/run.sh" ]
7 changes: 7 additions & 0 deletions client/tests/packaging/redhat-google-cloud-sdk.repo
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[google-cloud-cli]
name=Google Cloud CLI
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el9-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

0 comments on commit bab8a7d

Please sign in to comment.