The Gradle gretl plugin extends gradle for use as a sql-centric (geo)data etl. gretl = gradle etl.
A german user manual can be found here: docs/user/
The Publisher task is documented here: Publisher
GRETL is licensed under the MIT License.
GRETL is in stable state.
For the current version of GRETL, you will need a JRE (Java Runtime Environment) installed on your system, version 1.8 or later and Gradle, version 5.1.1 or later. For convenience use the gradle wrapper.
The GRETL repository is organized as Gradle multi-project:
gretl: GRETL source code with unit tests and integration tests.runtimeImage: Subproject for building the GRETL runtime (docker) image. The docker image is tested against the integration tests, too.
There are still signs and wonders taking place: Since fall 2019 the Oracle JDBC library can be found on maven central. Oracle database support is now straight forward.
./gradlew clean gretl:classes
./gradlew gretl:test gretl:dbTest
./gradlew gretl:s3Test -Ds3AccessKey=XXXXXXX -Ds3SecretKey=YYYYYYY -Ds3BucketName=ch.so.agi.gretl.test
./gradlew gretl:build gretl:publishPluginMavenPublicationToMavenLocal gretl:publishGretlPluginPluginMarkerMavenPublicationToMavenLocal -x test
./gradlew stageJars
cd runtimeImage/gretl
docker build -t sogis/gretl .
./gradlew gretl:jarTest
./gradlew gretl:jarS3Test -Ds3AccessKey=XXXXXXX -Ds3SecretKey=YYYYYYY -Ds3BucketName=ch.so.agi.gretl.test
./gradlew gretl:imageTest
./gradlew gretl:imageS3Test -Ds3AccessKey=XXXXXXX -Ds3SecretKey=YYYYYYY -Ds3BucketName=ch.so.agi.gretl.test
./gradlew clean gretl:build gretl:publishPluginMavenPublicationToMavenLocal gretl:jarTest -x test
E.g. if you want to test only the jar and only one test:
./gradlew clean gretl:build gretl:publishPluginMavenPublicationToMavenLocal gretl:jarTest -x test --tests ch.so.agi.gretl.jobs.Av2chTest.transformation_Ok
If you write a new custom task and the integration test thinks your new task type is not available, delete the artifacts in your local maven repository once. It seems that after this clean up, changes in the code will be deployed to the local maven repo. Not sure what the problem is. Maybe an deployed version with additional plugin meta data.
If you want to do some further testing with standalone jobs or use the plugin on your local machine and use the new plugin dsl syntax, you need to deploy the plugin as "plugin":
./gradlew gretl:publishGretlPluginPluginMarkerMavenPublicationToMavenLocal
Debugging the Docker image tests can be harder than debugging unit and jar tests. The docker run / start-gretl.sh command is printed on the console when the test is running. Use this command to run the test in the console manually. Most probably you will get more information.
Unit tests with "heavy" dependencies like PostgreSQL are categorized (ch.so.agi.gretl.testutil.DbTest) and can be run with ./gradlew gretl:dbTest. This will manage the PostgreSQL database with the Testcontainers framework . Testcontainers start a docker container before every test method or every test class.
The integrations are used for testing the resulting Jar file (jarTest) and the Docker image (imageTest).
Since the integration tests are not done with the Gradle TestKit framework, the resulting Jar has to be deployed to the local maven repository (./gradlew clean gretl:build gretl:publishPluginMavenPublicationToMavenLocal). Then the Gradle build jobs (= integration tests) are run from a Java class as an external process. The jobs share all the same init.gradle that defines the maven repositories. Since GRETL is not published in any other than the local maven repository (except as plugin (!= raw jar) in the Gradle plugin repo), it should really always use the local deployed one. With classpath 'ch.so.agi:gretl:latest.integration' it will use the latest anything - snapshot or release, whatever it finds newer.
The Docker image tests are done very similar. First the Docker image will be build with a shell script (TODO: with Gradle?). This build process will copy everything GRETL needs into the image. This includes all the dependencies of GRETL, the GRETL jar itself and any 3rd party plugin you want. The Docker image should be as offline capable as possible. The start-gretl.sh that will be used to start the Docker container is slightly different to the start-gretl.sh from sogis/gretljobs repository. The sogis/gretljobs one is more sophisticated.
If you use a Jenkins-Docker-Image for your CI/CD pipeline you will probably run into the "Docker-in-Docker" issue when doing the Docker image integration tests. It will not find the job directory you try to mount with the docker run command. Therefor you can simple create a symbolic link on the host machine, e.g. ln -s /opt/jenkins_home /var/jenkins_home.
Instead of using a mocking library we to some real world testing with S3. Therefore a bucket must exist and the access key and secret key must be known. The keys used for CI belong to the user gretl (group: gretl-group, policy: gretl-s3).
The pipeline cannot run parallel since some tests write to S3 buckets. If something goes wrong they need to be cleaned manually (needs to be fixed...).
It uses a simple release management and versioning mechanism: Local builds are tagged as 2.2.LOCALBUILD. Builds on Github Action will append the build number, e.g. 2.2.230. Major version will be increased after "major" changes. After every commit to the repository a docker image will be build and pushed to hub.docker.com. It will be tagged as latest, with the build number (2.2.230) and with major and minor version number (2 and 2.2).
You have to bump the version number in three files:
- versioning.gradle
- runtimeImage/build.gradle
- .github/workflows/gretl.yml