PLUGIN-6: Added Jethro source, action and post-run action plugins.

jethro-plugin/docs/JethroAction-action.md

@@ -0,0 +1,42 @@
+# Jethro Action
+
+
+Description
+-----------
+Action that runs a Jethro command.
+
+
+Use Case
+--------
+The action can be used whenever you want to run a Jethro command before a data pipeline.
+For example, you may want to run a sql command on a database before the pipeline source pulls data from tables.
+
+
+Properties
+----------
+**Driver Name:** Name of the JDBC driver to use.
+
+**Host:** Host that Jethro is running on.
+
+**Port:** Port that Jethro is running on.
+
+**Instance:** Jethro database name.
+
+**Database Command:** Database command to execute.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+Example
+-------
+Suppose you want to execute a query against a Jethro database instance named "prod" that is running on "localhost" 
+port 9112, then configure the plugin with:
+
+```
+Driver Name: "jethro"
+Host: "localhost"
+Port: 9112
+Database: "prod"
+Database Command: "TRUNCATE TABLE testTable"
+```

jethro-plugin/docs/JethroPostAction-postaction.md

@@ -0,0 +1,51 @@
+# Jethro Action
+
+
+Description
+-----------
+Runs a Jethro query at the end of the pipeline run.
+Can be configured to run only on success, only on failure, or always at the end of the run.
+
+
+Use Case
+--------
+The action is used whenever you need to run a query at the end of a pipeline run.
+For example, you may have a pipeline that imports data from a database table to
+hdfs files. At the end of the run, you may want to run a query that deletes the data
+that was read from the table.
+
+
+Properties
+----------
+**Run Condition:** When to run the action. Must be 'completion', 'success', or 'failure'. Defaults to 'success'.
+If set to 'completion', the action will be executed regardless of whether the pipeline run succeeded or failed.
+If set to 'success', the action will only be executed if the pipeline run succeeded.
+If set to 'failure', the action will only be executed if the pipeline run failed.
+
+**Driver Name:** Name of the JDBC driver to use.
+
+**Host:** Host that Jethro is running on.
+
+**Port:** Port that Jethro is running on.
+
+**Database:** MemSQL database name.
+
+**Query:** Query to run.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+Example
+-------
+Suppose you want to execute a query against a Jethro database instance named "prod" that is running on "localhost" 
+port 9112, then configure the plugin with:
+
+```
+Run Condition: "success" 
+Driver Name: "jethro"
+Host: "localhost"
+Port: 9112
+Database: "prod"
+Query: "TRUNCATE TABLE testTable"
+```

jethro-plugin/docs/JethroSource-batchsource.md

@@ -0,0 +1,60 @@
+# Jethro Batch Source
+
+
+Description
+-----------
+Reads from a Jethro Data instance using a configurable SQL query.
+Outputs one record for each row returned by the query.
+
+
+Use Case
+--------
+The source is used whenever you need to read from a Jethro Data instance. For example, you may want
+to create daily snapshots of a database table by using this source and writing to
+a TimePartitionedFileSet.
+
+
+Properties
+----------
+**Reference Name:** Name used to uniquely identify this source for lineage, annotating metadata, etc.
+
+**Driver Name:** Name of the JDBC driver to use.
+
+**Host:** Host that Jethro Data is running on.
+
+**Port:** Port that Jethro Data is running on.
+
+**Instance:** Jethro Data instance name.
+
+**Import Query:** The SELECT query to use to import data from the specified table.
+You can specify an arbitrary number of columns to import, or import all columns using \*. The Query should
+contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'.
+The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
+The '$CONDITIONS' string is not required if numSplits is set to one.
+
+**Bounding Query:** Bounding Query should return the min and max of the values of the 'splitBy' field.
+For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is set to one.
+
+**Username:** User identity for connecting to the specified database.
+
+**Password:** Password to use to connect to the specified database.
+
+**Split-By Field Name:** Field Name which will be used to generate splits. Not required if numSplits is set to one.
+
+**Number of Splits to Generate:** Number of splits to generate.
+
+**Schema:** The schema of records output by the source. This will be used in place of whatever schema comes
+back from the query. However, it must match the schema that comes back from the query,
+except it can mark fields as nullable and can contain a subset of the fields.
+
+Data Types Mapping
+----------
+
+    | Jethro Data Type               | CDAP Schema Data Type | Comment                                            |
+    | ------------------------------ | --------------------- | -------------------------------------------------- |
+    | INTEGER                        | int                   |                                                    |
+    | BIGINT                         | long                  |                                                    |
+    | FLOAT                          | float                 |                                                    |
+    | DOUBLE                         | double                |                                                    |
+    | STRING                         | string                |                                                    |
+    | TIMESTAMP                      | timestamp             |                                                    |

jethro-plugin/pom.xml

@@ -0,0 +1,117 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Copyright © 2019 CDAP
+
+  Licensed under the Apache License, Version 2.0 (the "License"); you may not
+  use this file except in compliance with the License. You may obtain a copy of
+  the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+  License for the specific language governing permissions and limitations under
+  the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <parent>
+    <artifactId>database-plugins</artifactId>
+    <groupId>io.cdap.plugin</groupId>
+    <version>1.3.0-SNAPSHOT</version>
+  </parent>
+
+  <name>Jethro Data plugin</name>
+  <artifactId>jethro-plugin</artifactId>
+  <version>1.3.0-SNAPSHOT</version>
+  <modelVersion>4.0.0</modelVersion>
+
+  <dependencies>
+    <dependency>
+      <groupId>io.cdap.cdap</groupId>
+      <artifactId>cdap-etl-api</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.cdap.plugin</groupId>
+      <artifactId>database-commons</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>io.cdap.plugin</groupId>
+      <artifactId>hydrator-common</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.cdap.cdap</groupId>
+      <artifactId>cdap-api</artifactId>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.jetbrains</groupId>
+      <artifactId>annotations</artifactId>
+      <version>RELEASE</version>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.jethro</groupId>
+      <artifactId>jethro-jdbc-driver</artifactId>
+      <version>3.9</version>
+      <scope>system</scope>
+      <systemPath>${project.basedir}/libs/jethro-jdbc-3.9.jar</systemPath>
+    </dependency>
+
+    <dependency>
+      <groupId>io.cdap.plugin</groupId>
+      <artifactId>database-commons</artifactId>
+      <version>${project.version}</version>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.cdap.cdap</groupId>
+      <artifactId>hydrator-test</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.cdap.cdap</groupId>
+      <artifactId>cdap-data-pipeline</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.felix</groupId>
+        <artifactId>maven-bundle-plugin</artifactId>
+        <version>3.3.0</version>
+        <extensions>true</extensions>
+        <configuration>
+          <instructions>
+            <_exportcontents>
+              io.cdap.plugin.db.*;
+            </_exportcontents>
+            <Embed-Dependency>*;inline=false;scope=compile</Embed-Dependency>
+            <Embed-Transitive>true</Embed-Transitive>
+            <Embed-Directory>lib</Embed-Directory>
+          </instructions>
+        </configuration>
+        <executions>
+          <execution>
+            <phase>package</phase>
+            <goals>
+              <goal>bundle</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+      <plugin>
+        <groupId>io.cdap</groupId>
+        <artifactId>cdap-maven-plugin</artifactId>
+      </plugin>
+    </plugins>
+  </build>
+</project>

jethro-plugin/src/main/java/io/cdap/plugin/db/jethro/JethroUtils.java

@@ -0,0 +1,28 @@
+/*
+ * Copyright © 2019 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.db.jethro;
+
+/**
+ * Jethro utils
+ */
+public class JethroUtils {
+
+  private static final String JETHRO_CONNECTION_STRING_FORMAT = "jdbc:JethroData://%s:%d/%s";
+
+  public static String getConnectionString(String host, int port, String database) {
+    return String.format(JETHRO_CONNECTION_STRING_FORMAT, host, port, database);
+  }
+}

jethro-plugin/src/main/java/io/cdap/plugin/db/jethro/action/JethroAction.java

@@ -0,0 +1,40 @@
+/*
+ * Copyright © 2019 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package io.cdap.plugin.db.jethro.action;
+
+import io.cdap.cdap.api.annotation.Description;
+import io.cdap.cdap.api.annotation.Name;
+import io.cdap.cdap.api.annotation.Plugin;
+import io.cdap.cdap.etl.api.action.Action;
+import io.cdap.plugin.db.batch.action.AbstractDBAction;
+
+/**
+ * Action that runs Jethro command.
+ */
+@Plugin(type = Action.PLUGIN_TYPE)
+@Name(JethroAction.NAME)
+@Description("Action that runs a Jethro command")
+public class JethroAction extends AbstractDBAction {
+  public static final String NAME = "JethroAction";
+
+  private final JethroActionConfig jethroActionConfig;
+
+  public JethroAction(JethroActionConfig jethroActionConfig) {
+    super(jethroActionConfig, false);
+    this.jethroActionConfig = jethroActionConfig;
+  }
+}

jethro-plugin/src/main/java/io/cdap/plugin/db/jethro/action/JethroActionConfig.java

@@ -0,0 +1,31 @@
+/*
+ * Copyright © 2019 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package io.cdap.plugin.db.jethro.action;
+
+import io.cdap.plugin.db.batch.config.DBSpecificQueryConfig;
+import io.cdap.plugin.db.jethro.JethroUtils;
+
+/**
+ * Jethro Action config
+ */
+public class JethroActionConfig extends DBSpecificQueryConfig {
+
+  @Override
+  public String getConnectionString() {
+    return JethroUtils.getConnectionString(host, port, database);
+  }
+}