E-MapReduce DataSources

Requirements

• Spark 1.3+

Introduction

• This project supports interaction with Aliyun's base service, e.g. OSS, ODPS, LogService and ONS, in Spark runtime environment.

Build and Install

	    git clone https://github.com/aliyun/aliyun-emapreduce-datasources.git
	    cd  aliyun-emapreduce-datasources
	    mvn clean package -DskipTests

Build emr-maxcompute with Spark3.2.0

	    git clone https://github.com/aliyun/aliyun-emapreduce-datasources.git
	    cd  aliyun-emapreduce-datasources/emr-maxcompute/
	    mvn clean package -Pspark3 -DskipTests

Use SDK in Eclipse project directly

• copy sdk jar to your project
• right click Eclipse project -> Properties -> Java Build Path -> Add JARs
• choose and import the sdk
• you can use the sdk in your Eclipse project

Maven

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-maxcompute_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-logservice_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-tablestore</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-ons_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-mns_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-redis_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-hbase_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-jdbc_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-dts_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-kudu_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-datahub_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-druid_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-sql_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-oss</artifactId>
            <version>2.0.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-common</artifactId>
            <version>2.2.0</version>
        </dependency>
        
        <dependency>
            <groupId>com.aliyun.emr</groupId>
            <artifactId>emr-kafka-client-metrics</artifactId>
            <version>2.2.0</version>
        </dependency>

Run tests

• How to run tests

JindoFS/OSS support

• Hadoop on JindoFS/OSS (Hive, Spark, Presto, Impala, Hbase and Flink are also supported)

MaxCompute support

• Spark on MaxCompute

ONS support

• Spark on ONS

LogService support

• Spark on LogService

TableStore support

• HadoopMR on TableStore
• Spark on TableStore
• Hive/SparkSQL on TableStore

License

Licensed under the Apache License 2.0