add spark-yarn dependency

[colorant]

Add Spark-yarn dependency when SHARK_YARN=true, which by default is false. Also fix Hadoop FileSystem Sevice Meta file's assembly rules.

[colorant]

with this patch and mesos/spark#868, I can run bin/shark on yarn cluster with : MASTER=yarn-client SPARK_JAR=./target/scala-2.9.3/shark-assembly-0.8.0-SNAPSHOT-hadoop2.0.5-alpha.jar SPARK_YARN_APP_JAR=./target/scala-2.9.3/shark_2.9.3-0.8.0-SNAPSHOT.jar bin/shark

SPARK_YARN_APP_JAR is actually not used in this case.

will look into shark-shell and shark server later.

[rxin]

Jenkins, test this please.

[rxin]

Thanks, Raymond! This is great.

[fxc123]

It seems the spark-yarn jar is removed from spark project. We may need to update the build conf

[colorant]

@fxc123 removed? do you set SPARK_YARN(different from ten days ago) to true when running sbt package?
I use SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true ./sbt/sbt clean package assembly publish-local on latest spark code. You can find spark yarn jar in both target and local ivy dir.