Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spark atlas connecter fails to track linege if an action is the last or if it's the only action in "spark-submit" #292

Open
har5havardhan opened this issue Feb 20, 2020 · 0 comments

Comments

@har5havardhan
Copy link

lineage gets tracked using the below code:
_from pyspark.sql import SparkSession
import time
spark = SparkSession.builder.appName('test').enableHiveSupport().getOrCreate()
df = spark.sql("select * from table limit 10")
df.write.mode("overwrite").saveAsTable("dummydb.test");
df.write.mode("overwrite").saveAsTable("dummydb.test");

lineage doesn't get tracked using the below code:
_from pyspark.sql import SparkSession
import time
spark = SparkSession.builder.appName('test').enableHiveSupport().getOrCreate()
df = spark.sql("select * from table limit 10")
df.write.mode("overwrite").saveAsTable("dummydb.test");

The only difference here is the number of actions and position of actions.
This is my spark-submit command:

spark-submit --files file:///atlas-application.properties --master yarn --deploy-mode cluster --jars file:///spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar --conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker --conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker a.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant