Skip to content

vexelenn/pyspark-dockerfile

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 

Repository files navigation

Dockerfile for pyspark isolated runs

I have created this dockerfile to run some simple pyspark applications on local machine. This is also helpfull to debug jobs on localhost on a sampled data sets.

Base image

Build

docker build -t pyspark .

Usage

The pyspark file can be directly passed into the dockerfile by using helping script docker-submit. This command will mount your existing directory (from where the command was run) into the /data folder in dockerfile. It will pass the asset file as an parameter to the spark-submit application inside the docker container.

docker-submit.sh asset.py

Vim

If you like (like me) editing python files in a vim editor, you may use the helping command command docker-submit.sh, by simply entering this command in vim:

:!sh docker-submit.sh %

This will run your asset file inside a docker

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published