Spark PR Review Board

This repository hosts the code for spark-prs.appspot.com, a tool for assisting in Apache Spark pull request review.

The backend is written in Python and hosted via Google App Engine. It uses a periodic cron job to access the GitHub API, fetch lists of recently-updated pull requests, and enqueue tasks to download PR information and persist it into a database. The backend has several other functions, including:

• Classification of pull requests based on which files they modify.
• Automatically linking JIRA issues to pull requests that reference them.
• Triggering Jenkins jobs to test pull requests (experimental / admin-only).

The frontend uses React.js to render UI components. It communicates with the backend via a REST API.

Development Instructions

Installing dependencies

1. Install the App Engine Python SDK.

2. Install library dependencies:

   pip install -r requirements.txt -t lib
   npm install .   
   

3. Create a settings.cfg file (see settings.cfg.template). For most user-facing feature development, it is not necessary to fill out the entire settings.cfg file. However, you may need to supply several of these configuration options in order to test certain backend functionality, such as GitHub data-fetching and authentication, JIRA integration, etc.

4. Run dev_appserver.py --datastore_path datastore . and browse to http://localhost:8080 to view the application.

Loading Data

Initially, the dashboard will be empty because the development appserver doesn't run the cron job that contacts GitHub to download pull requests. The easiest way to get started is to download a sample database dump and pass the --datastore_path /path/to/downloaded/datastore option to dev_appserver.py.

If you'd rather generate your own datastore, configure settings.cfg with proper GitHub API keys, then browse to http://localhost:8000/cron and hit "Run now" to manually trigger the cron job that refreshes pull requests.

In order to backfill the datastore with old pull requests, visit /tasks/github/backfill-prs and log in with AppEngine app admin credentials. This will enqueue update tasks for every pull request ever opened against the repository, using the slower old-prs task queue to avoid exceeding the GitHub API rate limit.

Front-end development

The front-end UI is implemented as a single-page web app using the React.js library. The majority of UI components are written in React's JSX Javascript dialect; these files have .jsx extensions. These JSX files are converted into plain Javascript using a Grunt task.

A good development workflow is to run grunt default watch in a separate terminal so that Grunt watches files for changes and automatically performs the JS -> JSX conversion and runs the code style checks.

For now, we commit both the .jsx source files and generated .js files. In the longer run, it would be a good idea to use something like Browserify or Uglify.js to compile our Javascript source files into a single file and to not check in generated sources (pull requests are welcome for this!).

Code Style

For Python code, we follow the PEP8 style, with the exception of a 100-character maximum line length instead 80.

For Javascript code, we roughly follow the Airbnb style guide.

To perform style checks:

pep8
grunt lint

License

This project is licensed under the Apache 2.0 License. See LICENSE for full license text.