Skip to content

Commit

Permalink
Merge pull request #37 from Eventual-Inc/docs
Browse files Browse the repository at this point in the history
docs: Update readme
  • Loading branch information
raunakab authored Jan 17, 2025
2 parents e96ecad + 3e9eadd commit 132ae79
Showing 1 changed file with 62 additions and 43 deletions.
105 changes: 62 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,13 @@
[![Latest](https://img.shields.io/github/v/tag/Eventual-Inc/daft-launcher?label=latest&logo=GitHub)](https://github.com/Eventual-Inc/daft-launcher/tags)
[![License](https://img.shields.io/badge/daft_launcher-docs-red.svg)](https://eventual-inc.github.io/daft-launcher)

# Daft Launcher CLI Tool
# Daft Launcher

`daft-launcher` is a simple launcher for spinning up and managing Ray clusters for [`daft`](https://github.com/Eventual-Inc/Daft).

## Goal

Getting started with Daft in a local environment is easy.
However, getting started with Daft in a cloud environment is substantially more difficult.
So much more difficult, in fact, that users end up spending more time setting up their environment than actually playing with our query engine.

Daft Launcher aims to solve this problem by providing a simple CLI tool to remove all of this unnecessary heavy-lifting.
It abstracts away all the complexities of dealing with Ray yourself, allowing you to focus on running `daft` in a distributed manner.

## Capabilities

What Daft Launcher is capable of:
1. Spinning up clusters.
2. Listing all available clusters (as well as their statuses).
3. Submitting jobs to a cluster.
Expand All @@ -41,24 +33,22 @@ What Daft Launcher is capable of:

## Usage

### Pre-requisites
You'll need a python package manager installed.
We highly recommend using [`uv`](https://astral.sh/blog/uv) for all things python!

You'll need some python package manager installed.
We recommend using `uv` for all things python.

#### AWS
### AWS

If you're using AWS, you'll need:
1. A valid AWS account with the necessary IAM role to spin up EC2 instances.
This IAM role can either be created by you (assuming you have the appropriate permissions).
Or this IAM role will need to be created by your administrator.
2. The [AWS CLI](https://aws.amazon.com/cli/) installed and configured on your machine.
2. The [AWS CLI](https://aws.amazon.com/cli) installed and configured on your machine.
3. To login using the AWS CLI.
For full instructions, please look [here](https://google.com).

### Installation
## Installation

Using `uv`:
Using `uv` (recommended):

```bash
# create project
Expand All @@ -74,39 +64,68 @@ source .venv/bin/activate
uv pip install daft-launcher
```

### Example

All interactions with Daft Launcher are primarily communicated via a configuration file.
By default, Daft Launcher will look inside your `$CWD` for a file named `.daft.toml`.
You can override this behaviour by specifying a custom configuration file.
## Example

```bash
```sh
# create a new configuration file
# will create a file named `.daft.toml` in the current working directory
daft init-config
# or optionally, pass in a custom name
daft init-config my-custom-config.toml
daft init
```
That should create a configuration file for you which should look like:

```toml
# This is a default configuration file that you can use to spin up a ray-cluster using `daft-launcher`.
# Change up some of the configurations in here, and then run `daft up`.
#
# For more information on the availale commands and configuration options, visit [here](https://eventual-inc.github.io/daft-launcher).
#
# Happy daft-ing!

[setup]
name = "daft-launcher-example"
version = "<VERSION>"
provider = "aws"
region = "us-west-2"
number-of-workers = 4

# The following configurations specify the type of servers in your cluster.
# The machine type below is what we usually use at Eventual, and the image id is Ubuntu based.
# If you want a smaller or bigger cluster, change the below two configurations accordingly.
instance-type = "i3.2xlarge"
image-id = "ami-04dd23e62ed049936"

# This is the user profile that ssh's into the head machine.
# This value depends upon the `image-id` value up above.
# For Ubuntu AMIs, keep it as 'ubuntu'; for AWS AMIs, change it to 'ec2-user'.
ssh-user = "ubuntu"

# Fill this out with your custom `.pem` key, or generate a new one by running `ssh-keygen -t rsa -b 2048 -m PEM -f my-key.pem`.
# Make sure the public key is uploaded to AWS.
ssh-private-key = "~/.ssh/my-keypair.pem"

# Fill in your python dependencies here.
# They'll be downloaded using `uv`.
dependencies = []
```

Some of the above values will need to be modified by you.
If you have any confusions on a value, you can always run `daft check` to check the syntax and schema of your configuration file.

Once you're content with your configuration file, go back to your terminal and run the following:

# spin up a cluster
```sh
# spin your cluster up
daft up
# or optionally, pass in a custom config file
daft up -c my-custom-config.toml

# list all the active clusters (can have multiple clusters running at the same time)
# list all the active clusters
daft list

# submit a directory and a command to run on the cluster
daft submit --working-dir <...> -- command arg1 arg2 ...
# or optionally, pass in a custom config file
daft submit -c my-custom-config.toml --working-dir $WORKING_DIR -- command arg1 arg2 ...
# submit a directory and command to run on the cluster
# (where `my-job-name` should be an entry in your .daft.toml file)
daft submit my-job-name

# run a direct SQL query against the daft query engine running in the remote cluster
daft sql -- "SELECT * FROM my_table WHERE column = 'value'"
# or optionally, pass in a custom config file
daft sql -c my-custom-config.toml -- "SELECT * FROM my_table WHERE column = 'value'"
# run a direct SQL query on daft
daft sql "SELECT * FROM my_table WHERE column = 'value'"

# spin down a cluster
# finally, once you're done, spin the cluster down
daft down
# or optionally, pass in a custom name
daft down -c my-custom-config.toml
```

0 comments on commit 132ae79

Please sign in to comment.