Skip to content

Commit

Permalink
make health-path unconfigured by default, add full k8s doc
Browse files Browse the repository at this point in the history
  • Loading branch information
@harshavardhana
harshavardhana committed Mar 19, 2020
1 parent a182a8d commit 9e3420c
Show file tree
Hide file tree
Showing 2 changed files with 120 additions and 18 deletions.
117 changes: 108 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,15 @@

*sidekick* is a high-performance sidecar load-balancer. By attaching a tiny load balancer as a sidecar to each of the client application processes, you can eliminate the centralized loadbalancer bottleneck and DNS failover management. *sidekick* automatically avoids sending traffic to the failed servers by checking their health via the readiness API and HTTP error returns.

**Table of Contents**

- [Download](#download)
- [Usage](#usage)
- [Examples](#examples)
- [Examples as Spark *executor* sidecar](#examples-as-spark-executor-sidecar)
- [Realworld Example with spark-orchestrator](#realworld-example-with-spark-orchestrator)
- [Configure *spark-orchestrator*](#configure-spark-orchestrator)
- [Install *MinIO*](#install-minio)
- [Run the spark job](#run-the-spark-job)
- [Roadmap](#roadmap)

# Download
Expand All @@ -20,7 +25,7 @@ USAGE:
FLAGS:
--address value, -a value listening address for sidekick (default: ":8080")
--health-path value, -p value health check path (default: "/health/ready")
--health-path value, -p value health check path
--health-duration value, -d value health check duration (default: 5s)
--insecure, -i disable TLS certificate verification
--log , -l enable logging
Expand All @@ -32,30 +37,124 @@ FLAGS:

- Load balance across a web service using DNS provided IPs.
```
$ sidekick http://myapp.myorg.dom
$ sidekick --health-path=/ready http://myapp.myorg.dom
```

- Load balance across 4 MinIO Servers (http://minio1:9000 to http://minio4:9000)
```
$ sidekick --address :8000 http://minio{1...4}:9000
$ sidekick --health-path=/minio/health/ready --address :8000 http://minio{1...4}:9000
```

- Load balance across 16 MinIO Servers (http://minio1:9000 to http://minio16:9000)
```
$ sidekick --health-path=/minio/health/ready http://minio{1...16}:9000
```

## Examples as Spark *executor* sidecar
## Realworld Example with spark-orchestrator

As spark *driver*, *executor* sidecars, to begin with install spark-operator and MinIO on your kubernetes cluster

**optional** create a kubernetes namespace `spark-operator`
```
kubectl create ns spark-operator
```

### Configure *spark-orchestrator*

We shall be using maintained spark operator by GCP at https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

```
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm install spark-operator incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=spark-operator --set enableWebhook=true
```

### Install *MinIO*
```
helm install minio-distributed stable/minio --namespace spark-operator --set accessKey=minio,secretKey=minio123,persistence.enabled=false,mode=distributed
```

> NOTE: persistence is disabled here for testing, make sure you are using persistence with PVs for production workload.
> For more details read our helm [documentation](https://github.com/helm/charts/tree/master/stable/minio)
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-sidecar-containers
Once minio-distributed is up and running configure `mc` and upload some data, we shall choose `mybucket` as our bucketname.

Port-forward to access minio-cluster locally.
```
kubectl port-forward pod/minio-distributed-0 9000
```

Create bucket named `mybucket` and upload some text data for spark word count sample.
```
mc config host add minio-distributed http://localhost:9000 minio minio123
mc mb minio-distributed/mybucket
mc cp /etc/hosts minio-distributed/mybucket/mydata/{1..4}.txt
```

### Run the spark job

```yml
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-minio-app
namespace: spark-operator
spec:
sparkConf:
spark.kubernetes.allocation.batch.size: "50"
hadoopConf:
"fs.s3a.endpoint": "http://127.0.0.1:9000"
"fs.s3a.access.key": "minio"
"fs.s3a.secret.key": "minio123"
"fs.s3a.path.style.access": "true"
"fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
type: Scala
sparkVersion: 2.4.5
mode: cluster
image: minio/spark:v2.4.5-hadoop-3.1
imagePullPolicy: Always
restartPolicy:
type: OnFailure
onFailureRetries: 3
onFailureRetryInterval: 10
onSubmissionFailureRetries: 5
onSubmissionFailureRetryInterval: 20

mainClass: org.apache.spark.examples.JavaWordCount
mainApplicationFile: "local:///opt/spark/examples/target/original-spark-examples_2.11-2.4.6-SNAPSHOT.jar"
arguments:
- "s3a://mytestbucket/mydata"
driver:
cores: 1
coreLimit: "1000m"
memory: "512m"
labels:
version: 2.4.5
sidecars:
- name: minio-lb
image: "minio/sidekick:v0.1.4"
imagePullPolicy: Always
args: ["--health-path", "/minio/health/ready", "--address", ":9000", "http://minio-distributed-{0...3}.minio-distributed-svc.spark-operator.svc.cluster.local:9000"]
ports:
- containerPort: 9000

executor:
cores: 1
instances: 4
memory: "512m"
labels:
version: 2.4.5
sidecars:
- name: minio-lb
image: "minio/sidekick:latest"
args: ["--address", ":8080", "--health-path", "/minio/health/ready", "http://minio{1...16}:9000"]
...
image: "minio/sidekick:v0.1.4"
imagePullPolicy: Always
args: ["--health-path", "/minio/health/ready", "--address", ":9000", "http://minio-distributed-{0...3}.minio-distributed-svc.spark-operator.svc.cluster.local:9000"]
ports:
- containerPort: 9000
```
```
kubectl create -f spark-job.yaml
kubectl logs -f --namespace spark-operator spark-minio-app-driver spark-kubernetes-driver
```

## Roadmap
Expand Down
21 changes: 12 additions & 9 deletions main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ func (b *Backend) ErrorHandler(w http.ResponseWriter, r *http.Request, err error

// healthCheck - background routine which checks if a backend is up or down.
func (b *Backend) healthCheck() {
healthCheckURL := b.endpoint + b.healthCheckPath
healthCheckURL := strings.TrimSuffix(b.endpoint, "/") + b.healthCheckPath
for {
req, err := http.NewRequest(http.MethodGet, healthCheckURL, nil)
if err != nil {
Expand Down Expand Up @@ -259,7 +259,15 @@ func clientTransport(ctx *cli.Context, enableTLS bool) http.RoundTripper {
return tr
}

func checkMain(ctx *cli.Context) {
if !ctx.Args().Present() {
console.Fatalln(fmt.Errorf("not arguments found, please check documentation '%s --help'", ctx.App.Name))
}
}

func sidekickMain(ctx *cli.Context) {
checkMain(ctx)

healthCheckPath := ctx.GlobalString("health-path")
healthCheckDuration := ctx.GlobalInt("health-duration")
addr := ctx.GlobalString("address")
Expand All @@ -269,10 +277,6 @@ func sidekickMain(ctx *cli.Context) {
healthCheckPath = "/" + healthCheckPath
}

if !ctx.Args().Present() {
console.Fatalln(fmt.Errorf("not arguments found, please use '%s --help'", ctx.App.Name))
}

var endpoints []string
if ellipses.HasEllipses(ctx.Args()...) {
argPatterns := make([]ellipses.ArgPattern, len(ctx.Args()))
Expand Down Expand Up @@ -352,7 +356,6 @@ func main() {
cli.StringFlag{
Name: "health-path, p",
Usage: "health check path",
Value: "/minio/health/ready",
},
cli.IntFlag{
Name: "health-duration, d",
Expand Down Expand Up @@ -383,13 +386,13 @@ VERSION:
EXAMPLES:
1. Load balance across 4 MinIO Servers (http://minio1:9000 to http://minio4:9000)
$ sidekick http://minio{1...4}:9000
$ sidekick --health-path "/minio/health/ready" --http://minio{1...4}:9000
2. Load balance across 4 MinIO Servers (http://minio1:9000 to http://minio4:9000), listen on port 8000
$ sidekick --address ":8000" http://minio{1...4}:9000
$ sidekick --health-path "/minio/health/ready" --address ":8000" http://minio{1...4}:9000
3. Load balance across 4 MinIO Servers using HTTPS and disable TLS certificate validation
$ sidekick --insecure https://minio{1...4}:9000
$ sidekick --health-path "/minio/health/ready" --insecure https://minio{1...4}:9000
`
app.Action = sidekickMain
app.Run(os.Args)
Expand Down

0 comments on commit 9e3420c

Please sign in to comment.