Skip to content
This repository has been archived by the owner on Apr 22, 2022. It is now read-only.

WIP: Run on kubernetes #36

Open
wants to merge 37 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
d480ec0
Initial setup for kubernetes using `kompose convert` and some manual …
Sep 24, 2019
58502df
Fix the key in `labels` to get the network between pods going
Sep 24, 2019
b3b4e4a
Change web app service type to be LoadBalancer
JCZuurmond Sep 24, 2019
38dd8d3
Make shop-api also of type loadbalancer to expose it on localhost
Sep 24, 2019
3ff4214
Start
Sep 24, 2019
ee26a59
Merge branch 'run-on-kubernetes' of github.com:divolte/shop into run-…
Sep 24, 2019
faa2216
First attempt to load shop content using Job pod, not working
Sep 24, 2019
9c1e9f0
Loading the shop content using job pod
Sep 24, 2019
8b405c7
Back the bare minimum config
Sep 25, 2019
7b41316
Change shop API service to ClusterIP
JCZuurmond Sep 25, 2019
8b89bb3
Change shop webapp service to ClusterIP
JCZuurmond Sep 25, 2019
1be73e4
Add ingress yaml
JCZuurmond Sep 25, 2019
601b808
Change path to webshop to be index
JCZuurmond Sep 25, 2019
966daac
Add set-up for Ingress nginx to README
JCZuurmond Sep 25, 2019
cc058ec
Add initContainers to delay the start of the put-categories pods.
Sep 25, 2019
505d7e8
Merge branch 'run-on-kubernetes' of github.com:divolte/shop into run-…
Sep 25, 2019
49d91b9
Bug fix: change index path was not committed correctly
JCZuurmond Sep 25, 2019
32dc2cc
add Liveness probe for API with own route
anibaldk Sep 25, 2019
0a9c94e
Merge branch 'live-probes' into run-on-kubernetes
anibaldk Sep 25, 2019
8cdee5d
Change init container to check if shop-api service is alive
JCZuurmond Sep 26, 2019
dc83947
Add init containers which checks if API is alive
JCZuurmond Sep 26, 2019
9f0c106
Solve flake8 issues
JCZuurmond Sep 26, 2019
76fed7b
Change input order according to PEP8
JCZuurmond Sep 26, 2019
4393bf4
Put json requests in separate function
JCZuurmond Sep 26, 2019
292fe7d
Retry json put if 10 times ConnectionError
JCZuurmond Sep 26, 2019
2a5d2f0
Bug fix: json.dump --> json.dumps
JCZuurmond Sep 26, 2019
3438536
Add retrying to be pip installed
JCZuurmond Sep 26, 2019
3e01af4
Add utility script to build the jars, without the need of sbt/gradle.
abij Sep 26, 2019
ffa2b94
Add comment explaining ttlSecondsAfterFinished
JCZuurmond Sep 26, 2019
be74dc2
Make catalog-builder an image to load data into the shop
anibaldk Sep 26, 2019
2fcd672
merge work with Cor
anibaldk Sep 26, 2019
b8ff7a1
Clean up deployment file
anibaldk Sep 26, 2019
d326724
Merge branch 'run-on-kubernetes' of github.com:divolte/shop into run-…
JCZuurmond Sep 26, 2019
b4831d5
Make sure the docker-compose also works.
abij Sep 26, 2019
3081478
Fix typo
abij Sep 26, 2019
8927773
Uniform the docker context of catalog builder.
abij Sep 26, 2019
9722410
Small readme update
abij Jan 29, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 56 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This application comprises a number of different processes:
- a **kafka** topic
- A **top-pick-service** that receives data from Kafka and continuously selects popular products, using...
- a **redis** store to tally all the clicks
- A **spark** streaming consumer of the events to calculate metrics like: nr of events per type per 2 minutes

```text
+
Expand All @@ -41,30 +42,39 @@ This application comprises a number of different processes:
+
```

## Prerequisite(s)
## Kubernetes: Ingress Nginx

The following package(s) are required;
- `sbt`.
Our setup uses an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) with Nginx to define reroutes to the webshop and the
API (aka service) used by the webshop. The following steps
are required:

Install them with your package manager:
For all deployments:

```
brew update
brew install sbt
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
```

Provider specific:

**For Docker for mac**
```
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
```

**For minikube**
```
apt update
apt install sbt
minikube addons enable ingress
```

## Running with Docker
For more info, see [here](https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md#installation-guide).

## Using docker containers

The easiest way to get started is with Docker Compose.

Make sure you have Docker running locally. You can download a proper version at the [Docker Store][ds].

We are running 6 containers, and the default size is not large enough. Boost the ram of Docker to at least 4GB,
We are running a couple of containers, and the default size is not large enough. Boost the ram of Docker to at least 4GB,
otherwise you will run into startup problems (Exit with code: 137) when running the `docker-compose up` command.

We are going to build these containers locally:
Expand All @@ -73,29 +83,60 @@ We are going to build these containers locally:
- shop/docker-shop-service
- shop/docker-shop-webapp

We will use these public containers:

- redis
- docker.elastic.co/elasticsearch/elasticsearch
- krisgeus/docker-kafka

[ds]:https://store.docker.com/

### Building Project Jars (required)

Build **jars** from the source-code, before you can wrap them into docker containers for:
- service
- spark-container

### Running with docker compose
```bash
# You can use the utility script, build inside a dockers containers:
bash build-jars.sh

# Or setup the tools (sbt) yourself and run:
service/gradlew -p service build
cd spark-container/streaming && sbt assembly
```

### (Option 1) Running with docker compose

When you have the containers up and running you can access the webshop
through [localhost:9011](http://localhost:9011/).

> These ports should be available: 9011, 8080, 8081, 9200, 9300, 8290, 9092, 2181, 6379, 8989

```bash
service/gradlew -p service build && (cd spark-container/streaming && sbt assembly) && docker-compose up -d --build
# note: make sure the jar's have been build

docker-compose up -d --build
```

#### Download new products
### (Option 2) Running with Kubernetes

```bash
# note: make sure the jar's have been build

Optionally you can download new image-data from flickr.
# Create docker images
docker-compose build

# Register all services
kubectl apply -f k8s
```

### Initial Data

> Note: Data will be loaded automatically, after all services are started.

#### Download new products (optional)

You can download new image-data from flickr. These are stored in `catalog-builder/categories` folder.

> Note: you need to fill in your own Flickr Api key, and wait a very long time...!

Expand All @@ -114,21 +155,4 @@ docker run --rm -it \
--key ${FLICKR_API_KEY}"
```

#### Loading products
The first time you start the docker composition, you have to load the product catalog, like this:

```text
docker run -it --rm --volume $PWD:/divolte-shop \
--workdir /divolte-shop \
--network host \
python:3.6 \
bash -c 'pip install requests && python catalog-builder/put-categories.py \
data/categories/animals.json \
data/categories/architecture.json \
data/categories/cars.json \
data/categories/cities.json \
data/categories/flowers.json \
data/categories/landscape.json \
data/categories/nautical.json'
```
Go to [localhost:9011](http://localhost:9011/).
33 changes: 33 additions & 0 deletions build-jars.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash
# Utility script to build the project code into jars using public containers
# instead of installing tools locally like SBT, gradle.
#
# Just requires docker.

set -e # exit on error
set -u # Need to bind specified variables


# Create local folders for java dependecies (jars)
mkdir -p ~/.ivy2 && mkdir -p ~/.m2 && mkdir -p ~/.sbt

echo '[1/2] Build "spark-streaming" fat jar using SBT ...'
docker run --rm \
-v ~/.m2:/root/.m2 \
-v ~/.ivy2:/root/.ivy2 \
-v ~/.sbt:/root/.sbt \
-v "${PWD}/spark-container/streaming":/app \
-w /app \
mozilla/sbt sbt assembly

echo '[2/2] Build "service" fat jar using Gradle...'
docker run --rm \
-u gradle \
-v ~/.m2:/root/.m2 \
-v ~/.ivy2:/root/.ivy2 \
-v "${PWD}/service":/home/gradle/project \
-w /home/gradle/project \
gradle:4.6.0-jdk8-alpine gradle build

echo ''
echo 'Done building the jars for the projects'
18 changes: 18 additions & 0 deletions catalog-builder/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Data population image for initial data
#
#
FROM python:3.6

# Copy all stuff from this dir into the default working directory
COPY . .

RUN pip install -r requirements.txt
CMD ["sh", "-c", "python catalog-builder/put-categories.py --api-base-url=$SHOP_API_URL \
categories/animals.json \
categories/architecture.json \
categories/cars.json \
categories/cities.json \
categories/flowers.json \
categories/landscape.json \
categories/nautical.json" \
]
4 changes: 3 additions & 1 deletion catalog-builder/download-category.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,9 @@ def parse_args():
parser = argparse.ArgumentParser(
description='Create a JSON description of a set of images retrieved from the Flickr API based on keyword searches.')
parser.add_argument('--key', '-k', metavar='API_KEY', type=str, required=True, help='The Flickr API key to use.')
parser.add_argument('--dest', '-d', metavar='DEST', type=str, help='The output directory to store the jsons', default="data/categories")
parser.add_argument('--dest', '-d', metavar='DEST', type=str, help='The output directory to '
'store the jsons',
default="catalog-builder/categories")
parser.add_argument('--searches', '-s', metavar='SEARCH_KEYWORD', type=str, required=True, default='categories.yml',
help='Yaml file containing the categories and keywords to search for.')
parser.add_argument('--fav-threshold', '-t', metavar='FAVOURITES_THRESHOLD', type=int, default=30,
Expand Down
58 changes: 44 additions & 14 deletions catalog-builder/put-categories.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
import argparse
import requests
import json
import os

from functools import reduce
from collections import Counter

import requests
from requests.exceptions import ConnectionError
from retrying import retry


def make_item(json, filename):
return {
Expand All @@ -27,10 +29,11 @@ def make_item(json, filename):
'flickr_url': size['url'],
'img_source': size['source']
}
for name,size in json['sizes'].items()
for name, size in json['sizes'].items()
}
}


def merge_items(*input):
items = list(filter(lambda x: x, input))

Expand All @@ -43,10 +46,12 @@ def merge_items(*input):
result = {}
result.update(items[0])
for item in items[1:]:
result['categories'] = list(set(result['categories'] + item['categories']))
result['categories'] = list(
set(result['categories'] + item['categories']))

return result


def with_price(item, min_favs, max_favs):
r = (max_favs - min_favs + 1) / 2.0
result = {
Expand All @@ -56,43 +61,68 @@ def with_price(item, min_favs, max_favs):

return result


@retry(stop_max_attempt_number=10,
wait_fixed=1000, # wait 1 sec
retry_on_exception=lambda e: isinstance(e, ConnectionError))
def _requests_put_json(url, data):
headers = {'Content-Type': 'application/json'}
return requests.put(url, data=data, headers=headers)


def main(args):
print('Items from:\n\t%s' % '\n\t'.join(args.files))

item_list = reduce(
lambda x,y: x+y,
[[
make_item(json.loads(line), fn)
for line in open(fn).readlines()]
for fn in args.files])
lambda x, y: x+y,
[[make_item(json.loads(line), fn) for line in open(fn).readlines()]
for fn in args.files])

min_favs = min([item['favs'] for item in item_list])
max_favs = max([item['favs'] for item in item_list])
priced_item_list = [with_price(item, min_favs, max_favs) for item in item_list]
priced_item_list = [
with_price(item, min_favs, max_favs) for item in item_list]

items = {}
for item in priced_item_list:
items[item['id']] = merge_items(item, items.get(item['id']))

responses = Counter()
for item in items.values():
r = requests.put(args.api_base_url + '/catalog/item', data=json.dumps(item), headers={'Content-Type': 'application/json'})
r = _requests_put_json(
args.api_base_url + '/catalog/item',
data=json.dumps(item)
)
print('PUT %s, %d' % (item['id'], r.status_code))
if r.status_code != 200:
print(json.dumps(item))
responses.update([r.status_code])

print('\nSummary:')
for status, count in responses.items():
print('\t%d times status code %d' % (count,status))
print('\t%d times status code %d' % (count, status))


def parse_args():
parser = argparse.ArgumentParser(description='Take downloaded category JSON files and push them into the Catalog API.')
parser.add_argument('--api-base-url', '-u', metavar='API_BASE_URL', type=str, default='http://localhost:8080/api', help='The base URL of the Catalog API.')
parser = argparse.ArgumentParser(
description=(
'Take downloaded category JSON files and push them into the '
'Catalog API.'
)
)
parser.add_argument(
'--api-base-url',
'-u',
metavar='API_BASE_URL',
type=str,
default='http://localhost:8080/api',
help='The base URL of the Catalog API.'
)
parser.add_argument('files', metavar='JSON_FILE', type=str, nargs='+')

return parser.parse_args()


if __name__ == '__main__':
args = parse_args()
main(args)
1 change: 1 addition & 0 deletions catalog-builder/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
requests
xmltodict
pyyaml
retrying
11 changes: 11 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,17 @@ services:
- kafka:kafka
networks:
- divolte.io

catalog-builder:
build:
context: ./catalog-builder
container_name: catalog-builder
environment:
SHOP_API_URL: "http://shop-api:8080/api"
depends_on:
- shop-api


# divolte-druid:
# # image: fokkodriesprong/docker-druid
# image: shop/divolte-druid
Expand Down
16 changes: 8 additions & 8 deletions install_without_docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,13 @@ To load the data from the data/catalog directory into elasticsearch the `catalog

```
python put-catagories.py \
../data/categories/animals.json \
../data/categories/architecture.json \
../data/categories/cars.json \
../data/categories/cities.json \
../data/categories/flowers.json \
../data/categories/landscape.json \
../data/categories/nautical.json
categories/animals.json \
categories/architecture.json \
categories/cars.json \
categories/cities.json \
categories/flowers.json \
categories/landscape.json \
categories/nautical.json
```

## Running the webapp
Expand All @@ -89,4 +89,4 @@ python consumer.py --schema ../divolte/schema/src/main/resources/ShopEventRecord
--client toppick-client --group toppick-group

python bandit_service.py
```
```
Loading