From 88a5e26287e267c4d25453dac28ee76ae57569f9 Mon Sep 17 00:00:00 2001 From: oak2278 Date: Wed, 18 Apr 2018 19:42:35 -0700 Subject: [PATCH] Update README.md --- README.md | 224 +++++++++++++++++++++++++++--------------------------- 1 file changed, 112 insertions(+), 112 deletions(-) diff --git a/README.md b/README.md index e674251..846c3b5 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ * [Lab 2 - Explore the Algorithmic Trading Concepts with Jupyter ](#lab2) * [Lab 3 - Deploy an Automated Trading Strategy with EC2 Spot Fleet](#lab3) * [Lab 4 - Leverage a Fully Managed Solution using AWS Batch](#lab4) -* [Clean Up](#cleanup) +* [Clean Up](#cleanup) * [Appendix](#appendix) @@ -32,11 +32,11 @@ If you'd like to learn more: [Basics of Algorithmic Trading: Concepts and Exampl ### Lab Overview: -These labs are designed to be completed in sequence. If you are reading this at a live AWS event, the workshop attendants will give you a high level run down of the labs. Then it's up to you to follow the instructions below to complete the labs. Don't worry if you're embarking on this journey in the comfort of your office or home - presentation materials can be found in the git repo in the top-level [presentation](https://github.com/awslabs/spot-montecarlo-workshop/tree/master/presentation) folder. +These labs are designed to be completed in sequence. If you are reading this at a live AWS event, the workshop attendants will give you a high level run down of the labs. Then it's up to you to follow the instructions below to complete the labs. Don't worry if you're embarking on this journey in the comfort of your office or home - presentation materials can be found in the git repo in the top-level [presentation](https://github.com/aws-samples/ec2-spot-montecarlo-workshop/tree/master/presentation) folder. **Lab 1:** Setup the workshop environment on AWS **Lab 2:** Explore the Algorithmic Trading Concepts with Jupyter -**Lab 3:** Deploy an Automated Trading Strategy +**Lab 3:** Deploy an Automated Trading Strategy **Lab 4:** Leverage a Fully Managed Solution using AWS Batch @@ -61,16 +61,16 @@ This section will appear again below as a reminder because you will be deploying ## Let's Begin! -### Lab 1 - Set up the Workshop Environment on AWS: - +### Lab 1 - Set up the Workshop Environment on AWS: + ![Lab 1 Architecture](images/lab1_arch.png) #### Create an SSH Key -First, you'll need to select a [region](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). For this lab, you will need to choose either the Oregon or Ohio Region. - -
-SSH Key Pair Instructions (expand for details)

+First, you'll need to select a [region](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). For this lab, you will need to choose either the Oregon or Ohio Region. + +

+SSH Key Pair Instructions (expand for details)

At the top right hand corner of the AWS Console, you'll see a **Support** dropdown. To the left of that is the region selection dropdown. 2. Then you'll need to create an SSH key pair which will be used to login to the instances once provisioned. Go to the EC2 Dashboard and click on **Key Pairs** in the left menu under Network & Security. Click **Create Key Pair**, provide a name (can be anything, make it something memorable) when prompted, and click **Create**. Once created, the private key in the form of .pem file will be automatically downloaded. @@ -79,9 +79,9 @@ At the top right hand corner of the AWS Console, you'll see a **Support** dropdo $ chmod 400 PRIVATE_KEY.PEM - >If you're on windows you'll need to convert the .pem file to .ppk to work with putty. Here is a link to instructions for the file conversion - [Connecting to Your Linux Instance from Windows Using PuTTY](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html) + >If you're on windows you'll need to convert the .pem file to .ppk to work with putty. Here is a link to instructions for the file conversion - [Connecting to Your Linux Instance from Windows Using PuTTY](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html) -

+

#### Launch the Workshop template For your convenience, we provide a CloudFormation template to stand up the core infrastructure. @@ -89,14 +89,14 @@ For your convenience, we provide a CloudFormation template to stand up the core The template sets up a VPC, IAM roles, S3 bucket, and an EC2 Instance. The EC2 instance will run a Jupyter Notebook which we will leverage in Lab 2 and a small website that we will use in Lab 3. The idea is to provide a contained environment, so as not to interfere with any other provisioned resources in your account. In order to demonstrate cost optimization strategies, the EC2 Instance is an [EC2 Spot Instance](https://aws.amazon.com/ec2/spot/) deployed by [Spot Fleet](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html). If you are new to [CloudFormation](https://aws.amazon.com/cloudformation/), take the opportunity to review the [template](https://github.com/awslabs/spot-motecarlo-workshop/blob/master/lab-1-setup/cfn-templates/spot-montecarlo-workshop.yaml) during stack creation. ->**Important:** Prior to launching a stack, be aware that a few of the resources launched need to be manually deleted when the workshop is over. When finished working, please review the "Workshop Cleanup" section to learn what manual teardown is required by you. - -1. Click on one of these CloudFormation templates that matches the region you created your keypair in to launch your stack: - - Region | Launch Template - ------------ | ------------- - **Ohio** (us-east-2) | [![Launch Monte Carlo Workshop into Ohio with CloudFormation](images/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=spot-montecarlo-stack&templateURL=https://s3-us-west-2.amazonaws.com/reinvent2017-cmp316/monte-carlo-workshop.yaml) - **Oregon** (us-west-2) | [![Launch Monte Carlo Workshop into Oregon with CloudFormation](images/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=spot-montecarlo-stack&templateURL=https://s3-us-west-2.amazonaws.com/reinvent2017-cmp316/monte-carlo-workshop.yaml) +>**Important:** Prior to launching a stack, be aware that a few of the resources launched need to be manually deleted when the workshop is over. When finished working, please review the "Workshop Cleanup" section to learn what manual teardown is required by you. + +1. Click on one of these CloudFormation templates that matches the region you created your keypair in to launch your stack: + + Region | Launch Template + ------------ | ------------- + **Ohio** (us-east-2) | [![Launch Monte Carlo Workshop into Ohio with CloudFormation](images/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=spot-montecarlo-stack&templateURL=https://s3-us-west-2.amazonaws.com/reinvent2017-cmp316/monte-carlo-workshop.yaml) + **Oregon** (us-west-2) | [![Launch Monte Carlo Workshop into Oregon with CloudFormation](images/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=spot-montecarlo-stack&templateURL=https://s3-us-west-2.amazonaws.com/reinvent2017-cmp316/monte-carlo-workshop.yaml) 2. The template will automatically bring you to the CloudFormation Dashboard and start the stack creation process in the specified region. Click **Next** on the page it brings you to. Do not change anything on the first screen. @@ -117,18 +117,18 @@ The template sets up a VPC, IAM roles, S3 bucket, and an EC2 Instance. The EC2 i ![CloudFormation Complete](images/cf-complete.png) -10. Under the CloudFormation Outputs, click on the URLs for **Jupyter** and **Web Server**. Each should load a web page confirming that the environment has been deployed correctly. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser. - +10. Under the CloudFormation Outputs, click on the URLs for **Jupyter** and **Web Server**. Each should load a web page confirming that the environment has been deployed correctly. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser. - Certificate Warning + + Certificate Warning ![Certificate Warning](images/cert_warning.png) - Jupyter + Jupyter ![CloudFormation Jupyter Complete](images/jupyter.png) - Web + Web ![CloudFormation Web Client Complete](images/web.png) @@ -152,21 +152,21 @@ The [Jupyter Notebook](http://jupyter.org/) allows you to create and share docum ### Lab 3 - Deploy an Automated Trading Strategy on EC2 Spot Fleet: -Now that we understand the basics of our trading strategy, lets get our hands dirty building out the batch processing pipeline. - +Now that we understand the basics of our trading strategy, lets get our hands dirty building out the batch processing pipeline. + ![Lab 13 Architecture](images/lab3_arch.png) -#### Create a standard SQS Queue -We will start by creating a managed message queue to store the batch job parameters. +#### Create a standard SQS Queue +We will start by creating a managed message queue to store the batch job parameters. + +1. Go to the SQS Console, if you haven't used the service in this region, click **Get Started Now**. Otherwise, click **Create New Queue**. +2. Name your queue *"workshop"*. Select **Standard Queue**. Click **Quick-Create Queue**. + > **NOTE: Queue Name is Case Sensitive** + + ![SQS Creation](images/sqs_create.png) -1. Go to the SQS Console, if you haven't used the service in this region, click **Get Started Now**. Otherwise, click **Create New Queue**. -2. Name your queue *"workshop"*. Select **Standard Queue**. Click **Quick-Create Queue**. - > **NOTE: Queue Name is Case Sensitive** - - ![SQS Creation](images/sqs_create.png) - ->For regions that don't yet support FIFO queues, the console may look different than shown. Just name the queue and accept the defaults. +>For regions that don't yet support FIFO queues, the console may look different than shown. Just name the queue and accept the defaults. 3. Save the queue **Name** and **URL** for later use. @@ -227,34 +227,34 @@ The CloudFormation template deployed a web server that will serve as the user in 7. For **Allocation Strategy**, pick **Diversified**. 8. For **Network**, pick the VPC we created for the Spot Monte Carlo Workshop. 9. Under Availability Zone, check the box next to the first two AZs. The Network Subnet should auto-populate. If the subnet dropdown box says *"No subnets in this zone*, uncheck and select another AZ -10. Select **Use automated bidding** +10. Select **Use automated bidding** ![Spot Request](images/request_spot_configuration.png) 11. Click **Next** 12. We will use User Data to bootstrap our work nodes. Copy and paste the [spotlabworker.sh](https://github.com/aws-samples/ec2-spot-montecarlo-workshop/blob/master/templates/spotlabworker.sh) code from the repo We recommend using grabbing the latest code from the repo, but you can review the script below. 
-	#!/bin/bash
-	# Install Dependencies
-	yum -y install git python-numpy python-matplotlib python-scipy
-	pip install pandas-datareader
-	pip install scipy 
-	pip install boto3
-	
-	#Populate Variables
-    echo 'Populating Variables'
-	REGION=`curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}'`
-	mkdir /home/ec2-user/spotlabworker
-	chown ec2-user:ec2-user /home/ec2-user/spotlabworker
-	cd /home/ec2-user/spotlabworker
-	WEBURL=$(aws cloudformation --region $REGION describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`WebInterface`].OutputValue' --output text)
-	echo 'Region is '$REGION
-    echo 'URL is '$WEBURL
-
-    echo "Downloading worker code"
-	wget $WEBURL/static/queue_processor.py
-	wget $WEBURL/static/worker.py
-	
-	echo 'Starting the worker processor'
+	#!/bin/bash
+	# Install Dependencies
+	yum -y install git python-numpy python-matplotlib python-scipy
+	pip install pandas-datareader
+	pip install scipy 
+	pip install boto3
+	
+	#Populate Variables
+    echo 'Populating Variables'
+	REGION=`curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}'`
+	mkdir /home/ec2-user/spotlabworker
+	chown ec2-user:ec2-user /home/ec2-user/spotlabworker
+	cd /home/ec2-user/spotlabworker
+	WEBURL=$(aws cloudformation --region $REGION describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`WebInterface`].OutputValue' --output text)
+	echo 'Region is '$REGION
+    echo 'URL is '$WEBURL
+
+    echo "Downloading worker code"
+	wget $WEBURL/static/queue_processor.py
+	wget $WEBURL/static/worker.py
+	
+	echo 'Starting the worker processor'
 	python /home/ec2-user/spotlabworker/queue_processor.py --region $REGION> stdout.txt 2>&1
12. Under **Tags**, Enter **Name** for *Key*. Enter **WorkerNode** for *Value*. @@ -264,63 +264,63 @@ The CloudFormation template deployed a web server that will serve as the user in * **Health Checks** * **Interruption behavior** * **Load Balancer registration** - * **EBS Optimized** - - ![Spot Request](images/spot_user_data.png) + * **EBS Optimized** + + ![Spot Request](images/spot_user_data.png) 15. Click **Review**, review your settings, and then click **Launch**. 16. Wait until the request is fulfilled, capacity shows the specified number of Spot instances, and the status is Active. -17. Once the workers come up, they should start processing the SQS messages automatically. Feel free to create some more jobs from the webpage. - -#### Optional: Auto-scale the Worker Fleet on EC2 Spot -In the previous step, we specified two Spot instances, but what if we need to process more than two jobs at once? In this optional section we will configure auto-scaling so that new spot instances are created as more jobs get added to the queue. - -1. Go to the CloudWatch console, and click on **Alarms**. -2. Click on **Create Alarm**. Select **SQS Metrics**. -3. Scroll down and select **ApproximateNumberOfMessagesVisible**. Click **Next** - - ![CW Alarm](images/spot_cw_alarm.png) - -4. We will create a threshold for scaling up. Name the alarm, set the threshold for **>= 2** messages for **2** consecutive periods. Delete the default notification actions and hit **Create**. - - ![CW Alarm](images/spot_cw_alarmfinal.png) - - 5. Repeat these steps for the scale down policy. Name the alarm appropriately. Set the threshold for **<= 1** message for **3** consecutive periods. - 6. Return to **Spot Requests** in the EC2 Console. - 7. Select your fleet and go to the **Auto Scaling** tab at the bottom pane. - 8. Click **Configure**. On the next screen, click on **Scale Spot Fleet using step or simple scaling policies** - - ![CW Alarm](images/spot_auto_scale.png) - - 9. Under the **ScaleUp** and **ScaleDown** policies, configure the appropriate alarms under **Policy trigger**. - +17. Once the workers come up, they should start processing the SQS messages automatically. Feel free to create some more jobs from the webpage. + +#### Optional: Auto-scale the Worker Fleet on EC2 Spot +In the previous step, we specified two Spot instances, but what if we need to process more than two jobs at once? In this optional section we will configure auto-scaling so that new spot instances are created as more jobs get added to the queue. + +1. Go to the CloudWatch console, and click on **Alarms**. +2. Click on **Create Alarm**. Select **SQS Metrics**. +3. Scroll down and select **ApproximateNumberOfMessagesVisible**. Click **Next** + + ![CW Alarm](images/spot_cw_alarm.png) + +4. We will create a threshold for scaling up. Name the alarm, set the threshold for **>= 2** messages for **2** consecutive periods. Delete the default notification actions and hit **Create**. + + ![CW Alarm](images/spot_cw_alarmfinal.png) + + 5. Repeat these steps for the scale down policy. Name the alarm appropriately. Set the threshold for **<= 1** message for **3** consecutive periods. + 6. Return to **Spot Requests** in the EC2 Console. + 7. Select your fleet and go to the **Auto Scaling** tab at the bottom pane. + 8. Click **Configure**. On the next screen, click on **Scale Spot Fleet using step or simple scaling policies** + + ![CW Alarm](images/spot_auto_scale.png) + + 9. Under the **ScaleUp** and **ScaleDown** policies, configure the appropriate alarms under **Policy trigger**. + ![CW Alarm](images/spot_auto_scale_final.png) - - 10. Click **Save** + + 10. Click **Save** #### Evaluate the Results 1. Check your S3 Bucket. In a few minutes, you should see results start appearing the bucket. -2. If you monitor the SQS queue for messages you should see them being picked up by the worker nodes. - -#### Terminate the Spot Fleet -In the next lab, we will use [AWS Batch](https://aws.amazon.com/batch/) to create a managed batch process pipeline. We will reuse our existing queue, so let's terminate our EC2 Spot worker fleet. - -1. From the EC2 Console, select **Spot Requests** and click **Request Spot Instances**. -2. Check the box beside the Spot fleet request containing your worker nodes. The correct request will have a capacity of 2 and the shortest time since it was created. - - > **IMPORTANT**: Take care not to cancel the Spot fleet request responsible for our workstation node (Jupyter/WebClient). It will have a capacity of 1 and the instance type will be m4.large. - +2. If you monitor the SQS queue for messages you should see them being picked up by the worker nodes. + +#### Terminate the Spot Fleet +In the next lab, we will use [AWS Batch](https://aws.amazon.com/batch/) to create a managed batch process pipeline. We will reuse our existing queue, so let's terminate our EC2 Spot worker fleet. + +1. From the EC2 Console, select **Spot Requests** and click **Request Spot Instances**. +2. Check the box beside the Spot fleet request containing your worker nodes. The correct request will have a capacity of 2 and the shortest time since it was created. + + > **IMPORTANT**: Take care not to cancel the Spot fleet request responsible for our workstation node (Jupyter/WebClient). It will have a capacity of 1 and the instance type will be m4.large. + 3. Under **Actions**, select **Cancel Spot request**. **You've completed Lab 3, Congrats!** #### Extra Credit * Each job is handled fully by one worker. Maybe you could look at adding more parallelism to task scheduler. - + -### Lab 4 - Leverage a Fully Managed Solution using AWS Batch - +### Lab 4 - Leverage a Fully Managed Solution using AWS Batch + ![Lab 4 Architecture](images/lab4_arch.png) @@ -330,8 +330,8 @@ In the next lab, we will use [AWS Batch](https://aws.amazon.com/batch/) to creat * **How would you like to run your job ?** : No job submission and hit Next * **Compute environment name** : montecarlo-batch-worker * **Service role** and **EC2 instance role** : Leave it defaulted to "Create a new Role" - * **Provisioning Model** : Spot - * **Maximum bid price** : 100 + * **Provisioning Model** : Spot + * **Maximum bid price** : 100 * **Spot fleet role** : Select the role containing your workshop name * **Allowed instance types** : optimal * **Minimum vCPUs** : 0 @@ -346,10 +346,10 @@ In the next lab, we will use [AWS Batch](https://aws.amazon.com/batch/) to creat 4. Go to **Job Definition** , hit **Create** and enter the following details * **Job definition name** : montecarlo-queue-processor * **Job role** : Select the one that appears in drop down, as created during setup - * **Container image** : anshrma/montecarlo-workshop-worker:latest - + * **Container image** : anshrma/montecarlo-workshop-worker:latest + > We have created a docker container image containing the required libraries and the Worker code that we used in the previous lab. This container image is stored on [Dockerhub](https://hub.docker.com/r/anshrma/montecarlo-workshop-worker/). This is the image that we are pulling for our batch job. - + * **Environment variables (Key)** : REGION * **Environment variables (Value)** : Name the region you are using, example us-east-1 * Leave everything as default and click **Create job Definition** @@ -367,12 +367,12 @@ This process may take 2-3 minutes. When you refresh the screen, you will see the #### Evaluate the Results 1. Once the job reaches **Running** state, check your S3 Bucket. In a few minutes you should see results start appearing the bucket. -2. If you monitor the SQS queue for messages you should see them being picked up by the worker container. - +2. If you monitor the SQS queue for messages you should see them being picked up by the worker container. + #### Extra Credit -* Use [AWS QuickSight](https://https://quicksight.aws/) to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. You will need to create a json manifest file with your Amazon S3 data location. Use the following template as a starting point: - -
+* Use [AWS QuickSight](https://https://quicksight.aws/) to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. You will need to create a json manifest file with your Amazon S3 data location. Use the following template as a starting point:
+
+
 {
     "fileLocations": [
         {
@@ -393,9 +393,9 @@ Hopefully you've enjoyed the workshop and learned a few new things. Now follow t
 
 1. In the EC2 Console > Spot Requests, click **Cancel Spot request** under **Actions**. Make sure **Terminate instances** is checked.
 2. In the SQS Console, delete the queue that you created earlier. This is located under **Queue Actions**.
-3. In the S3 Console, locate the resultsBucket that was created for your workshop. Click on the bucket and select **Empty bucket**. You will need to copy and paste the bucket name in to confirm the action. 
+3. In the S3 Console, locate the resultsBucket that was created for your workshop. Click on the bucket and select **Empty bucket**. You will need to copy and paste the bucket name in to confirm the action. 
 4. Under AWS Batch, click on your running job and click **Terminate job**. Under **Job definitions**, click on your job definition and select **deregister**. Go to **Job queues**, then disable, and delete the configured job queue.
-5. In the CloudFormation template, select the workshop stack and select **Actions** and then **Delete stack**.
+5. In the CloudFormation template, select the workshop stack and select **Actions** and then **Delete stack**.
 
 
 ##Appendix