Skip to content

Commit

Permalink
Updating readme for Batch
Browse files Browse the repository at this point in the history
  • Loading branch information
@anshrma
anshrma committed Nov 22, 2017
1 parent 1cbd943 commit f7e2797
Show file tree
Hide file tree
Showing 3 changed files with 146 additions and 101 deletions.
245 changes: 144 additions & 101 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,28 +94,28 @@ The template sets up a VPC, IAM roles, S3 bucket, and an EC2 Instance. The EC2 i
**Sydney** (ap-southeast-2) | [![Launch Monte Carlo Workshop into Sydney with CloudFormation](images/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=spot-montecarlo-stack&templateURL=https://s3-us-west-2.amazonaws.com/reinvent2017-cmp316/monte-carlo-workshop.yaml)

2. The template will automatically bring you to the CloudFormation Dashboard and start the stack creation process in the specified region. Click **Next** on the page it brings you to. Do not change anything on the first screen.


First Screen
![CloudFormation PARAMETERS](images/cf-initial.png)

Parameters Screen
![CloudFormation PARAMETERS](images/cf-initial.png)

Parameters Screen
![CloudFormation PARAMETERS](images/cf-params.png)

3. Select a password to use for the Jupyter Notebook. You will use this password in Lab 2.
4. The default port for the Jupyter Notebook is 8888. Some corporate firewalls and VPNs will block this port. You can change the **JupyterPort** to 443 to get around this.
5. Select your ssh key.
>**Important:** On the parameter selection page of launching your CloudFormation stack, make sure to choose the key pair that you created in step 1. If you don't see a key pair to select, check your region and try again.
3. Select a password to use for the Jupyter Notebook. You will use this password in Lab 2.
4. The default port for the Jupyter Notebook is 8888. Some corporate firewalls and VPNs will block this port. You can change the **JupyterPort** to 443 to get around this.
5. Select your ssh key.
>**Important:** On the parameter selection page of launching your CloudFormation stack, make sure to choose the key pair that you created in step 1. If you don't see a key pair to select, check your region and try again.
6. After you've selected your ssh key pair, click **Next**.
7. On the **Options** page, accept all defaults - you don't need to make any changes. Click **Next**.
8. On the **Review** page, under **Capabilities** check the box next to **"I acknowledge that AWS CloudFormation might create IAM resources."** and click **Create**. Your CloudFormation stack is now being created.
9. Periodically check on the stack creation process in the CloudFormation Dashboard. Your stack should show status **CREATE\_COMPLETE** in roughly 10-15 minutes. In the Outputs tab, take note of the **Jupyter** and **Web Server** values; you will need these in the next two labs.

![CloudFormation Complete](images/cf-complete.png)

10. Click on the Output URLs for **Jupyter** and **Web Server**. Each should load a web page confirming that the environment has been deployed correctly. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser.
10. Click on the Output URLs for **Jupyter** and **Web Server**. Each should load a web page confirming that the environment has been deployed correctly. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser.

Certificate Warning
![Certificate Warning](images/cert_warning.png)
Certificate Warning
![Certificate Warning](images/cert_warning.png)

Jupyter
![CloudFormation Jupyter Complete](images/jupyter.png)
Expand All @@ -124,70 +124,70 @@ The template sets up a VPC, IAM roles, S3 bucket, and an EC2 Instance. The EC2 i
![CloudFormation Web Client Complete](images/web.png)

If there was an error during the stack creation process, CloudFormation will rollback and terminate. You can investigate and troubleshoot by looking in the Events tab. Any errors encountered during stack creation will appear in the event log.


**You've completed Lab 1, Congrats!**

<a name="lab2"></a>
### Lab 2 - Explore the Algorithmic Trading Concepts with Jupyter:

The [Jupyter Notebook](http://jupyter.org/) allows you to create and share documents that contain live code, equations, visualizations and narrative text.

1. Log into the Jupyter Notebook using the **Jupyter** URL output from the CloudFormation Template using the password you configured when building the stack. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser.

![Certificate Warning](images/cert_warning.png)
1. Log into the Jupyter Notebook using the **Jupyter** URL output from the CloudFormation Template using the password you configured when building the stack. We have created a self-signed certificate for the Jupyter Notebook. You will see messages about an unsafe connection. It is safe to ignore these warnings and continue. The steps will differ depending on your browser.

![Certificate Warning](images/cert_warning.png)

2. Click on the notebook named *monte-carlo-workshop.ipynb* and it should open in a new tab.
3. Follow the instructions in the Notebook to complete Lab 2. If you're new to Jupyter, you press shift-enter to run code and/or proceed to the next section. When you're done with the Notebook, return here and we'll take the concepts we learned in this lab and build our own automated pipeline.

3. Follow the instructions in the Notebook to complete Lab 2. If you're new to Jupyter, you press shift-enter to run code and/or proceed to the next section. When you're done with the Notebook, return here and we'll take the concepts we learned in this lab and build our own automated pipeline.

**You've completed Lab 2, Congrats!**


<a name="lab3"></a>
### Lab 3 - Deploy an Automated Trading Strategy on EC2 Spot Fleet:

### Lab 3 - Deploy an Automated Trading Strategy on EC2 Spot Fleet:

Now that we understand the basics of our trading strategy, lets get our hands dirty building out the batch processing pipeline.

#### Create the SQS Queue
We will start by creating a managed message queue to store the batch job parameters.
#### Create the SQS Queue
We will start by creating a managed message queue to store the batch job parameters.

1. Go to the SQS Console, if you haven't used the service in this region, click **Get Started Now**. Otherwise, click **Create New Queue**.
2. Name your queue *"Workshop"*. Select **Standard Queue**. Click **Quick-Create Queue**.

![SQS Creation](images/sqs_create.png)

1. Go to the SQS Console, if you haven't used the service in this region, click **Get Started Now**. Otherwise, click **Create New Queue**.
2. Name your queue *"Workshop"*. Select **Standard Queue**. Click **Quick-Create Queue**.

![SQS Creation](images/sqs_create.png)

>For regions that don't yet support FIFO queues, the console may look different than shown. Just name the queue and accept the defaults.
>For regions that don't yet support FIFO queues, the console may look different than shown. Just name the queue and accept the defaults.
3. Save the queue **Name** and **URL** for later use.

![SQS Info](images/sqs_info.png)
3. Save the queue **Name** and **URL** for later use.


#### Edit the EC2 Instance Profile
Our EC2 instances run with an Instance Profile that contains an IAM role giving the instance permissions to interact with other AWS services. We need to edit the associated policy with permissions to access the SQS service.

1. Go to the EC2 Console.
2. Under **Instances**, select the instance named *montecarlo-workshop*.
3. Scroll down and select the **IAM Role**.
4. You should see two attached policies. One will be an inline policy named after the workshop. Click the arrow beside the policy and click **Edit policy**.

![IAM Role](images/iam_role.png)

5. Under *Actions*, place a comma after `"cloudformation:Describe*"`.
6. Add a new entry `"sqs:*"`which will give permission for all SQS actions. Click **Validate Policy**.

![IAM Validate](images/iam_validate.png)

7. If the policy is valid, click **Save**.

#### Configure the Web Client
The CloudFormation template deployed a web server that will serve as the user interface. We need to configure it with our SQS queue
![SQS Info](images/sqs_info.png)


#### Edit the EC2 Instance Profile
Our EC2 instances run with an Instance Profile that contains an IAM role giving the instance permissions to interact with other AWS services. We need to edit the associated policy with permissions to access the SQS service.

1. Go to the EC2 Console.
2. Under **Instances**, select the instance named *montecarlo-workshop*.
3. Scroll down and select the **IAM Role**.
4. You should see two attached policies. One will be an inline policy named after the workshop. Click the arrow beside the policy and click **Edit policy**.

![IAM Role](images/iam_role.png)

5. Under *Actions*, place a comma after `"cloudformation:Describe*"`.
6. Add a new entry `"sqs:*"`which will give permission for all SQS actions. Click **Validate Policy**.

![IAM Validate](images/iam_validate.png)

7. If the policy is valid, click **Save**.

#### Configure the Web Client
The CloudFormation template deployed a web server that will serve as the user interface. We need to configure it with our SQS queue

1. Launch Web Site using the URL from the CloudFormation output.
2. Click **Configuration**
3. Configure the **SQS URL**, **S3 Bucket Name**, and **AWS Region** using the output values from the CloudFormation stack.

![Web Config](images/web_config.png)


![Web Config](images/web_config.png)

4. Click **Submit** and then click **Home** to return to the home page.

#### Configure our Simulation
Expand All @@ -196,14 +196,14 @@ The CloudFormation template deployed a web server that will serve as the user in
* Short Window = 40 days
* Long Window = 100 days
* Trading Days = 1260 (5 years)
* Iterations = 2000
* Preview Only = *unchecked* - You can use this to see the json message placed on the queue.

#### View the messages in SQS
1. Go to the SQS Console and select your queue.
2. Under **Queue Actions**, select **View/Delete Messages**.
3. Click on **Start Polling for Messages**
4. You should see the message that was created by the web client. Explore the message attributes to see what we will be passing to the worker script
* Iterations = 2000
* Preview Only = *unchecked* - You can use this to see the json message placed on the queue.

#### View the messages in SQS
1. Go to the SQS Console and select your queue.
2. Under **Queue Actions**, select **View/Delete Messages**.
3. Click on **Start Polling for Messages**
4. You should see the message that was created by the web client. Explore the message attributes to see what we will be passing to the worker script
5. Now that we have messages on the queue, lets spin up some workers on EC2 spot instances.

#### Create the Spot Worker Fleet
Expand All @@ -217,35 +217,35 @@ The CloudFormation template deployed a web server that will serve as the user in
9. Under Availability Zone, check the box next to the first two AZs. The Network Subnet should auto-populate. If the subnet dropdown box says *"No subnets in this zone*, uncheck and select another AZ
10. Select **Use automated bidding**
11. Click **Next**
12. We will use User Data to bootstrap our work nodes. Copy and paste the [spotlabworker.sh](https://github.com/aws-samples/ec2-spot-montecarlo-workshop/blob/master/templates/spotlabworker.sh) code from the repo We recommend using grabbing the latest code from the repo, but you can review the script below.

<pre>
#!/bin/bash
# Install Dependencies
yum -y install git python-numpy python-matplotlib python-scipy
pip install pandas-datareader
pip install scipy
pip install boto3

#Populate Variables
REGION=`curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}'`
mkdir /home/ec2-user/spotlabworker
chown ec2-user:ec2-user /home/ec2-user/spotlabworker
cd /home/ec2-user/spotlabworker
WEBURL=$(aws cloudformation --region $REGION describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`WebInterface`].OutputValue' --output text)

#Download the worker code
wget $WEBURL/static/queue_processor.py
wget $WEBURL/static/worker.py

#Configure the worker with the proper inputs
echo "QUEUE = 'Workshop'" > config.py
echo "REGION = '$REGION'" >> config.py

#Start the worker processor
python /home/ec2-user/spotlabworker/queue_processor.py > stdout.txt 2>&1
12. We will use User Data to bootstrap our work nodes. Copy and paste the [spotlabworker.sh](https://github.com/aws-samples/ec2-spot-montecarlo-workshop/blob/master/templates/spotlabworker.sh) code from the repo We recommend using grabbing the latest code from the repo, but you can review the script below.

<pre>
#!/bin/bash
# Install Dependencies
yum -y install git python-numpy python-matplotlib python-scipy
pip install pandas-datareader
pip install scipy
pip install boto3

#Populate Variables
REGION=`curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}'`
mkdir /home/ec2-user/spotlabworker
chown ec2-user:ec2-user /home/ec2-user/spotlabworker
cd /home/ec2-user/spotlabworker
WEBURL=$(aws cloudformation --region $REGION describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`WebInterface`].OutputValue' --output text)

#Download the worker code
wget $WEBURL/static/queue_processor.py
wget $WEBURL/static/worker.py

#Configure the worker with the proper inputs
echo "QUEUE = 'Workshop'" > config.py
echo "REGION = '$REGION'" >> config.py

#Start the worker processor
python /home/ec2-user/spotlabworker/queue_processor.py > stdout.txt 2>&1
</pre>
12. Under **Tags**, Enter *Name* for **Key**. Enter **WorkerNode** for *Value*.
12. Under **Tags**, Enter *Name* for **Key**. Enter **WorkerNode** for *Value*.
13. Under **IAM instance profile**, pull the dropdown and select the profile beginning with the workshop name you configured in the CloudFormation Template.
13. Select the Security Group named after your Workshop.
14. We will accept the rest of the defaults, but take a moment at look at the options that you can configure for your Spot Fleet
Expand All @@ -260,7 +260,7 @@ The CloudFormation template deployed a web server that will serve as the user in
#### Evaluate the Results
1. Check your S3 Bucket. In a few minutes you should see results start appearing the bucket.
2. If you monitor the SQS queue for messages you should see them being picked up by the worker nodes.


**You've completed Lab 3, Congrats!**

#### Extra Credit
Expand All @@ -272,24 +272,67 @@ The CloudFormation template deployed a web server that will serve as the user in
<a name="lab4"></a>
### Lab 4 - Leverage a Fully Managed Solution using AWS Batch

Coming Soon!
1. From the AWS Console, enter URL as https://console.aws.amazon.com/batch/home?region=REPLACE_ME_WITH_REGION#/wizard .
Replace REPLACE_ME_WITH_REGION in the above URL with the region that you are using.

2. Select/Enter the following values
* **How would you like to run your job ?** : No job submission and hit Next
* **Compute environment name** : montecarlo-batch-worker
* **Service role** and **EC2 instance role** : Leave it defaulted to "Create a new Role"
* **Provisioning Model** : Spot
* **Allowed instance typese** : optimal
* **Spot fleet role** : select the role created earlier
* **Minimum vCPUs** : 0
* **Desired vCPUs** : 0
* **Maximum vCPUs** : 20
* **VPC Id** : VPC as created earlier
* **Subnets** : Any two subnets in the VPC
* **Security groups** : Security Group as created earlier
* **Job queue name** : montecarlo-batch-worker

3. Click **Create** . It will take less than one minute for the setup to complete. Once complete, click on **View Dashboard**
4. Go to **Job Definition** , hit **Create** and enter the following details
* **Job definition name** : montecarlo-queue-processor
* **Job role** : Select the one that appears in drop down, as created during setup
* **Container image** : anshrma/montecarlo-workshop-worker:latest
* **Memory (MiB)** : 500
* **Environment variables (Key)** : REGION
* **Environment variables (Value)** : Name the region you are using, example us-east-1
* Leave everything as default and click **Create job Definition**
5. Now we are ready to submit a job (with the definition created above) and run it against the compute environment created above.
Go to Jobs , select **Submit job** and enter the following details
* **Job name** : montecarlo-batch-first-run
* **Job definition** : Select the one created above
* **Job queue** : Select the one created above
* Leave everything as default and click **Submit Job**

This will create the EC2 Instances using Spot price as bid during creating the compute environment.
This process may take 2-3 minutes. When you refresh the screen, you will see the staus of the job getting transitioned from submitted > pending > runnable > starting > running.

![Job Status](images/batch-job-status.png)

#### Evaluate the Results
1. Once the job reaches **Running** state, check your S3 Bucket. In a few minutes you should see results start appearing the bucket.
2. If you monitor the SQS queue for messages you should see them being picked up by the worker container.




<a name="cleanup"></a>
## Clean Up
## Clean Up
Hopefully you've enjoyed the workshop and learned a few new things. Now follow these steps to make sure everything is cleaned up.

1. In the EC2 Console > Spot Requests, click **Cancel Spot request** under **Actions**. Make sure **Terminate instances** is checked.
1. In the EC2 Console > Spot Requests, click **Cancel Spot request** under **Actions**. Make sure **Terminate instances** is checked.
2. In the SQS Console, delete the queue that you created earlier. This is located under **Queue Actions**.
3. In the S3 Console, locate the resultsBucket that was created for your workshop. Click on the bucket and select **Empty bucket**. You will need to copy and paste the bucket name in to confirm the aciton.
4. In the CloudFormation template, select the workshop stack and select **Actions** and then **Delete stack**.

<a name="appendix"></a>
##Appendix

### Estimated Costs
The estimated cost for running this 2.5 hour workshop will be less than $5.

4. In the CloudFormation template, select the workshop stack and select **Actions** and then **Delete stack**.

<a name="appendix"></a>
##Appendix

### Estimated Costs
The estimated cost for running this 2.5 hour workshop will be less than $5.

### Learning Resources:


Binary file added images/batch-job-status.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions templates/monte-carlo-workshop.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,7 @@ Resources:
Principal:
Service:
- ec2.amazonaws.com
- ecs-tasks.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
Expand All @@ -380,6 +381,7 @@ Resources:
- ec2:CreateTags
- ec2:AssociateAddress
- cloudformation:Describe*
- logs:*
Path: /
Type: AWS::IAM::Role
spotFleetRole:
Expand Down

0 comments on commit f7e2797

Please sign in to comment.