Skip to content

Commit

Permalink
[IT-1729] A lambda to stop or terminate all EC2 instances in an account
Browse files Browse the repository at this point in the history
  • Loading branch information
ConsoleCatzirl committed Apr 10, 2024
1 parent 1409b12 commit d25da9d
Show file tree
Hide file tree
Showing 12 changed files with 546 additions and 323 deletions.
2 changes: 1 addition & 1 deletion .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
relative_files = True

# Use 'source' instead of 'omit' in order to ignore 'tests/unit/__init__.py'
source = hello_world
source = ec2_terminator
9 changes: 5 additions & 4 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: 3.9
- uses: pre-commit/[email protected].0
- uses: pre-commit/[email protected].1

pytest:
runs-on: ubuntu-latest
Expand All @@ -27,12 +27,13 @@ jobs:
- run: pip install -U pipenv
- run: pipenv install --dev
- run: pipenv run coverage run -m pytest tests/ -vv
- run: pipenv run coverage report -m
- name: upload coverage to coveralls
uses: coverallsapp/github-action@v2

sam-build-and-lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: ./.github/actions/sam-build
- run: sam validate --lint --template .aws-sam/build/template.yaml
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
rev: v4.6.0
hooks:
# On Windows, git will convert all CRLF to LF, but only after all hooks are done executing.
# yamllint will fail before git has a chance to convert line endings, so line endings must be explicitly converted before yamllint
Expand All @@ -11,16 +11,16 @@ repos:
- id: trailing-whitespace
- id: check-ast
- repo: https://github.com/adrienverge/yamllint
rev: v1.27.1
rev: v1.35.1
hooks:
- id: yamllint
- repo: https://github.com/awslabs/cfn-python-lint
rev: v0.63.2
rev: v0.86.2
hooks:
- id: cfn-python-lint
files: template\.(json|yml|yaml)$
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.3.1
rev: v1.5.5
hooks:
- id: remove-tabs
- repo: https://github.com/aristanetworks/j2lint.git
Expand Down
5 changes: 1 addition & 4 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,10 @@ url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "~=7.1"
pytest = "~=8.1"
pytest-mock = "~=3.8"
boto3 = "~=1.24"
coverage = "~=7.3"

[packages]
crhelper = "~=2.0"

[requires]
python_version = "3.9"
191 changes: 91 additions & 100 deletions Pipfile.lock

Large diffs are not rendered by default.

66 changes: 48 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,44 @@
# lambda-template
A GitHub template for quickly starting a new AWS lambda project.
# lambda-ec2-terminator

## Naming
Naming conventions:
* for a vanilla Lambda: `lambda-<context>`
* for a Cloudformation Transform macro: `cfn-macro-<context>`
* for a Cloudformation Custom Resource: `cfn-cr-<context>`
An AWS lambda to stop or terminate all EC2 instances in the current account.

## Operation

This lambda will scan the account it is deployed to for running or stopped EC2
instances and will stop or terminate them. Stop Protection or Termination
Protection should be used to prevent instances from being effected.

### Parameters

| Parameter Name | Default Value | Allowed Values |
|----------------|---------------|-----------------------|
| Ec2Action | "STOP" | "STOP" or "TERMINATE" |

#### Ec2Action

The EC2 action to take on running instances: either stop or terminate.

### Running

This lambda is triggered by a scheduled CloudWatch event at 2am UTC (8pm PST).

## Development

### Contributions

Contributions are welcome.

### Setup Development Environment

Install the following applications:
* [AWS CLI](https://github.com/aws/aws-cli)
* [AWS SAM CLI](https://github.com/aws/aws-sam-cli)
* [pre-commit](https://github.com/pre-commit/pre-commit)
* [pipenv](https://github.com/pypa/pipenv)

- [AWS CLI](https://github.com/aws/aws-cli)
- [AWS SAM CLI](https://github.com/aws/aws-sam-cli)
- [pre-commit](https://github.com/pre-commit/pre-commit)
- [pipenv](https://github.com/pypa/pipenv)

### Install Requirements

Run `pipenv install --dev` to install both production and development
requirements, and `pipenv shell` to activate the virtual environment. For more
information see the [pipenv docs](https://pipenv.pypa.io/en/latest/).
Expand All @@ -29,6 +47,7 @@ After activating the virtual environment, run `pre-commit install` to install
the [pre-commit](https://pre-commit.com/) git hook.

### Update Requirements

First, make any needed updates to the base requirements in `Pipfile`,
then use `pipenv` to regenerate both `Pipfile.lock` and
`requirements.txt`. We use `pipenv` to control versions in testing,
Expand All @@ -49,17 +68,21 @@ $ pipenv requirements > requirements.txt
```

Additionally, `pre-commit` manages its own requirements.

```shell script
$ pre-commit autoupdate
```

### Create a local build

Use a Lambda-like docker container to build the Lambda artifact

```shell script
$ sam build --use-container
```

### Run unit tests

Tests are defined in the `tests` folder in this project, and dependencies are
managed with `pipenv`. Install the development dependencies and run the tests
using `coverage`.
Expand All @@ -71,16 +94,18 @@ $ pipenv run coverage run -m pytest tests/ -vv
Automated testing will upload coverage results to [Coveralls](coveralls.io).

### Run integration tests

Running integration tests
[requires docker](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-api.html)

```shell script
$ sam local invoke HelloWorldFunction --event events/event.json
$ sam local invoke TerminatorFunction --event events/event.json
```

## Deployment

### Deploy Lambda to S3

Deployments are sent to the
[Sage cloudformation repository](https://bootstrap-awss3cloudformationbucket-19qromfd235z9.s3.amazonaws.com/index.html)
which requires permissions to upload to Sage
Expand All @@ -98,6 +123,7 @@ aws s3 cp .aws-sam/build/lambda-template.yaml s3://bootstrap-awss3cloudformation
## Publish Lambda

### Private access

Publishing the lambda makes it available in your AWS account. It will be accessible in
the [serverless application repository](https://console.aws.amazon.com/serverlessrepo).

Expand All @@ -106,6 +132,7 @@ sam publish --template .aws-sam/build/lambda-template.yaml
```

### Public access

Making the lambda publicly accessible makes it available in the
[global AWS serverless application repository](https://serverlessrepo.aws.amazon.com/applications)

Expand All @@ -118,6 +145,7 @@ aws serverlessrepo put-application-policy \
## Install Lambda into AWS

### Sceptre

Create the following [sceptre](https://github.com/Sceptre/sceptre) file
config/prod/lambda-template.yaml

Expand All @@ -133,20 +161,22 @@ stack_tags:
```
Install the lambda using sceptre:
```shell script
sceptre --var "profile=my-profile" --var "region=us-east-1" launch prod/lambda-template.yaml
```

### AWS Console

Steps to deploy from AWS console.

1. Login to AWS
2. Access the
[serverless application repository](https://console.aws.amazon.com/serverlessrepo)
-> Available Applications
3. Select application to install
4. Enter Application settings
5. Click Deploy
1. Access the
[serverless application repository](https://console.aws.amazon.com/serverlessrepo)
-> Available Applications
1. Select application to install
1. Enter Application settings
1. Click Deploy

## Releasing

Expand Down
File renamed without changes.
157 changes: 157 additions & 0 deletions ec2_terminator/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
import json
import logging
import os
import time

import boto3


logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s')
for lib in ["botocore", "urllib3"]:
log = logging.getLogger(lib)
log.setLevel(logging.WARNING)


def list_regions():
client = boto3.client('ec2')

region_desc = client.describe_regions()
regions = [region['RegionName'] for region in region_desc['Regions']]

logging.debug(f"Regions: {regions}")
return regions


def list_instances(region):
"""Return a list of instance IDs in the given region to stop or terminate"""
client = boto3.client('ec2', region_name=region)

# This intentionally reprocesses stopped instances for debugging
state_filters = [{
'Name': 'instance-state-name',
'Values': ['running', 'stopping', 'stopped'],
}]

instances = []

pager = client.get_paginator('describe_instances')
for page in pager.paginate(Filters=state_filters):
for rsvp in page['Reservations']:
for ec2 in rsvp['Instances']:
ec2_id = ec2['InstanceId']
logging.debug(f"EC2: {ec2}")
instances.append(ec2_id)

logging.debug(f"Instances found: {instances}")
return instances


def stop_instances(instances, region):
"""Stop all given instances"""
client = boto3.client(region_name=region)

stopped = []
resp = client.stop_instances(InstanceIds=instances)
if resp['StoppingInstances']:
stopped = [s['InstanceId'] for s in resp['StoppingInstances']]

logging.debug(f"Stopped: {stopped}")
return stopped


def terminate_instances(instances, region):
"""Terminate all given instances"""
client = boto3.client(region_name=region)

terminated = []
resp = client.terminate_instances(InstanceIds=instances)
if resp['TerminatingInstances']:
terminated = [t['InstanceId'] for t in resp['TerminatingInstances']]

logging.debug(f"Terminated :{terminated}")
return terminated


def lambda_handler(event, context):
"""Lambda function to stop or terminate all EC2 instances in the current account.
Parameters
----------
event: dict, required
API Gateway Lambda Proxy Input Format
Event doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format
context: object, required
Lambda Context runtime methods and attributes
Context doc: https://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html
Returns
------
API Gateway Lambda Proxy Output Format: dict
Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
"""

try:
# Are we stopping or terminating instances?
ec2_action = 'stop'
if os.environ.get('EC2_ACTION', '') == 'TERMINATE':
ec2_action = 'terminate'

# Get the list of regions
regions = list_regions()
if not regions:
raise ValueError("No available regions")

# List of stopped or terminated instances
found = False
processed = []

# Iterate over every region
for region in regions:
logging.info(f"Region: {region}")
ec2_instances = list_instances(region)

# Stop or terminate any instances found
if ec2_instances:
found = True
if ec2_action == 'terminate':
logging.info(f"Terminating Instances: {ec2_instances}")
terminated = terminate_instances(ec2_instances, region)
processed.extend(terminated)
else:
logging.info(f"Stopping Instances: {ec2_instances}")
stopped = stop_instances(ec2_instances, region)
processed.extend(stopped)
else:
logging.debug("No instances found")

# Report results
if found and not processed:
raise RuntimeError("Some instances failed to stop or terminate")
elif not found:
message = "No running or stopped instances found"
elif ec2_action == 'terminate':
message = f"Instances terminated: {processed}"
else:
message = f"Instances stopped: {processed}"

logging.info(message)
return {
"statusCode": 200,
"body": json.dumps({
"message": message,
})
}

except Exception as exc:
logging.exception(exc)
return {
"statusCode": 500,
"body": json.dumps({
"message": str(exc)
}),
}
Loading

0 comments on commit d25da9d

Please sign in to comment.