Skip to content

Latest commit

 

History

History
222 lines (149 loc) · 11 KB

README.md

File metadata and controls

222 lines (149 loc) · 11 KB
Raw

Quota Monitor for AWS

🚀Solution Landing Page | 🚧Feature request | 🐛Bug Report | 📜Documentation Improvement

Note: For any relevant information outside the scope of this readme, please refer to the solution landing page and implementation guide.

Table of content

Solution overview

Quota Monitor for AWS is a reference implementation that provides a foundation for monitoring AWS services' quotas usage. Customers can leverage the solution to monitor quotas across services supported by Amazon Trusted Advisor and AWS Service Quotas; in multiple regions and multiple AWS accounts. The solution integrates with Amazon SNS and Slack to notify customers for service quotas approaching thresholds.

Architecture

The architecture can be broken down into different components, few which are installed in the monitoring account and others are installed in the monitored accounts. The monitoring account also known as hub account, collects all usage events from the monitored accounts (spokes) and raises notifications. Additionally, all the usage events are put on DynamoDB table in the hub account, which can be used to view historical trend of resource usage across all accounts.

Deployment scenarios:

The solution follows hub-spoke model and supports different deployment scenarios

  • Environments where all AWS accounts are part of your AWS Organization
  • Hybrid environments with AWS Organization and independent AWS accounts
  • Environments not using AWS Organizations

hub: For the first two scenarios use quota-monitor-hub.template. For environments not using Organizations, use quota-monitor-hub-no-ou.template. Note: Hub template should be deployed in the monitoring account. For the first two scenarios, this account should also be your delegated administrator for StackSets in the organization.

spoke: Spoke templates are automatically deployed by StackSets for targeted Organizational Units. For hybrid environments and environments not using Organizations, deploy the spoke templates individually in the accounts where monitoring is needed. Note: ta-spoke.template should be deployed in us-east-1 ONLY. sq-spoke.template can be deployed in any region.

Installing pre-packaged solution template

Note: hub, hub-no-ou and sq-spoke templates can be deployed in ANY region; prerequisite and ta-spoke template can be deployed in us-east-1 ONLY.

Parameters for hub template

  • Deployment Configuration: Choose Organizations or Hybrid based on your use-case
  • Notification Configuration: Choose the notifications you want to receive

Note: Deployment Configuration parameter is not available in hub-no-ou template.

Parameters for spoke templates

  • EventBridge bus arn: Arn for the EventBridge bus where you want to send usage events

Note: You may leave rest of the parameters to default.

Customization

The steps given below can be followed if you are looking to customize the solution or extend the solution with newer capabilities

Setup

  • Javascript Pre-requisite: node=v16.17.0 | npm=8.15.0

Clone the repository and run the following commands to install dependencies

git clone https://github.com/aws-solutions/quota-monitor-for-aws.git
cd ./quota-monitor-for-aws
npm ci

(optional) Run the following commands to format and lint the project per the project standards

npm run prettier-format
npm run lint

Note: Following steps have been tested under above pre-requisites

Running unit tests for customization

Run unit tests to make sure added customization passes the tests.

cd ./deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh

✅ Ensure all unit tests pass. Review the generated coverage report.

Build

To build your customized distributable run build from the project root

npm run build:all

✅ All assets are now built.

Deploy

Run the following command from the root of the project

cd ./source/resources
npm ci

Bootstrap your CDK environment

  • This solution requires that you have bootstrapped your AWS account with CDK using the default stack name. If you haven't done so already, run:
npm run cdk -- bootstrap --profile <PROFILE_NAME>

npm run cdk -- deploy <STACK_NAME> --profile <PROFILE_NAME>
npm run orgHub:deploy -- deploy quota-monitor-hub --profile <PROFILE_NAME> (to deploy the hub template in org mode which works with stacksets)

Note:

  • STACK_NAME, substitute the name of the stack that you want to deploy, check cdk app
  • PROFILE_NAME, substitute the name of an AWS CLI profile that contains appropriate credentials for deploying in your preferred region
  • The deployment scripts assume the CDK bootstrap stack is named CDKToolkit. If you've used a custom name for your bootstrap stack, you'll need to modify the get-cdk-bucket script in package.json.
  • For the ORG/HYBRID mode, Ensure that the account or organization you're deploying stacks to has the necessary permissions to access the CDK assets bucket.
  • CDK creates an S3 bucket for assets in only one region. As a result, spoke stacks will deploy successfully only in the region where this bucket is created.

✅ Solution stack is deployed with your customized code.

Independent spoke templates

There are three spoke templates packaged with the solution:

  • ta-spoke: provisions resources to support Trusted Advisor quota checks
  • sq-spoke: provisions resources to support Service Quotas checks
  • sns-spoke: provisions resources to support spoke account-specific notifications

All three spoke templates (TA, SQ, and SNS) are independent standalone stacks that can be individually deployed. You can deploy these spoke stacks and route usage events and notifications to your preferred destinations.

For the SQ spoke stack, you can control which services to monitor by toggling the monitored status of the services in the DynamoDB table ServiceTable. The SNS spoke stack provides an additional option for routing notifications within spoke accounts. For deploying sq-spoke stack:

npm run cdk -- deploy quota-monitor-sq-spoke --parameters EventBusArn=<BUS_ARN> --profile <PROFILE_NAME>

Note: BUS_ARN, substitute the arn of the EventBridge bus where you want to send usage events

SSM Parameter Store based workflow

The solution provisions /QuotaMonitor/OUs and /QuotaMonitor/Accounts SSM Parameter Store. You can modify the parameters at any point after deployment to update the list of targeted organizational-units and accounts for monitoring.

  • /QuotaMonitor/OUs: Once you update the parameter, StackSets takes care of deploying the spoke templates in the targeted OUs

  • /QuotaMonitor/Accounts: Once you update the parameter, you need to deploy the spoke templates individually in the targeted accounts

File Structure

The project consists of several microservices, shared utility lambda layer and CDK resources

|-deployment/
  |-run-unit-test.sh              [ run all unit tests ]
  |-templates                     [ solution pre-baked templates ]
|-source/
  |-lambda
    |-services/
      |-cwPoller/                 [ microservice for polling CloudWatch metrics for quotas usage ]
      |-deploymentManager/        [ microservice for managing CloudFormation StackSet deployments ]
      |-helper/                   [ microservice for helper modules ]
      |-preReqManager/            [ microservice for fulfilling pre-requisites in the management account ]
      |-quotaListManager/         [ microservice for managing quota list that supports usage monitoring ]
      |-reporter/                 [ microservice for putting quota usage details on dynamodb ]
      |-slackNotifier/            [ microservice for raising alerts on slack ]
      |-snsPublisher/             [ microservice for publishing alerts to SNS ]
      |-taRefresher/              [ microservice for refreshing trusted advisor checks ]
    |-utilsLayer/                 [ lambda layer with shared modules, like logger, metrics, try/catch wrapper ]
  |-resources                     [ cdk resources to provision infrastructure ]
|-README.md
|-additional_files                [ CHANGELOG, CODE_OF_CONDUCT, LICENSE, NOTICE, sonar-project.properties etc.]

License

See license here

Collection of operational metrics

This solution collects anonymized operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the implementation guide.

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/LICENSE-2.0

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.