automatically increase service quotas if reached for routes and auth …

…rules (#13)

docs/certificate-users.md

@@ -85,5 +85,5 @@ Modify base2-ciinabox.config.ovpn to include the full location of your extracted
 	echo "key /<path>/user1.key" >> myvpn.config.ovpn
 	echo "cert /<path>/user1.crt" >> myvpn.config.ovpn
 
-Open myvpn.config.ovpn with your favourite openvpn client.
+Open myvpn.config.ovpn with your favorite openvpn client.
 ```

docs/getting-started.md

@@ -50,7 +50,7 @@ This is the default option when launching a ClientVPN using certificated based a
 The following command and required options will launch a new certificate based Client-VPN
 
 ```sh
-cfn-vpn init [name] --bucket [s3-bucket] --server-cn [server certificate name] --subnet-ids [list of subets to associate with the vpn]
+cfn-vpn init [name] --bucket [s3-bucket] --server-cn [server certificate name] --subnet-ids [list of subnets to associate with the vpn]
 ```
 
 
@@ -64,15 +64,15 @@ The following command and required option will launch a new federated based Clie
 
 ```sh
 cfn-vpn init [name] --server-cn [server certificate name] \
-  --subnet-ids [list of subets to associate with the vpn] \
-  --saml-arn [identity providor arn]
+  --subnet-ids [list of subnets to associate with the vpn] \
+  --saml-arn [identity provider arn]
 ```
 
 The default authorization rule for the associated subnets allows all. You can optionally change this by using the `--default-groups` flag to set groups on the default authorization rule. 
 
 ```diff
-! Group id's must be used if creating authorisation rules. 
-! Each SAML providor will have different group id's and means of retrieving them.
+! Group id's must be used if creating authorization rules. 
+! Each SAML provider will have different group id's and means of retrieving them.
 ```
 
 ```sh
@@ -104,16 +104,16 @@ The following command and required option will launch a new directory service ba
 
 ```sh
 cfn-vpn init simple-ad --server-cn [server certificate name] \
-  --subnet-ids [list of subets to associate with the vpn] \
-  --directory-id [aws directirory serivce id]
+  --subnet-ids [list of subnets to associate with the vpn] \
+  --directory-id [aws directory service id]
 ```
 
 The default authorization rule for the associated subnets allows all. You can optionally change this by using the `--default-groups` flag to set groups on the default authorization rule. The group Id is the Active Directory Group ID or SID.
 
 ```sh
 cfn-vpn init simple-ad --server-cn [server certificate name] \
-  --subnet-ids [list of subets to associate with the vpn] \
-  --directory-id [aws directirory serivce id] \
+  --subnet-ids [list of subnets to associate with the vpn] \
+  --directory-id [aws directory service id] \
   --default-groups [list of group ids]
 ```
 
@@ -128,27 +128,30 @@ When using a federated ClientVPN you can modify the default auth to only allow s
 
 ```
 Options:
-  r, [--region=REGION]                                 # AWS Region
-                                                       # Default: ap-southeast-2
-      [--verbose], [--no-verbose]                      # set log level to debug
-      --server-cn=SERVER_CN                            # server certificate common name
-      [--client-cn=CLIENT_CN]                          # client certificate common name
-      [--easyrsa-local], [--no-easyrsa-local]          # run the easyrsa executable from your local rather than from docker
-      [--bucket=BUCKET]                                # s3 bucket, if not set one will be generated for you
-      --subnet-ids=one two three                       # subnet id to associate your vpn with
-      [--default-groups=one two three]                 # groups to allow through the subnet associations when using federated auth
-      [--cidr=CIDR]                                    # cidr from which to assign client IP addresses
-                                                       # Default: 10.250.0.0/16
-      [--dns-servers=one two three]                    # DNS Servers to push to clients.
-      [--split-tunnel], [--no-split-tunnel]            # only push routes to the client on the vpn endpoint
-                                                       # Default: true
-      [--internet-route=INTERNET_ROUTE]                # [subnet-id] create a default route to the internet through a subnet
-      [--protocol=PROTOCOL]                            # set the protocol for the vpn connections
-                                                       # Default: udp
-                                                       # Possible values: udp, tcp
-      [--start=START]                                  # cloudwatch event cron schedule in UTC to associate subnets to the client vpn
-      [--stop=STOP]                                    # cloudwatch event cron schedule in UTC to disassociate subnets to the client vpn
-      [--saml-arn=SAML_ARN]                            # IAM SAML idenditiy providor arn if using SAML federated authentication
-      [--saml-self-service-arn=SAML_SELF_SERVICE_ARN]  # IAM SAML idenditiy providor arn for the self service portal
-      [--directory-id=DIRECTORY_ID]                    # AWS Directory Service directory id if using Active Directory authentication
+  r, [--region=REGION]                                     # AWS Region
+                                                           # Default: ap-southeast-2
+      [--verbose], [--no-verbose]                          # set log level to debug
+      --server-cn=SERVER_CN                                # server certificate common name
+      [--client-cn=CLIENT_CN]                              # client certificate common name
+      [--easyrsa-local], [--no-easyrsa-local]              # run the easyrsa executable from your local rather than from docker
+      [--bucket=BUCKET]                                    # s3 bucket, if not set one will be generated for you
+      --subnet-ids=one two three                           # subnet id to associate your vpn with
+      [--default-groups=one two three]                     # groups to allow through the subnet associations when using federated auth
+      [--cidr=CIDR]                                        # cidr from which to assign client IP addresses
+                                                           # Default: 10.250.0.0/16
+      [--dns-servers=one two three]                        # DNS Servers to push to clients.
+      [--split-tunnel], [--no-split-tunnel]                # only push routes to the client on the vpn endpoint
+                                                           # Default: true
+      [--internet-route=INTERNET_ROUTE]                    # [subnet-id] create a default route to the internet through a subnet
+      [--protocol=PROTOCOL]                                # set the protocol for the vpn connections
+                                                           # Default: udp
+                                                           # Possible values: udp, tcp
+      [--start=START]                                      # cloudwatch event cron schedule in UTC to associate subnets to the client vpn
+      [--stop=STOP]                                        # cloudwatch event cron schedule in UTC to disassociate subnets to the client vpn
+      [--saml-arn=SAML_ARN]                                # IAM SAML identity provider arn if using SAML federated authentication
+      [--saml-self-service-arn=SAML_SELF_SERVICE_ARN]      # IAM SAML identity provider arn for the self service portal
+      [--directory-id=DIRECTORY_ID]                        # AWS Directory Service directory id if using Active Directory authentication
+      [--slack-webhook-url=SLACK_WEBHOOK_URL]              # slack webhook url to send notifications from the scheduler and route populator
+      [--auto-limit-increase], [--no-auto-limit-increase]  # automatically request a AWS service quota increase if limits are hit for route entry and authorization rule limits
+                                                           # Default: true
 ```

docs/routes.md

@@ -121,14 +121,26 @@ Check out the AWS [docs](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/
 
 ### Increasing Limits
 
-Routes per Client VPN endpoint
+**Automatic**
+
+cfn-vpn supports automatically creating requests to increase the limits for `Routes per Client VPN endpoint` (by 10) and `Authorization rules per Client VPN endpoint` (by 20).
+
+This functionality is enabled by default but can be disabled by modifying the vpn setting the `--no-auto-limit-increase` flag
+
+```sh
+cfn-vpn modify [name] --no-auto-limit-increase
+```
+
+**Manual**
+
+`Routes per Client VPN endpoint`
 
 ```sh
-aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-401D78F7 --desired-value 20
+aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-401D78F7 --desired-value [value]
 ```
 
-Authorization rules per Client VPN endpoint
+`Authorization rules per Client VPN endpoint`
 
 ```sh
-aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-9A1BC94B --desired-value 75
+aws service-quotas request-service-quota-increase --service-code ec2 --quota-code L-9A1BC94B --desired-value [value]
 ```

docs/slack-notifications.md

@@ -26,6 +26,7 @@ cfn-vpn modify [name] --slack-webhook-url "https://hooks.slack.com/services/T000
 - `RESOLVE_FAILED`: failed to resolve the provided dns entry
 - `RATE_LIMIT_EXCEEDED`: concurrent modifications of the route table is being rated limited
 - `SUBNET_NOT_ASSOCIATED`: no subnets are associated with the Client VPN
+- `QUOTA_INCREASE_REQUEST`: automatic quota increase made 
 
 ## Scheduler Events
 

lib/cfnvpn/actions/init.rb

@@ -35,12 +35,13 @@ class Init < Thor::Group
     class_option :start, type: :string, desc: 'cloudwatch event cron schedule in UTC to associate subnets to the client vpn'
     class_option :stop, type: :string, desc: 'cloudwatch event cron schedule in UTC to disassociate subnets to the client vpn'
 
-    class_option :saml_arn, desc: 'IAM SAML idenditiy providor arn if using SAML federated authentication'
-    class_option :saml_self_service_arn, desc: 'IAM SAML idenditiy providor arn for the self service portal'
+    class_option :saml_arn, desc: 'IAM SAML identity provider arn if using SAML federated authentication'
+    class_option :saml_self_service_arn, desc: 'IAM SAML identity provider arn for the self service portal'
     class_option :directory_id, desc: 'AWS Directory Service directory id if using Active Directory authentication'
 
     class_option :slack_webhook_url, type: :string, desc: 'slack webhook url to send notifications from the scheduler and route populator'
-
+    class_option :auto_limit_increase, type: :boolean, default: true, desc: 'automatically request a AWS service quota increase if limits are hit for route entry and authorization rule limits'
+
     def self.source_root
       File.dirname(__FILE__)
     end
@@ -70,6 +71,7 @@ def initialize_config
         saml_self_service_arn: @options['saml_self_service_arn'],
         directory_id: @options['directory_id'],
         slack_webhook_url: @options['slack_webhook_url'],
+        auto_limit_increase: @options['auto_limit_increase'],
         routes: []
       }
     end

lib/cfnvpn/actions/modify.rb

@@ -38,6 +38,7 @@ class Modify < Thor::Group
     class_option :stop, type: :string, desc: 'cloudwatch event cron schedule in UTC to disassociate subnets to the client vpn'
 
     class_option :slack_webhook_url, type: :string, desc: 'slack webhook url to send notifications from the scheduler and route populator'
+    class_option :auto_limit_increase, type: :boolean, desc: 'automatically request a AWS service quota increase if limits are hit for route entry and authorization rule limits'
 
     class_option :param_yaml, type: :string, desc: 'pass in cfnvpn params through YAML file'
 

lib/cfnvpn/templates/lambdas/auto_route_populator/app.py

@@ -1,9 +1,11 @@
+import os
 import socket
 import boto3
 from botocore.exceptions import ClientError
 from lib.slack import Slack
 from states import *
 import logging
+from quotas import increase_quota, AUTH_RULE_TABLE_QUOTA_CODE, ROUTE_TABLE_QUOTA_CODE
 
 logger = logging.getLogger(__name__)
 logger.setLevel(logging.INFO)
@@ -90,7 +92,7 @@ def get_routes(client, event):
   )
 
   routes = [route for route in response['Routes'] if event['Record'] in route['Description']]
-  logger.info(f"found {len(routes)} exisiting routes for {event['Record']}")
+  logger.info(f"found {len(routes)} existing routes for {event['Record']}")
   return routes
 
 
@@ -141,6 +143,10 @@ def handler(event,context):
 
   routes = get_routes(client, event)
 
+  auto_limit_increase = os.environ.get('AUTO_LIMIT_INCREASE')
+  route_limit_increase_required = False
+  auth_rules_limit_increase_required = False
+
   for cidr in cidrs:
     route = next((route for route in routes if route['DestinationCidr'] == cidr), None)
 
@@ -158,6 +164,7 @@ def handler(event,context):
           logger.error(f"route for CIDR {cidr} already exists with a different endpoint")
           continue
         elif e.response['Error']['Code'] == 'ClientVpnRouteLimitExceeded':
+          route_limit_increase_required = True
           logger.error("vpn route table has reached the route limit", exc_info=True)
           slack.post_event(
             message=f"unable to create route {cidr} from {event['Record']}",
@@ -166,6 +173,7 @@ def handler(event,context):
           )
           continue
         elif e.response['Error']['Code'] == 'ClientVpnAuthorizationRuleLimitExceeded':
+          auth_rules_limit_increase_required = True
           logger.error("vpn has reached the authorization rule limit", exc_info=True)
           slack.post_event(
             message=f"unable add to authorization rule for route {cidr} from {event['Record']}",
@@ -228,8 +236,20 @@ def handler(event,context):
           revoke_route_auth(client, event, cidr)
           authorize_route(client, event, cidr)
 
-
+  # request limit increase
+  if route_limit_increase_required and auto_limit_increase:
+    case_id = increase_quota(10, ROUTE_TABLE_QUOTA_CODE, event['ClientVpnEndpointId'])
+    if case_id is not None:
+      slack.post_event(message=f"requested an increase for the routes per vpn service quota", state=QUOTA_INCREASE_REQUEST, support_case=case_id)
+    else:
+      logger.info(f"routes per vpn service quota increase request pending")
 
+  if auth_rules_limit_increase_required and auto_limit_increase:
+    case_id = increase_quota(20, AUTH_RULE_TABLE_QUOTA_CODE, event['ClientVpnEndpointId'])
+    if case_id is not None:
+      slack.post_event(message=f"requested an increase for the authorization rules per vpn service quota", state=QUOTA_INCREASE_REQUEST, support_case=case_id)
+    else:
+      logger.info(f"authorization rules per vpn service quota increase request pending")
 
   # clean up any expired routes when the ips for an endpoint change
   expired_routes = [route for route in routes if route['DestinationCidr'] not in cidrs]

lib/cfnvpn/templates/lambdas/auto_route_populator/quotas.py

@@ -0,0 +1,37 @@
+import boto3
+
+ROUTE_TABLE_QUOTA_CODE = 'L-401D78F7'
+AUTH_RULE_TABLE_QUOTA_CODE = 'L-9A1BC94B'
+EC2_SERVICE_CODE = 'ec2'
+IN_PROGRESS = ['PENDING', 'CASE_OPENED']
+
+def get_route_count(endpoint) -> int:
+  client = boto3.client('ec2')
+  response = client.describe_client_vpn_routes(
+    ClientVpnEndpointId=endpoint,
+  )
+  return len(response['Routes'])
+
+def quota_request_open(quota_code) -> bool:
+  client = boto3.client('service-quotas')
+  response = client.list_requested_service_quota_change_history_by_quota(
+    ServiceCode=EC2_SERVICE_CODE,
+    QuotaCode=quota_code
+  )
+  # Status='PENDING'|'CASE_OPENED'|'APPROVED'|'DENIED'|'CASE_CLOSED'
+  return any(req['status'] in IN_PROGRESS for req in response['RequestedQuotas'])
+
+def increase_quota(increase_value, quota_code, endpoint) -> str:
+  if quota_request_open(quota_code):
+    return None
+
+  current_route_count = get_route_count(endpoint)
+  desired_value = current_route_count + increase_value
+
+  client = boto3.client('service-quotas')
+  response = client.request_service_quota_increase(
+    ServiceCode=EC2_SERVICE_CODE,
+    QuotaCode=quota_code,
+    DesiredValue=desired_value
+  )
+  return response['CaseId']

lib/cfnvpn/templates/lambdas/auto_route_populator/states.py

@@ -9,6 +9,7 @@
 RESOLVE_FAILED: failed to resolve the provided dns entry
 RATE_LIMIT_EXCEEDED: concurrent modifications of the route table is being rated limited
 SUBNET_NOT_ASSOCIATED: no subnets are associated with the client vpn
+QUOTA_INCREASE_REQUEST: automatic quota increase made
 """
 
 FAILED = 'FAILED'
@@ -18,4 +19,5 @@
 AUTH_RULE_LIMIT_EXCEEDED = 'AUTH_RULE_LIMIT_EXCEEDED'
 RESOLVE_FAILED = 'RESOLVE_FAILED'
 RATE_LIMIT_EXCEEDED = 'RATE_LIMIT_EXCEEDED'
-SUBNET_NOT_ASSOCIATED = 'SUBNET_NOT_ASSOCIATED'
+SUBNET_NOT_ASSOCIATED = 'SUBNET_NOT_ASSOCIATED'
+QUOTA_INCREASE_REQUEST = 'QUOTA_INCREASE_REQUEST'

lib/cfnvpn/templates/lambdas/lib/slack.py

@@ -12,7 +12,7 @@ def __init__(self, username):
         self.username = username
         self.slack_url = os.environ.get('SLACK_URL')
 
-    def post_event(self, message, state, error=None):
+    def post_event(self, message, state, error=None, support_case=None):
         """Posts event to slack using an incoming webhook
         Parameters
         ----------
@@ -22,6 +22,8 @@ def post_event(self, message, state, error=None):
             the state of the event
         error: str
             error message to add to the message
+        support_case: str
+            displays a aws console link to the support case in the message
         """
 
         if not self.slack_url.startswith('https://hooks.slack.com'):
@@ -35,8 +37,12 @@ def post_event(self, message, state, error=None):
             colour = '#3ead3e'
 
         text = f'Message: {message}\nState: {state}'
+
         if error:
             text += f'\nError: {error}'
+
+        if support_case:
+            text += f'\nSupport Case: <https://console.aws.amazon.com/support/cases#/{support_case}|{support_case}>'
 
         payload = {
             'username': self.username,

lib/cfnvpn/templates/vpn.rb

@@ -285,7 +285,17 @@ def auto_route_populator(name, config)
           ])
         }
 
-        s3_key = CfnVpn::Templates::Lambdas.package_lambda(name: name, bucket: config[:bucket], func: 'auto_route_populator', files: ['auto_route_populator/app.py', 'lib/slack.py', 'auto_route_populator/states.py'])
+        s3_key = CfnVpn::Templates::Lambdas.package_lambda(
+          name: name,
+          bucket: config[:bucket],
+          func: 'auto_route_populator',
+          files: [
+            'auto_route_populator/app.py',
+            'auto_route_populator/quotas.py',
+            'lib/slack.py',
+            'auto_route_populator/states.py'
+          ]
+        )
 
         Lambda_Function(:CfnVpnAutoRoutePopulator) {
           Runtime 'python3.8'
@@ -299,7 +309,8 @@ def auto_route_populator(name, config)
           })
           Environment({
             Variables: {
-              SLACK_URL: config[:slack_webhook_url] || ''
+              SLACK_URL: config[:slack_webhook_url] || '',
+              AUTO_LIMIT_INCREASE: config[:auto_limit_increase]
             }
           })
           Tags([