[AWS EC2] Add support for configuring EBS volumes in EC2 lithops workers

lithops-cloud · Jan 18, 2025 · 41482bc · 41482bc
1 parent c616131
commit 41482bc
Show file tree

Hide file tree

Showing 7 changed files with 103 additions and 41 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,8 @@
 
 ### Added
 - [Core] Added support for python 3.13
+- [AWS EC2] Add support for configuring EBS volumes in EC2 lithops workers
+- [AWS EC2] Add support for specifying CIDR block in EC2 public subnet
 
 ### Changed
 - 

diff --git a/config/README.md b/config/README.md
@@ -133,12 +133,12 @@ config = {
     }
 }
 
-def hello_world(name):
-    return f'Hello {name}!'
+def hello_world(number):
+    return f'Hello {number}!'
 
 if __name__ == '__main__':
     fexec = lithops.FunctionExecutor(config=config)
-    fexec.call_async(hello_world, 'World')
+    fexec.map(hello_world, [1, 2, 3, 4])
     print(fexec.get_result())
 ```
 

diff --git a/docs/source/compute_config/aws_ec2.md b/docs/source/compute_config/aws_ec2.md
@@ -109,13 +109,14 @@ In summary, you can use one of the following settings:
 |---|---|---|---|---|
 |aws_ec2 | region | |no | Region name, for example: `eu-west-1`. Lithops will use the `region` set under the `aws` section if it is not set here |
 |aws_ec2 | instance_role | | yes | EC2 Instance role name created in the configuration section above. Do not use the full ARN here; only the role name is required. For example: `ec2LithopsInstanceRole`|
-|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:) |
-|aws_ec2 | subnet_id | | no | Subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:) |
-|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22 and 8080 open |
+|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:). If not provided, Lithops will create a new VPC |
+|aws_ec2 | public_subnet_id | | no | Public subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:). If not provided, Lithops will create a new public subnet |
+|aws_ec2 | public_subnet_cidr_block | 10.0.1.0/24 | no | In case a `public_subnet_id` is not provided, Lithops will create a new subnet with this CIDR block |
+|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22, 6379, 8080 and 8081 open. If not provided, Lithops will create a new security group |
 |aws_ec2 | ssh_key_name | | no | SSH Key name. You can find the available keys in the [EC2 console page](https://console.aws.amazon.com/ec2/v2/home#KeyPairs:). Create a new one or upload your own key if it does not exist|
 |aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
 |aws_ec2 | ssh_password |  |no | Password for accessing the worker VMs. If not provided, it is created randomly|
-|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
+|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. If not provided, Lithops will use the default path and create a new ssk key for the VPC |
 |aws_ec2 | request_spot_instances | True | no | Request spot instance for worker VMs|
 |aws_ec2 | target_ami | | no | Virtual machine image id. Default is Ubuntu Server 22.04 |
 |aws_ec2 | master_instance_type | t2.micro | no | Profile name for the master VM |
@@ -130,6 +131,47 @@ In summary, you can use one of the following settings:
 |aws_ec2 | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to  **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to **reuse** will try to reuse running workers if exist |
 
 
+## Additional configuration
+
+# Elastic Block Store (EBS)
+
+To attach EBS volumes to an EC2 instance in Lithops, you can configure the `aws_ec2` section as follows.
+
+```yaml
+aws_ec2:
+    execution_role: <EXECUTION_ROLE_ARN>
+    region: <REGION_NAME>
+    ...
+    ebs_volumes:
+        - device_name: /dev/xvda
+          ebs:
+              volume_size: 100
+              volume_type: gp2
+              delete_on_termination: true
+              encrypted: false
+              kms_key_id: <KMS_KEY_ARN>
+        - device_name: /dev/xvdf
+          ebs:
+              volume_size: 50
+              volume_type: gp3
+              delete_on_termination: true
+              encrypted: false
+              iops: 3000
+              throughput: 125
+        ...
+```
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+| ebs | volume_size | 8  | No | Size of the volume in GiB.                                                                     |
+| ebs | volume_type | 'gp2' | No | Type of volume. Options: 'gp2', 'gp3', 'io1', 'io2', 'sc1', 'st1', 'standard'.                |
+| ebs | delete_on_termination| True | No | Whether the volume is deleted automatically when the instance is terminated.                   |
+| ebs | encrypted | False | No | Whether the volume is encrypted.                                                               |
+| ebs | kms_key_i | | No | ARN of the KMS key used for encryption. If not provided, the default AWS-managed key is used.  |
+| ebs | iops | | No | Provisioned IOPS for 'io1', 'io2', or 'gp3' volumes.                                           |
+| ebs | throughput | | No | Throughput in MiB/s for 'gp3' volumes.                                                         |
+
+
 ## Consume mode
 
 In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.

diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
@@ -71,22 +71,30 @@ Here is an example of providing configuration keys for IBM Cloud Functions and I
 
 .. code:: python
 
-    import lithops
-
-
-    config = {'lithops': {'backend': 'ibm_cf', 'storage': 'ibm_cos'},
-              'ibm': {'region': 'REGION',
-                      'iam_api_key': 'IAM_API_KEY',
-                      'resource_group_id': 'RESOURCE_GROUP_ID'}
-              'ibm_cos': {'storage_bucket': 'STORAGE_BUCKET'}}
-
-    def hello_world(name):
-        return 'Hello {}!'.format(name)
-
-    if __name__ == '__main__':
-        fexec = lithops.FunctionExecutor(config=config)
-        fexec.call_async(hello_world, 'World')
-        print(fexec.get_result())
+   import lithops
+
+   config = {
+      'lithops': {
+         'backend': 'code_engine',
+         'storage': 'ibm_cos'
+      },
+      'ibm': {
+         'region': 'REGION',
+         'iam_api_key': 'IAM_API_KEY',
+         'resource_group_id': 'RESOURCE_GROUP_ID'
+      },
+      'ibm_cos': {
+         'storage_bucket': 'STORAGE_BUCKET'
+      }
+   }
+
+   def hello_world(number):
+      return f'Hello {number}!'
+
+   if __name__ == '__main__':
+      fexec = lithops.FunctionExecutor(config=config)
+      fexec.map(hello_world, [1, 2, 3, 4])
+      print(fexec.get_result())
 
 
 .. _config-reference-label:

diff --git a/docs/source/lithops_config_keys.csv b/docs/source/lithops_config_keys.csv
@@ -1,6 +1,6 @@
 Group;Key;Default;Mandatory;Additional info
-lithops;backend;``ibm_cf``;no;Compute backend implementation. IBM Cloud Functions is the default. If not set, Lithops will check the `mode` and use the `backend` set under the `serverless` or `standalone` sections described below.
-lithops;storage;``ibm_cos``;no;Storage backend implementation. IBM Cloud Object Storage is the default.
+lithops;backend;``aws_lambda``;no;Compute backend implementation. AWS Lambda is the default.
+lithops;storage;``aws_s3``;no;Storage backend implementation. AWS S3 is the default.
 lithops;data_cleaner;``True``;no;If set to True, then the cleaner will automatically delete all the temporary data that was written into `storage_bucket/lithops.jobs`.
 lithops;monitoring;``storage``;no;Monitoring system implementation. One of: **storage** or **rabbitmq**.
 lithops;monitoring_interval;``2``;no;Monitoring check interval in seconds in case of **storage** monitoring.

diff --git a/lithops/standalone/backends/aws_ec2/aws_ec2.py b/lithops/standalone/backends/aws_ec2/aws_ec2.py
@@ -182,7 +182,8 @@ def _create_subnets(self):
         if 'public_subnet_id' not in self.config:
             logger.debug(f'Creating new public subnet in VPC {self.vpc_name}')
             response = self.ec2_client.create_subnet(
-                CidrBlock='10.0.1.0/24', VpcId=self.config['vpc_id'],
+                CidrBlock=self.config['public_subnet_cidr_block'],
+                VpcId=self.config['vpc_id'],
             )
             public_subnet_id = response['Subnet']['SubnetId']
             self.config['public_subnet_id'] = public_subnet_id
@@ -1157,20 +1158,28 @@ def _create_instance(self, user_data=None):
         """
         Creates a new VM instance
         """
-        if self.fast_io:
-            BlockDeviceMappings = [
-                {
-                    'DeviceName': '/dev/xvda',
-                    'Ebs': {
-                        'VolumeSize': 100,
-                        'DeleteOnTermination': True,
-                        'VolumeType': 'gp2',
-                        # 'Iops' : 10000,
-                    },
-                },
-            ]
-        else:
-            BlockDeviceMappings = None
+        ebs_volumes = self.config.get('ebs_volumes', [])
+        BlockDeviceMappings = []
+
+        for volume in ebs_volumes:
+            ebs_config = volume.get('ebs', {})
+            block_device = {
+                'DeviceName': volume['device_name'],
+                'Ebs': {
+                    'VolumeSize': ebs_config.get('volume_size', 8),  # Default 8 GiB
+                    'VolumeType': ebs_config.get('volume_type', 'gp2'),
+                    'DeleteOnTermination': ebs_config.get('delete_on_termination', True),
+                    'Encrypted': ebs_config.get('encrypted', False),
+                }
+            }
+            if 'iops' in ebs_config:
+                block_device['Ebs']['Iops'] = ebs_config['iops']
+            if 'throughput' in ebs_config:
+                block_device['Ebs']['Throughput'] = ebs_config['throughput']
+            if 'kms_key_id' in ebs_config:
+                block_device['Ebs']['KmsKeyId'] = ebs_config['kms_key_id']
+
+            BlockDeviceMappings.append(block_device)
 
         LaunchSpecification = {
             "ImageId": self.config['target_ami'],

diff --git a/lithops/standalone/backends/aws_ec2/config.py b/lithops/standalone/backends/aws_ec2/config.py
@@ -28,7 +28,8 @@
     'request_spot_instances': True,
     'delete_on_dismantle': True,
     'max_workers': 100,
-    'worker_processes': 'AUTO'
+    'worker_processes': 'AUTO',
+    'public_subnet_cidr_block': '10.0.1.0/24'
 }
 
 REQ_PARAMS_1 = ('instance_id',)