-
Notifications
You must be signed in to change notification settings - Fork 16
Benchmarking Platform
This benchmarking platform is based on Docker and uses AWS EC2 instances to run benchmarks. Also GitHub Actions (GA) are involved to trigger the task of building and publishing Docker images and preparing the benchmarking VMs.
The system is based on two Github Action workflows that are described in the subsequent sections more in detail:
-
build_docker_imagesdefined in docker.yml -
run_benchmarksdefined in benchmark.yml
Credentials to AWS, Docker, and also the private deployment key for this repository are passed using the Secrets datastore of this repository.
The following explains the folder structure of this repository by the example of SEAL. Only files that are expected to be present for every tool are described.
└── SEAL <<< main folder, one for each tool
├── Dockerfile <<< Docker image based on image_base/Dockerfile + evaluation programs (source/ directory)
├── README.md <<< additional information (if required)
├── image_base <<< folder for base Docker image
│ ├── Dockerfile <<< Docker image containing SEAL + all required dependencies (is pushed to DockerHub)
└── source <<< the evaluation programs adapted to work with SEAL (are copied into container)
├── CMakeLists.txt
└── laplacian_sharpening
├── laplacian_sharpening_batched.cpp
└── laplacian_sharpening_batched.h
Workflow file: docker.yml
The workflow is executed every time any Dockerfile in a image_base/ directory is added or modified.
It builds all defined Dockerfiles and pushes the images to the given repository at DockerHub.
The workflow must be extended every time a new base image is added by copying the step that uses the mr-smithers-excellent/docker-build-push@v3 action and adapting image and dockerfile appropriately.
Workflow file: benchmark.yml
The workflow defines several variables (see env: section) that are available to all jobs and their steps.
For example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that once defined are automatically used by the AWS CLI as credentials.
Also, there is NUM_RUNS environment variable that defines the number of test runs to be executed for each application.
This variable is passed into the Docker containers wherein the benchmarking is performed and is used by the benchmarking application.
The workflow consists of three jobs: setup, run-benchmarks, and generate-plots that are described following.
This job generates a timestamp in the format YYYYMMDD_HHMMSS that is used for the name of the folder in the S3 bucket.
The timestamp is made available to all subsequent jobs to ensure that all save their results in the same folder.
For that, two things are important: First, the timestamp is mapped from the step output to the job output, otherwise it cannot be accessed.
Second, the run-benchmarks job specifies the setup job as a dependency, i.e., starts after the setup job finished.
First, the GitHub Action creates a new EC2 instance by using the AWS CLI that is pre-installed in the ubuntu-20.04 runner.
The step then waits (blocking) until the instance is ready to continue with the next step.
The command echo "##[set-output name=instance_id;]$instanceId" makes the instance ID accessible via ${{ steps.container_creation.outputs.instance_id }} in subsequent steps.
Instead of SSH, the GitHub Action uses the AWS Systems Manager Agent (SSM) to communicate with the VM. This has the advantage of not requiring any SSH key but instead only the AWS token that is required anyway. The step then adds the SoK repository SSH deployment key, clones the repository, and starts the respective evaluation docker container.
For ensuring that the VM is shut down in case of an error, we execute the command echo "sudo halt" | at now + ${{ env.instance-termination-timeout }} minutes that automatically shuts down the VM after the amount of minutes specified in the instance-termination-timeout workflow variable. Due to the defined termination behavior of the VM, the VM is terminated in case of a shutdown instead of stopped only.
This job waits until all benchmarks were executed and then copies all plotting files (*.py) from the repository folder (scripts/plotting/) to the plot/ folder in the respective S3 benchmark folder (e.g., 20200804_160933/plot) of the current run.
As AWS SSM just sends commands to the VM, i.e., the workflow does not know if the benchmark finished, there is a (blocking) waiting mechanism implemented. This mechanism checks the VM's state each minute using the AWS CLI and aborts if the value defined by the workflow variable instance-termination-timeout (e.g., 30 minutes) is reached. In that case, the workflow continues with the job's execution.
Then, the workflow installs Python and the dependencies by taking the requirements.txt. Finally, the plot_all.py is executed that generates and uploads the plots to S3.
This section quickly summarizes the required AWS configuration to run the GAs.
The GitHub Action in the run_benchmarks workflow uses a AWS user to spin up the EC2 VMs.
Also, the evaluation docker image (the one that includes the evaluation programs) uses the same user to upload the results to a S3 bucket.
As such, the AWS user sok-repository-github-action requires the following permissions:
- AmazonEC2FullAccess (AWS managed policy) – for creating and running EC2 VMs
- AmazonSSMFullAccess (AWS managed policy) - for accessing the VM via SSM
- AddRoleToInstance (managed policy) - for assigning the AmazonSSMFullAccess role to the EC2 VM
- AmazonS3FullAccess (AWS managed policy) - for listing and write access to the sok-repository-eval-benchmarks S3 bucket
For reproducibility, the policy summaries of the managed policies are given following.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:AssociateIamInstanceProfile",
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::400029883903:role/ec2-amazonSSMFullAccess"
}
]
}
A new AMI (ID: ami-0545c20ee4d4e0ea3) was created based on the official Ubuntu 20.04 LTS - Focal by Canonical.
Modifications made:
- Docker installed
The S3 bucket's URL is s3://sok-repository-eval-benchmarks.
The bucket contains a directory for each workflow run that is named according to the start date/time of the run-benchmarks workflow.
This timestamp is generated by the setup job in the workflow.
The following shows the content of the bucket as an example:
─── 20200805_181943
│ └── ...
├── 20200805_201905
│ └── ...
├── 20200805_203445
│ ├── ...
├── 20200806_001631 // benchmark started at August 6, 2020 at 00:16:31
│ ├── ALCHEMY // tool folder
│ │ └── alchemy_cardio.csv // result for benchmark program 'cardio'
│ ├── Cingulata
│ │ └── cingulata_cardio.csv
│ ├── Lobster
│ │ ├── lobster_cardio.csv
│ ├── SEAL
│ │ └── seal_cardio.csv
│ ├── ..
│ └── plot // plot scripts + plots generated by workflow
│ ├── plot_all.py
│ ├── plot_cardio.pdf
│ ├── plot_cardio.png
│ ├── plot_cardio.py
│ ├── requirements.txt
│ └── s3_utils.py
The benchmarking EC2 instances can either be accessed via SSM using the EC2 dashboard and by choosing Connect -> Session Manager or by using EC2 Instance Connect that automatically generates a new SSH key, transfers it to the VM with a 60 seconds validity, and connects to it.
A connection to a VM via EC2 Instance connect can be established either by using the EC2 dashboard and choosing Connect -> EC2 Instance Connect (browser-based SSH connection) or by using the EC2 Instance Connect CLI tool.
Install the EC2 Instance Connect CLI tool on your local machine: pip3 install ec2instanceconnectcli. If the command mssh works, the installation was successful.
Thereafter, simply run mssh ubuntu@instance-id where ubuntu is the username of the VM.
NOTE: The IAM user must have the required permission to send a public SSH key to the VM (and also the ec2:DescribeInstances permission as this is used by the EC2 Instance Connect tool). See the official docs, Step 4 for details. The required policy is already added to the Administrators group such that there's no additional change required if a user is in this group.
The benchmarks can be executed by triggering the repository_dispatch event.
This event cannot be executed via the GitHub website.
A POST request must be invoked, for example, via cURL.
For that, it is require to authenticate to the GitHub API using a personal access token, which can be created in the Developer Settings of your profile.
Access permissions for the repo scope is required for the token.
Replace your-username with your GitHub username and your-personal-access-token with your previously generated access token.
curl -X POST https://api.github.com/repos/MarbleHE/SoK/dispatches -H 'Accept: application/vnd.github.everest-preview+json' -u your-username:your-personal-access-token --data '{"event_type": "run_benchmarks"}'
- Install the GitHub Actions extension
- When prompted for a PAT, enter your personal access token.
- Right-click on the workflow and choose trigger workflow (see the workflow's description page for details).
The progress of the workflow execution can be observed on the workflow's status page. Note that the workflow finishes after sending the respective commands to the created EC2 instances, i.e., it does not not wait until the command execution finished.
At the end of the workflow execution, a command ID and a AWS CLI command is printed. This can be used to check the SSM task progress and its output is given as an example in the following:
> aws ssm list-command-invocations --command-id ad7a56c1-20f1-4413-b4eb-407171711a04 --details
{
"CommandInvocations": [
{
"CommandId": "ad7a56c1-20f1-4413-b4eb-407171711a04",
"InstanceId": "i-02c20258e4880cfe6",
"InstanceName": "",
"Comment": "Executed by Github Actions",
"DocumentName": "AWS-RunShellScript",
"DocumentVersion": "",
"RequestedDateTime": "2020-07-20T15:55:04.416000+02:00",
"Status": "InProgress", <<< shows the current status of command execution
"StatusDetails": "InProgress",
"StandardOutputUrl": "",
"StandardErrorUrl": "",
"CommandPlugins": [
{
"Name": "aws:runShellScript",
"Status": "InProgress",
"StatusDetails": "InProgress",
"ResponseCode": -1,
"Output": "", <<< shows the output of the executed command
"StandardOutputUrl": "",
"StandardErrorUrl": "",
"OutputS3Region": "",
"OutputS3BucketName": "",
"OutputS3KeyPrefix": ""
}
],
"ServiceRole": "",
"NotificationConfig": {
"NotificationArn": "",
"NotificationEvents": [],
"NotificationType": ""
},
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "",
"CloudWatchOutputEnabled": false
}
}
]
}
After connecting to an instance (see Connecting to Instances), you can run docker ps to determine the container's ID, followed by docker logs <instance ID> to see the containers stdout/stderr output. You can append the parameter -f to the docker logs to follow the output.
If you would like to execute a command in the running container, use docker exec -it <instance ID> /bin/bash to open a new bash session in the container.
The results must manually be retrieved from the S3 bucket sok-repository-eval-benchmarks.
The subfolder plot/ in a workflow's run folder contains plots generated during the workflow as PDF and PNG.
The interactive visualization website is currently work in progress, see Visualization
- Home
- Compilers & Optimizations
- Libraries
- Benchmark Programs
- Implementation Docs