Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updated support for benchmark language versions #198

Merged
merged 32 commits into from
Jul 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
2f09f26
updated support for benchmark language versions
octonawish-akcodes Mar 27, 2024
483fd88
added aws lambda python3.10 support
octonawish-akcodes Apr 17, 2024
a4ea1a8
Create requirements.txt.3.10 for Python3.10 for Benchamark - 210.thum…
prajinkhadka May 29, 2024
877be27
Create requirements.txt.3.10 for Python3.10 for Benchmark - 501.graph…
prajinkhadka May 29, 2024
0dc5287
Create requirements.txt.3.10 for Python3.10 for Benchmark - 502.graph…
prajinkhadka May 29, 2024
2e9daf7
Create requirements.txt.3.10 for Python 3.10 for Benchmark - 503.grap…
prajinkhadka May 29, 2024
dceb506
fix benchmarks run issue for Azure platform
octonawish-akcodes Jun 10, 2024
6f4e024
fix 210.thumbnailer dependency issue
octonawish-akcodes Jun 11, 2024
40fc9d2
Merge remote-tracking branch 'origin/master' into sebs4
mcopik Jul 9, 2024
a04fd2b
[regression] Exclude 411 for new Python versions on AWS
mcopik Jul 9, 2024
f553d8e
[aws] Support Python 3.11 on AWS
mcopik Jul 9, 2024
7d76edf
[aws] Deprecate Node.js 14
mcopik Jul 9, 2024
2731e73
[azure] Fix bug in regression - use different directories inside Dock…
mcopik Jul 9, 2024
a498708
[azure] Document problem with large archives
mcopik Jul 9, 2024
bbe0873
[benchmarks] Update PyTorch inference to Python 3.10
mcopik Jul 9, 2024
5e8aefd
[regression] Fix excluding 411 benchmark for aws and gcp
mcopik Jul 9, 2024
52097e8
[benchmarks] Upgrade 411.image-recognition to Python 3.11
mcopik Jul 9, 2024
598c4ce
[benchmarks] Support 210.thumbnailer on Node.js 16, 18 & 20
mcopik Jul 9, 2024
7c77e2c
[docs] Update info about Python versions on Google Cloud
mcopik Jul 9, 2024
587c1a1
[azure] Add documentation on the problematic upload of archive to Docker
mcopik Jul 9, 2024
ea153a4
[regression] Use single deployment for gcp
mcopik Jul 9, 2024
8ee7b80
[benchmarks] Update benchmarks for Python 3.12
mcopik Jul 9, 2024
8713e9e
[gcp] Update supported versions
mcopik Jul 9, 2024
de412d4
[gcp] Update Docker images to work with Node.js 18 and 20
mcopik Jul 9, 2024
8cdff2d
[openwhisk] Update Openwhisk images
mcopik Jul 9, 2024
917e2b4
[openwhisk] Revert old Python version
mcopik Jul 10, 2024
86920aa
[gcp] Revert older Python version
mcopik Jul 10, 2024
82b66c4
[local] Add GET method to control if server is alive
mcopik Jul 10, 2024
131145f
[local] Wait until container is running
mcopik Jul 10, 2024
6d05bb2
[local] Adapt the build image to Python3
mcopik Jul 10, 2024
8cb9807
[local] Update supported languge versions
mcopik Jul 10, 2024
e0c2915
[local] Extend node.js server with a GET alive method
mcopik Jul 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions benchmarks/200.multimedia/210.thumbnailer/nodejs/package.json.18
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"name": "",
"version": "1.0.0",
"description": "",
"author": "",
"license": "",
"dependencies": {
"sharp": "^0.33"
}
}
10 changes: 10 additions & 0 deletions benchmarks/200.multimedia/210.thumbnailer/nodejs/package.json.20
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"name": "",
"version": "1.0.0",
"description": "",
"author": "",
"license": "",
"dependencies": {
"sharp": "^0.33"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pillow==10.3.0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pillow==10.3.0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pillow==10.3.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pillow==10.3.0
https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp310-cp310-linux_x86_64.whl
torchvision==0.12
# prevent installing numpy 2.0
numpy==1.22.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pillow==10.3.0
https://download.pytorch.org/whl/cpu/torch-2.0.0%2Bcpu-cp311-cp311-linux_x86_64.whl
torchvision==0.15.1
# prevent installing numpy 2.0
numpy==1.24.0
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
igraph==0.11.4
57 changes: 34 additions & 23 deletions config/systems.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@
"python": {
"base_images": {
"3.7": "python:3.7-slim",
"3.8": "python:3.8-slim"
"3.8": "python:3.8-slim",
"3.9": "python:3.9-slim",
"3.10": "python:3.10-slim",
"3.11": "python:3.11-slim"
},
"images": [
"run",
Expand All @@ -34,8 +37,10 @@
},
"nodejs": {
"base_images": {
"12": "node:12-slim",
"14": "node:14-slim"
"14": "node:14-slim",
"16": "node:16-slim",
"18": "node:18-slim",
"20": "node:20-slim"
},
"images": [
"run",
Expand All @@ -55,15 +60,11 @@
"languages": {
"python": {
"base_images": {
"3.11": "amazon/aws-lambda-python:3.11",
"3.10": "amazon/aws-lambda-python:3.10",
"3.9": "amazon/aws-lambda-python:3.9",
"3.8": "amazon/aws-lambda-python:3.8",
"3.7": "amazon/aws-lambda-python:3.7"
"3.8": "amazon/aws-lambda-python:3.8"
},
"versions": [
"3.7",
"3.8",
"3.9"
],
"images": [
"build"
],
Expand All @@ -77,12 +78,8 @@
},
"nodejs": {
"base_images": {
"14": "amazon/aws-lambda-nodejs:14",
"16": "amazon/aws-lambda-nodejs:16"
},
"versions": [
"14", "16"
],
"images": [
"build"
],
Expand All @@ -104,7 +101,9 @@
"base_images": {
"3.7": "mcr.microsoft.com/azure-functions/python:3.0-python3.7",
"3.8": "mcr.microsoft.com/azure-functions/python:3.0-python3.8",
"3.9": "mcr.microsoft.com/azure-functions/python:3.0-python3.9"
"3.9": "mcr.microsoft.com/azure-functions/python:3.0-python3.9",
"3.10": "mcr.microsoft.com/azure-functions/python:4-python3.10",
"3.11": "mcr.microsoft.com/azure-functions/python:4-python3.11"
},
"images": [
"build"
Expand All @@ -122,7 +121,9 @@
},
"nodejs": {
"base_images": {
"14": "mcr.microsoft.com/azure-functions/node:3.0-node14"
"16": "mcr.microsoft.com/azure-functions/node:4-node16",
"18": "mcr.microsoft.com/azure-functions/node:4-node18",
"20": "mcr.microsoft.com/azure-functions/node:4-node20"
},
"images": [
"build"
Expand Down Expand Up @@ -152,7 +153,10 @@
"base_images": {
"3.7": "ubuntu:22.04",
"3.8": "ubuntu:22.04",
"3.9": "ubuntu:22.04"
"3.9": "ubuntu:22.04",
"3.10": "ubuntu:22.04",
"3.11": "ubuntu:22.04",
"3.12": "ubuntu:22.04"
},
"images": [
"build"
Expand All @@ -170,10 +174,12 @@
},
"nodejs": {
"base_images": {
"10": "gcr.io/google-appengine/nodejs",
"12": "gcr.io/google-appengine/nodejs",
"14": "gcr.io/google-appengine/nodejs",
"16": "gcr.io/google-appengine/nodejs"
"10": "ubuntu:18.04",
"12": "ubuntu:18.04",
"14": "ubuntu:18.04",
"16": "ubuntu:18.04",
"18": "ubuntu:22.04",
"20": "ubuntu:22.04"
},
"images": [
"build"
Expand All @@ -197,7 +203,9 @@
"python": {
"base_images": {
"3.7": "openwhisk/action-python-v3.7",
"3.9": "openwhisk/action-python-v3.9"
"3.9": "openwhisk/action-python-v3.9",
"3.10": "openwhisk/action-python-v3.10",
"3.11": "openwhisk/action-python-v3.11"
},
"images": [
"function"
Expand All @@ -217,7 +225,10 @@
"nodejs": {
"base_images": {
"10": "openwhisk/action-nodejs-v10",
"12": "openwhisk/action-nodejs-v12"
"12": "openwhisk/action-nodejs-v12",
"14": "openwhisk/action-nodejs-v14",
"18": "openwhisk/action-nodejs-v18",
"20": "openwhisk/action-nodejs-v20"
},
"images": [
"function"
Expand Down
9 changes: 6 additions & 3 deletions dockerfiles/gcp/nodejs/Dockerfile.build
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
ARG VERSION
ENV HOME=/home/${USER}
ENV NVM_DIR=/nvm

RUN install_node --ignore-verification-failure v${VERSION}
RUN apt-get update && apt-get install -y gosu
#RUN install_node --ignore-verification-failure v${VERSION}
RUN apt-get update && apt-get install -y gosu wget
RUN mkdir -p ${NVM_DIR} && wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
RUN . ${NVM_DIR}/nvm.sh && nvm install ${VERSION} && nvm alias default ${VERSION} && nvm use default

RUN mkdir -p /sebs/
COPY dockerfiles/nodejs_installer.sh /sebs/installer.sh
Expand All @@ -16,3 +18,4 @@ ENV PATH=/usr/sbin:$PATH
ENV SCRIPT_FILE=/mnt/function/package.sh
CMD /bin/bash /sebs/installer.sh
ENTRYPOINT ["/sebs/entrypoint.sh"]

6 changes: 6 additions & 0 deletions dockerfiles/local/nodejs/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ const { v4: uuidv4 } = require('uuid');
var app = express();
app.use(express.json());

app.post('/alive', function (req, res) {
res.send(JSON.stringify({
status: "ok"
}));
});

app.post('/', function (req, res) {

let begin = Date.now();
Expand Down
2 changes: 1 addition & 1 deletion dockerfiles/local/python/Dockerfile.build
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ ARG VERSION
ENV PYTHON_VERSION=${VERSION}

RUN apt-get update\
&& apt-get install -y --no-install-recommends gcc build-essential python-dev libxml2 libxml2-dev zlib1g-dev gosu\
&& apt-get install -y --no-install-recommends gcc build-essential python3-dev libxml2 libxml2-dev zlib1g-dev gosu\
&& apt-get purge -y --auto-remove

RUN mkdir -p /sebs/
Expand Down
8 changes: 7 additions & 1 deletion dockerfiles/local/python/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,14 @@

CODE_LOCATION='/function'

@route('/alive', method='GET')
def alive():
return {
"result:" "ok"
}
Comment on lines +11 to +15
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the syntax error in the /alive route handler.

There is a missing comma in the dictionary returned by the alive function.

-        "result:" "ok"
+        "result": "ok"
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@route('/alive', method='GET')
def alive():
return {
"result:" "ok"
}
@route('/alive', method='GET')
def alive():
return {
"result": "ok"
}


@route('/', method='POST')
def flush_log():
def process_request():
begin = datetime.datetime.now()
from function import function
end = datetime.datetime.now()
Expand Down
3 changes: 3 additions & 0 deletions dockerfiles/nodejs_installer.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
#!/bin/bash

if [ -f $FILE ]; then
. /nvm/nvm.sh
fi
cd /mnt/function && npm install && rm -rf package-lock.json
2 changes: 1 addition & 1 deletion docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Below, we discuss the most important implementation details of each benchmark. F
> Benchmark `411.image-recognition` does not work on AWS with Python 3.9 due to excessive code size. While it is possible to ship the benchmark by zipping `torchvision` and `numpy` (see `benchmarks/400.inference/411.image-recognition/python/package.sh`), this significantly affects cold startup. On the lowest supported memory configuration of 512 MB, the cold startup can reach 30 seconds, making HTTP trigger unusable due to 30 second timeout of API gateway. In future, we might support Docker-based deployment on AWS that are not affected by code size limitations.

> [!WARNING]
> Benchmark `411.image-recognition` does not work on GCP with Python 3.8 and 3.9 due to excessive code size. To the best of our knowledge, there is no way of circumventing that limit, as Google Cloud offers neither layers nor custom Docker images.
> Benchmark `411.image-recognition` does not work on GCP with Python 3.8+ due to excessive code size. To the best of our knowledge, there is no way of circumventing that limit, as Google Cloud offers neither layers nor custom Docker images.

## Webapps

Expand Down
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
testtools>=2.4.0
docker>=4.2.0
tzlocal>=2.1
requests
#linting
flake8
black==22.8.0
Expand Down
17 changes: 11 additions & 6 deletions sebs/azure/azure.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import os
import shutil
import time
import uuid
from typing import cast, Dict, List, Optional, Set, Tuple, Type # noqa

import docker
Expand Down Expand Up @@ -202,7 +203,7 @@ def package_code(
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)",
"version": "[4.0.0, 5.0.0)",
},
}
json.dump(default_host_json, open(os.path.join(directory, "host.json"), "w"), indent=2)
Expand All @@ -215,6 +216,7 @@ def publish_function(
self,
function: Function,
code_package: Benchmark,
container_dest: str,
repeat_on_failure: bool = False,
) -> str:
success = False
Expand All @@ -223,7 +225,7 @@ def publish_function(
while not success:
try:
ret = self.cli_instance.execute(
"bash -c 'cd /mnt/function "
f"bash -c 'cd {container_dest} "
"&& func azure functionapp publish {} --{} --no-build'".format(
function.name, self.AZURE_RUNTIMES[code_package.language_name]
)
Expand Down Expand Up @@ -286,8 +288,8 @@ def publish_function(
def update_function(self, function: Function, code_package: Benchmark):

# Mount code package in Docker instance
self._mount_function_code(code_package)
url = self.publish_function(function, code_package, True)
container_dest = self._mount_function_code(code_package)
url = self.publish_function(function, code_package, container_dest, True)

trigger = HTTPTrigger(url, self.config.resources.data_storage_account(self.cli_instance))
trigger.logging_handlers = self.logging_handlers
Expand All @@ -299,8 +301,10 @@ def update_function_configuration(self, function: Function, code_package: Benchm
"Updating function's memory and timeout configuration is not supported."
)

def _mount_function_code(self, code_package: Benchmark):
self.cli_instance.upload_package(code_package.code_location, "/mnt/function/")
def _mount_function_code(self, code_package: Benchmark) -> str:
dest = os.path.join("/mnt", "function", uuid.uuid4().hex)
self.cli_instance.upload_package(code_package.code_location, dest)
return dest

def default_function_name(self, code_package: Benchmark) -> str:
"""
Expand Down Expand Up @@ -366,6 +370,7 @@ def create_function(self, code_package: Benchmark, func_name: str) -> AzureFunct
" --os-type Linux --consumption-plan-location {region} "
" --runtime {runtime} --runtime-version {runtime_version} "
" --name {func_name} --storage-account {storage_account}"
" --functions-version 4 "
).format(**config)
)
self.logging.info("Azure: Created function app {}".format(func_name))
Expand Down
14 changes: 14 additions & 0 deletions sebs/azure/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,24 @@ def login(self, appId: str, tenant: str, password: str) -> bytes:
return result

def upload_package(self, directory: str, dest: str):

"""
This is not an efficient and memory-intensive implementation.
So far, we didn't have very large functions that require many gigabytes.

Since docker-py does not support a straightforward copy, and we can't
put_archive in chunks.

If we end up having problems because of the archive size, there are two
potential solutions:
(1) manually call docker cp and decompress
(2) commit the docker container and restart with a new mount volume.
"""
handle = io.BytesIO()
with tarfile.open(fileobj=handle, mode="w:gz") as tar:
for f in os.listdir(directory):
tar.add(os.path.join(directory, f), arcname=f)
# shutil.make_archive(, 'zip', directory)
# move to the beginning of memory before writing
handle.seek(0)
self.execute("mkdir -p {}".format(dest))
Expand Down
4 changes: 4 additions & 0 deletions sebs/local/function.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ def __init__(

self._measurement_pid = measurement_pid

@property
def url(self) -> str:
return self._url

@property
def memory_measurement_pid(self) -> Optional[int]:
return self._measurement_pid
Expand Down
Loading