diff --git a/docs/deploy-im-dashboard.md b/docs/deploy-im-dashboard.md index a7fe7867..a6c2c426 100644 --- a/docs/deploy-im-dashboard.md +++ b/docs/deploy-im-dashboard.md @@ -57,12 +57,29 @@ These are the steps: In this panel you can specify the number of Working Nodes (WNs) of the cluster together with the computational requirements for each node. We leave the default values. + - Number of WNs in the oscar cluster: Number of working nodes. + - Number of CPUs for the front-end node: Number of CPUs in the primary node. + - Amount of Memory for the front-end node: RAM in the primary node. + - Flavor name of the front-end node. Only required in case of special flavors i.e. with GPUs: Type of instance that will be selected in the front node. + - Number of CPUs for the WNs: number of CPUs per working node. + - Amount of Memory for the WNs: RAM per working node. + - Flavor name of the WNs. Only required in case of special flavors i.e. with GPUs: Type of instance that will be selected in the working nodes. + - Size of the extra HD added to the instance: Extra memory in the primary node. ![template-hw](images/im-dashboard/im-dashboard-02.png) In this panel, specify the passwords to be employed to access the Kubernetes Web UI (Dashboard), to access the OSCAR web UI and to access the MinIO dashboard. These tokens can also be used for programmatic access to the respective services. + + - Access Token for the Kubernetes admin user: It is the token to connect to the Dashboard of Kubernetes. + - OSCAR password: password to OSCAR. + - MinIO password 8 characters min.: password to MinIO. + - Email to be used in the Lets Encrypt issuer: It is an Email linked with the certificates in case the user has any questions. + - ID of the user that creates the infrastructure: unique identifier. Do not touch. + - VO to support: It supports OIDC log in. If there is nothing, only can connect the user who deploys, in case there is a VO, it can be the user who deploys and all people in the VO. + - Flag to add NVIDIA support: if you want to use NVIDIA. + - Flag to install Apache YuniKorn: if you are going to use YuniKorn. ![template-param](images/im-dashboard/im-dashboard-03.png) Now, choose the Cloud provider. The ID specified when creating the Cloud diff --git a/docs/faq.md b/docs/faq.md index 71ae8d8d..8c5a0fee 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -15,4 +15,21 @@ You can check if you are using the correct default cluster with the following co and set a new default cluster with the following command: -`oscar-cli cluster default -s CLUSTER_ID` \ No newline at end of file +`oscar-cli cluster default -s CLUSTER_ID` + +- **How do I use a secret image?** + +In case it is required the use of secret images, you should create a [secret with the docker login configuration](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) with a structure like this: + +``` +apiVersion: v1 +kind: Secret +metadata: + name: dockersecret + namespace: oscar-svc +data: + .dockerconfigjson: {base64 .docker/config.json} +type: kubernetes.io/dockerconfigjson +``` + +Apply the file through kubectl into the Kubernetes OSCAR cluster to create the secret. To use it in OSCAR services, you must add the secret name (`dockersecret` in this example) in the definition of the service, using the API or a FDL, under the `image_pull_secrets` parameter, or through the "Docker secret" field in OSCAR-UI. \ No newline at end of file diff --git a/docs/images/im-dashboard/im-dashboard-03.png b/docs/images/im-dashboard/im-dashboard-03.png index 38740349..d83c9abb 100644 Binary files a/docs/images/im-dashboard/im-dashboard-03.png and b/docs/images/im-dashboard/im-dashboard-03.png differ diff --git a/examples/body-pose-detection/README.md b/examples/body-pose-detection/README.md new file mode 100644 index 00000000..5dfd8b01 --- /dev/null +++ b/examples/body-pose-detection/README.md @@ -0,0 +1,21 @@ +# Body Pose Invocation asynchronous invocation + +This example uses the pre-trained classification model by DEEP-Hybrid-DataCloud +[Body Pose Detection](https://marketplace.deep-hybrid-datacloud.eu/modules/deep-oc-posenet-tf.html) +and is prepared to be used with asynchronous invocations. + + +In order to invoke the function, first you have to do is create a service, +either by the OSCAR UI or by using the FDL within the following command. + +``` sh +oscar-cli apply body-pose-detection-async.yaml +``` + +Once the service is created you can make the invocation with the following +command, which will store the output on a minio bucket. + +``` sh +oscar-cli service put-file body-pose-detection-async minio images/001.jpg body-pose-detection-async/input/001.jpg +``` + diff --git a/examples/body-pose-detection/body-pose-detection-async.yaml b/examples/body-pose-detection/body-pose-detection-async.yaml new file mode 100644 index 00000000..4f637af6 --- /dev/null +++ b/examples/body-pose-detection/body-pose-detection-async.yaml @@ -0,0 +1,14 @@ +functions: + oscar: + - oscar-cluster: + name: body-pose-detection-async + memory: 2Gi + cpu: '1.0' + image: deephdc/deep-oc-posenet-tf + script: script.sh + input: + - storage_provider: minio.default + path: body-pose-detection-async/input + output: + - storage_provider: minio.default + path: body-pose-detection-async/output diff --git a/examples/body-pose-detection/images/001.jpg b/examples/body-pose-detection/images/001.jpg new file mode 100644 index 00000000..9bea8a94 Binary files /dev/null and b/examples/body-pose-detection/images/001.jpg differ diff --git a/examples/body-pose-detection/images/002.jpg b/examples/body-pose-detection/images/002.jpg new file mode 100644 index 00000000..79226684 Binary files /dev/null and b/examples/body-pose-detection/images/002.jpg differ diff --git a/examples/body-pose-detection/images/003.jpg b/examples/body-pose-detection/images/003.jpg new file mode 100644 index 00000000..7fc2d19d Binary files /dev/null and b/examples/body-pose-detection/images/003.jpg differ diff --git a/examples/body-pose-detection/script.sh b/examples/body-pose-detection/script.sh new file mode 100644 index 00000000..0e403292 --- /dev/null +++ b/examples/body-pose-detection/script.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +IMAGE_NAME=`basename "$INPUT_FILE_PATH"` +OUTPUT_IMAGE="$TMP_OUTPUT_DIR/" + +deepaas-predict -i "$INPUT_FILE_PATH" -ct application/zip -o $OUTPUT_IMAGE diff --git a/examples/compss/README.md b/examples/compss/README.md new file mode 100644 index 00000000..bb994f98 --- /dev/null +++ b/examples/compss/README.md @@ -0,0 +1,47 @@ +# COMPSS with OSCAR + +## Dockerfile + +The first step is the creation of the Docker image +The Docker image has to start from an image allocated in the [compss profile at dockerhub](https://hub.docker.com/u/compss) +with all the dependencies + +``` Docker +FROM compss/compss:{version} +``` + +Copy the program inside the Docker container: + +- C/C++ Applications should be the entire project folder (the compilation will create the binary in execution time) compss_build_app +- Java Application, the `.jar` file should be copy +- In Python Applications, introduce all the code + +## Script + +Create the name of the output file and save it into the `$OUTPUT_FILE` variable. + +``` bash +FILE_NAME=`basename "$INPUT_FILE_PATH" | cut -f 1 -d '.'` +OUTPUT_FILE="$TMP_OUTPUT_DIR/$FILE_NAME.txt" +``` + +Some services receive a compressed file that needs to be uncompressed, while others require parsing the input file. Once the input is parsed, the ssh server needs to be initialized. + +``` bash +/etc/init.d/ssh start +``` + +Finally, run COMPSs. Select the command more appropriate according to your language program. +C programs need to be built first with `compss_build_app increment`. +Redirect the output to `$OUTPUT_FILE`. + +``` bash +runcompss --pythonpath=$(pwd) --python_interpreter=python3 {path_to_the_python_program.py} {input_variables} > $OUTPUT_FILE +runcompss --classpath={path_to_the_jar.jar} {in_java_folder/MainClass} {input_variables} > $OUTPUT_FILE +runcompss --lang=c --project=./xml/templates/project.xml master/{name_program} {input_variables} > $OUTPUT_FILE +``` + +### Output warning redirect + +When COMPSs starts the execution, a warning message will appear in the logs. +This message can be ignored. `WARNING: COMPSs Properties file is null. Setting default values` diff --git a/examples/compss/c/Dockerfile b/examples/compss/c/Dockerfile new file mode 100644 index 00000000..d01e922b --- /dev/null +++ b/examples/compss/c/Dockerfile @@ -0,0 +1,2 @@ +FROM compss/compss:latest +COPY increment /opt/increment \ No newline at end of file diff --git a/examples/compss/c/README.md b/examples/compss/c/README.md new file mode 100644 index 00000000..fea3ef40 --- /dev/null +++ b/examples/compss/c/README.md @@ -0,0 +1,29 @@ +# Increment in C with COMPSS and OSCAR + +The source code can be found in [github](https://github.com/bsc-wdc/tutorial_apps/tree/stable/c/increment) +This example works by introducing four numbers: the first is the number of times to increase the counters the other three are the counters. +The return will be the three counters incremented by the first number. + +The first step in this example is to parse the input file and set the arguments: + +```bash +file=$(cat $INPUT_FILE_PATH) +incrementNumber=$(echo "$file" | cut -f 1 -d ';') +counter1=$(echo "$file" | cut -f 2 -d ';') +counter2=$(echo "$file" | cut -f 3 -d ';') +counter3=$(echo "$file" | cut -f 4 -d ';') +``` + +In C language, it is necessary to change the directory and compile the project in a run time: + +``` bash +cd /opt/increment +compss_build_app increment +``` + +The compilation process will create a folder with the name `master`. +This folder contains binary program: + +``` bash +runcompss --lang=c --project=./xml/templates/project.xml master/increment $incrementNumber $counter1 $counter2 $counter3 > $OUTPUT_FILE +``` diff --git a/examples/compss/c/buildAndPush.sh b/examples/compss/c/buildAndPush.sh new file mode 100755 index 00000000..25538477 --- /dev/null +++ b/examples/compss/c/buildAndPush.sh @@ -0,0 +1,3 @@ +#!/bin/bash +docker build -t ghcr.io/grycap/increment-compss-c . +docker push ghcr.io/grycap/increment-compss-c \ No newline at end of file diff --git a/examples/compss/c/increment/Readme b/examples/compss/c/increment/Readme new file mode 100644 index 00000000..e7f39554 --- /dev/null +++ b/examples/compss/c/increment/Readme @@ -0,0 +1,31 @@ +This is the Readme for: +Increment + +[Name]: Increment +[Contact Person]: support-compss@bsc.es +[Access Level]: public +[License Agreement]: Apache2 +[Platform]: COMPSs + +[Body] +== Description == +Increment is an application that takes three different values and increases them a number of given times. The purpose of this application is to show parallelism between the different increments. + +== Execution instructions == +Usage: +runcompss --lang=c /home/compss/tutorial_apps/c/increment/master/increment + +where: + * - N: Number of times to increase the counters + * - initValue1: Initial value for counter 1 + * - initValue2: Initial value for counter 2 + * - initValue3: Initial value for counter 3 + + +== Execution Example == +runcompss --lang=c /home/compss/tutorial_apps/c/increment/master/increment 10 1 2 3 + + +== Build == +compss_build_app increment + diff --git a/examples/compss/c/increment/increment-functions.cc b/examples/compss/c/increment/increment-functions.cc new file mode 100644 index 00000000..b63f0caa --- /dev/null +++ b/examples/compss/c/increment/increment-functions.cc @@ -0,0 +1,54 @@ +/* + * Copyright 2002-2015 Barcelona Supercomputing Center (www.bsc.es) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include +#include +#include +#include"increment.h" + + +void increment(file fileName) { + std::cout << "INIT TASK" << std::endl; + std::cout << "Param: " << fileName << std::endl; + + int value; + // Read value + std::ifstream fis(fileName); + if (fis.is_open()) { + if (fis >> value) { + fis.close(); + } else { + std::cerr << "[ERROR] Unable to read final value" << std::endl; + fis.close(); + } + fis.close(); + } else { + std::cerr << "[ERROR] Unable to open file" << std::endl; + } + + // Increment + std::cout << "INIT VALUE: " << value << std::endl; + std::cout << "FINAL VALUE: " << ++value << std::endl; + + // Write new value + std::ofstream fos (fileName); + if (fos.is_open()) { + fos << value << std::endl; + fos.close(); + } else { + std::cerr << "[ERROR] Unable to open file" << std::endl; + } + std::cout << "END TASK" << std::endl; +} diff --git a/examples/compss/c/increment/increment.cc b/examples/compss/c/increment/increment.cc new file mode 100644 index 00000000..f91aa15e --- /dev/null +++ b/examples/compss/c/increment/increment.cc @@ -0,0 +1,161 @@ +/* + * Copyright 2002-2015 Barcelona Supercomputing Center (www.bsc.es) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include +#include +#include + +#include "increment.h" + +using namespace std; + +#define FILE_NAME1 "file1.txt" +#define FILE_NAME2 "file2.txt" +#define FILE_NAME3 "file3.txt" + +void usage() { + cerr << "[ERROR] Bad number of parameters" << endl; + cout << " Usage: increment " << endl; +} + +void initializeCounters(string counter1, string counter2, string counter3, file fileName1, file fileName2, file fileName3) { + // Write file1 + ofstream fos1 (fileName1); + if (fos1.is_open()) { + fos1 << counter1 << endl; + fos1.close(); + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } + + // Write file2 + ofstream fos2 (fileName2); + if (fos2.is_open()) { + fos2 << counter2 << endl; + fos2.close(); + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } + + // Write file3 + ofstream fos3 (fileName3); + if (fos3.is_open()) { + fos3 << counter3 << endl; + fos3.close(); + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } +} + +void printCounterValues(file fileName1, file fileName2, file fileName3) { + // Read new value from file 1 + string value1; + ifstream fis1; + compss_ifstream(fileName1, fis1); + if (fis1.is_open()) { + if (getline(fis1, value1)) { + cout << "- Counter1 value is " << value1 << endl; + fis1.close(); + } else { + cerr << "[ERROR] Unable to read counter1 value" << endl; + fis1.close(); + return; + } + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } + + // Read new value from file 2 + string value2; + ifstream fis2; + compss_ifstream(fileName2, fis2); + if (fis2.is_open()) { + if (getline(fis2, value2)) { + cout << "- Counter2 value is " << value2 << endl; + fis2.close(); + } else { + cerr << "[ERROR] Unable to read counter2 value" << endl; + fis2.close(); + return; + } + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } + + // Read new value from file 3 + string value3; + ifstream fis3; + compss_ifstream(fileName3, fis3); + if (fis3.is_open()) { + if (getline(fis3, value3)) { + cout << "- Counter3 value is " << value3 << endl; + fis3.close(); + } else { + cerr << "[ERROR] Unable to read counter3 value" << endl; + fis3.close(); + return; + } + } else { + cerr << "[ERROR] Unable to open file" << endl; + return; + } +} + +int main(int argc, char *argv[]) { + // Check and get parameters + if (argc != 5) { + usage(); + return -1; + } + int N = atoi( argv[1] ); + string counter1 = argv[2]; + string counter2 = argv[3]; + string counter3 = argv[4]; + + // Init COMPSs + compss_on(); + + // Initialize counter files + file fileName1 = strdup(FILE_NAME1); + file fileName2 = strdup(FILE_NAME2); + file fileName3 = strdup(FILE_NAME3); + initializeCounters(counter1, counter2, counter3, fileName1, fileName2, fileName3); + + // Print initial counters state + cout << "Initial counter values: " << endl; + printCounterValues(fileName1, fileName2, fileName3); + + // Execute increment tasks + for (int i = 0; i < N; ++i) { + increment(fileName1); + increment(fileName2); + increment(fileName3); + } + + // Print final state + cout << "Final counter values: " << endl; + printCounterValues(fileName1, fileName2, fileName3); + + // Stop COMPSs + compss_off(); + + return 0; +} + diff --git a/examples/compss/c/increment/increment.idl b/examples/compss/c/increment/increment.idl new file mode 100644 index 00000000..212df79e --- /dev/null +++ b/examples/compss/c/increment/increment.idl @@ -0,0 +1,3 @@ +interface increment { + void increment(inout File filename); +}; diff --git a/examples/compss/c/increment/pom.xml b/examples/compss/c/increment/pom.xml new file mode 100644 index 00000000..46411eed --- /dev/null +++ b/examples/compss/c/increment/pom.xml @@ -0,0 +1,183 @@ + + 4.0.0 + + + increment + pom + + es.bsc.compss.tutorial-apps.c + compss-tutorial-apps-c + 3.1.rc2301 + .. + + + + increment + Increment application + + + Apache License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.html + + + + BSC + http://www.bsc.es + + + + + + + + + + + maven-resources-plugin + ${maven-resources-plugin.version} + + + copy-resources + validate + + copy-resources + + + ${project.basedir}/xml/ + + + ${project.basedir}/xml/templates/ + true + + project.xml + resources.xml + + + + + + + + + + + org.codehaus.mojo + exec-maven-plugin + + + application-execution + + exec + + + + + application-execution-clean + clean + + exec + + + bash + ${basedir}/ + + -c + rm -rf file?.txt compss-serialized-obj* master worker lib src/*.o xml/*.xml + + false + + + + + application-build + compile + + exec + + + compss_build_app + ${basedir}/ + + increment + + false + + + + + application-xml-generation + compile + + exec + + + ${basedir}/xml/templates/replace.sh + ${basedir}/xml/templates/ + + ${basedir} + + false + + + + + + runcompss + ${basedir}/ + + --lang=c + --project=${basedir}/xml/project.xml + --resources=${basedir}/xml/resources.xml + ${basedir}/master/increment + 5 + 1 + 2 + 3 + + false + + + + + + + + + org.eclipse.m2e + lifecycle-mapping + 1.0.0 + + + + + + + org.apache.maven.plugins + + + maven-dependency-plugin + + + [2.1,) + + + + copy-dependencies + + + + + + + + + + + + + + + + diff --git a/examples/compss/c/increment/src/Makefile b/examples/compss/c/increment/src/Makefile new file mode 100644 index 00000000..d20f8b59 --- /dev/null +++ b/examples/compss/c/increment/src/Makefile @@ -0,0 +1,27 @@ +GCC=g++ +#CFLAGS=-std=c++11 -DDEBUG_BINDING -g -I. -I/opt/COMPSs/Bindings/c/include -I/opt/COMPSs/Bindings/bindings-common/include +CFLAGS=-std=c++11 -g -I. -I/opt/COMPSs/Bindings/c/include -I/opt/COMPSs/Bindings/bindings-common/include + +all: clean compile_master compile_worker deploy + +compile_master: + @echo "Building application for master..." + $(GCC) -DCOMPSS_MASTER $(CFLAGS) -c example.cc + ar rvs libmaster.a example.o + ranlib libmaster.a + +compile_worker: + @echo "Building application for workers..." + $(GCC) -DCOMPSS_WORKER $(CFLAGS) -c example.cc + ar rvs libworker.a example.o + ranlib libworker.a + +deploy: + @echo "Deploying lib files..." + @rm -rf ../lib + @mkdir ../lib + @mv *.a ../lib + +clean: + @echo "Cleaning intermediate files..." + rm -f *.o diff --git a/examples/compss/c/increment/src/example.cc b/examples/compss/c/increment/src/example.cc new file mode 100644 index 00000000..55bbfc3e --- /dev/null +++ b/examples/compss/c/increment/src/example.cc @@ -0,0 +1,16 @@ +/* + * Copyright 2002-2015 Barcelona Supercomputing Center (www.bsc.es) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "example.h" diff --git a/examples/compss/c/increment/src/example.h b/examples/compss/c/increment/src/example.h new file mode 100644 index 00000000..5b5bee8a --- /dev/null +++ b/examples/compss/c/increment/src/example.h @@ -0,0 +1,33 @@ +/* + * Copyright 2002-2015 Barcelona Supercomputing Center (www.bsc.es) + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef EXAMPLE_H +#define EXAMPLE_H + +#include + +using namespace std; + + +class example { + +public: + example(){}; + +private: + +}; + +#endif diff --git a/examples/compss/c/increment/xml/templates/project.xml b/examples/compss/c/increment/xml/templates/project.xml new file mode 100644 index 00000000..5280d5b9 --- /dev/null +++ b/examples/compss/c/increment/xml/templates/project.xml @@ -0,0 +1,13 @@ + + + + + + /opt/COMPSs/ + /tmp/WorkerLocalhost/ + + + + + + diff --git a/examples/compss/c/increment/xml/templates/replace.sh b/examples/compss/c/increment/xml/templates/replace.sh new file mode 100755 index 00000000..f77e673c --- /dev/null +++ b/examples/compss/c/increment/xml/templates/replace.sh @@ -0,0 +1,17 @@ +#!/bin/bash + + basedir=$1 + + # Add double // to the basedir path + parsedbasedir=${basedir//\//\\\/} + + # Debug + echo "BASEDIR: $basedir" + echo "PARSED BASEDIR: $parsedbasedir" + + # Sed new value on AppDir on project file + sed -i "s//'${parsedbasedir}'/g" ../project.xml + + # Exit with last command value + exit + diff --git a/examples/compss/c/increment/xml/templates/resources.xml b/examples/compss/c/increment/xml/templates/resources.xml new file mode 100644 index 00000000..749385c7 --- /dev/null +++ b/examples/compss/c/increment/xml/templates/resources.xml @@ -0,0 +1,28 @@ + + + + + 4 + + + + + + + + 43001 + 43002 + + + + + + sequential + + + + sshtrilead + + + + diff --git a/examples/compss/c/input.txt b/examples/compss/c/input.txt new file mode 100644 index 00000000..a849c940 --- /dev/null +++ b/examples/compss/c/input.txt @@ -0,0 +1 @@ +25;4;6;7 diff --git a/examples/compss/c/script.sh b/examples/compss/c/script.sh new file mode 100644 index 00000000..23dab151 --- /dev/null +++ b/examples/compss/c/script.sh @@ -0,0 +1,13 @@ +#!/bin/bash +FILE_NAME=`basename "$INPUT_FILE_PATH" | cut -f 1 -d '.'` +OUTPUT_FILE="$TMP_OUTPUT_DIR/$FILE_NAME.txt" + +file=$(cat $INPUT_FILE_PATH) +incrementNumber=$(echo "$file" | cut -f 1 -d ';') +counter1=$(echo "$file" | cut -f 2 -d ';') +counter2=$(echo "$file" | cut -f 3 -d ';') +counter3=$(echo "$file" | cut -f 4 -d ';') +/etc/init.d/ssh start +cd /opt/increment +compss_build_app increment +runcompss --lang=c --project=./xml/templates/project.xml master/increment $incrementNumber $counter1 $counter2 $counter3 > $OUTPUT_FILE diff --git a/examples/compss/c/workflow.yaml b/examples/compss/c/workflow.yaml new file mode 100644 index 00000000..f63159aa --- /dev/null +++ b/examples/compss/c/workflow.yaml @@ -0,0 +1,15 @@ +functions: + oscar: + - oscar-cluster: + name: increment-compss-c + memory: 3Gi + cpu: '3.0' + image: ghcr.io/grycap/increment-compss-c + script: script.sh + log_level: INFO + input: + - path: increment/in + storage_provider: minio + output: + - path: increment/out + storage_provider: minio diff --git a/examples/compss/java/Dockerfile b/examples/compss/java/Dockerfile new file mode 100644 index 00000000..c245bec1 --- /dev/null +++ b/examples/compss/java/Dockerfile @@ -0,0 +1,3 @@ +FROM compss/compss:latest +COPY simple.jar /opt/simple.jar + diff --git a/examples/compss/java/README.md b/examples/compss/java/README.md new file mode 100644 index 00000000..7321571c --- /dev/null +++ b/examples/compss/java/README.md @@ -0,0 +1,11 @@ +# Simple in java with COMPSs and OSCAR + +The source code can be found in [github](https://github.com/bsc-wdc/tutorial_apps/tree/stable/python/wordcount) +This example works by introducing a number and returning the number increased by one and module 255. + +In the use of the java example program, it needs a `.jar` file generated with maven `mvn clean package`. +The input file is a flat file that contains a number, and it will be read directly as the example command shows: + +``` bash +runcompss --classpath=/opt/simple.jar simple.Simple `cat $INPUT_FILE_PATH` > $OUTPUT_FILE +``` diff --git a/examples/compss/java/buildAndPush.sh b/examples/compss/java/buildAndPush.sh new file mode 100755 index 00000000..0a69f667 --- /dev/null +++ b/examples/compss/java/buildAndPush.sh @@ -0,0 +1,3 @@ +#!/bin/bash +docker build -t ghcr.io/grycap/simple-compss-java . +docker push ghcr.io/grycap/simple-compss-java \ No newline at end of file diff --git a/examples/compss/java/file.tar.xz b/examples/compss/java/file.tar.xz new file mode 100644 index 00000000..204a4ffb Binary files /dev/null and b/examples/compss/java/file.tar.xz differ diff --git a/examples/compss/java/input.txt b/examples/compss/java/input.txt new file mode 100644 index 00000000..4d0e90cb --- /dev/null +++ b/examples/compss/java/input.txt @@ -0,0 +1 @@ +512 diff --git a/examples/compss/java/script.sh b/examples/compss/java/script.sh new file mode 100644 index 00000000..a8ac9d43 --- /dev/null +++ b/examples/compss/java/script.sh @@ -0,0 +1,5 @@ +#!/bin/bash +FILE_NAME=`basename "$INPUT_FILE_PATH" | cut -f 1 -d '.'` +OUTPUT_FILE="$TMP_OUTPUT_DIR/$FILE_NAME.txt" +/etc/init.d/ssh start +runcompss --classpath=/opt/simple.jar simple.Simple `cat $INPUT_FILE_PATH` > $OUTPUT_FILE \ No newline at end of file diff --git a/examples/compss/java/simple.jar b/examples/compss/java/simple.jar new file mode 100644 index 00000000..6a3c2fa8 Binary files /dev/null and b/examples/compss/java/simple.jar differ diff --git a/examples/compss/java/workflow.yaml b/examples/compss/java/workflow.yaml new file mode 100644 index 00000000..2ea32732 --- /dev/null +++ b/examples/compss/java/workflow.yaml @@ -0,0 +1,15 @@ +functions: + oscar: + - oscar-cluster: + name: simple-compss-java + memory: 3Gi + cpu: '3.0' + image: ghcr.io/grycap/simple-compss-java + script: script.sh + log_level: INFO + input: + - path: simple-java/in + storage_provider: minio + output: + - path: simple-java/out + storage_provider: minio diff --git a/examples/compss/python/Dockerfile b/examples/compss/python/Dockerfile new file mode 100644 index 00000000..7975929e --- /dev/null +++ b/examples/compss/python/Dockerfile @@ -0,0 +1,3 @@ +FROM compss/compss:latest +COPY wordcount_merge.py /opt/wordcount_merge.py +RUN mkdir -p /opt/folder \ No newline at end of file diff --git a/examples/compss/python/README.md b/examples/compss/python/README.md new file mode 100644 index 00000000..92ba437f --- /dev/null +++ b/examples/compss/python/README.md @@ -0,0 +1,18 @@ +# Wordcount in python with COMPSs and OSCAR + +The source code can be found in [github](https://github.com/bsc-wdc/tutorial_apps/tree/stable/python/wordcount) +This example works by introducing a folder name, and it will read the files inside the folder and count the words. + +The input file that will be put in a MinIO bucket is a `tar` file that contains all the files. + +A has been created a folder inside the Docker `/opt/folder` +In this auxiliary folder will be extracted all the input files +and it will be an argument to run COMPSs. + +**This example does not support zip files, just tar** + +The complete command to run the COMPSs is: + +``` bash +runcompss --pythonpath=$(pwd) --python_interpreter=python3 /opt/wordcount_merge.py /opt/folder > $OUTPUT_FILE +``` diff --git a/examples/compss/python/buildAndPush.sh b/examples/compss/python/buildAndPush.sh new file mode 100755 index 00000000..ae48e769 --- /dev/null +++ b/examples/compss/python/buildAndPush.sh @@ -0,0 +1,3 @@ +#!/bin/bash +docker build -t ghcr.io/grycap/wordcount-compss-python . +docker push ghcr.io/grycap/wordcount-compss-python \ No newline at end of file diff --git a/examples/compss/python/file.tar.xz b/examples/compss/python/file.tar.xz new file mode 100644 index 00000000..204a4ffb Binary files /dev/null and b/examples/compss/python/file.tar.xz differ diff --git a/examples/compss/python/script.sh b/examples/compss/python/script.sh new file mode 100644 index 00000000..99cb5005 --- /dev/null +++ b/examples/compss/python/script.sh @@ -0,0 +1,10 @@ +#!/bin/bash +FILE_NAME=`basename "$INPUT_FILE_PATH" | cut -f 1 -d '.'` +OUTPUT_FILE="$TMP_OUTPUT_DIR/$FILE_NAME.txt" + +tar --extract -f $INPUT_FILE_PATH -C /opt/folder + + +/etc/init.d/ssh start + +runcompss --pythonpath=$(pwd) --python_interpreter=python3 /opt/wordcount_merge.py /opt/folder > $OUTPUT_FILE \ No newline at end of file diff --git a/examples/compss/python/wordcount_merge.py b/examples/compss/python/wordcount_merge.py new file mode 100755 index 00000000..8228b4e5 --- /dev/null +++ b/examples/compss/python/wordcount_merge.py @@ -0,0 +1,124 @@ +#!/usr/bin/python +# +# Copyright 2002-2019 Barcelona Supercomputing Center (www.bsc.es) +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# -*- coding: utf-8 -*- + +import sys +import os +import time + +from pycompss.api.task import task +from pycompss.api.parameter import * + + +@task(file_path=FILE_IN, returns=list) +def read_file(file_path): + """ Read a file and return a list of words. + :param file_path: file's path + :return: list of words + """ + data = [] + with open(file_path, 'r') as f: + for line in f: + data += line.split() + return data + + +@task(returns=dict) +def wordCount(data): + """ Construct a frequency word dictorionary from a list of words. + :param data: a list of words + :return: a dictionary where key=word and value=#appearances + """ + partialResult = {} + for entry in data: + if entry in partialResult: + partialResult[entry] += 1 + else: + partialResult[entry] = 1 + return partialResult + + +@task(returns=dict, priority=True) +def merge_two_dicts(dic1, dic2): + """ Update a dictionary with another dictionary. + :param dic1: first dictionary + :param dic2: second dictionary + :return: dic1+=dic2 + """ + for k in dic2: + if k in dic1: + dic1[k] += dic2[k] + else: + dic1[k] = dic2[k] + return dic1 + + +def mergeReduce(function, data): + """ Apply function cumulatively to the items of data, + from left to right in binary tree structure, so as to + reduce the data to a single value. + :param function: function to apply to reduce data + :param data: List of items to be reduced + :return: result of reduce the data to a single value + """ + from collections import deque + q = deque(list(range(len(data)))) + while len(q): + x = q.popleft() + if len(q): + y = q.popleft() + data[x] = function(data[x], data[y]) + q.append(x) + else: + return data[x] + + +if __name__ == "__main__": + from pycompss.api.api import compss_wait_on + + # Start counting time... + start_time = time.time() + + # Get the dataset path + pathDataset = sys.argv[1] + + # Construct a list with the file's paths from the dataset + paths = [] + for fileName in os.listdir(pathDataset): + paths.append(os.path.join(pathDataset, fileName)) + + # Read file's content execute a wordcount on each of them + partialResult = [] + for p in paths: + data = read_file(p) + partialResult.append(wordCount(data)) + + # Accumulate the partial results to get the final result. + result = mergeReduce(merge_two_dicts, partialResult) + + # Wait for result + result = compss_wait_on(result) + + elapsed_time = time.time() - start_time + + # Print the results and elapsed time + print("Word appearances:") + from pprint import pprint + pprint(result) + print("Elapsed Time (s): " + str(elapsed_time)) + print("Words: " + str(sum(result.values()))) diff --git a/examples/compss/python/workflow.yaml b/examples/compss/python/workflow.yaml new file mode 100644 index 00000000..f54d2c9d --- /dev/null +++ b/examples/compss/python/workflow.yaml @@ -0,0 +1,15 @@ +functions: + oscar: + - oscar-cluster: + name: wordcount-compss-python + memory: 3Gi + cpu: '3.0' + image: ghcr.io/grycap/wordcount-compss-python + script: script.sh + log_level: INFO + input: + - path: wordcount-python/in + storage_provider: minio + output: + - path: wordcount-python/out + storage_provider: minio