Skip to content

Commit

Permalink
Merge pull request #1 from grycap/dev-calarcon
Browse files Browse the repository at this point in the history 
Changes to release
  • Loading branch information
@catttam
catttam authored Nov 23, 2022
2 parents dc447bd + e3a9c4c commit 5b0bcbe
Show file tree
Hide file tree
Showing 5 changed files with 125 additions and 111 deletions.
37 changes: 22 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
[![Build](https://github.com/grycap/oscar_python/actions/workflows/main.yaml/badge.svg)](https://github.com/grycap/oscar_python/actions/workflows/main.yaml)
![PyPI](https://img.shields.io/pypi/v/oscar_python)

This package provides an API to interact with OSCAR (https://oscar.grycap.net) clusters and services. It is available on Pypi with the name [oscar-python](https://pypi.org/project/oscar-python/).

### Contents
- [Python OSCAR API](#python-oscar-api)
- [Contents](#contents)
Expand All @@ -23,22 +25,27 @@ from oscar_python.client import Client
client = Client("cluster-id","https://cluster-endpoint", "username", "password", True)

# get the cluster information
info = client.get_cluster_info()
print(info.text)
try:
info = client.get_cluster_info()
print(info.text)
except Exception as err:
print("Failed with: ", err)
```

- Sample code to create a simple service with the [cowsay example](https://github.com/grycap/oscar/tree/master/examples/cowsay) and afterwards make a synchronous invocation.
- Sample code to create a simple service with the [cowsay example](https://github.com/grycap/oscar/tree/master/examples/cowsay) and make a synchronous invocation.

``` python
from oscar_python.client import Client

client = Client("cluster-id","https://cluster-endpoint", "username", "password", True)

err = client.create_service("/absolute_path/cowsay.yaml")
if not err:
res = client.run_service("cowsay", '{"message": "Hi there"}')
if res.status_code == 200:
print(res.text)
try:
client.create_service("/absolute_path/cowsay.yaml")
res = client.run_service("cowsay", '{"message": "Hi there"}')
if res.status_code == 200:
print(res.text)
except Exception as err:
print("Failed with: ", err)
```

### API methods
Expand All @@ -48,27 +55,27 @@ if not err:
**get_cluster_info**
``` python
# get the cluster information
info = client.get_cluster_info() # returns an http response
info = client.get_cluster_info() # returns an HTTP response or an HTTPError
```

**get_cluster_config**
``` python
# get the cluster config
config = client.get_cluster_config() # returns an http response
config = client.get_cluster_config() # returns an http response or an HTTPError
```

#### Service methods

**get_service**
``` python
# get the definition of a service
service = client.get_service("service_name") # returns an http response
service = client.get_service("service_name") # returns an http response or an HTTPError
```

**list_services**
``` python
# get a list of all the services deployed
services = client.list_services() # returns an http response
services = client.list_services() # returns an http response or an HTTPError
```

> _Note_ : Both `path_to_fdl` and the script path inside the fdl must be absolute.
Expand All @@ -93,7 +100,7 @@ response = client.remove_service("service_name") # returns an http response

**run_service**

The `input` parameter may not be passed if the function doesn't require an input.
The `input` parameter may not be passed if the function doesn't require input.

``` python
# make a synchronous execution
Expand Down Expand Up @@ -129,7 +136,7 @@ response = client.remove_all_jobs("service_name") # returns an http response

#### Storage usage

You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as it follows:
You can create a storage object to operate over the different storage providers defined on a service with the method `create_storage_client` as follows:

``` python
storage_service = client.create_storage_client("service_name") # returns a storage object
Expand All @@ -138,7 +145,7 @@ storage_service = client.create_storage_client("service_name") # returns a stora
**list_files_from_path**

This method returns a json with the info except for Onedata, which returns an http response.
This method returns a JSON with the info except for OneData, which returns an HTTP response.

``` python
# get a list of the files of one of the service storage provider
Expand Down
130 changes: 61 additions & 69 deletions jupyter_example/oscar_notebook.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,119 +1,111 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### OSCAR API notebook example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is a sample of the usage of the python API with jupyter notebooks step by step. You can modify the parameters in the code and the service to test it with your own OSCAR cluster.\n",
"\n",
"The first step is to install the package from Pypi."
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: oscar-python in /home/caterina/.local/lib/python3.9/site-packages (1.0.1b2)\n",
"Requirement already satisfied: webdavclient3==3.14.6 in /home/caterina/.local/lib/python3.9/site-packages (from oscar-python) (3.14.6)\n",
"Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from oscar-python) (2.25.1)\n",
"Requirement already satisfied: boto3 in /home/caterina/.local/lib/python3.9/site-packages (from oscar-python) (1.24.0)\n",
"Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from oscar-python) (5.3.1)\n",
"Requirement already satisfied: setuptools>=40.8.0 in /usr/lib/python3/dist-packages (from oscar-python) (52.0.0)\n",
"Requirement already satisfied: aiohttp in /home/caterina/.local/lib/python3.9/site-packages (from oscar-python) (3.8.1)\n",
"Requirement already satisfied: python-dateutil in /home/caterina/.local/lib/python3.9/site-packages (from webdavclient3==3.14.6->oscar-python) (2.8.2)\n",
"Requirement already satisfied: lxml in /home/caterina/.local/lib/python3.9/site-packages (from webdavclient3==3.14.6->oscar-python) (4.9.0)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->oscar-python) (21.4.0)\n",
"Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (2.1.1)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (4.0.2)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (1.2.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (6.0.2)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (1.3.1)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /home/caterina/.local/lib/python3.9/site-packages (from aiohttp->oscar-python) (1.8.1)\n",
"Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.0->aiohttp->oscar-python) (2.10)\n",
"Requirement already satisfied: botocore<1.28.0,>=1.27.0 in /home/caterina/.local/lib/python3.9/site-packages (from boto3->oscar-python) (1.27.28)\n",
"Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/caterina/.local/lib/python3.9/site-packages (from boto3->oscar-python) (0.6.0)\n",
"Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/caterina/.local/lib/python3.9/site-packages (from boto3->oscar-python) (1.0.0)\n",
"Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.28.0,>=1.27.0->boto3->oscar-python) (1.26.5)\n",
"Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->webdavclient3==3.14.6->oscar-python) (1.16.0)\n"
]
}
],
"outputs": [],
"source": [
"# Install a pip package in the current Jupyter kernel\n",
"# Install oscar-python package in the current Jupyter kernel\n",
"import sys\n",
"!{sys.executable} -m pip install oscar-python"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the package installed you can now create the client with your cluster credentials, enpoint and ID or cluster name."
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Import oscar_python library\n",
"from oscar_python.client import Client\n",
"\n",
"client = Client(\"cluster-id\",\"https://your-cluster-endpoint.net\", \"user\", \"password\", True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once the client is created, you can call the function `create_service` with the path to the FDL file wich contains the function definition. To make it easier we recommend to have a folder for each service with their respective fdl and script within the path of the jupyter notebook. \n",
"\n",
"Also, the cluster ID on the FDL file has to match the one provided on the client definition, otherwise the function will throw an exception."
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Service created!\n"
]
}
],
"outputs": [],
"source": [
"# Create a service with the function create_service, which returns an error if something goes wrong or nothing if the service is created correctly\n",
"try:\n",
" client.create_service(\"services/cowsay_example/cowsay.yaml\")\n",
" print(\"Service created\")\n",
"except Exception as ex:\n",
" print(\"Error creating service: \", ex)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"err = client.create_service(\"services/cowsay_example/cowsay.yaml\")\n",
"if not err:\n",
" print(\"Service created!\")"
"Then, you can run a synchronous invocation with the function `run_service` and the input parameters that the function requires."
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" __________\n",
"< Hi there >\n",
" ----------\n",
" \\ ^__^\n",
" \\ (oo)\\_______\n",
" (__)\\ )\\/\\\n",
" ||----w |\n",
" || ||\n",
"\n",
"\n"
]
}
],
"outputs": [],
"source": [
"res = client.run_service(\"cowsay\", input = '{\"message\": \"Hi there\"}')\n",
"if res.status_code == 200:\n",
" print(res.text)"
"# Test the synchronous invocation of the service\n",
"try:\n",
" res = client.run_service(\"cowsay\", input = '{\"message\": \"Hi there\"}')\n",
" if res.status_code == 200:\n",
" print(res.text)\n",
"except Exception as ex:\n",
" print(\"Error running service: \", ex)"
]
},
{
Expand Down
25 changes: 18 additions & 7 deletions oscar_python/_utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,22 @@
import requests

""" Generic http request """
def make_request(c , path, method, data="", file="", token=""):
def make_request(c , path, method, **kwargs):
url = c.endpoint+path
headers = get_headers(c)
if method == "post" or "put":
if token: headers = get_headers_with_token(token)
if data: return requests.request(method, url, headers=headers, verify=c.ssl, data=data)
if file: return requests.request(method, url, headers=headers, verify=c.ssl, files=file)
return requests.request(method, url, headers=headers, verify=c.ssl)
if method in ["post", "put"]:
if "token" in kwargs.keys() and kwargs["token"]:
headers = get_headers_with_token(kwargs["token"])
if "data" in kwargs.keys() and kwargs["data"]:
result = requests.request(method, url, headers=headers, verify=c.ssl, data=kwargs["data"])
else:
result = requests.request(method, url, headers=headers, verify=c.ssl)

if "handle" in kwargs.keys() and kwargs["handle"] == False:
return result

result.raise_for_status()
return result

""" Function to generate headers with basic authentication """
def get_headers(c):
Expand All @@ -33,4 +41,7 @@ def get_headers(c):

""" Function to generate headers with token auth """
def get_headers_with_token(token):
return {"Authorization": "Bearer "+ str(token)}
return {"Authorization": "Bearer "+ str(token)}

def raise_http_errors(response):
response.raise_for_status()
Loading

0 comments on commit 5b0bcbe

Please sign in to comment.