Skip to content

Commit

Permalink
Merge pull request #78 from rabix/feature/copy_app_improvements
Browse files Browse the repository at this point in the history 
Feature/copy app improvements
  • Loading branch information
@pavlemarinkovic
pavlemarinkovic authored Nov 13, 2023
2 parents 44aefc2 + 94e3056 commit 852bd05
Show file tree
Hide file tree
Showing 7 changed files with 487 additions and 390 deletions.
80 changes: 53 additions & 27 deletions sbpack/noncwl/Readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,39 +53,48 @@ to the Platform.
```
$ sbpack_nf -h
usage: sbpack_nf [-h] [--profile PROFILE] --appid APPID --workflow-path WORKFLOW_PATH [--entrypoint ENTRYPOINT] [--sb-package-id SB_PACKAGE_ID] [--sb-inputs SB_INPUTS] [--sb-outputs SB_OUTPUTS]
[--sb-doc SB_DOC] [--dump-sb-app] [--no-package] [--json] [--sb-schema SB_SCHEMA]
usage: nextflow.py [-h] [--profile PROFILE] --appid APPID --workflow-path
WORKFLOW_PATH [--entrypoint ENTRYPOINT]
[--sb-package-id SB_PACKAGE_ID] [--sb-doc SB_DOC]
[--dump-sb-app] [--no-package]
[--executor-version EXECUTOR_VERSION]
[--execution-mode {single-instance,multi-instance}]
[--json] [--sb-schema SB_SCHEMA]
[--revision-note REVISION_NOTE]
optional arguments:
-h, --help show this help message and exit
--profile PROFILE SB platform profile as set in the SB API credentials file.
--profile PROFILE SB platform profile as set in the SB API credentials
file.
--appid APPID Takes the form {user or division}/{project}/{app_id}.
--entrypoint ENTRYPOINT
Relative path to the workflow from the main workflow directory. If not provided, 'main.nf' will be used if available. If not available, any '.nf' located in the workflow-path will be used.
--workflow-path WORKFLOW_PATH
Path to the main workflow directory
--entrypoint ENTRYPOINT
Relative path to the workflow from the main workflow
directory. If not provided, 'main.nf' will be used if
available. If not available, but a single '*.nf' is
located in the workflow-path will be used. If more
than one '*.nf' script is detected, an error is
raised.
--sb-package-id SB_PACKAGE_ID
Id of an already uploaded package
--sb-inputs SB_INPUTS
Path to pre built sb app inputs schema
--sb-outputs SB_OUTPUTS
Path to pre build sb app outputs schema
--sb-doc SB_DOC Path to a doc file for sb app. If not provided, README.md will be used if available
--sb-doc SB_DOC Path to a doc file for sb app. If not provided,
README.md will be used if available
--dump-sb-app Dump created sb app to file if true and exit
--no-package Only provide a sb app schema and a git URL for entrypoint
--no-package Only provide a sb app schema and a git URL for
entrypoint
--executor-version EXECUTOR_VERSION
Version of the Nextflow executor to be used with the app.
Version of the Nextflow executor to be used with the
app.
--execution-mode {single-instance,multi-instance}
Execution mode for your application.
--json Dump sb app schema in JSON format (YAML by default)
--sb-schema SB_SCHEMA
Do not create new schema, use this schema file. It is sb_nextflow_schema in JSON or YAML format.
--output-schema-files OUTPUT_SCHEMA_FILES [OUTPUT_SCHEMA_FILES ...]
Additional output schema files in CWL or tower.yml format.
--input-schema-files INPUT_SCHEMA_FILES [INPUT_SCHEMA_FILES ...]
Additional input schema files in CWL format.
--revision-note REVISION_NOTE [REVISION_NOTE ...]
Revision note to be placed in the CWL schema if the app is uploaded to the sbg platform.
--manual-validation You will have to provide validation for all 'string' type inputs if are string (str), file (file), directory (dir), list of file (files), or list of directory (dirs) type inputs.
Do not create new schema, use this schema file. It is
sb_nextflow_schema in JSON or YAML format.
--revision-note REVISION_NOTE
Revision note to be placed in the CWL schema if the
app is uploaded to the sbg platform.
```
### Example

Expand All @@ -97,24 +106,41 @@ $ sbpack_nf --profile default --workflow-path /path/to/workflow_dir
# sbcopy

Developed to enable deep copying of Nextflow and WDL apps on the SB platform between projects.
Note: The link between the original, and the new app will not be available.
This tool also allows copying between divisions and platforms. When used to perform this copy action, it requires credentials for both the source and the destination environment. Both are provided to the `--profile` argument in the source, target order.
Note: The link between the original, and the new app will not be available.

### Usage

```
sbcopy -h
usage: sbcopy [-h] [--profile PROFILE] --appid APPID --projectid PROJECTID
usage: copy_app.py [-h] [--profile PROFILE [PROFILE ...]] --appid APPID
--projectid PROJECTID
optional arguments:
-h, --help show this help message and exit
--profile PROFILE SB platform profile as set in the SB API credentials file.
--appid APPID What to copy? Takes the form {user or division}/{project}/{app_id} or {user or division}/{project}/{app_id}/{revision_no}.
--profile PROFILE [PROFILE ...]
SB platform profile as set in the SB API credentials
file. If you are using sbcopy to copy an app from one
division to another, please provide two profiles -
first profile for the source app (appid), and second
for the destination project (projectid).
--appid APPID What to copy? Takes the form {user or
division}/{project}/{app_id} or {user or
division}/{project}/{app_id}/{revision_no}.
--projectid PROJECTID
Where to copy? Takes the form {user or division}/{project}
Where to copy? Takes the form {user or
division}/{project}
```

### Example
### Examples

Copying an app from one project to another within the same division.
```
sbcopy --appid division-name/project-name/app-name --projectid division-name/destination-project-name
```

Copying an app from one division to another.
```
sbcopy --profile source_division target_division --appid source-division-name/project-name/app-name --projectid target-division-name/destination-project-name
```
98 changes: 98 additions & 0 deletions sbpack/noncwl/constants.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
from enum import Enum

# ############################## Generic Bits ############################### #
PACKAGE_SIZE_LIMIT = 100 * 1024 * 1024 # 100 MB


# keep track of what extensions are applicable for processing
class EXTENSIONS:
yaml = 'yaml'
yml = 'yml'
json = 'json'
cwl = 'cwl'

yaml_all = [yaml, yml, cwl]
json_all = [json]
all_ = [yaml, yml, json, cwl]


# ############################ CWL Standard Bits ############################ #
# A generic SB input array of files that should be available on the
# instance but are not explicitly provided to the execution as wdl params.
GENERIC_FILE_ARRAY_INPUT = {
"id": "auxiliary_files",
"type": "File[]?",
"label": "Auxiliary files",
"doc": "List of files not added as explicit workflow inputs but "
"required for workflow execution."
}

GENERIC_NF_OUTPUT_DIRECTORY = {
"id": "nf_workdir",
"type": "Directory?",
"label": "Work Directory",
"doc": "This is a template output. "
"Please change glob to directories specified in "
"publishDir in the workflow.",
"outputBinding": {
"glob": "work"
}
}

GENERIC_WDL_OUTPUT_DIRECTORY = {
"id": "output_txt",
"doc": "This is a template output. "
"Please modify to collect final outputs using "
"glob inside the working directory.",
"type": "File[]",
"outputBinding": {
"glob": "*.txt"
}
}

# Requirements to be added to sb wrapper
WRAPPER_REQUIREMENTS = [
{
"class": "InlineJavascriptRequirement"
},
{
"class": "InitialWorkDirRequirement",
"listing": [
"$(inputs.auxiliary_files)"
]
}
]

# ############################## Nextflow Bits ############################## #
# Keys that should be skipped when parsing nextflow tower yaml file

# Mappings of nextflow input fields to SB input fields
# nextflow_key: cwl_key mapping
NF_TO_CWL_PORT_MAP = {
'default': 'sbg:toolDefaultValue',
'description': 'label',
'help_text': 'doc',
'mimetype': 'format',
'fa_icon': 'sbg:icon',
'pattern': 'sbg:pattern',
'hidden': 'sbg:hidden',
}

# Mappings of nextflow definition fields to SB category fields
# nextflow_key: cwl_key mapping
NF_TO_CWL_CATEGORY_MAP = {
'title': 'sbg:title',
'description': 'sbg:doc',
'fa_icon': 'sbg:icon',
}

# What keys to skip from the tower.yml file
SKIP_NEXTFLOW_TOWER_KEYS = [
'tower',
'mail',
]


class ExecMode(Enum):
single = 'single-instance'
multi = 'multi-instance'
92 changes: 75 additions & 17 deletions sbpack/noncwl/copy_app.py
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import argparse
import logging
import sbpack.lib as lib
# from sevenbridges.errors import NotFound
from sbpack.noncwl.utils import install_or_upgrade_app
from sbpack.noncwl.utils import install_or_upgrade_app, push_zip

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
Expand All @@ -11,34 +10,93 @@
def main():
# CLI parameters
parser = argparse.ArgumentParser()
parser.add_argument("--profile", default="default",
help="SB platform profile as set in the SB API "
"credentials file.")
parser.add_argument("--appid", required=True,
help="What to copy? Takes the form {user or division}/{project}/{app_id} "
"or {user or division}/{project}/{app_id}/{revision_no}.")
parser.add_argument("--projectid", required=True,
help="Where to copy? Takes the form "
"{user or division}/{project}")
parser.add_argument(
"--profile", default="default", nargs="+",
help="SB platform profile as set in the SB API credentials file. If "
"you are using sbcopy to copy an app from one division to "
"another, please provide two profiles - first profile for the "
"source app (appid), and second for the destination project "
"(projectid)."
)
parser.add_argument(
"--appid", required=True,
help="What to copy? Takes the form "
"{user or division}/{project}/{app_id} or "
"{user or division}/{project}/{app_id}/{revision_no}."
)
parser.add_argument(
"--projectid", required=True,
help="Where to copy? Takes the form {user or division}/{project}"
)
args = parser.parse_args()

# Preprocess CLI parameter values

# Init api
api = lib.get_profile(args.profile)
if len(args.profile) > 1:
api_source = lib.get_profile(args.profile[0])
api_dest = lib.get_profile(args.profile[1])
else:
api_source = lib.get_profile(args.profile[0])
api_dest = api_source

# Source and destination apps
source_app = api.apps.get(args.appid)
source_app = api_source.apps.get(args.appid)
sb_app_raw = source_app.raw
destination_app_id = args.projectid + '/' + args.appid.split('/')[2]

# Copy the code package
source_package = source_app.raw.get('app_content', {}).get('code_package', '')
source_package = source_app.raw.get(
'app_content', {}
).get('code_package', '')

if source_package:
new_file = api.files.get(api.files.get(source_package).copy(project=args.projectid))
sb_app_raw['app_content']['code_package'] = new_file.id
# The app_content.code_package field exists and contains the id of the
# code package file

source_package_file = api_source.files.get(source_package)
if api_source == api_dest:
# Copy has been performed in the same division/env
# Copy the file to the destination through the API
new_file_id = api_source.files.get(
source_package_file.copy(project=args.projectid)
).id
else:
# Copy has been performed between two different divisions/envs
# Download the file
name = source_package_file.name
source_package_file.download(
path=name,
overwrite=True
)

# Find out if the parent folder is the root of the project
parent = source_package_file.parent
project = api_source.projects.get(source_package_file.project)
if parent == project.root_folder:
# If the parent is the root, then set folder name to None
# This means that the code package will go into the root of the
# destination project
folder_name = None
else:
# Parent is not the root, so use it when pushing the zip file
# to preserve the folder structure
folder_name = api_source.files.get(parent).name

# Push the zip to the destination project
new_file_id = push_zip(
api=api_dest,
zip_path=name,
project_id=args.projectid,
folder_name=folder_name
)
# With this complete the code package is now at the destination

# Change the id of the code package to the new file
sb_app_raw['app_content']['code_package'] = new_file_id

install_or_upgrade_app(api,destination_app_id, sb_app_raw)
# Use the install_or_upgrade_app function to copy the app
install_or_upgrade_app(api_dest, destination_app_id, sb_app_raw)


if __name__ == "__main__":
Expand Down
Loading

0 comments on commit 852bd05

Please sign in to comment.