Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create an appliance for ray framework #157

Closed
3 tasks
rsmontero opened this issue Dec 12, 2024 · 0 comments
Closed
3 tasks

Create an appliance for ray framework #157

rsmontero opened this issue Dec 12, 2024 · 0 comments
Labels
category: appliances Generic category for new appliances status: accepted The issue is valid and will be planned for fix or implementation type: feature A new feature planned for implementation
Milestone
Release 6.10.0-3

Comments

@rsmontero
Copy link
Member

This issue is to create an appliance with pre-installed ray

  • Create base appliance with a curated set of models and pre-defined API endpoints. Thing to be defined:
    • Base OS
    • Ray dependencies to be installed
    • model.py API definition
    • YAML configuration file: relevant parameters and options
  • Air-gaped version of the appliance
  • Flow service for deploying ray as a cluster
@rsmontero rsmontero changed the title Create an appliance for ray framewor Create an appliance for ray framework Dec 12, 2024
rsmontero pushed a commit that referenced this issue Dec 20, 2024
@aleixrm
f-#157: Ray appliance (#160)
ddb4508 
- ONE_APP_RAY_MODEL_URL or ONE_APP_RAY_MODEL64: the python model file url for download it or the file content encoded in base64, respectively.

- ONE_APP_RAY_CONFIG_FILE or ONE_APP_RAY_CONFIG64 will contain the Ray configuration yaml file content (in yaml plaintext or encoded in base64). If not provided, the config.yaml.templatefile will be parameterized with the following attributes:

- ONE_APP_RAY_SERVE_PORT: The port where the chatbot API will be served. Defaults to '8000'
- ONE_APP_RAY_MODEL_ID = The language model id to use for the chatbot app. Defaults to meta-llama/Llama-3.2-1B
- ONE_APP_RAY_TOKEN: The token used for calling the model
- ONE_APP_RAY_TEMPERATURE: String with language model temperature float number (from 0.0 to 1.0). Defaults to 0.1
- ONE_APP_RAY_CHATBOT_REPLICAS: num of chatbot deployed replicas
- ONE_APP_RAY_CHATBOT_CPUS: chatbot app allocated CPU

Signed-off-by: Aleix Ramírez <[email protected]>
rsmontero added a commit that referenced this issue Dec 20, 2024
@rsmontero
F #157: Some changes to Ray VM interface
e9ac511 
- Linting
- Update vatiable naming
- Update VM with chatbot endpoint
- Add default model.py

Template configuration

- DISK should be no less than 15Gb
- Memory should be 8Gb or greater
- User inputs

USER_INPUTS=[
  ONEAPP_RAY_API_PORT="O|number|port for api server| |8000",
  ONEAPP_RAY_CONFIG_FILE64="O|text|Configuration file raw| |",
  ONEAPP_RAY_MODEL_FILE64="O|text64|Python script to use as model| |",
  ONEAPP_RAY_MODEL_ID="O|list-multiple|AI model to use|meta-llama/Llama-3.2-1B-Instruct,Qwen/Qwen2.5-1.5B-Instruct|",
  ONEAPP_RAY_MODEL_TEMPERATURE="O|number-float|| |0.1",
  ONEAPP_RAY_MODEL_TOKEN="O|text|token for the model| | ******",
  ONEAPP_RAY_MODEL_URL="O|text|URL to download de model| |" ]

- Context

CONTEXT=[
  NETWORK="YES",
  ONEAPP_RAY_API_PORT="$ONEAPP_RAY_API_PORT",
  ONEAPP_RAY_CHATBOT_CPUS="$VCPU",
  ONEAPP_RAY_CONFIG_FILE64="$ONEAPP_RAY_CONFIG_FILE64",
  ONEAPP_RAY_MODEL_FILE64="$ONEAPP_RAY_MODEL_FILE64",
  ONEAPP_RAY_MODEL_ID="$ONEAPP_RAY_MODEL_ID",
  ONEAPP_RAY_MODEL_TEMPERATURE="$ONEAPP_RAY_MODEL_TEMPERATURE",
  ONEAPP_RAY_MODEL_TOKEN="$ONEAPP_RAY_MODEL_TOKEN",
  ONEAPP_RAY_MODEL_URL="$ONEAPP_RAY_MODEL_URL",
  SET_HOSTNAME="$NAME",
  SSH_PUBLIC_KEY="$USER[SSH_PUBLIC_KEY]",
  TOKEN="YES" ]
rsmontero added a commit that referenced this issue Dec 26, 2024
@rsmontero
F #157: Add application section.
9b9f31c 
Moves some of the configuration options under its own section
(APPLICATION). Template configuration:

CONTEXT=[
  NETWORK="YES",
  ONEAPP_RAY_API_PORT="$ONEAPP_RAY_API_PORT",
  ONEAPP_RAY_API_ROUTE="$ONEAPP_RAY_API_ROUTE",
  ONEAPP_RAY_APPLICATION_FILE64="$ONEAPP_RAY_MODEL_FILE64",
  ONEAPP_RAY_APPLICATION_URL="$ONEAPP_RAY_MODEL_URL",
  ONEAPP_RAY_CHATBOT_CPUS="$VCPU",
  ONEAPP_RAY_MODEL_ID="$ONEAPP_RAY_MODEL_ID",
  ONEAPP_RAY_MODEL_TEMPERATURE="$ONEAPP_RAY_MODEL_TEMPERATURE",
  ONEAPP_RAY_MODEL_TOKEN="$ONEAPP_RAY_MODEL_TOKEN",
  SET_HOSTNAME="$NAME",
  SSH_PUBLIC_KEY="$USER[SSH_PUBLIC_KEY]",
  TOKEN="YES" ]

INPUTS_ORDER="ONEAPP_RAY_API_PORT,ONEAPP_RAY_MODEL_ID,ONEAPP_RAY_MODEL_TEMPERATURE,ONEAPP_RAY_MODEL_TOKEN"

USER_INPUTS=[
  ONEAPP_RAY_API_PORT="O|number|Port number for the API endpoint.| |8000",
  ONEAPP_RAY_API_ROUTE="O|text|Route path for the REST API exposed by the RAY application.| |/chat",
  ONEAPP_RAY_APPLICATION_FILE64="O|text64|Python application to be deployed in the RAY framework (base64).| |",
  ONEAPP_RAY_APPLICATION_URL="O|text|URL to download the Python application for the RAY framework.| |",
  ONEAPP_RAY_MODEL_ID="M|list-multiple|Determines the AI model(s) to use for inference.|meta-llama/Llama-3.2-1B-Instruct,Qwen/Qwen2.5-1.5B-Instruct|",
  ONEAPP_RAY_MODEL_TEMPERATURE="M|number-float|Temperature parameter for model outputs, controlling the randomness of generated text.| |0.1",
  ONEAPP_RAY_MODEL_TOKEN="M|text|Provides the authentication token required to access the specified AI model. | |********" ]
rsmontero added a commit that referenced this issue Dec 31, 2024
@rsmontero
F #157: Automatically configure the number of GPUs to use
a8575fd
rsmontero added a commit that referenced this issue Jan 2, 2025
@rsmontero
F #157: Add a simple test client for the default application for testing
eceb678
rsmontero added a commit to OpenNebula/marketplace that referenced this issue Jan 3, 2025
@rsmontero
F OpenNebula/one-apps#157: Appliance metadata
a8179f5
@rsmontero rsmontero added this to the Release 6.10.0-3 milestone Jan 8, 2025
@rsmontero rsmontero added type: feature A new feature planned for implementation status: accepted The issue is valid and will be planned for fix or implementation category: appliances Generic category for new appliances labels Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
category: appliances Generic category for new appliances status: accepted The issue is valid and will be planned for fix or implementation type: feature A new feature planned for implementation
Projects
None yet
Milestone
Release 6.10.0-3
Development

No branches or pull requests
1 participant
@rsmontero