Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to Container CI #397

Closed
wants to merge 18 commits into from
Closed

Improvements to Container CI #397

wants to merge 18 commits into from

Conversation

tylertitsworth
Copy link
Contributor

Description

Update CI to follow DevOps and DevSecOps best practices.

How it works

Given an example(s) (AudioQnA), this workflow creates jobs that perform container CI on that example on a pull request based on whether or not those examples were modified in that PR.

# .github/workflows/example-ci.yaml
on:
  pull_request: null
...
# This job creates a list of examples as a matrix to test based on a git diff command
- name: Output Modified example Directories
  id: example-list
  run: |
    # Get diff array filtered by specific filetypes
    DIFF=$(git diff --diff-filter=d \
        --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} \
        -- '*/*Dockerfile' '*.py' '*.yaml' '*.yml' '*.sh' '*/*requirements.txt' '*.json' '*.ts' '*.js' | \
        jq -R '.' | jq -sc '.' \
    )
    # Search for compose files in each file to determine the container examples
    DOCKER_COMPOSE_PATHS=()
    for path in $(echo $DIFF | jq -r '.[]'); do
      while [[ "$path" != "." ]]; do
        DIR_PATH=$(dirname "$path")
        if [ -n "$(find "$DIR_PATH" -maxdepth 1 -name 'docker-compose.yaml' -print -quit)" ] && [ "$DIR_PATH" != "." ]; then
          DOCKER_COMPOSE_PATHS+=("$DIR_PATH")
          path="."
        else
          path="$DIR_PATH"
        fi
      done
    done
    # Convert the array to a JSON array
    DOCKER_COMPOSE_PATHS_JSON=$(printf '%s\n' "${DOCKER_COMPOSE_PATHS[@]}" | jq -R '.' | jq -sc 'unique_by(.)')
    echo "FOLDERS=$DOCKER_COMPOSE_PATHS_JSON" >> $GITHUB_OUTPUT

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)

Dependencies

n/a

Tests

I don't have the runners required to do the validation portion, however when previously testing with ubuntu-latest I was able to launch the test script from the correct directory on that runner before it ran out of space.
https://github.com/tylertitsworth/GenAIExamples/actions/runs/9896085740?pr=3

tylertitsworth added 17 commits July 10, 2024 12:21
Signed-off-by: tylertitsworth <[email protected]>

add chatqna support

Signed-off-by: tylertitsworth <[email protected]>

update tests and repo path

Signed-off-by: tylertitsworth <[email protected]>

add actions.json file for ci configuration

Signed-off-by: tylertitsworth <[email protected]>

add codegen

Signed-off-by: tylertitsworth <[email protected]>

add weekly test

Signed-off-by: tylertitsworth <[email protected]>

remove ref

Signed-off-by: tylertitsworth <[email protected]>

change runner label

Signed-off-by: tylertitsworth <[email protected]>

do the rest of them

Signed-off-by: tylertitsworth <[email protected]>

copy workflow call into repo due to gha bug #2877542

Signed-off-by: tylertitsworth <[email protected]>

configure no start

Signed-off-by: tylertitsworth <[email protected]>

patch scan and test wfs

Signed-off-by: tylertitsworth <[email protected]>

change to more descriptive name for reusable workflow

Signed-off-by: tylertitsworth <[email protected]>

additional bugfix

Signed-off-by: tylertitsworth <[email protected]>

update scan workflow

Signed-off-by: tylertitsworth <[email protected]>

see help menus

Signed-off-by: tylertitsworth <[email protected]>

add missing env

Signed-off-by: tylertitsworth <[email protected]>

use downcase method

Signed-off-by: tylertitsworth <[email protected]>

remove test

Signed-off-by: tylertitsworth <[email protected]>

use valid group dir for rest of scan workflow

Signed-off-by: tylertitsworth <[email protected]>

return import step

Signed-off-by: tylertitsworth <[email protected]>

add and validate smoke tests

Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: Tyler Titsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
Signed-off-by: tylertitsworth <[email protected]>
@tylertitsworth tylertitsworth self-assigned this Jul 12, 2024
@tylertitsworth tylertitsworth added the feature New feature or request label Jul 12, 2024
Signed-off-by: tylertitsworth <[email protected]>
JakubLedworowski pushed a commit to JakubLedworowski/GenAIExamples that referenced this pull request Jan 28, 2025
* helm: remove unused values

Signed-off-by: Lianhao Lu <[email protected]>

* change GenAIExamples to download models by iteself

Changes the default example for K8S manifests for GenAIExamples to
always download the model to avoid extra setup requirements for the end
users.

Signed-off-by: Lianhao Lu <[email protected]>

---------

Signed-off-by: Lianhao Lu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant