Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More flexible build process for Athena #325

Merged
merged 61 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
01d2528
update build process
dmytropolityka Aug 4, 2024
0226987
add a test job, reinstall prospector
dmytropolityka Aug 4, 2024
5a59529
renew dependencies for common athena
dmytropolityka Aug 4, 2024
aa122d0
remove text output
dmytropolityka Aug 4, 2024
63009ee
add modeling llm, change lint workflow
dmytropolityka Aug 4, 2024
f47d446
change the name of a job
dmytropolityka Aug 4, 2024
e443196
attempt to get rid of the warning
dmytropolityka Aug 4, 2024
fede060
make it work again with warning
dmytropolityka Aug 4, 2024
f324319
Update pyproject.toml
dmytropolityka Aug 4, 2024
5f25edc
re-lock master project
dmytropolityka Aug 4, 2024
a640238
move modules and create scripts folder: comments of Felix
dmytropolityka Aug 8, 2024
e625cc5
move module example to programming modules
dmytropolityka Aug 10, 2024
815699b
Merge branch 'develop' into feature/change-build-process
dmytropolityka Aug 10, 2024
d95f836
fix testing server url; resolve merge conflict issues
dmytropolityka Aug 10, 2024
c195e40
fix further occurrences
dmytropolityka Aug 10, 2024
3b56177
remove duplicated value
dmytropolityka Aug 10, 2024
3452773
address athena correctly
dmytropolityka Aug 10, 2024
2215385
change the version of langsmith for text_llm
dmytropolityka Aug 10, 2024
d4bc0ed
update path for linting profile
dmytropolityka Aug 10, 2024
51e7017
update path for linting profile
dmytropolityka Aug 10, 2024
14dcc7e
change the way of addressing the prospector profile
dmytropolityka Aug 10, 2024
24b951c
fix linting errors for module text llm
dmytropolityka Aug 10, 2024
7580d37
change the script to find modules to build
dmytropolityka Aug 10, 2024
411b3e7
change the script to find modules to build
dmytropolityka Aug 10, 2024
85c244f
update docker files to not depend on common athena
dmytropolityka Aug 10, 2024
c8d8d1e
revert deploy changes
dmytropolityka Aug 10, 2024
31222fd
extract only the dir name in the image to build script
dmytropolityka Aug 10, 2024
79d60a2
change only the image name
dmytropolityka Aug 10, 2024
20be2a7
use the module name for uploads
dmytropolityka Aug 10, 2024
6655906
attempt to solve problem
dmytropolityka Aug 10, 2024
2d6095b
attempt to solve the problem
dmytropolityka Aug 10, 2024
dacb67d
attempt to solve the problem
dmytropolityka Aug 10, 2024
db508ca
attempt to solve the problem
dmytropolityka Aug 11, 2024
61ff6ac
resolve versioning problem
dmytropolityka Aug 11, 2024
939c0e3
fix a problem in module modeling llm
dmytropolityka Aug 11, 2024
e936ecc
replace lms url
dmytropolityka Aug 11, 2024
e386066
create a new virtual env by default
dmytropolityka Aug 12, 2024
faf4c41
create a new virtual env by default
dmytropolityka Aug 12, 2024
a8f905c
the same effect but using a better approach
dmytropolityka Aug 12, 2024
b7eb56a
apply the same for linting script
dmytropolityka Aug 12, 2024
0a1a0dd
refactoring
dmytropolityka Aug 12, 2024
464abdc
use the same env vars for prospector step
dmytropolityka Aug 12, 2024
a3813ee
revert changes
dmytropolityka Aug 12, 2024
76279bd
move lint job to the same step
dmytropolityka Aug 12, 2024
1f56d08
revert changes
dmytropolityka Aug 12, 2024
37b8dfa
modify scripts
dmytropolityka Aug 12, 2024
8e0b1e4
modify scripts
dmytropolityka Aug 12, 2024
133e1e0
revert changes
dmytropolityka Aug 12, 2024
df0b4d1
install to specific location
dmytropolityka Aug 12, 2024
f38f3ae
use the same local environment
dmytropolityka Aug 12, 2024
fcdc7e0
use another approach to set custom path
dmytropolityka Aug 12, 2024
d1c2d34
install prospector for log_viewer
dmytropolityka Aug 12, 2024
5ffda52
Merge branch 'develop' into feature/change-build-process
maximiliansoelch Aug 12, 2024
64f922a
Merge branch 'develop' into feature/change-build-process
maximiliansoelch Aug 13, 2024
054a6d0
Merge branch 'develop' into feature/change-build-process
dmytropolityka Sep 2, 2024
641a71a
remove idea file
dmytropolityka Sep 2, 2024
8b2bdcb
Merge remote-tracking branch 'origin/feature/change-build-process' in…
dmytropolityka Sep 2, 2024
d586061
adapt path for module text cofee
dmytropolityka Sep 2, 2024
004fcec
update poetry lock files
dmytropolityka Sep 2, 2024
0a0a1da
move prospector and pydantic to dev dependencies
dmytropolityka Sep 4, 2024
487007a
update vs code paths
dmytropolityka Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions .github/scripts/images-to-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ fi
# Check if athena folder was changed (then we need to build all module_* images)
ATHENA_CHANGED=$(echo "$CHANGED_FILES" | grep -q "^athena" && echo "true" || echo "false")

# Loop over all root level directories
for DIR in */; do
# Loop over all root level directories and modules
for DIR in modules/*/*/ */; do
# If a Dockerfile exists in the directory
if [[ -e "${DIR}Dockerfile" ]]; then
DIR=${DIR%/} # Remove trailing slash
Expand All @@ -44,6 +44,7 @@ for DIR in */; do
continue
fi


# Build all images on develop branch
if [[ "$GITHUB_REF" == "refs/heads/develop" ]]; then
DIRS+=("$DIR")
Expand All @@ -56,8 +57,11 @@ for DIR in */; do
continue
fi

# Extract just the final directory name (e.g., "module_example") from the full path
IMAGE_NAME_SUFFIX=$(basename "$DIR")

# Construct Docker image name and tag
IMAGE_NAME="athena/$DIR"
IMAGE_NAME="athena/$IMAGE_NAME_SUFFIX"
IMAGE_TAG="pr-$PR_NUMBER"

# Check if any file has changed in that directory since the pull request was created
Expand Down
9 changes: 6 additions & 3 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,17 @@ jobs:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}


- name: Set IMAGE_NAME
run: echo "IMAGE_NAME=$(basename ${{ matrix.image }})" >> $GITHUB_ENV

- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
images: |
ghcr.io/${{ github.repository_owner }}/Athena/${{ matrix.image }}
ls1tum/athena_${{ matrix.image }}
ghcr.io/${{ github.repository_owner }}/Athena/${{ env.IMAGE_NAME }}
ls1tum/athena_${{ env.IMAGE_NAME }}
tags: |
type=raw,value=${{ github.ref == 'refs/heads/develop' && 'develop' || github.sha }}
type=raw,value=${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.sha }}
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,6 @@ jobs:
run: |
echo "$HOME/.local/bin" >> $GITHUB_PATH
poetry install
poetry run install_all
- name: Run Prospector
run: poetry run prospector
run: poetry run lint_all
38 changes: 38 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Tests

on:
push:
branches:
- develop
pull_request:

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Check out code
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install Poetry
run: |
curl -sSL https://install.python-poetry.org | python3 -
- name: Cache Poetry dependencies
uses: actions/cache@v2
with:
path: ~/.cache/pypoetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-poetry-
- name: Install dependencies
run: |
echo "$HOME/.local/bin" >> $GITHUB_PATH
poetry install
poetry run install_all
- name: Run tests
run: poetry run test_all
4 changes: 3 additions & 1 deletion .idea/runConfigurations/module_programming_llm.xml
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions .prospector.yaml
dmytropolityka marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ autodetect: false
ignore-paths:
- docs
- module_text_cofee/module_text_cofee/protobuf
- module_programming_themisml/module_programming_themisml/extract_methods/languages
- module_programming_themisml/module_programming_themisml/extract_methods/method_parser_listener.py
- module_programming_apted/module_programming_apted/convert_code_to_ast/languages
- module_programming_themisml/extract_methods/languages
- module_programming_themisml/extract_methods/method_parser_listener.py
- module_programming_apted/convert_code_to_ast/languages

mypy:
run: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ class Module(BaseModel):
url: AnyHttpUrl = Field(example="http://localhost:5001")
type: ExerciseType = Field(example=ExerciseType.text)
supports_evaluation: bool = Field(description="Whether the module supports evaluation", example=True)
supports_non_graded_feedback_requests: bool = Field(description="Whether the module supports non-graded feedback requests", example=True),
supports_non_graded_feedback_requests: bool = Field(description="Whether the module supports non-graded feedback requests", example=True)
supports_graded_feedback_requests: bool = Field(description="Whether the module supports graded feedback requests", example=True)
Loading
Loading