Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix download programming repositories #70

Merged
merged 10 commits into from
Aug 12, 2023

Conversation

FelixTJDietrich
Copy link
Collaborator

@FelixTJDietrich FelixTJDietrich commented Aug 11, 2023

Motivation and Context

@bassner had some issues running the script and I'm addressing those issues in this PR.

Description

  • Use axios for requests now (this probably also lowers the node version requirement to node 16)
  • Improve general error handling/logging
  • Remove parallelization in favor of a more long running script that is more transparent with what is going on
  • Add participation ids to the evaluation data
  • Download the repositories in batches of 25 instead of having one request downloading potentially ~1000 repos at a time
  • Write download-<timestamp>.txt and submission-<timestamp>.txt for logging downloaded/failed exercise/participation ids

Steps for Testing

  1. cd playground
  2. npm install
  3. Use the data from below in data/evaluation/programming-exercises.json
  4. Run npm run export:artemis:3-download-programming-repositories
    a. Login with instructor test account on Artemis (production)
    b. Select all exercises
    c. Download
  5. It should now run in batches
{
  "TEST": {
    "semester": "WS20/21",
    "course_id": 86,
    "exercises": [
      {
        "id": 10573,
        "title": "TEST 01",
        "participations": [
          7945271,
          8082848,
          7936574
        ],
        "is_exam_exercise": 0
      },
      {
        "id": 10573,
        "title": "TEST 02",
        "participations": [
          7945271,
          8082848,
          7936574
        ],
        "is_exam_exercise": 0
      }
    ]
  }
}

Screenshots

image_720

@FelixTJDietrich FelixTJDietrich self-assigned this Aug 11, 2023
@FelixTJDietrich FelixTJDietrich added bug Something isn't working enhancement New feature or request playground Pull requests that update the playground labels Aug 11, 2023
@FelixTJDietrich FelixTJDietrich changed the title Hotfix download programming repositories Fix download programming repositories Aug 12, 2023
Copy link
Contributor

@pal03377 pal03377 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good! I have a few suggestions below. Feel free to apply them as you wish / think is useful and then merge the PR.

@FelixTJDietrich FelixTJDietrich force-pushed the hot-fix/export-script-download branch from f238ae8 to fec1402 Compare August 12, 2023 15:30
@FelixTJDietrich FelixTJDietrich merged commit 192372d into develop Aug 12, 2023
@FelixTJDietrich FelixTJDietrich mentioned this pull request Aug 13, 2023
16 tasks
@pal03377 pal03377 deleted the hot-fix/export-script-download branch August 14, 2023 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request playground Pull requests that update the playground
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants