Clean up model related fields in the launcher #202
Labels
Comments
Leon-Leyang
added a commit
that referenced
this issue
Feb 18, 2024
4 tasks
PaulCCCCCCH
added a commit
that referenced
this issue
May 9, 2024
…age's dependencies & Update supported CUDA versions (#236) * Modify the UI * Add functionality of selecting dev set path * Change the type of dialog of saveProfile * Skip checking config when it's in manageTab * Print complete logs of the running container * fix: influence calculation fixed; implemented with RBuffer to interact with DB; need debugging * Print logs on logBrowser * Let text browsers auto scroll to the bottom * Let text browsers auto scroll to the bottom * Allow to save config in newly created file * Stop all threads when the launcher is closed * Stop all threads when the launcher is closed * Remove color info in log with regex * Check if Docker is running before starting the launcher * Check if Docker is running before starting the launcher * Check if the device matches the image version * fix: influence function fixed; fixed conflict with flashtorch; forced influence calculation to use dataLoaders with num_workers=0; need testing * feat: implemented instance-level influence calculation * influence calculation ready for test * fix: influence calculation bug fixed; OOM issue resolved as suggested by (need further checking) * Refactor the structure * Refactor the structure * fix: remove unique constraint for influ_rel.path in db because joint PK is used * fix: now correctly using image paths after calculating influence * fix: influence calculation done * Refactor the structure * changed influence scale * Refactor to mvc pattern * Refactor to mvc pattern * Fix the bug in displaying log * Add new config in the view * Connect new signals to slot functions * Complete slot functions * Save model relevant configs in a json file and pass it as the config file * Fix the bug of displaying wrong image version * Fix the bug of displaying wrong image version * Close #80 * Remove useless port setting #81 * Change the layout and close #81 * Change the layout and close #81 * Remove deprecated port map during container creation and close #81 * Dynamically fetch the image versions * Modify key names in launcher model's profile to match the backend code * Provide weight file options according to the checkpoint path * Add tool tips for each column in the launcher #92 * Package the launcher and add the documentation in README.md #70 * Checkout all files from dev except those under launcher/ * Supplement checks for model relevant settings * Add local .gitignore for launcher/ * Assign default value to model architecture * Fix the size of the main window * Fix the sizes of the components * Resolve resource issues by using the .qrc file and update the packaging approach #109 * Create `RecordData` folder to store temp files #108 * Add prompt in popup window about network failure #107 * Display image download progress in `Details` page #113 * Initialize `apiClient` according to the OS #113 * Fix the bug of connecting `shuffleCheckBox` signal to wrong controller function * Add the container name and port used for each line of logs. * Update requirements.txt * Update requirements.txt * Update the setup * Fix the bug when there is no local image * Check if the image has a tag * Print specific exception info on the popup window * Fix the bug of adding the stopped container to `created` after failing to start it due to port conflict * Fix #125 * added support for validation set (#129) * Added support for `validation set` #99 * Refactor `launcher_v2.ui` * Refactor `popup.ui` * Refactor `model.py` * Generate new `main_view_ui.py` and `popup_view_ui.py` * Refactor controllers * Synchronize record data at the beginning #130 * Wujc/generated for review and layout adjustment (#134) * commit before pull and merge dev * passed all backend tests * add mkdir for the generated file in the app setup * make validation, paired, generated changeable on UI and add paired and genreated boxs to UI * add mount bindings for paired and gen in docker_ctrl.py * Updated the about page (#131) * Updated the about page * feat: improved and unified some styles Co-authored-by: Desmond <[email protected]> --------- Co-authored-by: Chonghan Chen <[email protected]> Co-authored-by: Desmond <[email protected]> * Adjust ui to display all contents in `data_group_box` * Adjust ui of `manage_widget` * Rename 'Generated Files' to 'Output Folder' and add relevant check in `check_profile` * Create global folder for records #139 * Modify the path of the global record folder for mac os #139 * Change the global record data folder on mac os to `~/Library/Application Support/RobustarLauncher` #139 * Init `api_client` for mac os #138 * Force the selection of a container for stop or deletion to be done on Manage Tab Page * Set line edits for the data paths read-only * Add the input check for port, batch size, worker number, class number and image size #142 * Implement the new way of specifying the device #141 * Separate creation and other operations on existing containers on two tab pages * Add `LoggerManager` * Move `LoggerManager` to a separated file and add func `append_log` * Add logs for operations in `main_ctrl` * Change the path to save the config records and assume the directory 'configs' has been created with 'config_record.json' in it * Add logs for operations in `docker_ctrl` * Fix typo * Ignore the operation if the user select nothing in the file dialog * Fix typo * Fix typo * Add a global try exception for the launcher * Modify the global try exception hook to redirect the uncaught exception to app log * Set the global exception hook in `App`'s `__init__` method * Customize the exception hook for threading * Prompt users when the launcher cannot work properly * Refresh the server list widgets whenever switching to the `manage_widget` * Add match dict to align ui padding text for backend use * Stop searching ckpt files when the ckpt root is not properly set * Get gpu info with `nvgpu.gpu_info` and append the index of the gpu that is not fully used to the device combo box #141 * Handle the case when the item to be removed does not belong to the list widget * Add log info * Manually restore the changes in files not under the /launcher directory * Catch all exceptions when using `nvgpu.gpu_info()` * Add timestamp to logs output * Modify the check for port unavailable error * Changed prompt format * Remove `batch_size` and `num_worker` configs in launcher * Clean up terms in `config` * Clean up terms in `config` * Clean up model related fields in the launcher #202 * Align Docker image's package versions with backend package versions * Align Docker image's package versions with backend package versions * Update base docker image's dependencies & base docker image version * Update base docker image's dependencies & base docker image version(for testing) * Modify default VCUDA version(for testing) * Modify default VCUDA version(for testing) * Finish package dependency testing --------- Co-authored-by: Chonghan <[email protected]> Co-authored-by: paulcccccch <[email protected]> Co-authored-by: Chonghan Chen <[email protected]> Co-authored-by: Jingcheng Wu（吴 璟承） <[email protected]> Co-authored-by: Desmond <[email protected]>
PaulCCCCCCH
added a commit
that referenced
this issue
May 9, 2024
* Model switch and other CRUD APIs implemented. (#193) * Add db setup doc; remove useless db files * add database schema change note * add db schema design diagram and code * Create 'model.py' for initial model APIs sketch * Add docstrings for the UploadModel API * Add 'model_utils.py' for utilities & outline 'UploadModel' API * Get all metadata of the model according to the db design * Complete `init_model` function implementation * Extend `UploadModel` to handle uploading models after training * refactor: implemented orm models * fixed db model definition bugs * change db types in function signatures * Enhance UploadModel API with ID assignment and architecture metadata * Complete `val_model` function implementation * fix: new database operations with SQLAlchemy partially working * most functionalities working; not passing test cases * test cases passing * chore: update docker image version; include SQLAlchemy dependency * feat: implemented model operations * Update save path for model definition and checkpoint files to /generated/models * Refactor 'UploadModel' API for enhanced model handling - Improve API to handle predefined models. - Remove logic handling post-training model uploads. - Utilize `create_model` from `RModelWrapper` for saving. * Migrate from importlib_resources to importlib.resources * Refresh requirements.txt with updated packge versions * Correct some mistakes to let 'UploadModel' API work for basic custom model upload - Change `Models` table id type from BigInteger to Integer for autoincrement - Correct typos * Fix type errors of var `pretrained` and `num_classes` * Save the weight of pretrained predefined model to a local file * Raise an exception if pretrained weight is required for "ResNet18-32x32" * Save `num_classes` of predefined models into local files for later restoration * Save current model weights to a local file if the weight file is not provided * Comment db clean up temporarily. * feat: add model switch with dummy model id and test * feat: model apis implemented; need testing * model switch and CRUD api implemented * chore: resolve conflict in requirements.txt * fixed things to pass test_predict * model name attribute name fix * api fixes * fix: add a serializer for db models * add model api error handling and use serializer everywhere * update test case * commenting out test case for now * chore: fix nit * fix problem in set_current_model --------- Co-authored-by: Leon-Leyang <[email protected]> * Merge/model db upload val (#205) * Add db setup doc; remove useless db files * add database schema change note * add db schema design diagram and code * Create 'model.py' for initial model APIs sketch * Add docstrings for the UploadModel API * Add 'model_utils.py' for utilities & outline 'UploadModel' API * Get all metadata of the model according to the db design * Complete `init_model` function implementation * Extend `UploadModel` to handle uploading models after training * refactor: implemented orm models * fixed db model definition bugs * change db types in function signatures * Enhance UploadModel API with ID assignment and architecture metadata * Complete `val_model` function implementation * fix: new database operations with SQLAlchemy partially working * most functionalities working; not passing test cases * test cases passing * chore: update docker image version; include SQLAlchemy dependency * feat: implemented model operations * Update save path for model definition and checkpoint files to /generated/models * Refactor 'UploadModel' API for enhanced model handling - Improve API to handle predefined models. - Remove logic handling post-training model uploads. - Utilize `create_model` from `RModelWrapper` for saving. * Migrate from importlib_resources to importlib.resources * Refresh requirements.txt with updated packge versions * Correct some mistakes to let 'UploadModel' API work for basic custom model upload - Change `Models` table id type from BigInteger to Integer for autoincrement - Correct typos * Fix type errors of var `pretrained` and `num_classes` * Save the weight of pretrained predefined model to a local file * Raise an exception if pretrained weight is required for "ResNet18-32x32" * Save `num_classes` of predefined models into local files for later restoration * Save current model weights to a local file if the weight file is not provided * Comment db clean up temporarily. * Permit optional transmission of description and tags in metadata from frontend * Update model metadata design: Introduce `nickname` and Rename `name` to `class_name` * Extract 'tags' column from 'Models' table into a new table, and maintain the relationship * Enhance `UploadModel` docstring documentation * Close DB connection explicitly for .db file removal; Set fixture scope to 'function' for fresh instances per test. * Change the base directory in pytest for debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Restore for a full test * Capture the 'stdout' and 'stderr' in pytest for debugging * Debugging * Debugging * Debugging * Solve the problem by specifying the db file address as absolute path * Restore the config of CircleCI test * fix: provide app context to all thread creations * fix: correctly getting context for child threads * Resolve code review feedback: - Improve robustness by checking for 'code' and 'pretrained' in request - Remove 'back-end/database/db_ops.py' file (double-checked for necessity) Closes: #192 * Add `predefined` to the model's metadata * Cherry-pick model refactors from refactor/model-db-upload-val Selected changes from: - ./backend/apis/model.py - ./backend/utils/model_utils.py * Add one TODO * Remove one TODO * Add `pretrained` as a new field in 'models' table & Stop saving weights for models uploaded without an explicit weight file * Set `pretrained` as required field for metadata --------- Co-authored-by: Chonghan <[email protected]> * Resolved #210: Ensured correct device mapping for model weights loading * Resolved #209: Added unique constraint to 'nickname' column in 'Models' table to ensure uniqueness * Resolved #207: Initialized `num_classes` in RDataManager's constructor * Reverted `num_classes` initialization in `RModelWrapper` due to circular import issue with `RDataManager` in `RServer` [#207] * Refactored `init_predefined_model` and `init_custom_model` in `RModelWrapper` from static to non-static methods for member variables access * Implemented check for model's idle state before attempting model switch * Removed unnecessary import for code cleanup * Moved `check_configs` function to `utils/train_utils.py` for better modularity * Added model acquisition step prior to initiating training process * Switch to target model before training setup initiation * Perform basic code refactor for clarity and efficiency * Utilize `image_size` from `data_manager` for initializing `test_set` * Remove redundant model release in `start_train` exception handling * feat: more APIs implemented and manually tested (#211) Co-authored-by: Leon-Leyang <[email protected]> * Fix wrong type match for `auto_save` * Fix wrong type match for values in training configs * Fix wrong type match for values in training configs * Replace test set with validation set for training validation; rename `net` to `model` in Trainer class * Refactor backend training logic for v0.3 architecture (#208) * Dispose db engine and delete db file; update `new_server_object` to accept `app` and `socket` arguments for flexible server instantiation in tests * Use 'Robustar2-test' as dataset during backend test * Use copying instead of unzipping for test set up * Test: implemented basic model swtich and model CRUD tests. (#214) * feat: more APIs implemented and manually tested * add utility functions and some test cases * test: implemented basic model switch CRUD tests * temp commit * uncomment a test case for edit * update edit test case * add another edit test case * refactored db init logic * fix backend unit tests * use same db across test cases * change scope to function * fix failed tests * Temporarily fix the end-to-end training test (#216) * Modify the default configs for train to pass the e2e test * Debugging * Upload a model during the initialization of the `RModelWrapper` to pass e2e test temporarily * Fix typo * Pass `app` to `RModelWrapper` for backend test compatibility, ensuring e2e test success * If `validation_root` does not exist, set it to `test_root` * Delete the model uploaded during last time's init * Update back-end/objects/RModelWrapper.py Co-authored-by: Chonghan Chen <[email protected]> * Ignore `None` fields and `0` fields when constructing model metadata --------- Co-authored-by: Chonghan Chen <[email protected]> * Update back-end/apis/model.py Co-authored-by: Leon-Leyang <[email protected]> * Update back-end/apis/model.py Co-authored-by: Leon-Leyang <[email protected]> --------- Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> * Model Page (model uploader, model list) (#218) * fix visualizer in annotate page and add the relevant front end test * add predit button test * add front-end test which catch the bug of calling api twice * cannot find the reason why predict api was called twice so i fix the visulizer display bug in annotate page * modify wait time(500) * fix the load issue in annotate page * delete one comment * add initialized model page * feat: model upload dialog * adjust page layout * Add db setup doc; remove useless db files * add database schema change note * add db schema design diagram and code * Create 'model.py' for initial model APIs sketch * Add docstrings for the UploadModel API * Add 'model_utils.py' for utilities & outline 'UploadModel' API * change button positions * feat: improve styles of model uploader; add data binding * Get all metadata of the model according to the db design * Complete `init_model` function implementation * Extend `UploadModel` to handle uploading models after training * refactor: implemented orm models * fixed db model definition bugs * delete misupload content * delete misapplied content * change db types in function signatures * feat: add code file input; read content from code file * Enhance UploadModel API with ID assignment and architecture metadata * Complete `val_model` function implementation * fix: new database operations with SQLAlchemy partially working * most functionalities working; not passing test cases * test cases passing * chore: update docker image version; include SQLAlchemy dependency * feat: implemented model operations * Update save path for model definition and checkpoint files to /generated/models * Refactor 'UploadModel' API for enhanced model handling - Improve API to handle predefined models. - Remove logic handling post-training model uploads. - Utilize `create_model` from `RModelWrapper` for saving. * Migrate from importlib_resources to importlib.resources * Refresh requirements.txt with updated packge versions * build: update lerna to 7; use npm workspaces; reorg deps * fix: build script * Correct some mistakes to let 'UploadModel' API work for basic custom model upload - Change `Models` table id type from BigInteger to Integer for autoincrement - Correct typos * Fix type errors of var `pretrained` and `num_classes` * Save the weight of pretrained predefined model to a local file * Raise an exception if pretrained weight is required for "ResNet18-32x32" * Save `num_classes` of predefined models into local files for later restoration * Save current model weights to a local file if the weight file is not provided * Comment db clean up temporarily. * feat: add model switch with dummy model id and test * feat: model apis implemented; need testing * Permit optional transmission of description and tags in metadata from frontend * Update model metadata design: Introduce `nickname` and Rename `name` to `class_name` * Extract 'tags' column from 'Models' table into a new table, and maintain the relationship * Enhance `UploadModel` docstring documentation * feat: improve styles of model page; add data binding * Close DB connection explicitly for .db file removal; Set fixture scope to 'function' for fresh instances per test. * Change the base directory in pytest for debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Restore for a full test * Capture the 'stdout' and 'stderr' in pytest for debugging * Debugging * Debugging * Debugging * Solve the problem by specifying the db file address as absolute path * Restore the config of CircleCI test * fix: provide app context to all thread creations * feat: add api functions * fix: correctly getting context for child threads * Resolve code review feedback: - Improve robustness by checking for 'code' and 'pretrained' in request - Remove 'back-end/database/db_ops.py' file (double-checked for necessity) Closes: #192 * model switch and CRUD api implemented * chore: resolve conflict in requirements.txt * fixed things to pass test_predict * feat: integrate model upload API * model name attribute name fix * feat: add code and weight file when uploading model * api fixes * fix: add a serializer for db models * feat: integrate get model list api * add model api error handling and use serializer everywhere * Add `predefined` to the model's metadata * update test case * commenting out test case for now * chore: fix nit * Cherry-pick model refactors from refactor/model-db-upload-val Selected changes from: - ./backend/apis/model.py - ./backend/utils/model_utils.py * Add one TODO * Remove one TODO * fix problem in set_current_model * Add `pretrained` as a new field in 'models' table & Stop saving weights for models uploaded without an explicit weight file * feat(model): integrate existing APIs * feat(model ): add tag field * feat: more APIs implemented and manually tested * add the model list panel * add utility functions and some test cases * modify the model list panel * modify the model list panel * fix(model ): uploader and model list issues * feat(model): integrate update, duplicate and get predefined API * test: implemented basic model switch CRUD tests * temp commit * modify the style of modellist component * uncomment a test case for edit * update edit test case * add another edit test case * refactored db init logic * fix backend unit tests * use same db across test cases * change scope to function * fix failed tests * feat(model): improve styles and fix bugs of list data table * re-organize files * chore: re-organize files and made some other small changes * fix merge conflicts * Fix training without current model set to pass tests for now * restoring package-lock since no library was updated --------- Co-authored-by: lxn <[email protected]> Co-authored-by: Desmond <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Xinnuo Li <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Desmond <[email protected]> * Test: implemented model upload tests. (#215) * feat: more APIs implemented and manually tested * add utility functions and some test cases * test: implemented basic model switch CRUD tests * temp commit * Remove 'Robustar2-test-copy' prior to setup if it exists * Uncomment the previously disabled test script for model apis * Skip downloading frontend dependencies for debugging * Remove `tags` in test metadata for debugging * Debugging * Check `response.status_code` instead of `response_data["code"]` * Debugging * Debugging * Revert settings from debug mode to standard configuration * Add tags in the test metadata * Test predefined model * Add the missing comma to the test metadata * Add test case for uploading custom model with weights * Set `basedir` as a fixture * Change the way of setting the path of the test weight file * Pass `basedir` to `test_upload_model` * Add test metadata for predefined and pretrained model * Add test metadata for custom model without code * uncomment a test case for edit * update edit test case * add another edit test case * refactored db init logic * fix backend unit tests * use same db across test cases * change scope to function * fix failed tests * Modify the default configs for train to pass the e2e test * Debugging * Upload a model during the initialization of the `RModelWrapper` to pass e2e test temporarily * Fix typo * Pass `app` to `RModelWrapper` for backend test compatibility, ensuring e2e test success * If `validation_root` does not exist, set it to `test_root` * Add `base_dir` as fixture * Fix typo in the test metadata * Remove duplicate check for `predefined` or not * Delete the model uploaded during last time's init * Remove the outermost `try...except` in `upload_model` which was used for catching unexpected errors * Add a test case for uploading custom model with the wrong code(wrong class number in the def) * Add check for model output dimension in `val_model` * Complete test cases for model upload * Remove resolved `TODO` --------- Co-authored-by: Chonghan <[email protected]> * fix: deleting current model (#223) * Fixed issue #219: enhanced exception handling in upload_model API for clean-up of temporary files. (#220) * feat: more APIs implemented and manually tested * add utility functions and some test cases * test: implemented basic model switch CRUD tests * temp commit * Remove 'Robustar2-test-copy' prior to setup if it exists * Uncomment the previously disabled test script for model apis * Skip downloading frontend dependencies for debugging * Remove `tags` in test metadata for debugging * Debugging * Check `response.status_code` instead of `response_data["code"]` * Debugging * Debugging * Revert settings from debug mode to standard configuration * Add tags in the test metadata * Test predefined model * Add the missing comma to the test metadata * Add test case for uploading custom model with weights * Set `basedir` as a fixture * Change the way of setting the path of the test weight file * Pass `basedir` to `test_upload_model` * Add test metadata for predefined and pretrained model * Add test metadata for custom model without code * uncomment a test case for edit * update edit test case * add another edit test case * refactored db init logic * fix backend unit tests * use same db across test cases * change scope to function * fix failed tests * Modify the default configs for train to pass the e2e test * Debugging * Upload a model during the initialization of the `RModelWrapper` to pass e2e test temporarily * Fix typo * Pass `app` to `RModelWrapper` for backend test compatibility, ensuring e2e test success * If `validation_root` does not exist, set it to `test_root` * Add `base_dir` as fixture * Fix typo in the test metadata * Remove duplicate check for `predefined` or not * Delete the model uploaded during last time's init * Remove the outermost `try...except` in `upload_model` which was used for catching unexpected errors * Add a test case for uploading custom model with the wrong code(wrong class number in the def) * Add check for model output dimension in `val_model` * Complete test cases for model upload * Fix #219: Implement customized context manager for clean-up in `upload_model` API * Move `traceback.print_exc()` to `__exit__` in `ContextManager` & Raise `ValueError` in `precheck_request_4_upload_model` instead of returning the error string * Update the download link for Robustar2-test.zip * Specify the version of gdown to install to avoiid download error --------- Co-authored-by: Chonghan <[email protected]> * Move /database/scripts to /back-end/scripts and deleted /database, close #183 (#221) * Removing an useless tab. (#227) * Fixed issue #213: allowed variable `num_classes` for pre-trained models (#229) * Update model loading for partial pre-trained weights support, addressing issue #213 * Fix tests * Use `IMAGENET_OUTPUT_SIZE` instead of `1000` * v0.3 bug fixes from manual testing (#230) * fix deleting annotation bug * fix: bug when copying a model twice * Completed issue #224: used `id` instead of `nickname` as the operation handle for models (#232) * Refactor backend and tests to identify models by `id` instead of `nickname` (#224) * Use `id` as the model handle in training * Fix typo * Fix typo * fix frontend model list operation bugs (#234) --------- Co-authored-by: Chonghan Chen <[email protected]> * Clean up model related fields in the launcher & Update base docker image's dependencies & Update supported CUDA versions (#236) * Modify the UI * Add functionality of selecting dev set path * Change the type of dialog of saveProfile * Skip checking config when it's in manageTab * Print complete logs of the running container * fix: influence calculation fixed; implemented with RBuffer to interact with DB; need debugging * Print logs on logBrowser * Let text browsers auto scroll to the bottom * Let text browsers auto scroll to the bottom * Allow to save config in newly created file * Stop all threads when the launcher is closed * Stop all threads when the launcher is closed * Remove color info in log with regex * Check if Docker is running before starting the launcher * Check if Docker is running before starting the launcher * Check if the device matches the image version * fix: influence function fixed; fixed conflict with flashtorch; forced influence calculation to use dataLoaders with num_workers=0; need testing * feat: implemented instance-level influence calculation * influence calculation ready for test * fix: influence calculation bug fixed; OOM issue resolved as suggested by (need further checking) * Refactor the structure * Refactor the structure * fix: remove unique constraint for influ_rel.path in db because joint PK is used * fix: now correctly using image paths after calculating influence * fix: influence calculation done * Refactor the structure * changed influence scale * Refactor to mvc pattern * Refactor to mvc pattern * Fix the bug in displaying log * Add new config in the view * Connect new signals to slot functions * Complete slot functions * Save model relevant configs in a json file and pass it as the config file * Fix the bug of displaying wrong image version * Fix the bug of displaying wrong image version * Close #80 * Remove useless port setting #81 * Change the layout and close #81 * Change the layout and close #81 * Remove deprecated port map during container creation and close #81 * Dynamically fetch the image versions * Modify key names in launcher model's profile to match the backend code * Provide weight file options according to the checkpoint path * Add tool tips for each column in the launcher #92 * Package the launcher and add the documentation in README.md #70 * Checkout all files from dev except those under launcher/ * Supplement checks for model relevant settings * Add local .gitignore for launcher/ * Assign default value to model architecture * Fix the size of the main window * Fix the sizes of the components * Resolve resource issues by using the .qrc file and update the packaging approach #109 * Create `RecordData` folder to store temp files #108 * Add prompt in popup window about network failure #107 * Display image download progress in `Details` page #113 * Initialize `apiClient` according to the OS #113 * Fix the bug of connecting `shuffleCheckBox` signal to wrong controller function * Add the container name and port used for each line of logs. * Update requirements.txt * Update requirements.txt * Update the setup * Fix the bug when there is no local image * Check if the image has a tag * Print specific exception info on the popup window * Fix the bug of adding the stopped container to `created` after failing to start it due to port conflict * Fix #125 * added support for validation set (#129) * Added support for `validation set` #99 * Refactor `launcher_v2.ui` * Refactor `popup.ui` * Refactor `model.py` * Generate new `main_view_ui.py` and `popup_view_ui.py` * Refactor controllers * Synchronize record data at the beginning #130 * Wujc/generated for review and layout adjustment (#134) * commit before pull and merge dev * passed all backend tests * add mkdir for the generated file in the app setup * make validation, paired, generated changeable on UI and add paired and genreated boxs to UI * add mount bindings for paired and gen in docker_ctrl.py * Updated the about page (#131) * Updated the about page * feat: improved and unified some styles Co-authored-by: Desmond <[email protected]> --------- Co-authored-by: Chonghan Chen <[email protected]> Co-authored-by: Desmond <[email protected]> * Adjust ui to display all contents in `data_group_box` * Adjust ui of `manage_widget` * Rename 'Generated Files' to 'Output Folder' and add relevant check in `check_profile` * Create global folder for records #139 * Modify the path of the global record folder for mac os #139 * Change the global record data folder on mac os to `~/Library/Application Support/RobustarLauncher` #139 * Init `api_client` for mac os #138 * Force the selection of a container for stop or deletion to be done on Manage Tab Page * Set line edits for the data paths read-only * Add the input check for port, batch size, worker number, class number and image size #142 * Implement the new way of specifying the device #141 * Separate creation and other operations on existing containers on two tab pages * Add `LoggerManager` * Move `LoggerManager` to a separated file and add func `append_log` * Add logs for operations in `main_ctrl` * Change the path to save the config records and assume the directory 'configs' has been created with 'config_record.json' in it * Add logs for operations in `docker_ctrl` * Fix typo * Ignore the operation if the user select nothing in the file dialog * Fix typo * Fix typo * Add a global try exception for the launcher * Modify the global try exception hook to redirect the uncaught exception to app log * Set the global exception hook in `App`'s `__init__` method * Customize the exception hook for threading * Prompt users when the launcher cannot work properly * Refresh the server list widgets whenever switching to the `manage_widget` * Add match dict to align ui padding text for backend use * Stop searching ckpt files when the ckpt root is not properly set * Get gpu info with `nvgpu.gpu_info` and append the index of the gpu that is not fully used to the device combo box #141 * Handle the case when the item to be removed does not belong to the list widget * Add log info * Manually restore the changes in files not under the /launcher directory * Catch all exceptions when using `nvgpu.gpu_info()` * Add timestamp to logs output * Modify the check for port unavailable error * Changed prompt format * Remove `batch_size` and `num_worker` configs in launcher * Clean up terms in `config` * Clean up terms in `config` * Clean up model related fields in the launcher #202 * Align Docker image's package versions with backend package versions * Align Docker image's package versions with backend package versions * Update base docker image's dependencies & base docker image version * Update base docker image's dependencies & base docker image version(for testing) * Modify default VCUDA version(for testing) * Modify default VCUDA version(for testing) * Finish package dependency testing --------- Co-authored-by: Chonghan <[email protected]> Co-authored-by: paulcccccch <[email protected]> Co-authored-by: Chonghan Chen <[email protected]> Co-authored-by: Jingcheng Wu（吴 璟承） <[email protected]> Co-authored-by: Desmond <[email protected]> * Implement first test case for model training with other necessary code changes (#235) * Adding first test for training with necessary refactors * minor cleanup * minor cleanup * add poll for task creation with some refactoring * fix: upload model before training start * add logging; training thread not registered in task center for some reason * Model Page UI Improvements (#228) * fix visualizer in annotate page and add the relevant front end test * add predit button test * add front-end test which catch the bug of calling api twice * cannot find the reason why predict api was called twice so i fix the visulizer display bug in annotate page * modify wait time(500) * fix the load issue in annotate page * delete one comment * add initialized model page * feat: model upload dialog * adjust page layout * Add db setup doc; remove useless db files * add database schema change note * add db schema design diagram and code * Create 'model.py' for initial model APIs sketch * Add docstrings for the UploadModel API * Add 'model_utils.py' for utilities & outline 'UploadModel' API * change button positions * feat: improve styles of model uploader; add data binding * Get all metadata of the model according to the db design * Complete `init_model` function implementation * Extend `UploadModel` to handle uploading models after training * refactor: implemented orm models * fixed db model definition bugs * delete misupload content * delete misapplied content * change db types in function signatures * feat: add code file input; read content from code file * Enhance UploadModel API with ID assignment and architecture metadata * Complete `val_model` function implementation * fix: new database operations with SQLAlchemy partially working * most functionalities working; not passing test cases * test cases passing * chore: update docker image version; include SQLAlchemy dependency * feat: implemented model operations * Update save path for model definition and checkpoint files to /generated/models * Refactor 'UploadModel' API for enhanced model handling - Improve API to handle predefined models. - Remove logic handling post-training model uploads. - Utilize `create_model` from `RModelWrapper` for saving. * Migrate from importlib_resources to importlib.resources * Refresh requirements.txt with updated packge versions * build: update lerna to 7; use npm workspaces; reorg deps * fix: build script * Correct some mistakes to let 'UploadModel' API work for basic custom model upload - Change `Models` table id type from BigInteger to Integer for autoincrement - Correct typos * Fix type errors of var `pretrained` and `num_classes` * Save the weight of pretrained predefined model to a local file * Raise an exception if pretrained weight is required for "ResNet18-32x32" * Save `num_classes` of predefined models into local files for later restoration * Save current model weights to a local file if the weight file is not provided * Comment db clean up temporarily. * feat: add model switch with dummy model id and test * feat: model apis implemented; need testing * Permit optional transmission of description and tags in metadata from frontend * Update model metadata design: Introduce `nickname` and Rename `name` to `class_name` * Extract 'tags' column from 'Models' table into a new table, and maintain the relationship * Enhance `UploadModel` docstring documentation * feat: improve styles of model page; add data binding * Close DB connection explicitly for .db file removal; Set fixture scope to 'function' for fresh instances per test. * Change the base directory in pytest for debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Debugging * Restore for a full test * Capture the 'stdout' and 'stderr' in pytest for debugging * Debugging * Debugging * Debugging * Solve the problem by specifying the db file address as absolute path * Restore the config of CircleCI test * fix: provide app context to all thread creations * feat: add api functions * fix: correctly getting context for child threads * Resolve code review feedback: - Improve robustness by checking for 'code' and 'pretrained' in request - Remove 'back-end/database/db_ops.py' file (double-checked for necessity) Closes: #192 * model switch and CRUD api implemented * chore: resolve conflict in requirements.txt * fixed things to pass test_predict * feat: integrate model upload API * model name attribute name fix * feat: add code and weight file when uploading model * api fixes * fix: add a serializer for db models * feat: integrate get model list api * add model api error handling and use serializer everywhere * Add `predefined` to the model's metadata * update test case * commenting out test case for now * chore: fix nit * Cherry-pick model refactors from refactor/model-db-upload-val Selected changes from: - ./backend/apis/model.py - ./backend/utils/model_utils.py * Add one TODO * Remove one TODO * fix problem in set_current_model * Add `pretrained` as a new field in 'models' table & Stop saving weights for models uploaded without an explicit weight file * feat(model): integrate existing APIs * feat(model ): add tag field * feat: more APIs implemented and manually tested * add the model list panel * add utility functions and some test cases * modify the model list panel * modify the model list panel * fix(model ): uploader and model list issues * feat(model): integrate update, duplicate and get predefined API * test: implemented basic model switch CRUD tests * temp commit * modify the style of modellist component * uncomment a test case for edit * update edit test case * add another edit test case * refactored db init logic * fix backend unit tests * use same db across test cases * change scope to function * fix failed tests * feat(model): improve styles and fix bugs of list data table * re-organize files * chore: re-organize files and made some other small changes * fix merge conflicts * Fix training without current model set to pass tests for now * restoring package-lock since no library was updated * fix(model): Set Current Model Button; Current Model Visibility * add test of current model * feat(model): add error messages * add front end model test * add front end model test * add front end model test * Refactor backend and tests to identify models by `id` instead of `nickname` (#224) * Use `id` as the model handle in training * Fix typo * Fix typo * add new test files * update test * resolve conflict * use id as primary key; all test passed * fix tag issue * enable tag selction * add predefined prefix to tag --------- Co-authored-by: lxn <[email protected]> Co-authored-by: Chonghan <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Xinnuo Li <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: monmonli <[email protected]> --------- Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: Leon-Leyang <[email protected]> Co-authored-by: lxn <[email protected]> Co-authored-by: Desmond <[email protected]> Co-authored-by: Xinnuo Li <[email protected]> Co-authored-by: Desmond <[email protected]> Co-authored-by: Jingcheng Wu（吴 璟承） <[email protected]> Co-authored-by: monmonli <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As we can now dynamically load models in the frontend, we no longer need model configs in the launcher.
The text was updated successfully, but these errors were encountered: