Combined pre-AGU GUI and Calibration Message Work #240

Fixing logic where pip installs the updated packages (to avoid the entire image rebuild), so make sure deps are ignored (as this was the slow part).

Adding type GET_SERIALIZED_FORM to get the entire serialized state of a dataset.

Refactor to ensure it takes advantage of connection handling via its async context manager logic.

Add new function to get serialized dataset details to DatasetExternalClient.

Fix issue with not acquiring a session, and wrapping some things in try block to catch and log exceptions.

Fixing bug in response to LIST_FILES query, where reason text was not being assembled entirely correctly.

Adding optional offset and length params to abstract interface definition of get_data, and adding get_file_stat abstract method.

Adding support for querying about dataset items.

Updating to support indicating start and size of data for partial transfers.

Optimizing the reloading of datasets on object store manager startup.

Accounting for changes to get_data parameters and implementing get_file_stat.

Updating service to respond to GET_DATASET_ITEMS queries and requests for data with an offset start (i.e. partials).

Adding initial second view display and manage existing datasets, along with navigation functionality to toggle between "manage" and "create" views.

Sending serialized datasets to HTML template as list/array rather than dict/map.

Implementing layout and initial details-viewing behavior for dataset management GUI view.

Adding commented-out line for host bind-mount volume to GUI Docker service config to make debugging Django template and Javascript code faster by mounted code from the host into the running service.

Make DatasetManagementView inherit from AbstractDatasetView so certain required pieces could be centrally located and reused.

Adding several classes used by Dataset management GUI view.

Cleanup, a bit of additional behavior, and attempts at getting downloads working.

Adding incomplete DatasetFileWebsocketFilelike class, since it is referenced by other things.

Adding abstract function to Hydrofabric interface, and implementing in GeoJsonHydrofabric subtype; also adjusting MappedGraphHydrofabric to be abstract since it cannot implement generate_domain currently (as there is no applicable DataFormat value to use for it).

Adding new GeoJsonBackedSubsetHandler extended from SubsetHandler to facilitate getting subset-specific GeoJSON data.

Updating versions of dmod.core and dmod.modeldata.

Adjust service to use GeoJsonBackedSubsetHandler handlers specifically and to potentially support multiple hydrofabrics when not in CLI mode.

Generalizing existing behavior and adding specialized function for worker volumes.

Adding function to manage the availability of hydrofabric datasets to entities that may need them (e.g., making sure special volumes are created for access to Docker containers).

Adding optional support for overwriting existing files when downloading.

Using common settings for SSL dir and MaaS endpoint URI.

Updating DockerS3FSPluginHelper with method to check whether volumes for datasets already exist.

Updating DockerS3FSPluginHelper with method to remove an existing Docker service of the utilized name if it exists already.

Updating DockerS3FSPluginHelper to refactor and improve how sanity checking is done for creating new S3FS plugin volumes for object store datasets, in particular moving logic to its own method for cleaner reading.

Updating and attempting to fix/improve logic in DockerS3FSPluginHelper for managing the volume-creating Docker service, making sure it has started properly, and cleaning it up after it is done.

Updating logic in data-service service manager to have the Docker S3FS helper member remove any existing service of the expected name before starting a new volume-create service.

Fixing issue where somehow (despite this working before) the scripts directory was not recursively being copied (i.e., just the top-level).

Accounting for the change in name for the hydrofabric root directory in MapView.py and crosswalk.py modules.

Commenting out start of work to load hydrofabrics based on whether the map had been zoomed into to a sufficent level, which was to be combined with loading just a subset of the desired hydrofabric (and may still be in the future).

Fixing issue with map.js where the schema of newer versions of hydrofabrics was not allowing the id of catchment (and other) features to be read and used correctly.

Updating to start logic for parsing what comes from form to the beginnings of a realization config, that can (later) be used to create a dataset for the job being requested.

Fixing bugs related to init of client member objects deriving from WebSocketClient.

Fixing issue from some upstream changes where migrations were necessary for dataset management GUI to function (even though they weren't used there) but were not run automatically.

Adding env vars that are used to populate static Django settings, along with Docker secrets for minio credentials.

Having GUI's DatasetApiView download and cache dataset files directly from MinIO and then bundle in downloadable zip file to support downloading entire datasets.

Updating to support downloading of entire zipped datasets.

Updating to fix function for individual dataset file downloads.

Fixing issue where a previously existing cache subfolder for a dataset would not be cleaned up correctly when cleanup was needed and the function received an explicit collection of files to cache (note that this is not a normal state that will occur in the cache directory given the current code, but it could/did occur due to older versions of the code, and code changes could make this possible again in the future).

Partially fixes NOAA-OWP#198. DOM elements were being removed using `forEach` this mutates the underling array and causes undesired side-effects.

first rough cut of forms. input types and validation still needs to be verified and shorted out.

… name

…e python imports happy

…, should be changed once django version has been updated. NOAA-OWP#209 is tracking this

…d label's interact

…mit when display is set to none

Moving factory class method within DatasetApiView for creating a MinIO client object up a level to abstract superclass, to facilitate reuse.

Reordering methods and removing unused imports.

Fixing template form for creating new datasets, both to make sure form has the right "action" destination and to ensure form handles multiple uploaded files.

Updating Django form components for dataset creation form to support uploading multiple files and to utilize centralized setting for datetime format string.

Ensure GUI app_server image installs latest dmod-core package, used directly as part of dataset creation.

Allowing underscores in dataset file names, which were incorrectly being forbidden by validation (only dataset names/buckets cannot have underscores).

Adding the now-required dependency to list of installed apk packages in foundation stage of py-deps image.

Enhancing factory_init_from_deserialized_json function by adding and using new _additional_deserialized_args function (empty for this type) to allow separate mechanism for subclasses to use to prepare their own specific init params and still use base deserialization logic of NGENRequest.factory_init_from_deserialized_json().

Adding new NgenCalibrationRequest and NgenCalibrationResponse classes.

Fixing a bug in the ModelExecRequestHandler class's service_client property method, where the order of the positional arguments was incorrect.

Adding a function to ModelExecRequestHandler for any preprocessing that is needed before sending things to the scheduler (with it being an empty implementation for now), to allow subclasses a way to extend behavior.

Updating requestservice service class to account for changes in the names of init params for utilized request handler classes.

this also includes some commented out code that *might* need to be added in the future to support data_requirement dissemination

…bjectLockConfiguration error

…en calibration jobs

Fixing bug in deserialization function where processed (i.e., by this type) optional kwargs were being combined into list of additional (i.e., potentially by subtypes) additional kwargs to be passed to __init__, with both lists being passed instead of just the combined.

Fixing setup for TestSimpleHydrofabricSubset that used the MappedGraphHydrofabric class after it was made abstract, replacing its use with its GeoJsonHydrofabric subtype.

Fixing argument handling for those args related to communicating with evaluation service, including the dest name for args, defaults, and use when creating main service handler class.

Fixing imports, class usage, and code style spacing for several things related to evaluation requests within main request-service handler class.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Combined pre-AGU GUI and Calibration Message Work #240

Combined pre-AGU GUI and Calibration Message Work #240

Commits on Jan 18, 2023

Commits on Jan 20, 2023

Combined pre-AGU GUI and Calibration Message Work #240

Are you sure you want to change the base?

Combined pre-AGU GUI and Calibration Message Work #240

Commits on Jan 18, 2023

Commits on Jan 20, 2023