-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improved logging and added new data ingestion workflows (#140)
Here's a summary of the changes from the last pull request: ## Backend: - [🎉 NEW] We enhanced the logging capabilities of FarmVibes.AI, providing more control over log file management, file size restrictions and log rotation. When setting up a new cluster with `farmvibes-ai <local/remote> setup`, users can pass the `--max-log-file-bytes` argument to define the maximum number of bytes for a log file, and the `--log-backup-count` that determines the number of log files to keep per service (e.g., orchestrator, cache, data-ops) with old log files being deleted. The default values for these arguments are 100MB and 5, respectively. - [📈 IMPROVEMENT] We implemented several enhancements in our caching service increasing the speed in which cached results are recovered. - [📈 IMPROVEMENT] We added a `restart` subcommand to `farmvibes-ai remote`, allowing AKS remote clusters to be restarted from the CLI. - [⚒️ FIX] We fixed a few encoding bugs that would break the setup process on Windows machines. Related to issue #136. ## Workflows: - [🎉 NEW] We added a new `workflows/data_ingestion/weather/download_herbie` workflow for downloading numerical weather prediction from [Herbie](https://github.com/blaylockbk/Herbie). - [🎉 NEW] We added new data ingestion workflows to download Forest Extent data from [Global Land Analysis & Discovery (GLAD)](https://glad.umd.edu/). `workflows/data_ingestion/glad/glad_forest_extent_download` downloads individual tiles and `workflows/data_ingestion/glad/glad_forest_extent_download_merge` provides the merged rasters. - [🎉 NEW] We added new data ingestion workflows to download global 25m resolution SAR mosaics and forest/non-forest maps from [ALOS Forest/Non-Forest Annual Mosaic](https://planetarycomputer.microsoft.com/dataset/alos-fnf-mosaic). Similarly, `workflows/data_ingestion/alos/alos_forest_extent_download` downloads individual tiles and `workflows/data_ingestion/alos/alos_forest_extent_download_merge` provides the merged rasters. - [🎉 NEW] We added new data ingestion workflows for downloading basemaps from Bing Maps. Similarly, `workflows/data_ingestion/bing/basemap_download` downloads multiple BingMaps tiles separately, and `workflows/data_ingestion/bing/basemap_download_merge` merge the individual tiles into a single image. - [📈 IMPROVEMENT] We extended the Segment Anything Model (SAM) workflow (`workflows/farm_ai/segmentation/segment_basemap`) to process basemap inputs, allowing users to segment maps with higher resolution than the previous version that operated on Sentinel-2 images. - [⚒️ FIX] We fixed an issue with the `workflows/data_processing/heatmap/classification` workflow that would cause the output heatmap to have a different range of values than the input data. Related to #134. ## Notebooks: - [🎉 NEW] We added three notebooks that perform field-level analysis, combining field boundary detection with index computation (`notebooks/sentinel/field_level_spectral_indices.ipynb`), irrigation classification (`notebooks/irrigation/field_level_irrigation_classification.ipynb`) and optimal sensor placement (`notebooks/sensor/optimal_locations_segmentation.ipynb`). - [🎉 NEW] We added a notebook that performs SAM segmentation over BingMaps basemaps (`notebooks/segment_anything/basemap_segmentation.ipynb`). ## Documentation: - [📈 IMPROVEMENT] We improved our [Remote Cluster User Guide](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/AKS.html), adding details on the requirements expected from the user's role, subscription, and machine for setting up a remote cluster. ---- Co-authored-by: Bruno Silva <[email protected]> Co-authored-by: Eduardo Rodrigues <[email protected]> Co-authored-by: Jessica Wolk <[email protected]> Co-authored-by: Naga Bilwanth Gangarapu <[email protected]> Co-authored-by: Rafael Padilha <[email protected]> Co-authored-by: Renato Luiz de Freitas Cunha <[email protected]> Co-authored-by: Rick Gutierrez <[email protected]>
- Loading branch information
1 parent
93e7d65
commit 6ba8bd4
Showing
60 changed files
with
3,919 additions
and
584 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.