Merge pull request #6 from getwilds/dev

PROOF v1.1 Updates

NEWS.md

@@ -1,5 +1,27 @@
+### September 2024 - PROOF v1.1
+**What's New**
+
+- **Visualization Tabs**: To aid users in visualizing their analyses, we've added two new tabs to the PROOF user interface:
+    - **WDL**: produces [mermaid diagrams](https://github.com/chanzuckerberg/miniwdl-viz?tab=readme-ov-file#simple) for a submitted WDL script.
+    - **Viewer**: displays [input json's](https://github.com/timelyportfolio/listviewer) in a much more intuitive and collapsible way.
+- **Login Persistence**: Many users have reported issues with accidentally navigating away from PROOF and having to log back in with their Fred Hutch credentials. Through a limited utilization of cookies, authentication in PROOF will now persist much longer, regardless of any navigation away from the page.
+- **PROOF Analytics Dashboard**: To better assess how the Fred Hutch community is using the PROOF platform, we have created the initial framework for a Shiny-based analytics dashboard centered around PROOF.
+
+**Fixes**
+
+- **Cromwell Scratch Directory**: In anticipation of the [retirement](https://sciwiki.fredhutch.org/scicomputing/store_scratch/) of the `/fh/scratch/` directories of the Fred Hutch Rhino cluster, the default value for a user's Cromwell scratch directory will now be set to `/hpc/temp/` whenever possible.
+- **Zombie Tasks**: On rare occassions, tasks are cancelled by the Slurm job scheduler and Cromwell has [trouble](https://github.com/broadinstitute/cromwell/issues/1499) recognizing these cancellations. This results in the task being marked as "Running" indefinitely in the "Track Jobs" tab. We have added functionality in PROOF to identify these "zombie" tasks and correctly update their status to "Cancelled".
+- **Concurrent Docker Pulls**: Measures were taken to ensure that tasks running in parallel do not interfere with each other when pulling down the same Docker image.
+- **Other Minor Improvements**:
+    - Wording clarifications in documentation
+    - Linting corrections identified by the linting GitHub Action
+    - Version upgrades for underlying dependencies
+    - Additional testing for improved CI/CD
+    - Easier local instance creation for development purposes
+
 ### June 2024 - PROOF v1.0
 **What's New**
+
 - Users can now interact with data stored in S3 buckets within WDL workflows executed in PROOF! Just make sure to have your AWS CLI credentials [established in Rhino](https://sciwiki.fredhutch.org/scicomputing/access_credentials/#configure-aws-cli) and you can use the S3 path just like any other input.
 - GPU analysis is now possible via the `gpu` argument in your WDL task's runtime section! For an example, try running [this example script](https://github.com/getwilds/ww-test-workflows/blob/main/gpuMatrixMult/gpuMatrixMult.wdl) in the [ww-test-workflows](https://github.com/getwilds/ww-test-workflows) repo of the [DaSL WILDS](https://github.com/getwilds).
 - Additional features have been introduced to increase transparency in terms of the functionality of PROOF:
@@ -8,6 +30,7 @@
     - Documentation links have been added to the "Welcome" tab and automated server creation email for increased visibility.
 
 **Fixes**
+
 - The default location of the Apptainer cache directory has been moved from `scratch` to `/hpc/temp` to avoid previously reported linkage issues associated with `scratch`.
 - PROOF's underlying WSGI server has been switched from Waitress to Gunicorn for better scalability and efficiency.
 - Session persistence or "stickiness" has been added to the Shiny app to ensure users only speak to one instance at a time.
@@ -16,6 +39,7 @@
 
 ### February 2024 - PROOF v0.1
 **What's New!**
+
 - PROOF has now grouped together all the components for setting up and configuring your database and Cromwell server so you don’t have to manually configure each part!
 - You can interact with PROOF in two improved ways when running your workflows:
     - [PROOF Shiny App](https://cromwellapp.fredhutch.org/) - an easy to use interface for accessing and configuring workflows to run on our cluster
@@ -25,4 +49,5 @@
     - Each user can only run one Cromwell server at a time so that you don’t lose track of your workflow sessions
 
 **Fixes**
+
 - No fixes just yet, we’re launching our product

README.md

@@ -1,36 +1,40 @@
 # PROOF (PRoduction On-ramp for Optimization and Feasibility)
 PROOF (PRoduction On-ramp for Optimization and Feasibility) is a user-friendly tool designed for managing and executing WDL (Workflow Description Language) workflows using the Cromwell workflow manager. PROOF allows users to:
+
 - Automate all the backend configurations necessary to run your workflows instantly.
 - Validate, troubleshoot, assess performance, and run their workflows all under one roof.
 - Refine their workflows before potential transitions to cloud-based infrastructures, providing a ‘proofing’ resource of sorts.
+
 While it is currently configured to run on the Fred Hutch cluster, it is readily adaptable for local instances and/or instances hosted at other institutions. If interested, please email [email protected] with any questions.
 
-## Repos:
+## Repos
 
 - [proof-api](https://github.com/FredHutch/proof-api): REST API for PROOF used to automate many/most of the startup/configuration steps of WDL workflow submission via Cromwell.
-    - Latest release: [v1.0.0](https://github.com/FredHutch/proof-api/releases/tag/v1.0.0)
+    - Latest release: [v1.1.0](https://github.com/FredHutch/proof-api/releases/tag/v1.1.0)
 - [shiny-cromwell](https://github.com/FredHutch/shiny-cromwell): Shiny-based front end for PROOF that provides a point-and-click interface for new users.
-    - Latest release: [v1.0.0](https://github.com/FredHutch/shiny-cromwell/releases/tag/v1.0.0)
+    - Latest release: [v1.1.0](https://github.com/FredHutch/shiny-cromwell/releases/tag/v1.1.0)
 - [rcromwell](https://github.com/getwilds/rcromwell): R package for interacting with Cromwell servers and the WDL workflows they manage.
-    - Latest release: [v3.2.1](https://github.com/getwilds/rcromwell/releases/tag/v3.2.1)
+    - Latest release: [v3.3.0](https://github.com/getwilds/rcromwell/releases/tag/v3.3.0)
 - [proofr](https://github.com/getwilds/proofr): R package for interacting with the PROOF API mentioned above.
-    - Latest release: [v0.2](https://github.com/getwilds/proofr/releases/tag/v0.2)
+    - Latest release: [v0.3.0](https://github.com/getwilds/proofr/releases/tag/v0.3.0)
+- [proofad](https://github.com/getwilds/proofad): collection of scripts used for the user analytics dashboard of PROOF.
+    - Latest release: [v0.1.0](https://github.com/getwilds/proofad/releases/tag/v0.1.0)
 
-## Docs:
+## Docs
 
 - Product Description: https://sciwiki.fredhutch.org/datascience/proof/
 - How to Use PROOF: https://sciwiki.fredhutch.org/dasldemos/proof-how-to/
 - PROOF Troubleshooting: https://sciwiki.fredhutch.org/dasldemos/proof-troubleshooting/
 
-## Support:
+## Support
 
 For general questions, reach out to the Fred Hutch Data Science Lab (DaSL) at [email protected]. For bugs and/or feature requests, if you know which repository is the most relevant for your bug/feature, please open an issue directly in the corresponding repository. If you are unclear which repository it belongs in, or if the bug/feature spans across multiple repositories, [open an issue](https://github.com/getwilds/proof/issues) in this overarching PROOF repository and the PROOF development team will file issues in the corresponding locations.
 
-## Contributing:
+## Contributing
 
 If you would like to contribute to this project, see the [contribution guidelines](CONTRIBUTING.md) as well out our [WILDS Contributor Guide](https://getwilds.org/guide/) for more details.
 
-## License:
+## License
 
 Distributed under the MIT License. See [LICENSE](LICENSE) for details.