Ideas about Workstream 2: Mapping of HTA Analytics Related R Packages

[hoppanda]

Please share your idea about the Workstream 2 below. particularly on tangible actions

The proposal is as below

Objectives
To develop a comprehensive inventory of R packages related to HTA analytics with useful information and application examples, facilitating their use and integration into daily practice

Proposed Activities

• Identify and categorize relevant HTA analytics R packages available in the open-source domain, with useful annotation
• Establish and apply criteria to evaluate and monitor the production readiness of identified HTA R packages
• Work with other initiatives (e.g. openstatsware, R for HTA consortium) to fill gaps of high-quality tools in key areas of HTA analytics
• Fill in gaps of document and application examples for highly production-ready HTA R packages
• Develop an online directory to host these work and facilitate ongoing engagement with the HTA R community

Proposed Outcomes
A publicly accessible and user-friendly directory offering a categorized inventory of HTA analytic-related R packages with annotation, practical examples, assessment report of production readiness evaluated based on transparent criteria

[hoppanda]

• Where to crawl HTA-related open-source R packages?

  • CRAN
  • R Universe
  • Github Organizations nominated by this WG via this discussion

• What could be a good initial bag of labels to search for HTA related?

  • These labels can be used as key words to search for relevant R packages from Open Source R Repository,
    and later annotation of the packages

• What aspects of an R package are important to quantify risk and readiness for production usage?

1. Software Engineering (SWE) Aspect:

• Number of Dependencies: Understand the package’s dependency tree. Fewer dependencies generally mean reduced risk of conflicts and easier maintenance, but essential dependencies should be stable and well-maintained.
• Testing and Coverage: Evaluate the extent of unit tests, integration tests, and overall test coverage. High test coverage ensures that the package functions as expected and helps prevent regressions.
• Continuous Integration (CI): Check if the package uses CI tools to automate testing and deployment. This ensures consistent quality and reliability across different environments.
• Completeness of Technical Documentation: Ensure that the package documentation is thorough, covering installation instructions, API reference, function usage, parameters, and return values. Well-documented packages are easier to adopt and troubleshoot.
• Code Quality: Review the source code for adherence to coding standards, readability, and maintainability. High-quality code is essential for long-term maintenance and reliability.
• Security Practices: Assess whether the package follows best practices for secure coding, such as avoiding the use of deprecated functions and libraries, and handling data securely.
• Vulnerability Scanning: Check for known security vulnerabilities in the package and its dependencies. Tools like rOpenSci can help with automated scans.
• Data Privacy Compliance: Ensure that the package complies with data privacy regulations like GDPR if it handles personal data.
• Error Handling: Check for robust error handling and meaningful error messages that make troubleshooting easier for users.

2. Scientific Aspect:

• Correctness of Output: Verify that the package produces accurate and scientifically valid results. This could involve reviewing validations, comparisons with known benchmarks, or peer-reviewed publications.
• Computational Performance: Assess the package’s performance in handling large datasets and complex calculations. Efficient algorithms and optimization are crucial for high-performance computing tasks.
• Clarity of Output: Ensure that the package outputs are clear, well-formatted, and easy to interpret. This includes both visual outputs (e.g., plots) and numerical results.
• Reproducibility: Check if results generated by the package can be consistently reproduced under the same conditions. Reproducibility is fundamental for scientific rigor and credibility.

3. Adoption Aspect:

• Tutorials and User Guides: Look for comprehensive tutorials, user guides, and vignettes that help users understand how to effectively utilize the package. These resources are essential for onboarding new users and facilitating widespread adoption.
• User Community and Support: Evaluate the size and activity of the user community. Active user communities can provide valuable support, share use cases, and contribute to the package’s development and improvement.
• History of Support: Examine how responsive the maintainers are to issues and user queries. Timely and effective support is crucial for resolving problems and maintaining user trust.
• Licensing: Ensure that the package's license is compatible with your organization’s policies and intended use. Open-source licenses can vary significantly in terms of permissions and restrictions.
• Frequency of Updates and Backwards Compatibility: Check the release frequency and update history. Regular updates indicate active maintenance. Additionally, assess whether new versions maintain backwards compatibility to avoid disrupting existing workflows.
• Compatibility with Upcoming Regulations: Consider the package's compliance with relevant regulations, such as the upcoming EU Cyber Resilience Act. Ensure that the package adheres to security and resilience standards specified by such regulations.

[RobertASmith]

Love this.

Agree with the concept and approach. 1. Software Engineering is an easy win so would suggest to get a database of packages used in prior submissions/early models and start with this before building in 2 and 3.

I am working on something related in assertHE which is to identify which functions in a model depend on external packages and what those packages are. This is surprisingly tricky.