Improving FATES development throughput

[glemieux]

@rgknox and I have had some discussions recently about ways in which to improve how quickly we can move software updates, both large and small, through the FATES development pipeline. I'm starting up this discussion thread so that we can solicit more ideas and discuss the trade offs with different options.

Please contribute any ideas for any stage of development process: planning, design, coding, review, testing, validation, maintenance, etc. It may be helpful to frame the idea using a template highlighting a specific issue, proposed solution(s), and pro/cons, similar to the fates design document template categories. Please feel free to also suggest references and other best practice resources.

[glemieux]

Challenge

Soliciting and conducting pull request reviews

Issues contributing to the challenge:

• Domain challenge: do we have people with the correct expertise reviewing the update? For example, I as a software developer can review the implementation details, but may not have the ability to validate the scientific merit.
• Coordination challenge: Candidates for reviewing are distributed across multiple institutions and time zones and are not intimately aware of all pipeline stages
• Prioritization challenge: candidates for reviewing do not necessarily have the task as a priority related to their day-to-day work or broader job performance metrics

Potential improvements summary

1. Conduct "staged" reviews during the development process
2. Document design and solicit design review prior to coding
3. Conducting more regular review walk-throughs
4. Public impact metrics based on review contributions (i.e. "h-index" for reviews)

Details and considerations

1. Staging more reviews
   For large reviews, it may be easier to move a PR through the pipeline if we draft PRs early and solicit reviews, particularly scientific reviews, prior to full functionality.

2. Design documentation reviews
   @rgknox noted that having a design document for large updates may help reviewers more quickly orient to the code during review. I suggest that we could even review the design document itself and perhaps more quickly sign off on the actual code review if we can show that the code meets the design requirements.

3. Regular review walk-throughs
   We already do this to some extent to help people get up to speed on new PRs. My suggestion here is that we make this more routine so we get in the practice of conducting reviews which would help us iterate on the process and improve our skills. This would require more explicit coordination however.

4. Review metric
   @rgknox suggested a quantitative metric similar to the "h-index" for publications in which reviewers are given a score based on their review contributions. He noted that this might be susceptible to gaming, but it could more concretely elevate the importance of this task in performance reviews resulting greater prioritization.