-
Notifications
You must be signed in to change notification settings - Fork 383
ReviewerQueuePolicy
When a secretary assigns a review, they are given a list of reviewers in a recommended order. Often they assign a review to the reviewer on the top of the list, but not always. This concept and the code behind it form the reviewer queue policy.
Different teams may have different policies. Each policy has a default rotation order. This is the order in which reviewers are listed in the recommended assignment order, if all other things are equal.
In practice, all other things are rarely equal. If a reviewer has received a previous version of a document, they should be high in the list. If they are the author of the document, they should be at the very end. Reviewers may be unavailable for some periods, and different reviewers have different preferences on how often they'd like to review a document. These considerations are reflected in the recommended assignment order, and these two may be different. The recommended assignment order is the order in which reviewers are shown in the dropdown when a secretary assigns a reviewer.
There are two policies that differ in their default rotation order:
- Rotate alphabetically: rotate reviewers by alphabetical last name.
- Least recently used: order reviewers with the least recent to have assigned/accepted/completed a review at the top.
The default rotation list is also included in the email drafted for current review assignment summaries.
In the rotate alphabetically policy, reviewers are rotated by last name. A pointer is kept on who the next reviewer should be in the NextReviewerInTeam object. For example, if the current state in the database is:
A, B, C, D, E
^
The default rotation order will be: C, D, E, A, B. If a review is assigned to C, the pointer will move to D.
A reviewer may also be unavailable. If D would be unavailable, they are entirely skipped over in the pointer change when C is assigned, and it is instead assigned to E.
In the least recently used policy, the default rotation order places the reviewers who least recently assigned/accepted/completed a review at the top. The reviewer who did that most recently is placed at the bottom. This is based on the date of assigning the review - not the completion date.
Reviewers that have never assigned/accepted/completed a review are placed at the top of the list, ordered by lowest primary key first. There will likely be one or none in most cases, and so the main priority is that the ordering is consistent.
The recommended assignment order reflects the order in which reviewers are shown in the dropdown of reviewers, when a secretary is assigning a review.
The default rotation order is one of the items used in determining the recommended assignment order for a specific review request, but there are other considerations. Each consideration applies a negative or positive score to a reviewer, and in the recommendation, the reviewer with the top score is listed first. In addition, each applied consideration may produce explanations which are visible to the secretary, to help them in making their choice.
The considerations and their scoring are as follows:
| Consideration for reviewer | Score |
|---|---|
| Has rejected review of document or ancestors before | -1 if rejected before, +1 otherwise |
| Has selected "Select me next for an assignment" | +1 if selected, -1 otherwise |
| Has completed review of document or ancestors before | +1 if reviewed before, -1 otherwise |
| Has review wish | +1 if wish exists, -1 otherwise |
| Has connection with document | -1 if connected, +1 otherwise |
| Filter regex matches | -1 if match, +1 if no match |
| Needs more days between reviews (per the max interval set by the reviewer) | minus the number of days still needed |
| Set "skip next X" in ReviewerSettings | minus the value of skip next X reviews |
| Index in the default rotation order | minus the index |
"Has connection with document" can mean that the reviewer:
- is the associated Area Director
- has a role in the group the document belongs too
- is shepherd of the document
- is an author of the document
In addition, information is included regarding the reviewer, if present, about:
- the number of open reviews and their total page count
- the number of completed, partially completed and no-response reviews, in the last year
The scores are not added up: they are considered in the order listed. So if a reviewer has a skip_next of 10, but has reviewed the document before, they are still above someone who has not reviewed the document but has a skip_next of zero.
The score table and algorithm therefore mean that having reviewed before and having a review wish have a very high impact on ordering, whereas the index in the default rotation order has an impact only if all other things are equal.
If a reviewer is unavailable, they are not included in the default rotation order at all. They are never part of the list, and therefore never included in the recommended assignment order. They can not be assigned to a review.
Note that a reviewer may be in an unavailability period with an availability of "canfinish". Those reviewers would be considered for assignments to documents they have previously reviewed.
Each reviewer has a skip count, which means "skip me for the next X reviews". Reviewer team secretaries can set this manually, but it may also be incremented when a secretary assigns a review, by ticking "Skip next time".
The skip count is decreased every time a reviewer is skipped over. A reviewer is never considered skipped over if they are in an unavailability period, i.e. their skip count is not decreased.
A reviewer is considered skipped over, when they were earlier in the default rotation order than the assigned reviewer. Additionally, skip counts are only decreased if the assignment was in the default rotation order, after accounting for skips.
An example:
- the current default rotation order is
C, D, E, A, B - reviewers C and D have a skip count
- the recommended assignment order is
E, A, B, C, D - reviewer E is assigned
- the assignment is considered in default rotation order, because E was the first skip_next=0 in the default rotation order
- the skip count for C and D is decreased by one
Another, similar, example:
- the current default rotation order is
C, D, E, A, B - reviewer C has a skip count
- the recommended assignment order is
E, A, B, C, D - reviewer E is assigned
- the assignment is considered out of default rotation order, because D was the first skip_next=0 in the default rotation order (even though it may be at the end of the recommended assignment order, e.g. because D is the document author)
- no skip counts are decreased, due to the assignment being out of default rotation order
Out of order assignments are assignments that do not follow the default rotation order. This might be because the person on top of the default rotation order has a lower score because they are the document author, or because the secretary chooses a different reviewer.
In the least recently used policy, out of order assignments do not require special considerations, because the default rotation order is always based on the current state of review assignments.
For the rotate alphabetically policy, this is a bit more complex: the NextReviewerInTeam will only advance, if a review is assigned to the reviewer with the current NextReviewerInTeam pointer.
An example:
- the current default rotation order is
C, D, E, A, B, i.e. the pointer is set to C - the recommended assignment order is
E, A, B, C, D - reviewer C is assigned
- this assignment is considered within the default rotation order
- the NextReviewerInTeam pointer moves to D
Another example:
- the current default rotation order is
C, D, E, A, B, i.e. the pointer is set to C - the recommended assignment order is
E, A, B, C, D - reviewer E is assigned
- this assignment is considered out of the default rotation order - even it was according to the recommended assignment order
- the NextReviewerInTeam pointer remains at C