SJIP 14: Clarify High and Medium severities

[WangSecurity]

Description

Rephrase high and medium severities to limit the subjectivity

Judging Guidelines PR

https://github.com/sherlock-protocol/sherlock-v2-docs/pull/38/files

Rationale

The first change is for Medium severity to give an understanding of what means "exceeds small and finite amounts". It causes subjectivity since in different contexts it's hard to decide if the loss is small. For example, the protocol takes a fee of 0.00006 ETH, but the attacker can bypass it. On the one hand, the loss is only 0.00006 ETH, which is relatively small. On the other hand, if the attacker bypasses the fee, the protocol loses 100%, which is relatively high.

It leads to cases when Watsons and Judges have different views on the loss, resulting in unnecessary long discussions during the escalations and one of the sides losing (either escalation ratio or escalation penalty for Judges).

I propose to change it the following way:

Causes a loss of funds but requires certain external conditions or specific states, or a loss is highly constrained. The loss of the affected party must exceed 0.01%.

Additionally, I propose to remove the part "and any amount relevant based on the precision or significance of the loss" since now we have a concrete number and this line only complicates the rule.

Hence, if the loss is <0.01%, then it can never be Medium or High severity.

The second change is for High severity. The main issue here is how to consider if the loss is highly constrained (Medium) or not. The result is the same as in the first case: unnecessary long discussions during the escalations and one of the sides losing (either escalation ratio or escalation penalty for Judges).

I propose the following change:

Definite loss of funds without (extensive) limitations of external conditions. The loss of the affected party must exceed 1%.

Hence, any loss less than 1% will never be a High severity impact, while the loss >=1% can be either High or Medium based on the constraints.

Additionally, I propose to remove the following line because it only creates more confusion and sounds like "serious insignificant losses":

Inflicts serious non-material losses (doesn't include contract simply not working).

I believe it solves the problem and will allow Watsons to objectively prove high severity and allow Judges to objectively decide if the loss is highly constrained or not.

Relevant Issue Discussions

sherlock-audit/2024-03-zivoe-judging#304

[panprog]

I have some concerns with judging the severity based only on percentage of funds lost. There are some cases when the amount lost is fixed (and small) regardless of the affected party balance / action size. So it's somewhat hard to judge the percentage loss. Some extreme example: there is a loss of 1 wei on most deposits. Watson can argue that on a deposit of 10 wei this will be a 10% loss, thus valid high (or medium). Same example not as extreme: the loss is fixed 0.001 USDC for some deposits. If user deposits 10 USDC (which is reasonable), this is a 0.01% loss. Can this be considered medium? What if the loss is 0.0001 USDC? Can a deposit of 1 USDC be considered reasonable with 0.01% loss? Where to draw the line?

I believe the language should include both percentage and fixed amounts. For example:

2. Breaks core contract functionality, rendering the contract useless or leading to loss of funds of the affected party larger than 0.01% but not less than 0.01 USD value.

I think this should clear all questions regarding the dust losses.

[WangSecurity]

It's unclear whether $9 counts as a percentage of what the attacker spends, or if it's the percentage of the total fee that is lost by the protocol

I meant that for example the protocol is about NFTs and takes a fee on each mint. For instance, the fee is always $9, but the attacker can make the protocol lose this fee. On the one hand, the loss of $9 doesn't qualify for medium. On the other hand, it's the 100% loss of that fee for the protocol, which should be high (assuming little to no constraints). Does it clear my question?

[IllIllI000]

Is there a specific reason a high doesn't have a dollar amount minimum? If you're intending to cover tiny protocol fees when there are no constraints, then shouldn't the medium rule also allow for that?

[WangSecurity]

Nope, there's no specific reason. As you saw we didn't have a dollar amount for Medium initially and was only introduced as a part of this thread.

If you're intending to cover tiny protocol fees when there are no constraints, then shouldn't the medium rule also allow for that?

It's a fair point, but if we don't have the dollar amount for Medium, then it allows for the case that @panprog expressed in the very start of this thread.

I think the optimal way is to cover both of these cases, i.e. prevent the situation that @panprog expressed, but also account for the loss of the protocol/user fees if they're fixed and less than 10 USD.

Firstly, I'm not sure if we need to cover that scenario with fees. Hence, I asked you two as LSWs so you could provide feedback.
Secondly, if you agree we need to cover this case, I think the following can be added:

The loss of the affected party must exceed 0.01% and 10 USD or the amount relevant based on the significance of the loss. Example: if the protocol has fixed fee of 9 USD and the attack causes 100% loss of this fee, it's considered valid.

Does it add more friction and ambiguity to the rules and it would be better with the current state?
Thirdly, I don't think we need to add a dollar amount for high, because from my point of view this least dollar amount for Medium is only to prevent reports with tiny losses (several wei) from being valid and playing the rules. Hence, if the loss satisfy this criteria, then it can be high depending on the % of the loss. Does it make sense?

[WangSecurity]

But, another thought that I have in mind is to add the following under medium severity definition:

Note: if the loss is fixed and below $10, but is 100% loss of the affected party, then it's considered valid. Example: if the protocol has fixed fee of 9 USD and the attack causes 100% loss of this fee.

I'm moving the SJIPs to the last call now, please let me know if this should be clarified and which was you think is the best (this comment or the one above). I think this comment is better, so if no feedback, I'll add the "Note" line.

[WangSecurity]

Updated

[IllIllI000]

I think Inflicts serious non-material losses (doesn't include contract simply not working). is meant to cover cases where the protocol doesn't work with user funds. For example this finding was for an on-chain twitter replacement, and in that case there was a way to poison light clients. Are we saying that those sorts of protocols can never have a med/high?

[WangSecurity]

Yeah, we discussed such types of protocols before implementing this change, but believe this is more of a broken core contract functionality and should be med at max (so it can be medium).

[IllIllI000]

breaking implies the feature is non-functional, whereas the above case is the contract still functioning. Perhaps update the rendering the contract useless somehow

[WangSecurity]

I also want to change that part, but will leave it to the next SJIP, this one is to make the losses more objective.

[IllIllI000]

why not undo the non-material losses change until then?

[WangSecurity]

It wasn't initially in the SJIP, to be honest. But it was something we discussed in Sherlock internally a couple of days before the SJIP and since we came up with the conclusion to remove this line, I decided it's better to add this line here, while we don't have a conclusion about how to change the rendering contract useless part of Medium.

[IllIllI000]

Are there any limitations on how long it may take to reach the $10 loss? If it takes one year for minuscule amounts to add up to $10, but would require not much other activity to happen on the project, is that still valid?

[WangSecurity]

oh, so you mean that this line is not clarified enough, correct?

Note: If a single attack can cause a 0.01% loss but can be replayed indefinitely (assuming little to no costs), it will considered a 100% loss and a high-severity issue (assuming little to no constraints).

And you think it needs clarification that this issue can be either low, or med, or high, depending on the constraints?

[IllIllI000]

correct

[WangSecurity]

Updated to the following:

Note: If a single attack can cause a 0.01% loss but can be replayed indefinitely, it will be considered a 100% loss and can be invalid, medium or high, depending on the constraints.

Do you think it's enough or it's still a bit vague for you?

[IllIllI000]

I'd remove the invalid, part, since I can't see how constraints would cause a 100% loss to be invalid, and that opens the door to the judge invalidating based on likelihood, which currently isn't allowed.

[WangSecurity]

good point, removed invalid

[IllIllI000]

If a single attack can cause a 0.01% loss but can be replayed indefinitely (assuming little to no costs), it will considered a 100% loss. What is the implication of a 100% loss? Does that mean it becomes a high? If not, what is it meant to affect?

[WangSecurity]

Exactly, it means that if the single iteration of the attack is 0.01% loss, which is only Med, but you can repeat it and cause the loss over 1%, then it can be high severity. Does it need to be additionally clarified?

[IllIllI000]

I think so, since it may be Med due to lots of constraints

[WangSecurity]

Updated.