Skip to content

Commit

Permalink
test: Sort tested dimensions dict by keys on A/B test failures
Browse files Browse the repository at this point in the history
When A/B test fails, it prints the tested dimensions, but it was not
sorted and was not easy for human to read.

Signed-off-by: Takahiro Itazuri <[email protected]>
  • Loading branch information
zulinx86 committed Feb 12, 2024
1 parent 41091e0 commit 5a39100
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tools/ab_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ def ab_performance_test(
f"for metric \033[1m{metric}\033[0m with \033[0;31m\033[1mp={result.pvalue}\033[0m. "
f"This means that observing a change of this magnitude or worse, assuming that performance "
f"characteristics did not change across the tested commits, has a probability of {result.pvalue:.2%}. "
f"Tested Dimensions:\n{json.dumps(dict(dimension_set), indent=2)}"
f"Tested Dimensions:\n{json.dumps(dict(dimension_set), indent=2, sort_keys=True)}"
)
messages.append(msg)

Expand Down

0 comments on commit 5a39100

Please sign in to comment.