-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ignore test_diff_management
#1074
ignore test_diff_management
#1074
Conversation
|
Report | Wed, September 4, 2024 at 13:37:28 UTC |
Project | Stratum V2 (SRI) |
Branch | ignore-test_diff_management |
Testbed | sv2 |
Click to view all benchmark results
Benchmark | Latency | Latency Results nanoseconds (ns) | (Δ%) | Latency Lower Boundary nanoseconds (ns) | (%) | Latency Upper Boundary nanoseconds (ns) | (%) |
---|---|---|---|---|
client_sv2_handle_message_common | ✅ (view plot) | 44.76 (+0.32%) | 43.63 (97.47%) | 45.61 (98.14%) |
client_sv2_handle_message_mining | ✅ (view plot) | 75.86 (+2.33%) | 65.05 (85.75%) | 83.21 (91.17%) |
client_sv2_mining_message_submit_standard | ✅ (view plot) | 14.66 (+0.02%) | 14.60 (99.56%) | 14.72 (99.60%) |
client_sv2_mining_message_submit_standard_serialize | ✅ (view plot) | 268.86 (+0.88%) | 248.88 (92.57%) | 284.16 (94.62%) |
client_sv2_mining_message_submit_standard_serialize_deserialize | ✅ (view plot) | 604.43 (+1.70%) | 561.78 (92.94%) | 626.84 (96.43%) |
client_sv2_open_channel | ✅ (view plot) | 166.10 (+0.05%) | 160.05 (96.36%) | 172.00 (96.57%) |
client_sv2_open_channel_serialize | ✅ (view plot) | 268.05 (-4.55%) | 266.62 (99.47%) | 295.04 (90.85%) |
client_sv2_open_channel_serialize_deserialize | ✅ (view plot) | 372.64 (-1.26%) | 331.71 (89.02%) | 423.09 (88.07%) |
client_sv2_setup_connection | ✅ (view plot) | 155.47 (-4.62%) | 151.15 (97.22%) | 174.87 (88.91%) |
client_sv2_setup_connection_serialize | ✅ (view plot) | 418.63 (-10.58%) | 416.94 (99.60%) | 519.35 (80.61%) |
client_sv2_setup_connection_serialize_deserialize | ✅ (view plot) | 978.26 (-0.53%) | 896.65 (91.66%) | 1,070.32 (91.40%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
Report | Wed, September 4, 2024 at 13:37:32 UTC |
Project | Stratum V2 (SRI) |
Branch | ignore-test_diff_management |
Testbed | sv1 |
Click to view all benchmark results
Benchmark | Estimated Cycles | Estimated Cycles Results estimated cycles | (Δ%) | Estimated Cycles Lower Boundary estimated cycles | (%) | Estimated Cycles Upper Boundary estimated cycles | (%) | Instructions | Instructions Results instructions | (Δ%) | Instructions Lower Boundary instructions | (%) | Instructions Upper Boundary instructions | (%) | L1 Accesses | L1 Accesses Results accesses | (Δ%) | L1 Accesses Lower Boundary accesses | (%) | L1 Accesses Upper Boundary accesses | (%) | L2 Accesses | L2 Accesses Results accesses | (Δ%) | L2 Accesses Lower Boundary accesses | (%) | L2 Accesses Upper Boundary accesses | (%) | RAM Accesses | RAM Accesses Results accesses | (Δ%) | RAM Accesses Lower Boundary accesses | (%) | RAM Accesses Upper Boundary accesses | (%) |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
get_authorize | ✅ (view plot) | 8,428.00 (-0.17%) | 8,197.96 (97.27%) | 8,686.41 (97.03%) | ✅ (view plot) | 3,746.00 (-0.04%) | 3,652.42 (97.50%) | 3,842.39 (97.49%) | ✅ (view plot) | 5,253.00 (-0.05%) | 5,125.56 (97.57%) | 5,386.22 (97.53%) | ✅ (view plot) | 5.00 (-31.73%) | 4.61 (92.26%) | 10.04 (49.82%) | ✅ (view plot) | 90.00 (+0.01%) | 86.44 (96.04%) | 93.54 (96.21%) |
get_submit | ✅ (view plot) | 95,341.00 (-0.16%) | 94,993.35 (99.64%) | 96,001.46 (99.31%) | ✅ (view plot) | 59,439.00 (-0.05%) | 59,238.78 (99.66%) | 59,703.30 (99.56%) | ✅ (view plot) | 85,371.00 (-0.05%) | 85,088.25 (99.67%) | 85,742.02 (99.57%) | ✅ (view plot) | 41.00 (-16.99%) | 34.81 (84.90%) | 63.97 (64.10%) | ✅ (view plot) | 279.00 (-0.72%) | 274.60 (98.42%) | 287.42 (97.07%) |
get_subscribe | ✅ (view plot) | 8,021.00 (+0.23%) | 7,754.76 (96.68%) | 8,251.02 (97.21%) | ✅ (view plot) | 2,841.00 (+0.20%) | 2,744.56 (96.61%) | 2,926.11 (97.09%) | ✅ (view plot) | 3,971.00 (+0.19%) | 3,841.96 (96.75%) | 4,084.75 (97.22%) | ✅ (view plot) | 12.00 (-10.93%) | 6.77 (56.45%) | 20.17 (59.49%) | ✅ (view plot) | 114.00 (+0.45%) | 109.42 (95.98%) | 117.57 (96.97%) |
serialize_authorize | ✅ (view plot) | 12,279.00 (+0.27%) | 11,979.77 (97.56%) | 12,512.27 (98.14%) | ✅ (view plot) | 5,317.00 (-0.03%) | 5,223.42 (98.24%) | 5,413.39 (98.22%) | ✅ (view plot) | 7,414.00 (-0.05%) | 7,287.56 (98.29%) | 7,548.55 (98.22%) | ✅ (view plot) | 7.00 (-26.89%) | 6.04 (86.26%) | 13.11 (53.39%) | ✅ (view plot) | 138.00 (+1.04%) | 131.98 (95.63%) | 141.17 (97.75%) |
serialize_deserialize_authorize | ✅ (view plot) | 24,701.00 (+0.48%) | 24,236.77 (98.12%) | 24,929.20 (99.08%) | ✅ (view plot) | 9,868.00 (-0.31%) | 9,788.05 (99.19%) | 10,008.92 (98.59%) | ✅ (view plot) | 13,931.00 (-0.28%) | 13,807.93 (99.12%) | 14,131.83 (98.58%) | ✅ (view plot) | 33.00 (-7.76%) | 30.18 (91.46%) | 41.37 (79.76%) | ✅ (view plot) | 303.00 (+1.64%) | 289.38 (95.50%) | 306.86 (98.74%) |
serialize_deserialize_handle_authorize | ✅ (view plot) | 30,360.00 (+0.41%) | 29,938.92 (98.61%) | 30,532.26 (99.44%) | ✅ (view plot) | 12,071.00 (-0.17%) | 11,994.53 (99.37%) | 12,189.17 (99.03%) | ✅ (view plot) | 17,095.00 (-0.14%) | 16,982.35 (99.34%) | 17,254.29 (99.08%) | ✅ (view plot) | 56.00 (-2.06%) | 50.31 (89.84%) | 64.04 (87.44%) | ✅ (view plot) | 371.00 (+1.20%) | 358.58 (96.65%) | 374.64 (99.03%) |
serialize_deserialize_handle_submit | ✅ (view plot) | 126,494.00 (+0.03%) | 125,944.10 (99.57%) | 126,959.38 (99.63%) | ✅ (view plot) | 73,280.00 (+0.01%) | 72,989.22 (99.60%) | 73,553.60 (99.63%) | ✅ (view plot) | 105,059.00 (+0.02%) | 104,628.44 (99.59%) | 105,441.43 (99.64%) | ✅ (view plot) | 101.00 (-9.93%) | 91.93 (91.02%) | 132.34 (76.32%) | ✅ (view plot) | 598.00 (+0.35%) | 590.12 (98.68%) | 601.66 (99.39%) |
serialize_deserialize_handle_subscribe | ✅ (view plot) | 28,012.00 (+1.30%) | 27,108.29 (96.77%) | 28,198.08 (99.34%) | ✅ (view plot) | 9,659.00 (+0.17%) | 9,548.75 (98.86%) | 9,737.17 (99.20%) | ✅ (view plot) | 13,662.00 (+0.16%) | 13,509.14 (98.88%) | 13,770.38 (99.21%) | ✅ (view plot) | 63.00 (-2.16%) | 56.64 (89.90%) | 72.14 (87.33%) | ✅ (view plot) | 401.00 (+2.51%) | 376.87 (93.98%) | 405.50 (98.89%) |
serialize_deserialize_submit | ✅ (view plot) | 115,211.00 (+0.04%) | 114,669.57 (99.53%) | 115,663.72 (99.61%) | ✅ (view plot) | 68,057.00 (-0.04%) | 67,811.17 (99.64%) | 68,355.90 (99.56%) | ✅ (view plot) | 97,656.00 (-0.04%) | 97,292.35 (99.63%) | 98,105.39 (99.54%) | ✅ (view plot) | 60.00 (-7.76%) | 53.89 (89.81%) | 76.20 (78.74%) | ✅ (view plot) | 493.00 (+0.66%) | 484.06 (98.19%) | 495.52 (99.49%) |
serialize_deserialize_subscribe | ✅ (view plot) | 23,409.00 (+1.43%) | 22,535.92 (96.27%) | 23,620.86 (99.10%) | ✅ (view plot) | 8,211.00 (+0.16%) | 8,102.42 (98.68%) | 8,292.69 (99.01%) | ✅ (view plot) | 11,564.00 (+0.15%) | 11,415.92 (98.72%) | 11,677.89 (99.02%) | ✅ (view plot) | 38.00 (-0.61%) | 33.57 (88.34%) | 42.89 (88.59%) | ✅ (view plot) | 333.00 (+2.77%) | 310.24 (93.17%) | 337.78 (98.59%) |
serialize_submit | ✅ (view plot) | 99,798.00 (-0.09%) | 99,414.30 (99.62%) | 100,358.89 (99.44%) | ✅ (view plot) | 61,483.00 (-0.05%) | 61,279.84 (99.67%) | 61,750.79 (99.57%) | ✅ (view plot) | 88,213.00 (-0.05%) | 87,926.62 (99.68%) | 88,592.02 (99.57%) | ✅ (view plot) | 42.00 (-16.97%) | 37.88 (90.19%) | 63.29 (66.36%) | ✅ (view plot) | 325.00 (+0.01%) | 319.14 (98.20%) | 330.83 (98.24%) |
serialize_subscribe | ✅ (view plot) | 11,418.00 (+0.35%) | 11,114.38 (97.34%) | 11,642.18 (98.07%) | ✅ (view plot) | 4,188.00 (+0.14%) | 4,091.56 (97.70%) | 4,273.11 (98.01%) | ✅ (view plot) | 5,828.00 (+0.13%) | 5,698.85 (97.78%) | 5,941.69 (98.09%) | ✅ (view plot) | 12.00 (-16.01%) | 9.19 (76.62%) | 19.38 (61.92%) | ✅ (view plot) | 158.00 (+0.79%) | 151.58 (95.94%) | 161.94 (97.57%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
Report | Wed, September 4, 2024 at 13:37:29 UTC |
Project | Stratum V2 (SRI) |
Branch | 1074/merge |
Testbed | sv1 |
🚨 1 ALERT: Threshold Boundary Limit exceeded!
Benchmark | Measure (units) | View | Value | Lower Boundary | Upper Boundary |
---|---|---|---|---|---|
client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize | Latency (nanoseconds (ns)) | 🚨 (view plot | view alert) | 666.77 (+7.37%) | 588.64 (88.28%) | 653.38 (102.05%) |
Click to view all benchmark results
Benchmark | Latency | Latency Results nanoseconds (ns) | (Δ%) | Latency Lower Boundary nanoseconds (ns) | (%) | Latency Upper Boundary nanoseconds (ns) | (%) |
---|---|---|---|---|
client-submit-serialize | ✅ (view plot) | 6,967.20 (+1.59%) | 6,416.56 (92.10%) | 7,299.90 (95.44%) |
client-submit-serialize-deserialize | ✅ (view plot) | 8,020.30 (+2.52%) | 7,310.50 (91.15%) | 8,335.64 (96.22%) |
client-submit-serialize-deserialize-handle/client-submit-serialize-deserialize-handle | ✅ (view plot) | 8,523.20 (+1.52%) | 7,953.19 (93.31%) | 8,838.05 (96.44%) |
client-sv1-authorize-serialize-deserialize-handle/client-sv1-authorize-serialize-deserialize-handle | ✅ (view plot) | 931.70 (+2.62%) | 869.33 (93.31%) | 946.49 (98.44%) |
client-sv1-authorize-serialize-deserialize/client-sv1-authorize-serialize-deserialize | ✅ (view plot) | 732.66 (+3.61%) | 675.97 (92.26%) | 738.36 (99.23%) |
client-sv1-authorize-serialize/client-sv1-authorize-serialize | ✅ (view plot) | 253.86 (+1.93%) | 242.25 (95.43%) | 255.87 (99.22%) |
client-sv1-get-authorize/client-sv1-get-authorize | ✅ (view plot) | 158.49 (+0.70%) | 152.52 (96.23%) | 162.25 (97.69%) |
client-sv1-get-submit | ✅ (view plot) | 6,611.70 (-0.38%) | 6,203.14 (93.82%) | 7,070.22 (93.51%) |
client-sv1-get-subscribe/client-sv1-get-subscribe | ✅ (view plot) | 277.79 (+0.31%) | 262.61 (94.54%) | 291.27 (95.37%) |
client-sv1-subscribe-serialize-deserialize-handle/client-sv1-subscribe-serialize-deserialize-handle | ✅ (view plot) | 787.43 (+3.79%) | 719.92 (91.43%) | 797.43 (98.75%) |
client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize | 🚨 (view plot | view alert) | 666.77 (+7.37%) | 588.64 (88.28%) | 653.38 (102.05%) |
client-sv1-subscribe-serialize/client-sv1-subscribe-serialize | ✅ (view plot) | 210.14 (+1.14%) | 195.83 (93.19%) | 219.69 (95.65%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
Report | Wed, September 4, 2024 at 13:37:31 UTC |
Project | Stratum V2 (SRI) |
Branch | ignore-test_diff_management |
Testbed | sv2 |
Click to view all benchmark results
Benchmark | Estimated Cycles | Estimated Cycles Results estimated cycles | (Δ%) | Estimated Cycles Lower Boundary estimated cycles | (%) | Estimated Cycles Upper Boundary estimated cycles | (%) | Instructions | Instructions Results instructions | (Δ%) | Instructions Lower Boundary instructions | (%) | Instructions Upper Boundary instructions | (%) | L1 Accesses | L1 Accesses Results accesses | (Δ%) | L1 Accesses Lower Boundary accesses | (%) | L1 Accesses Upper Boundary accesses | (%) | L2 Accesses | L2 Accesses Results accesses | (Δ%) | L2 Accesses Lower Boundary accesses | (%) | L2 Accesses Upper Boundary accesses | (%) | RAM Accesses | RAM Accesses Results accesses | (Δ%) | RAM Accesses Lower Boundary accesses | (%) | RAM Accesses Upper Boundary accesses | (%) |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
client_sv2_handle_message_common | ✅ (view plot) | 2,107.00 (+1.70%) | 1,986.25 (94.27%) | 2,157.24 (97.67%) | ✅ (view plot) | 473.00 (+0.31%) | 459.53 (97.15%) | 483.53 (97.82%) | ✅ (view plot) | 732.00 (-0.17%) | 715.13 (97.69%) | 751.41 (97.42%) | ✅ (view plot) | 9.00 (+44.50%) | 0.44 (4.92%) | 12.01 (74.91%) | ✅ (view plot) | 38.00 (+1.73%) | 34.99 (92.07%) | 39.72 (95.67%) |
client_sv2_handle_message_mining | ✅ (view plot) | 8,212.00 (+0.14%) | 8,093.01 (98.55%) | 8,307.58 (98.85%) | ✅ (view plot) | 2,137.00 (+0.20%) | 2,099.15 (98.23%) | 2,166.21 (98.65%) | ✅ (view plot) | 3,157.00 (+0.12%) | 3,099.06 (98.16%) | 3,207.43 (98.43%) | ✅ (view plot) | 38.00 (+3.61%) | 30.60 (80.52%) | 42.76 (88.88%) | ✅ (view plot) | 139.00 (+0.03%) | 136.54 (98.23%) | 141.38 (98.32%) |
client_sv2_mining_message_submit_standard | ✅ (view plot) | 6,297.00 (+0.28%) | 6,185.12 (98.22%) | 6,373.96 (98.79%) | ✅ (view plot) | 1,750.00 (-0.01%) | 1,738.79 (99.36%) | 1,761.51 (99.35%) | ✅ (view plot) | 2,547.00 (-0.25%) | 2,535.32 (99.54%) | 2,571.38 (99.05%) | ✅ (view plot) | 22.00 (+27.13%) | 12.01 (54.60%) | 22.60 (97.35%) | ✅ (view plot) | 104.00 (+0.01%) | 101.42 (97.51%) | 106.57 (97.59%) |
client_sv2_mining_message_submit_standard_serialize | ✅ (view plot) | 14,690.00 (-0.47%) | 14,552.72 (99.07%) | 14,966.35 (98.15%) | ✅ (view plot) | 4,694.00 (-0.00%) | 4,682.79 (99.76%) | 4,705.51 (99.76%) | ✅ (view plot) | 6,755.00 (-0.02%) | 6,739.38 (99.77%) | 6,773.69 (99.72%) | ✅ (view plot) | 47.00 (+4.75%) | 37.64 (80.09%) | 52.09 (90.23%) | ✅ (view plot) | 220.00 (-1.01%) | 216.45 (98.39%) | 228.04 (96.47%) |
client_sv2_mining_message_submit_standard_serialize_deserialize | ✅ (view plot) | 27,476.00 (-0.12%) | 27,231.40 (99.11%) | 27,784.33 (98.89%) | ✅ (view plot) | 10,585.00 (+0.21%) | 10,514.55 (99.33%) | 10,610.80 (99.76%) | ✅ (view plot) | 15,401.00 (+0.20%) | 15,299.89 (99.34%) | 15,439.84 (99.75%) | ✅ (view plot) | 84.00 (+2.61%) | 75.78 (90.22%) | 87.95 (95.51%) | ✅ (view plot) | 333.00 (-0.63%) | 327.38 (98.31%) | 342.83 (97.13%) |
client_sv2_open_channel | ✅ (view plot) | 4,337.00 (-2.21%) | 4,272.88 (98.52%) | 4,597.54 (94.33%) | ✅ (view plot) | 1,461.00 (+0.05%) | 1,449.36 (99.20%) | 1,471.23 (99.30%) | ✅ (view plot) | 2,162.00 (+0.29%) | 2,138.44 (98.91%) | 2,173.22 (99.48%) | ✅ (view plot) | 8.00 (-23.15%) | 5.16 (64.48%) | 15.66 (51.08%) | ✅ (view plot) | 61.00 (-4.15%) | 59.29 (97.20%) | 67.98 (89.73%) |
client_sv2_open_channel_serialize | ✅ (view plot) | 13,948.00 (-1.31%) | 13,864.49 (99.40%) | 14,402.67 (96.84%) | ✅ (view plot) | 5,064.00 (+0.01%) | 5,052.36 (99.77%) | 5,074.23 (99.80%) | ✅ (view plot) | 7,328.00 (+0.08%) | 7,302.71 (99.65%) | 7,341.40 (99.82%) | ✅ (view plot) | 36.00 (+2.16%) | 29.43 (81.75%) | 41.05 (87.70%) | ✅ (view plot) | 184.00 (-2.94%) | 182.06 (98.95%) | 197.10 (93.35%) |
client_sv2_open_channel_serialize_deserialize | ✅ (view plot) | 22,613.00 (-0.13%) | 22,362.61 (98.89%) | 22,922.23 (98.65%) | ✅ (view plot) | 8,027.00 (+0.29%) | 7,956.80 (99.13%) | 8,050.84 (99.70%) | ✅ (view plot) | 11,673.00 (+0.25%) | 11,574.18 (99.15%) | 11,713.42 (99.65%) | ✅ (view plot) | 81.00 (+10.17%) | 65.45 (80.80%) | 81.60 (99.26%) | ✅ (view plot) | 301.00 (-0.90%) | 295.82 (98.28%) | 311.66 (96.58%) |
client_sv2_setup_connection | ✅ (view plot) | 4,673.00 (-0.34%) | 4,624.54 (98.96%) | 4,753.19 (98.31%) | ✅ (view plot) | 1,502.00 (+0.05%) | 1,490.36 (99.23%) | 1,512.23 (99.32%) | ✅ (view plot) | 2,278.00 (+0.05%) | 2,259.48 (99.19%) | 2,294.35 (99.29%) | ✅ (view plot) | 10.00 (+6.49%) | 4.87 (48.66%) | 13.92 (71.86%) | ✅ (view plot) | 67.00 (-0.85%) | 65.74 (98.11%) | 69.41 (96.53%) |
client_sv2_setup_connection_serialize | ✅ (view plot) | 16,118.00 (-0.58%) | 16,000.09 (99.27%) | 16,424.10 (98.14%) | ✅ (view plot) | 5,963.00 (+0.01%) | 5,951.36 (99.80%) | 5,973.23 (99.83%) | ✅ (view plot) | 8,663.00 (+0.04%) | 8,638.63 (99.72%) | 8,679.65 (99.81%) | ✅ (view plot) | 42.00 (+0.18%) | 32.70 (77.85%) | 51.15 (82.11%) | ✅ (view plot) | 207.00 (-1.34%) | 204.16 (98.63%) | 215.46 (96.07%) |
client_sv2_setup_connection_serialize_deserialize | ✅ (view plot) | 35,572.00 (+0.07%) | 35,384.23 (99.47%) | 35,712.15 (99.61%) | ✅ (view plot) | 14,855.00 (+0.16%) | 14,783.28 (99.52%) | 14,879.31 (99.84%) | ✅ (view plot) | 21,817.00 (+0.15%) | 21,705.69 (99.49%) | 21,864.21 (99.78%) | ✅ (view plot) | 98.00 (+3.25%) | 79.72 (81.34%) | 110.11 (89.00%) | ✅ (view plot) | 379.00 (-0.18%) | 375.88 (99.18%) | 383.47 (98.83%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
Have you tried the values suggested here #988 (comment) ? |
I did some investigation, which is described in detail here: #995 (comment) my findings were:
it seems your #988 (comment) suggests in summary, bigger values for if we merge #1075 and have to re-open #988 again, then we should consider this PR, which completely disables |
reverting this to draft status in favor of #1075 |
yep measure hashrate above 10s seems to not improve significantly anymore. Higher total run time seems to always improve 40s seems to be good enough 60 would be better, but is 20s more every time we run the test |
if we hit that only once in weeks I would leave it there as the advantage of having the test overtake the hassle of re run it very few time. Otherwise we can just remove it. |
also why we have all those warnings in CI? |
#1051 |
6b111e2
to
0ce9217
Compare
cf7601d
to
33c0faf
Compare
33c0faf
to
ca6ba30
Compare
c57c5f2
to
8c4091e
Compare
8c4091e
to
6f3d0cd
Compare
after some failed attempts on #995, this PR disables the
test_diff_management
unit test as a temporary fix to #988we should not close #988 until a more sustainable fix is found for the long term
we should also be extra careful when modifying
roles/translator/src/lib/downstream_sv1/diff_management.rs