Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCP/PROTO: Consider RNDV_PERF_DIFF #10401

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

UCP/PROTO: Consider RNDV_PERF_DIFF #10401

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
47d06cd
UCP/PROTO: Consider RNDV_PERF_DIFF
rakhmets Dec 30, 2024
e7dae44
UCP/PROTO: Fixed code format issues.
rakhmets Dec 30, 2024
4a2dae0
UCP/PROTO: Moved function call out of the loop.
rakhmets Dec 30, 2024
b0a9f99
TEST/GTEST/UCP: Updated gold data in test_ucp_proto_mock_rcx.mock_ifa…
rakhmets Jan 13, 2025
2b89e12
TEST/GTEST/UCP: Added mock test with RNDV_PERF_DIFF=0.
rakhmets Jan 14, 2025
db2470a
Merge remote-tracking branch 'upstream/master' into topic/proto-consi…
rakhmets Jan 15, 2025
bed8784
TEST/GTEST/UCP: Fixed env variables.
rakhmets Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions src/ucp/proto/proto_perf.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,28 @@ ucs_status_t ucp_proto_perf_aggregate2(const char *name,
return ucp_proto_perf_aggregate(name, perf_elems, 2, perf_p);
}

void ucp_proto_perf_apply_bias(ucp_proto_perf_t *perf, double bias)
{
ucs_linear_func_t bias_func = ucs_linear_func_make(0.0, 1.0 - bias);
ucp_proto_perf_node_t *bias_node;
ucp_proto_perf_factor_id_t fid;
ucp_proto_perf_segment_t *seg;

if (fabs(bias) <= UCP_PROTO_PERF_EPSILON) {
return;
}

ucp_proto_perf_segment_foreach(seg, perf) {
for (fid = 0; fid < UCP_PROTO_PERF_FACTOR_LAST; ++fid) {
seg->perf_factors[fid] =
ucs_linear_func_compose(bias_func, seg->perf_factors[fid]);
}
ucp_proto_perf_node_update_factors(seg->node, seg->perf_factors);
bias_node = ucp_proto_perf_node_new_data("bias", "%.2f %%", bias);
ucp_proto_perf_node_own_child(seg->node, &bias_node);
}
}

/* TODO:
* Reconsider correctness of PPLN perf estimation logic since in case of async
* operations it seems wrong to choose the longest factor without paying
Expand Down
10 changes: 10 additions & 0 deletions src/ucp/proto/proto_perf.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,16 @@ ucs_status_t ucp_proto_perf_aggregate2(const char *name,
ucp_proto_perf_t **perf_p);


/**
* Apply a bias change to the given perf structure.
*
* @param [in] perf Performance data structure to update.
* @param [in] bias Bias to apply. A bias equal to 0.1 indicates a 10%
* performance improvement.
*/
void ucp_proto_perf_apply_bias(ucp_proto_perf_t *perf, double bias);


/**
* Expand given perf by estimation that all messages on interval
* [end of @a frag_seg + 1, @a max_length] would be sent in a pipeline async
Expand Down
1 change: 1 addition & 0 deletions src/ucp/rndv/proto_rndv.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,7 @@ static void ucp_proto_rndv_ctrl_variant_probe(
cfg_thresh = remote_proto->cfg_thresh;
}

ucp_proto_perf_apply_bias(perf, params->perf_bias);
Copy link
Contributor

@iyastreb iyastreb Jan 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems applying this bias leads to slightly different performance numbers and therefore breaks the mock test:
log1, log2, log3

Expected protocol:
{8247, 377094, "multi-frag copy-in"},
Actual protocol:
8247-363935 multi-frag copy-in

Could it be due to adding this bias? If so, we should update here the numbers in mock test as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is because of adding the bias. UCX selects rendezvous earlier with this changes.

brminich marked this conversation as resolved.
Show resolved Hide resolved
ucp_proto_select_add_proto(&params->super.super, cfg_thresh, cfg_priority,
perf, rpriv, priv_size);

Expand Down
39 changes: 39 additions & 0 deletions test/gtest/ucp/test_ucp_proto.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,45 @@ UCS_TEST_F(test_proto_perf, intersect_first)
expect_empty_range(5000, SIZE_MAX);
}

UCS_TEST_F(test_proto_perf, apply_zero_bias) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if it makes sense to add a mock test with RNDV_PERF_DIFF and make an assertion on selected protocols

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean remove this test, and add test case to test_ucp_proto_mock_rcx?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't insist on removing this test
But adding new test to test_ucp_proto_mock_rcx looks like superior alternative, because we test the entire chain

m_perf = create();
add_func(0, SIZE_MAX, UCP_PROTO_PERF_FACTOR_LOCAL_TL, local_tl_func);
add_func(0, SIZE_MAX, UCP_PROTO_PERF_FACTOR_REMOTE_TL, remote_tl_func);

/* Apply zero bias */
ucp_proto_perf_apply_bias(m_perf.get(), 0);

make_flat_perf();
print_perf();

expect_perf(0, SIZE_MAX,
{{UCP_PROTO_PERF_FACTOR_LOCAL_TL, local_tl_func},
{UCP_PROTO_PERF_FACTOR_REMOTE_TL, remote_tl_func}});
}

UCS_TEST_F(test_proto_perf, apply_bias) {
m_perf = create();
add_func(0, SIZE_MAX, UCP_PROTO_PERF_FACTOR_LOCAL_TL, local_tl_func);
add_func(0, SIZE_MAX, UCP_PROTO_PERF_FACTOR_REMOTE_TL, remote_tl_func);

/* Apply 10% bias */
double bias = 0.1;
ucp_proto_perf_apply_bias(m_perf.get(), bias);

make_flat_perf();
print_perf();

/* Calculate expected */
auto bias_func = ucs_linear_func_make(0, 1 - bias);
auto exp_local_tl_func = ucs_linear_func_compose(bias_func, local_tl_func);
auto exp_remote_tl_func = ucs_linear_func_compose(bias_func,
remote_tl_func);

expect_perf(0, SIZE_MAX,
{{UCP_PROTO_PERF_FACTOR_LOCAL_TL, exp_local_tl_func},
{UCP_PROTO_PERF_FACTOR_REMOTE_TL, exp_remote_tl_func}});
}

UCS_TEST_F(test_proto_perf, intersect_last)
{
/*
Expand Down
15 changes: 15 additions & 0 deletions test/gtest/ucp/test_ucp_proto_mock.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,21 @@ UCS_TEST_P(test_ucp_proto_mock_rcx, mock_iface_attr, "IB_NUM_PATHS?=1")
key.param.op_id_flags = UCP_OP_ID_AM_SEND;
key.param.op_attr = 0;

check_ep_config(sender(), {
{0, 200, "short"},
{201, 8246, "copy-in"},
{8247, 363935, "multi-frag copy-in"},
{363936, UCS_MEMUNITS_INF, "rendezvous zero-copy read from remote"},
}, key);
}

UCS_TEST_P(test_ucp_proto_mock_rcx, zero_rndv_perf_diff, "IB_NUM_PATHS?=1",
"RNDV_PERF_DIFF=0")
{
ucp_proto_select_key_t key = any_key();
key.param.op_id_flags = UCP_OP_ID_AM_SEND;
key.param.op_attr = 0;

check_ep_config(sender(), {
{0, 200, "short"},
{201, 8246, "copy-in"},
Expand Down
Loading