discard unused uct_ep when the pre-created uct_ep more than needed

[changchengx]

What

keep NUM_PATHS compatible between server (e.g. UCX_IB_NUM_PATHS=1) and client (e.g. UCX_IB_NUM_PATHS=2)
This PR is mainly ported from:

1. UCP/CORE: Implement flush+destroy for UCT EPs on UCP Worker openucx/ucx#5608
2. UCP/WIREUP: Implement support for SHM/multi-lane in case of CM openucx/ucx#5696

[changchengx]

1. UCP: fail if the server cannot connect to all eps addrs from the client #31 is reverted
2. There's no uct_ep resource leak as being reported in https://redmine.mellanox.com/issues/2284793 when verifying it on local two server.
3. verify path compatible function on local two serves & every server has one NIC with two ports(HCA MT4125, 40Gbps x 2)
   

[changchengx]

@yosefe
Merge all the ucp_ep_config_lanes_intersect related functions in 422a29c

[yosefe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[changchengx]

How to run the below io_demo Test tag match on CX4 test case locally to check the problem?
https://dev.azure.com/ucfconsort/ucx-int3/_build/results?buildId=35022&view=logs&j=f6213e06-5fba-58e7-a041-fa7d17b490d4&t=0687ad30-25a0-56ac-90ea-d6a36c068c41&l=30

2022-01-12T11:54:39.4062959Z + python /hpc/noarch/git_projects/hpc-mtt-conf/scripts/iodemo_analyzer.py -d drop_35022/tag_cx4_rc --duration 600 -t 3 --allow_list integration3
2022-01-12T11:54:39.4523334Z ==============================
2022-01-12T11:54:39.4524537Z Client Errors:
2022-01-12T11:54:39.4525117Z ==============================
2022-01-12T11:54:39.4526337Z Log: drop_35022/tag_cx4_rc/iodemo_swx-rain02_client_00.log
2022-01-12T11:54:39.4527104Z Error iodemo analyzer: [1641988008.295210] [DEMO] timeout waiting for 1 replies on 1 connections
2022-01-12T11:54:39.4527679Z 
2022-01-12T11:54:39.4528179Z ==============================
2022-01-12T11:54:39.4528814Z Server Errors:
2022-01-12T11:54:39.4529401Z ==============================
2022-01-12T11:54:39.4530206Z Log: drop_35022/tag_cx4_rc/iodemo_swx-rain01_server_00.log
2022-01-12T11:54:39.4531328Z Error iodemo analyzer: [1641987928.295231] [swx-rain01:438277:0]         wireup.c:318  UCX  ERROR ep 0x7f2042f71000: no remote ep address for lane[0]->remote_lane[0]
2022-01-12T11:54:39.4532016Z 
2022-01-12T11:54:39.4633260Z ##[error]Bash exited with code '1'.
2022-01-12T11:54:39.4648668Z ##[section]Finishing: Analyze for tag_cx4_rc

[yosefe]

@changchengx the relevant script is https://github.com/yosefe/ucx/blob/integration3/buildlib/pr/io_demo/az-stage-io-demo.yaml

[changchengx]

io_demo Test tag match on CX4

1. reserve 2 swx-ucx servers: e.g. swx-ucx03 & swx-ucx04

2. Login in swx-ucx03

3. Build ucx/integration3 with PR in swx-ucx03

4. Start network corrupter in background
$ #Confirm whether it works swx-ucx03 & swx-ucx04
$ #Start network corrupter
[swx-ucx03]$ buildlib/az-network-corrupter.sh initial_delay=20 cycles=100 downtime=5 uptime=180 interface=bond0

5. Run below command on swx-ucx03 server
[swx-ucx03]$ export UCX_NET_DEVICES=mlx5_bond_0:1
[swx-ucx03]$ export UCX_TLS=rc_x
[swx-ucx03]$ export UCX_RNDV_THRESH=4k
[swx-ucx03]$ export LD_LIBRARY_PATH=drop_35176/install/lib:
[swx-ucx03]$ LD_LIBRARY_PATH=drop_35176/install/lib:
[swx-ucx03]$ test/apps/iodemo/run_io_demo.sh -H swx-ucx03,swx-ucx04 --tasks-per-node 1 --duration 600 -v --num-clients 1 --num-servers 1 --map-by slot --log-dir drop_35176/tag_cx4_rc -i bond0 drop_35176/install/bin/io_demo -d 512:524288 -P 2 -o read,write -i 0 -w 16 -t 60

6. Run below command on swx-ucx03 server
[swx-ucx03] python /hpc/noarch/git_projects/hpc-mtt-conf/scripts/iodemo_analyzer.py -d drop_35176/tag_cx4_rc --duration 600 -t 3 --allow_list integration3

7. Kill the corrupter

8. restore interface
[swx-ucx03]$ drop_35176/buildlib/az-network-corrupter.sh reset=yes interface=bond0

take care of the direcotry

drop_35176/install/bin/io_demo
--log-dir drop_35176/tag_cx4_rc
-d drop_35176/tag_cx4_rc

[yosefe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[changchengx]

@yosefe Is it possible to review this PR before fixing the CI self problem?

[changchengx]

CI has been passed.

[yosefe]

@dmitrygx can you pls review?

[dmitrygx]

@changchengx did you use the latest code from master?
I see that some bugs came from the initial implementation.
So, I suggest you align this PR with the master branch.

[changchengx]

@dmitrygx I've followed your 1st round CR comments to refine the implmentation. Please check it.

[dmitrygx]

shall we add ucp_worker_discard_uct_ep_cleanup to cleanup discarding hash in ucp_worker_destroy?

[changchengx]

shall we add ucp_worker_discard_uct_ep_cleanup to cleanup discarding hash in ucp_worker_destroy?

As we've talked offline, I'll add warning if the worker->discard_uct_ep_hash isn't empty to minimize the change when doing ucp_worker_destroy

[changchengx]

@yosefe Dmitry has approved this PR and he's in Hackathon these days.

[changchengx]

@yosefe Do you have any comments about this PR?

[yosefe]

can we add CI to test this?
this branch has minimal ci infrastructure to run io_demo

[changchengx]

can we add CI to test this? this branch has minimal ci infrastructure to run io_demo

@yosefe Do you mean to add test into io-demo.yaml or MTT?

[yosefe]

2 short tests in io-demo.yaml , without interference, with different num_paths (2->1 and 1->2)

[changchengx]

@yosefe Does azure CI stop working? There's no CI triggered here.
I've added two test cases to verify path compatible function.

[yosefe]

@changchengx see https://dev.azure.com/ucfconsort/ucx-int3/_build/results?buildId=40860&view=results

[changchengx]

Two test cases have been added.
Pass all the CI test cases.

[yosefe]

@changchengx pls squash