[v1.22.x] prov/efa: Handle recv cancel for zero copy recv

[shijin-aws]

backport #10215

[shijin-aws]

@j-xiong does this have to be merged by 5 pm to catch 1.22.0 release? Or merging #10215 is enough

[j-xiong]

@shijin-aws No. It's enough to have the PR is here. I can monitor the CI and get it merged before I create my updated version of #10211

[shijin-aws]

@j-xiong Awesome, thx!

[j-xiong]

@shijin-aws What is the AWS CI error about? the same "out of memory error" as we saw in #10216?

[shijin-aws]

Yeah, not sure why, anyway, we may need to reduce the message size in this test

server_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o BatchMode=yes 172.31.66.252 'timeout 1800 /bin/bash --login -c '"'"'FI_LOG_LEVEL=warn /home/ec2-user/PortaFiducia/build/libraries/libfabric/pr10218-debug/install/fabtests/bin/fi_unexpected_msg -e rdm -M 1024 -I 5 -v -S 1048576 -D cuda -i 0 -p shm -E=9228'"'"''

client_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o BatchMode=yes 172.31.66.252 'timeout 1800 /bin/bash --login -c '"'"'FI_LOG_LEVEL=warn /home/ec2-user/PortaFiducia/build/libraries/libfabric/pr10218-debug/install/fabtests/bin/fi_unexpected_msg -e rdm -M 1024 -I 5 -v -S 1048576 -D cuda -i 0 -p shm -E=9228 172.31.66.252'"'"''
client_stdout:
[error] fabtests:common/hmem_cuda.c:305: cudaMalloc failed: cudaErrorMemoryAllocation out of memory
timeout: the monitored command dumped core

client returncode: 255
server_stdout:

server returncode: 124

[shijin-aws]

I opened a PR #10219 to remove that 1M test, will see how it goes

[j-xiong]

bot:aws:retest