spdk cherry pick: bdev/rbd: Do not submit IOs through thread sending.

[baum]

Cherry pick SPDK change

Cherry pick SHA c441e270231f11ad6cd1b5c9f60bf598b7fcb757

    bdev/rbd: Do not submit IOs through thread sending.

    Currently, we send IOs to the main_td thread.
    It is not needed, because all the read/write functions
    provided by librbd are thread safe, so we can eliminate the
    thread send messaging policy for read/write related functions.

    And with this patch, users can observe the load balance
    distribution of I/Os on each CPU core owned by spdk applications
    through spdk_top tool.

    In this patch, we did the following work:

    1 Move rbd_open when create the bdev since we will create once.
    2 Simplify the channel management.
    3 Do not use thread send messaging to do the read/write I/Os.

    According to our experiment results showed in
    https://github.com/spdk/spdk/issues/2204

    There will be more than 15% performance improvment in IOPS aspect
    for different write I/O patterns, and it also addresses the I/O Load
    balance issues.

    Fixes issue: #2204

    Change-Id: I9d2851c3d772261c131f9678f4b1bf722328aabb
    Signed-off-by: Ziye Yang <[email protected]>
    Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17644
    Reviewed-by: Shuhei Matsumoto <[email protected]>
    Tested-by: SPDK CI Jenkins <[email protected]>
    Reviewed-by: Jim Harris <[email protected]>