Skip to content

Large data transfer

Jump to bottom
Fanda Vacek edited this page Jun 14, 2018 · 18 revisions

Large data transfer, streaming, rsh, tunneling

Tunneling?

Tunnel creation

  1. client1 should call RPC method like tunnel or something on remote client2 node, return value will be new tunnel handle. There is no special name for the method, every method which sends revCallerIds in response meta data and revRequestId inr result is considered to be tunnel creator

Writing to the tunnel

  • client1 send fake RpcResponse with revRequestId and revClientIds returned when tunnel was created.
  • when client2 receives fake RpcResponse, then it checks if requestId == revRequestId sent to client1
  • the same fake RpcResponse is used by client1 for every chunk of data written to the tunnel

Reading from the tunnel

  • client2 send again RpcResponse with requestId and clientIds from create tunnel RPC call request.
  • when client1 receives RpcResponse with requestId from the create tunnel RPC call, then it considers response to be tunneled data
  • the same requestId is used by client2 for every chunk of data written to the tunnel

Closing tunnel

  • WIP

tunnels with dedicated connection

Sometimes it is convenient to create dedicated RPC connection for tunnel.

Example session

  • client7 opens tunnel calling RPC method runPtyCmd
  • client8 resposes with revRequestId == 3 stored in the RPC request result. Reverse caller ids is set to 8, since fake RpcRecponse used to write to tunnel should be sent to the client8
<== client:7 <T:RpcMessage, id:5u, shvPath:".broker/clients/8/app", method:"runPtyCmd">i{params:["/bin/sh", 278, 92]}
==> client:8 <T:RpcMessage, id:5u, shvPath:"", method:"runPtyCmd", cid:7u>i{params:["/bin/sh", 278, 92]}
<== client:8 <T:RpcMessage, id:5u, cid:7u, rcid:null>i{result:3u}
==> client:7 <T:RpcMessage, id:5u, rcid:8u>i{result:3u}
  • client8 writes launched process stdout ("$ sh: turning off NDELAY mode\r\n") to tunnel channel 1
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"$ sh: turning off NDELAY mode\r\n"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"$ sh: turning off NDELAY mode\r\n"]}
  • client 7 write "ls\r\n" to the tunnel with terminal echo tunneled back
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"l"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"l"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"l"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"l"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"s"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"s"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"s"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"s"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"\r"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"\r"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"\r\n"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"\r\n"]}
  • client8 writes ls command result to tunnel
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"bfsview        cp2cp\t    revitestdevice\t  shvagent\t  shvbroker\t   shvrexec\t   shvrsh\t shvspy        tests\t    tst_chainpack_rpcmessage\t    tst_chainpack_rpcvalue\t  tst_crypt\t"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"bfsview        cp2cp\t    revitestdevice\t  shvagent\t  shvbroker\t   shvrexec\t   shvrsh\t shvspy        tests\t    tst_chainpack_rpcmessage\t    tst_chainpack_rpcvalue\t  tst_crypt\t"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"   tst_stringview\r\nbfsview.debug  cp2cp.debug  revitestdevice.debug  shvagent.debug  shvbroker.debug  shvrexec.debug  shvrsh.debug  shvspy.debug  tests.debug  tst_chainpack_rpcmessage.debug  tst_chainpack_rpcvalue.debug  tst_crypt.debug  tst_stringview.debug\r\n"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"   tst_stringview\r\nbfsview.debug  cp2cp.debug  revitestdevice.debug  shvagent.debug  shvbroker.debug  shvrexec.debug  shvrsh.debug  shvspy.debug  tests.debug  tst_chainpack_rpcmessage.debug  tst_chainpack_rpcvalue.debug  tst_crypt.debug  tst_stringview.debug\r\n"]}
  • client8 writes shell prompt to the tunnel
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"$ "]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"$ "]}
  • client7 writes "exit" command to terminate remote shell and close tunnel
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"e"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"e"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"e"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"e"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"x"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"x"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"x"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"x"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"i"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"i"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"i"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"i"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"t"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"t"]}
<== client:8 <T:RpcMessage, id:5u, cid:7u>i{result:[1, b"t"]}
==> client:7 <T:RpcMessage, id:5u>i{result:[1, b"t"]}
<== client:7 <T:RpcMessage, id:3u, cid:8u>i{result:[0, b"\r"]}
==> client:8 <T:RpcMessage, id:3u>i{result:[0, b"\r"]}
Clone this wiki locally