Fix dtrace system level scripts

[leftwo]

While debugging system level issues on London, we found the dtrace scripts did not handle multiple session IDs correctly and were not showing correct results.

To fix, I did a refactor the scripts get-ds-state and get-lr-state to correctly handle a PID with multiple sessions and improved the printing time for getting a system summary output.

There is now a single dtrace sub-script that gathers all output from all process, then we sort and display a line for
each session in post-processing. This speeds up run time overall.

Sample output:

root@oxz_switch0:~# pilot host exec -c 'hostname && /opt/oxide/crucible_dtrace/get-lr-state.sh' 8-9 
 8  BRM44220011        ok: BRM44220011
oxz_propolis-server_7f810e9c 19793 e1a38744 0 0 0 0 0 0
oxz_propolis-server_68932d85 19836 29d41aea 0 0 0 0 0 0
oxz_propolis-server_68932d85 19836 ccb3ce4c 0 0 0 0 0 0
oxz_propolis-server_501befcb 20559 7ef8a2b6 0 0 0 0 0 0
 9  BRM44220005        ok: BRM44220005
oxz_propolis-server_3354b7e3 20062 8f594433 0 0 0 0 0 0
oxz_propolis-server_54f5c356 21042 ce250057 0 0 0 0 0 0
oxz_propolis-server_0e091218 22706 405c57b7 0 0 0 0 0 0
oxz_propolis-server_057ef9d1 22713 f60d7a47 0 0 0 0 0 0
oxz_propolis-server_52e30c4d 22777 a9649821 0 0 0 0 0 0
oxz_propolis-server_52e30c4d 22777 fe4ae0b9 0 0 0 0 0 0

root@oxz_switch0:~# pilot host exec -c 'hostname && /opt/oxide/crucible_dtrace/get-ds-state.sh' 8-9                                                                   
 8  BRM44220011        ok: BRM44220011
oxz_propolis-server_7f810e9c 19793 e1a38744            active            active            active
oxz_propolis-server_68932d85 19836 29d41aea            active            active            active
oxz_propolis-server_68932d85 19836 ccb3ce4c            active            active            active
oxz_propolis-server_501befcb 20559 7ef8a2b6            active            active            active
 9  BRM44220005        ok: BRM44220005
oxz_propolis-server_3354b7e3 20062 8f594433            active            active            active
oxz_propolis-server_54f5c356 21042 ce250057            active            active            active
oxz_propolis-server_0e091218 22706 405c57b7            active            active            active
oxz_propolis-server_057ef9d1 22713 f60d7a47            active            active            active
oxz_propolis-server_52e30c4d 22777 a9649821            active            active            active
oxz_propolis-server_52e30c4d 22777 fe4ae0b9            active            active            active
root@oxz_switch0:~# 

[faithanalog]

some minor nits but looks fine to me

[faithanalog]

nit: probably doesnt matter too much here, but should be while read -r p to avoid escape character expansion by the shell

[leftwo]

Done

[faithanalog]

question in the header comment: this prints for all PIDs rather than a given PID right?

[leftwo]

That was copy cruft from where I copied the script from. I've fixed the comment to address reality.

[faithanalog]

same thing about read -r from above

[leftwo]

Done, thanks for the comments.

[leftwo]

The test flake is #1498

[faithanalog]

lets merge it!