monitor-pv behaviour across different OpenEBS storage engines

[ksatchit]

Test Params:

• Sample size is 2 GB.
• Sample active data size is 4 MB.
• Source command is dd.

Results:

	Monitor PV	maya exporter	kubelet metrics
	(pv_utilization_bytes)	(openebs_actual_used)	(kubelet_volume_capacity_bytes - kubelet_volume_available_bytes)
Jiva	3.9258 MB	69.0703 MB	25.9141 MB
Local PV Hostpath	1.2142 GB
	(Sample size is 1GB + 220MB)
Local PV device	2.4079 GB
	(Sample size is 2GB + 418MB)

Notes:

• Jiva shows more utilization as can be seen: two possible reasons:

  • Filesystem metadata upon format (however, this additional usage seems more for a jiva vol compared to a higer sized raw disk attached to the VM. Needs confirmation from Sumit)
  • Smaller blocks treated as 4k blocks/packing of blocks (however, there is some fs optimization activity here which minimizes this impact for contiguous sequential workload like dd, v/s staggered/random writes)
  • Might be worth trying out comparing different workloads across local-pv / jiva from same relative start-point (i.e., 20K local PV = 65 MB from jiva and see if same increase in util is observed)
  • All above questions are important if monitor-pv is used as metrics source for both jiva as well as local PV.

• In case of Local PV hostpath & device, monitor-pv mirrors expected app usage via du.

Thoughts based on above observations:

• It is better to rely on different metrics sources for different PVs (jiva, cstor, local pv, zfs local pv) -- i.e. grafana panels uses different queries against diff metrics sources.

• For jiva/cstor where there is possibility of much divergence b/w user/app usage and actual usage, we can show different graphs or lines to highlight the difference and set the right expectation.

[ksatchit]

cc: @slalwani97 @utkarshmani1997 @kmova