add cache client read and write otel tracing

[pvditt]

Tracking issue

https://github.com/flyteorg/flyte/issues/

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.09%. Comparing base (24a6e4e) to head (9263d71).
Report is 9 commits behind head on master.

❗ Current head 9263d71 differs from pull request most recent head 889b8e2. Consider uploading reports for the commit 889b8e2 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5184      +/-   ##
==========================================
+ Coverage   58.88%   59.09%   +0.21%     
==========================================
  Files         568      646      +78     
  Lines       47967    55744    +7777     
==========================================
+ Hits        28243    32940    +4697     
- Misses      17309    20208    +2899     
- Partials     2415     2596     +181     

Flag	Coverage Δ
unittests	59.09% <100.00%> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.