chore: moving to otel from opencensus

[JaydipGabani]

What this PR does / why we need it:
As opencensus is EOL, moving to opentelemetry for metrics.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #2465

Special notes for your reviewer:

• Here is the gist of metrics export using opentelemetry exporter.
• Here is the diff of the metrics export with opencensus/prometheus vs otel/prometheus

[maxsmythe]

Thanks for figuring this out!

I left some comments on the first registry, but wanted to avoid repeating comments.

With proper encapsulation and separation-of-concerns, I think we can get this PR to the point where there are no changes outside of stats_reporter.go/tests, pkg/metrics.

[codecov-commenter]

Codecov Report

Attention: 87 lines in your changes are missing coverage. Please review.

Comparison is base (d2b6fcd) 53.78% compared to head (b5c3c1a) 54.44%.
Report is 1 commits behind head on master.

Files	Patch %	Lines
pkg/syncutil/stats_reporter.go	47.05%	27 Missing ⚠️
pkg/webhook/stats_reporter.go	78.57%	8 Missing and 4 partials ⚠️
...etrics/exporters/prometheus/prometheus_exporter.go	35.71%	9 Missing ⚠️
pkg/audit/stats_reporter.go	89.33%	6 Missing and 2 partials ⚠️
pkg/controller/mutators/stats_reporter.go	87.87%	4 Missing and 4 partials ⚠️
pkg/metrics/exporters/common/common.go	77.41%	6 Missing and 1 partial ⚠️
pkg/watch/stats_reporter.go	86.04%	4 Missing and 2 partials ⚠️
...kg/controller/constrainttemplate/stats_reporter.go	90.00%	3 Missing and 2 partials ⚠️
pkg/controller/constraint/stats_reporter.go	87.50%	2 Missing and 1 partial ⚠️
pkg/controller/expansion/stats_reporter.go	96.00%	0 Missing and 1 partial ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3011      +/-   ##
==========================================
+ Coverage   53.78%   54.44%   +0.65%     
==========================================
  Files         136      134       -2     
  Lines       12195    12329     +134     
==========================================
+ Hits         6559     6712     +153     
+ Misses       5135     5122      -13     
+ Partials      501      495       -6     

Flag	Coverage Δ
unittests	54.44% <80.75%> (+0.65%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maxsmythe]

Getting closer!

90% of comments are in the same general areas:

• files adjacent to stats_reporter.go and stats_reporter_test.go should not have deltas (I think there was one potential exception to this)

• All metrics state for callbacks should be maintained by reporter structs

• All callbacks should be named like observeX

• reportX functions should be maintained, with the same function signature (necessary for the zero-delta bullet point). In the case where callbacks are involved, this means updating reporter to have internal state and handling concurrency (locking/atomic variables).

• Callbacks should be handled via constructing the reporter (better separation of concerns and necessary for zero delta)

[JaydipGabani]

@maxsmythe Thanks for providing detailed feedback. I think I addressed all of your feedback. PTAL!

[maxsmythe]

Getting close! Biggest themes are:

• observe functions need to acquire a read mutex
• I think we can consolidate a lot of test helper code

Some other, less global comments. Once this is done, we can help make sure stackdriver doesn't have a diff.

[sozercan]

LGTM pending @acpana's approval for stackdriver

[acpana]

api nits: if we keep this structure, it may be useful to at least spell out the "required conditions". Alternatively splitting this into a AddReader and SetMetricsProvider might offer a cleaner/ more concise api

[acpana]

As of b5c3c1a, I don't see a delta to metrics sent to GCM (formerly stackdriver). Thank you loads Jay for working on this 💯 🥇

Max LGTM'ed in a comment

[sozercan]

LGTM