[BUG] AKS Managed prometheus - Big discrepancy between portal metrics and prometheus metrics

[grzesuav]

Describe the bug
There are few metrics:

• Active time series in Azure portal
• Active time series % utilization in Azure portal
• scrape_samples_scraped from prometheus - which is the number of samples the target exposed.

I cannot correlate the first two (from azure portal) with prometheus one.

To Reproduce

Expected behavior

Those metrics should be in line, there is no other way currently to see the number of metrics in each job other than scrape_samples_scraped metric.

Which should be trusted ?

Additional context

Related #4159

[grzesuav]

Continuation of topic started in #4689 @vishiy @aritraghosh

[microsoft-github-policy-service]

@vishiy, @saaror would you be able to assist?

[vishiy]

@grzesuav - how is scrape_samples_scraped equal to time-series count ? one is samples scraped (not ingested, as many could be dropped due to relabelings) and another is unique time-series (not samples).
To compare samples scraped with per minute AMW ingestion quota usage -
You should try -

sum(sum_over_time(scrape_samples_post_metric_relabeling [1m] ))

and compare with the metrics chart in the portal (for the samples/min ingested metric and not time-series).