[exporter/awsemfexporter]Split EMF log with larger than 100 buckets.

[zzhlogin]

Description:
In Application Signals, we utilize Base2 Exponential Bucket Histogram to aggregate and send latency data, with a default max number of buckets 160. In EMF exporter, these buckets are mapped to "Target members" in EMF log entries.
However, CloudWatch EMF logs impose a limit of 100 target members, beyond which EMF processors will mark the record as invalid, resulting in missing metrics and customer-facing errors reported via the EMFValidationErrors metric.

In this PR, we split histograms to two sub EMF logs with the following change:

1. Add an extra attribute metricIndex to groupedMetricMetadata : Current EMF exporter aggregate incoming metrics into groupedMetrics before converting to log events, where the groupKey is generated based on the groupedMetricMetadata including: metric namespace, timestamp, log group name, etc. After splitting, the two new metrics will share exactly the same key. Adding an extra metric metadata for key generation can prevent the second metric from dropping.
2. If the total buckets exceed 100, the exponential histogram metric are split into into multiple data points as needed,
   each containing a maximum of 100 buckets, to comply with CloudWatch EMF log constraints.
   For each split data point:

• Min and Max values are recalculated based on the bucket boundary within that specific split.
• Sum is only assigned to the first split to ensure the total sum of the datapoints after aggregation is correct.
• Count is accumulated based on the bucket counts within each split.

Testing:
The change is tested by generating traffic with more than 100 buckets, and the emf log with larger than 100 values are eliminated after the change:

Compare the added Benchmark test before vs after the code change:
Benchmark test with 100 bucket length:

goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter
cpu: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
                                                  │ before_100.txt │           after_100.txt            │
                                                  │     sec/op     │   sec/op     vs base               │
GetAndCalculateDeltaDataPointsInclude100Buckets-16      19.68µ ± 5%   20.55µ ± 6%  +4.41% (p=0.015 n=10)

                                                  │ before_100.txt │            after_100.txt            │
                                                  │      B/op      │     B/op      vs base               │
GetAndCalculateDeltaDataPointsInclude100Buckets-16     11.50Ki ± 0%   12.02Ki ± 0%  +4.48% (p=0.000 n=10)

                                                  │ before_100.txt │           after_100.txt           │
                                                  │   allocs/op    │ allocs/op   vs base               │
GetAndCalculateDeltaDataPointsInclude100Buckets-16       126.0 ± 0%   132.0 ± 0%  +4.76% (p=0.000 n=10)

Benchmark test with 200 bucket length:

goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter
cpu: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
                                                  │ before_200.txt │           after_200.txt            │
                                                  │     sec/op     │   sec/op     vs base               │
GetAndCalculateDeltaDataPointsInclude200Buckets-16      26.36µ ± 6%   28.01µ ± 5%  +6.24% (p=0.011 n=10)

                                                  │ before_200.txt │            after_200.txt            │
                                                  │      B/op      │     B/op      vs base               │
GetAndCalculateDeltaDataPointsInclude200Buckets-16     15.50Ki ± 0%   16.59Ki ± 0%  +7.06% (p=0.000 n=10)

                                                  │ before_200.txt │           after_200.txt            │
                                                  │   allocs/op    │ allocs/op   vs base                │
GetAndCalculateDeltaDataPointsInclude200Buckets-16       128.0 ± 0%   152.0 ± 0%  +18.75% (p=0.000 n=10)

Benchmark test with 300 bucket length:

goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter
cpu: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
                                                  │ before_300.txt │           after_300.txt            │
                                                  │     sec/op     │   sec/op     vs base               │
GetAndCalculateDeltaDataPointsInclude300Buckets-16      37.04µ ± 6%   39.17µ ± 5%  +5.73% (p=0.029 n=10)

                                                  │ before_300.txt │            after_300.txt             │
                                                  │      B/op      │     B/op      vs base                │
GetAndCalculateDeltaDataPointsInclude300Buckets-16     23.50Ki ± 0%   20.98Ki ± 0%  -10.70% (p=0.000 n=10)

                                                  │ before_300.txt │           after_300.txt            │
                                                  │   allocs/op    │ allocs/op   vs base                │
GetAndCalculateDeltaDataPointsInclude300Buckets-16       130.0 ± 0%   171.0 ± 0%  +31.54% (p=0.000 n=10)

Benchmark test with 500 bucket length:

goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter
cpu: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
                                                  │ before_500.txt │            after_500.txt            │
                                                  │     sec/op     │   sec/op     vs base                │
GetAndCalculateDeltaDataPointsInclude500Buckets-16      52.51µ ± 3%   58.15µ ± 5%  +10.74% (p=0.000 n=10)

                                                  │ before_500.txt │            after_500.txt             │
                                                  │      B/op      │     B/op      vs base                │
GetAndCalculateDeltaDataPointsInclude500Buckets-16     23.50Ki ± 0%   30.14Ki ± 0%  +28.26% (p=0.000 n=10)

                                                  │ before_500.txt │           after_500.txt            │
                                                  │   allocs/op    │ allocs/op   vs base                │
GetAndCalculateDeltaDataPointsInclude500Buckets-16       130.0 ± 0%   210.0 ± 0%  +61.54% (p=0.000 n=10)

[mxiamxia]

Please fix the PR checks

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[zzhlogin]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

Adding @Aneurysm9 for help on review.

[Aneurysm9]

Where are the benchmarks? I don't see them in the PR. The benchstat output is also missing the after and comparison data.

[zzhlogin]

Where are the benchmarks? I don't see them in the PR. The benchstat output is also missing the after and comparison data.

Sorry, failed to execute the comparison command before, added the after.txt output in the description now. The benchmark test is located in file "exporter/awsemfexporter/datapoint_test.go" at line 2075, 2076.

[songy23]

This leads to test failures, see #36727