fix: broken store config in grafana loki

[takirala]

What problem does this PR solve?:

When looking at the production cluster due to an unrelated issue @fatz raised, i found that the loki compactor was printing the following error

❯ k logs -nkommander grafana-loki-loki-distributed-compactor-0 -f
level=error ts=2024-11-19T23:16:56.703832344Z caller=compactor.go:523 msg="failed to run compaction" err="index store client not found for aws"
level=error ts=2024-11-19T23:26:56.703352606Z caller=compactor.go:523 msg="failed to run compaction" err="index store client not found for aws"
level=error ts=2024-11-19T23:36:56.702274812Z caller=compactor.go:523 msg="failed to run compaction" err="index store client not found for aws"
level=error ts=2024-11-19T23:46:56.704304251Z caller=compactor.go:523 msg="failed to run compaction" err="index store client not found for aws"

it is printed regularly at an interval of 10m which happens to be the default compaction interval - indicating that we aren't compacting. Acc. to grafana/loki#10554:

The issue is indeed caused by adding the multi-store support to the compactor. The compactor does not treat aws as an alias for s3 and tries to find an index store client for aws (which does not exist).
The object_type value in the period config needs to match the value of shared_store in the *_shipper config. In your case, either both aws or both s3.

I looked at the daily cluster and we still have this bug in 2.13 (so 2.8 to 2.13 - all are affected). This is before the fix:

sh-4.4$ ./s5cmd --endpoint-url http://rook-ceph-rgw-dkp-object-store.kommander.svc:80 --credentials-file creds ls -H s3://dkp-loki/index/loki_index_20046/
2024/11/19 18:02:50            184.5K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732038592.gz
2024/11/19 18:17:50            180.0K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732039200.gz
2024/11/19 18:32:50             77.5K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732040100.gz
2024/11/19 18:47:50             82.5K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732041000.gz
2024/11/19 19:02:50            101.4K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732041900.gz
2024/11/19 19:17:50            114.8K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732042800.gz
2024/11/19 19:32:50            444.3K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732043700.gz
2024/11/19 19:47:50            168.1K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732044600.gz
2024/11/19 20:02:50            102.6K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732045500.gz
2024/11/19 20:17:50            101.2K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732046400.gz
2024/11/19 20:32:50             94.7K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732047300.gz
2024/11/19 20:47:50            104.6K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732048200.gz
2024/11/19 21:02:50             93.2K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732049100.gz
2024/11/19 21:17:50             97.7K  grafana-loki-loki-distributed-ingester-0-1732036730465239319-1732050000.gz
...
...
..

After applying the patch from this PR, the compactor runs successfully and this is the update:

./s5cmd --endpoint-url http://rook-ceph-rgw-dkp-object-store.kommander.svc:80 --credentials-file creds ls -H s3://dkp-loki/index/loki_index_20046/
2024/11/20 04:12:03              3.9M  compactor-1732075922.gz

(for those curious, in my cluster setup - it compacted from 5.2Mb to 3.9M approx.)

Not compacting will not cause any data loss but consumes unncessary storage. This needs to be backported all the way back to 2.8 (and 2.7 even TBD)

Which issue(s) does this PR fix?:

https://jira.nutanix.com/browse/NCN-104281

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Checklist

• If the PR adds a version bump, ensure there is no breaking change in Licensing model (or NA).
• If a chart is changed or app configuration is significantly changed, the chart version is correctly incremented (so that apps are not automatically upgraded from a previous version of DKP).

[coveralls]

Pull Request Test Coverage Report for Build 11926650890

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 51.703%

---

Totals
Change from base Build 11923434883:	0.0%
Covered Lines:	167
Relevant Lines:	323

---

💛 - Coveralls

[msdolbey]

👍
Love this so much

[mesosphere-ci]

💚 All backports created successfully

Status	Branch	Result
✅	release-2.7
✅	release-2.8
✅	release-2.12

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details