Tempo can't connect to memcached: err="lookup _._tcp.memcached:11211: no such host"

[Defozo]

Describe the bug
After running the example docker-compose.yml, the tempo can't connect to Memcached:
2024-11-19 17:29:33 level=warn ts=2024-11-19T16:29:33.631013695Z caller=memcached_client.go:259 msg="error updating memcache servers" err="lookup _._tcp.memcached:11211: no such host"

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo using docker-compose.yml from the examples
2. Check the logs

Expected behavior
I'd expect Tempo to connect to Memcached and properly store the traces.

Environment:

• Infrastructure: docker-compose, Windows, PC
• Deployment tool: docker-compose

Additional Context
I've tried connecting to Memcached from the tempo docker container and this worked.

[javiermolinar]

Hi, good catch!

The problem is this DNS SRV record lookup:

tempo/pkg/cache/memcached_client.go

Line 282 in f71c4c6

_, addrs, err := net.LookupSRV(c.service, "tcp", c.hostname)

This doesn't work when Memcached runs in docker-compose because, as far as I know, Docker’s DNS does not create SRV service records. Instead, it only creates basic A records (IP address mappings).

We have a workaround by setting addresses instead of host in the tempo.yaml when configuring memcached

https://grafana.com/docs/tempo/latest/configuration/#cache

Using this mode, it won't do a DNS SRV lookup; instead, it will connect directly using the Memcached client.

If you fancy a PR with this change, that would be great!