Debugging redis connections, where are the "bull:" connections coming from?

[tommoor]

When running CLIENT LIST it shows a bunch of connections from bull in the form:

id=826 addr=172.18.0.1:59494 fd=11 name=bull:d2Vic29ja2V0cw== age=100 idle=5 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=brpoplpush user=default
id=827 addr=172.18.0.1:59496 fd=12 name=bull:Z2xvYmFsRXZlbnRz age=100 idle=5 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=brpoplpush user=default
id=828 addr=172.18.0.1:59498 fd=13 name=bull:dGFza3M= age=100 idle=5 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=brpoplpush user=default
id=829 addr=172.18.0.1:59500 fd=14 name=bull:cHJvY2Vzc29yRXZlbnRz age=100 idle=5 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=brpoplpush user=default

I don't understand this for a couple of reasons:

• We're using the pattern here to reuse connections. (source)
• connectionName does not appear anywhere in the bull codebase, so how can it be naming these connections in this format.

Any pointers appreciated

[manast]

Could you be a bit more specific on what is your question?

[manast]

Here some information that can help you understand: https://docs.bullmq.io/guide/connections

[tommoor]

Hi @manast appreciate your time but that doesn't answer the question and also is documentation for the other project not this one.

I understand the connection management aspect and am providing Bull with the Redis connections (see "source" link in original question), however these are additional connections that cannot be accounted for. If you're able to point to where these bull:<id> connections are created perhaps that would be enough info to debug further

[manast]

It is a different project but the information regarding connections is still relevant as it works pretty much the same. It would be much easier if you can paste a simple piece of code that you expect to have certain number of connections and then we can take it from there. The source is also open so you can peek to see where the connections are created and so on.

[tommoor]

It would be much easier if you can paste a simple piece of code

I'll link it again, this is the code with all context – it changed a little since yesterday as I added connectionName to try and aid in debugging this issue: https://github.com/outline/outline/blob/main/server/utils/queue.ts#L16-L32

My understanding was that this would result in a maximum of 3 connections per process, and those would be the three listed inside createClient, but that seems to be wrong, because I'm seeing 6 connections – client, subscriber and 4 bull: connections that I have no idea where they are coming from 😄 . This is blowing over our redis connection limit when multiplied by the amount of servers and threads, hence my investigation.

id=27563 addr=5.99.37.225:51694 fd=48 name=outline:worker:subscriber age=40862 idle=1 flags=N db=0 sub=4 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=27558 addr=5.99.37.225:51670 fd=40 name=outline:worker:client age=40862 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=27559 addr=5.99.37.225:51674 fd=41 name=bull:Z2xvYmFsRXZlbnRz age=40862 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=27560 addr=5.99.37.225:51676 fd=42 name=bull:cHJvY2Vzc29yRXZlbnRz age=40862 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=27561 addr=5.99.37.225:51678 fd=46 name=bull:c3Vic2NyaXB0aW9ucw== age=40862 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=27562 addr=5.99.37.225:51680 fd=47 name=bull:dGFza3M= age=40862 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush

[tommoor]

Ah, it's one per queue. I'm not sure why the naming is ignoring the connectionName I'm passing to ioredis though, that was really the source of the confusion here.

I need to find a good pattern for not calling createQueue in processes where the individual queues aren't used then