Inconsistent 2 way audio

[JakeMHughes]

I am trying to configure two way audio for a camera I picked up off amazon from a no-name brand Anpviz with varying amounts of success, but I finally got it working but the issue I am seeing now is that it only works sometimes

Basically after I make changes to the config and restart I'll have about 40% chance for the mic portion ( me speaking to computer and it plays on camera speaker) to work. then if I restart again, even if it was working before it may not work again despite the config being the same.

Something Ive observed is if I quickly probe the camera immediately after a restart the probe will include the line "audio, sendonly, PCMU/8000, G726/8000, MPEG4-GENERIC/16000" - but in the times where the mic does not work Ill notice that no longer shows up in the probe. Another observation is if I ave two streams with the exact same config, sometimes it will work on one stream but not the other

Heres everyting I have

Camera audio: G.711ulaw (tried multiple encodings this is the only one that came out clear, others like AAC nothing worked and some just crackled)
go2rtc version: 1.9.8 via frigate (I was using 1.9.2 but noticed a bit better consistency on he newer version)
go2rtc config:

streams:
    # one:
    #   - rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
  two:
  - rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
webrtc:
  candidates:
      #- frigate.security:8555
  - 192.168.0.21:8555
  - stun:8555

  listen: :8555
api:
  origin: '*'
hass:
  config: /config
log:
  format: text
rtsp:
  default_query: mp4
ffmpeg:
  bin: /usr/lib/ffmpeg/7.0/bin/ffmpeg

camera probe when working:

  "producers": [
    {
      "id": 2,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "192.168.0.15:554",
      "url": "rtsp://admin:[email protected]:554/cam/realmonitor?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif",
      "sdp": "v=0\r\no=- 1737993392252127 1737993392252127 IN IP4 192.168.0.15\r\ns=Media Presentation\r\ne=NONE\r\nb=AS:5150\r\nt=0 0\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\nm=video 0 RTP/AVP 96\r\nc=IN IP4 0.0.0.0\r\nb=AS:5000\r\na=recvonly\r\na=x-dimensions:1920,1080\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/trackID=1?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 profile-level-id=420029; packetization-mode=1; sprop-parameter-sets=Z00AM4qKcDwBE/L/4AAgAC2QAAA+gAAJxABA,aO48gA==\r\nm=audio 0 RTP/AVP 0\r\nc=IN IP4 0.0.0.0\r\nb=AS:50\r\na=recvonly\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/trackID=2?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\na=rtpmap:0 PCMU/8000\r\nm=audio 0 RTP/AVP 0 102 104\r\na=rtpmap:0 PCMU/8000/1\r\na=rtpmap:102 G726/8000/1\r\nc=IN IP4 0.0.0.0\r\nb=AS:50\r\na=sendonly\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/trackID=4?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\na=rtpmap:104 mpeg4-generic/16000/1\r\na=fmtp:104 profile-level-id=15; streamtype=5; mode=AAC-hbr; config=1408;SizeLength=13; IndexLength=3; IndexDeltaLength=3; Profile=1;\r\na=Media_header:MEDIAINFO=494D4B48010300000400000110710110401F000000FA000000000000000000000000000000000000;\r\na=appversion:1.0\r\n",
      "user_agent": "go2rtc/1.9.8",
      "medias": [
        "video, recvonly, H264",
        "audio, recvonly, PCMU/8000",
        "audio, sendonly, PCMU/8000, G726/8000, MPEG4-GENERIC/16000"
      ],
      "receivers": [
        {
          "id": 3,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 51,
            "profile": "Main"
          },
          "childs": [
            4
          ]
        },
        {
          "id": 5,
          "codec": {
            "codec_name": "pcm_mulaw",
            "codec_type": "audio",
            "sample_rate": 8000
          },
          "childs": [
            6
          ]
        }
      ],
      "senders": [
        {
          "id": 8,
          "codec": {
            "codec_name": "pcm_mulaw",
            "codec_type": "audio",
            "sample_rate": 8000
          },
          "parent": 7
        }
      ]
    }
  ],
  "consumers": [
    {
      "id": 1,
      "format_name": "probe",
      "protocol": "http",
      "remote_addr": "[::1]:45210",
      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
      "medias": [
        "video, sendonly, ALL",
        "audio, sendonly, ALL",
        "audio, recvonly, ANY"
      ],
      "receivers": [
        {
          "id": 7,
          "codec": {
            "codec_name": "ANY",
            "codec_type": ""
          },
          "childs": [
            8
          ]
        }
      ],
      "senders": [
        {
          "id": 4,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 51,
            "profile": "Main"
          },
          "parent": 3
        },
        {
          "id": 6,
          "codec": {
            "codec_name": "pcm_mulaw",
            "codec_type": "audio",
            "sample_rate": 8000
          },
          "parent": 5
        }
      ]
    }
  ]
}

and finally a probe when it is not working

{
  "producers": [
    {
      "id": 46,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "192.168.0.15:554",
      "url": "rtsp://admin:[email protected]:554/cam/realmonitor?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif",
      "sdp": "v=0\r\no=- 1737994836403324 1737994836403324 IN IP4 192.168.0.15\r\ns=Media Presentation\r\ne=NONE\r\nb=AS:5100\r\nt=0 0\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\nm=video 0 RTP/AVP 96\r\nc=IN IP4 0.0.0.0\r\nb=AS:5000\r\na=recvonly\r\na=x-dimensions:1920,1080\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/trackID=1?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 profile-level-id=420029; packetization-mode=1; sprop-parameter-sets=Z00AM4qKcDwBE/L/4AAgAC2QAAA+gAAJxABA,aO48gA==\r\nm=audio 0 RTP/AVP 0\r\nc=IN IP4 0.0.0.0\r\nb=AS:50\r\na=recvonly\r\na=control:rtsp://192.168.0.15:554/cam/realmonitor/trackID=2?channel=1\u0026subtype=0\u0026unicast=true\u0026proto=Onvif\r\na=rtpmap:0 PCMU/8000\r\na=Media_header:MEDIAINFO=494D4B48010300000400000110710110401F000000FA000000000000000000000000000000000000;\r\na=appversion:1.0\r\n",
      "user_agent": "go2rtc/1.9.8",
      "medias": [
        "video, recvonly, H264",
        "audio, recvonly, PCMU/8000"
      ],
      "receivers": [
        {
          "id": 47,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 51,
            "profile": "Main"
          },
          "childs": [
            10,
            54
          ],
          "bytes": 450955379,
          "packets": 325541
        },
        {
          "id": 48,
          "codec": {
            "codec_name": "pcm_mulaw",
            "codec_type": "audio",
            "sample_rate": 8000
          },
          "childs": [
            55
          ],
          "bytes": 7280640,
          "packets": 22752
        }
      ],
      "bytes_recv": 462443484
    }
  ],
  "consumers": [
    {
      "id": 9,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:44660",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.8\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 profile-level-id=420029; packetization-mode=1; sprop-parameter-sets=Z00AM4qKcDwBE/L/4AAgAC2QAAA+gAAJxABA,aO48gA==\r\na=control:trackID=0\r\n",
      "user_agent": "FFmpeg Frigate/0.15.0-5943fc1",
      "medias": [
        "video, sendonly, H264, H265",
        "audio, sendonly, MPEG4-GENERIC"
      ],
      "senders": [
        {
          "id": 10,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 51,
            "profile": "Main"
          },
          "parent": 47,
          "bytes": 505094161,
          "packets": 364635
        }
      ],
      "bytes_send": 510928321
    },
    {
      "id": 53,
      "format_name": "probe",
      "protocol": "http",
      "remote_addr": "10.1.36.144:49660 forwarded 192.168.0.30",
      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
      "medias": [
        "video, sendonly, ALL",
        "audio, sendonly, ALL",
        "audio, recvonly, ANY"
      ],
      "senders": [
        {
          "id": 54,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 51,
            "profile": "Main"
          },
          "parent": 47
        },
        {
          "id": 55,
          "codec": {
            "codec_name": "pcm_mulaw",
            "codec_type": "audio",
            "sample_rate": 8000
          },
          "parent": 48
        }
      ]
    }
  ]
}

[JakeMHughes]

Forgot to note, the only error/warning I see in logs for go2rtc is 16:20:19.641 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:170 > error="start from CONN state" url=rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif

[AlexxIT]

Only one RTSP source can take two way audio. Even if you don't need a microphone, it will still be busy.

Setting RTSP not to occupy the microphone:

streams:
  camera1: rtsp://...#backchannel=0

[JakeMHughes]

Only one RTSP source can take two way audio. Even if you don't need a microphone, it will still be busy.

Setting RTSP not to occupy the microphone:

streams:
  camera1: rtsp://...#backchannel=0

While that is good to know, the inconsistency issue still occurs with just a single camera in the stream. Maybe I'll retry it with ffmpeg since theres a snippet in the docs regarding glitchy cameras and when I tested it previously, I did have multiple streams configured on one camera

[AlexxIT]

With ffmpeg source you won't get two way audio.

[JakeMHughes]

Ohhh, I was under the impression that the snippet below for RTSP was still in regards of two way audio

If the stream from your camera is glitchy, try using ffmpeg source. It will not add CPU load if you won't use transcoding

[JakeMHughes]

So curiously, I think frigate may be the cause for the issue. On frigate I configured the go2rtc stream but I never configured it in frigates "Cameras" section and its consistently allowing my mic to work.

If I configure it in the cameras section in frigate, that's when the inconsistency issue happens. im curious if frigates "detect" or "record" roles are consuming the receive audio function (since you mentioned only one consumer at a time) so then I am unable to access it in frigates live view

I'll play with it some more, but in the mean time I just set it up with the mic in home assistant directly (skipping frigate) so it will work like that for now

[AlexxIT]

One more time. It doesn't matter if you consume sound. And whether you need two way audio or not.

Simply connecting to RTSP in "two way audio support mode" will already block the ability to connect to the camera in that mode from another location.

By default go2rtc will always connects to RTSP camera in two way audio mode.

[JakeMHughes]

And I understood that.

Ultimately I want two way audio to work through frigate, but for a camera to appear on frigates "live view" you need to add it to the cameras section in frigate config. By doing so you have to configure a role - detect,record,or audio. no matter which one you pick frigate will always be connected to the camera and it counts as that one connection

So for example, if I configure the camera in frigate like below with no other connection

go2rtc:
  streams:
    twoway:
      - rtsp://admin:[email protected]:554/cam/realmonitor?channel=1&subtype=0
  cam:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/twoway
          roles:
            - record
    detect:
      enabled: false

then if I dont load the UI and just go to go2rtc page - gortc will always show 1 consumer on the stream. If I do load frigate UI it will show 2 connected consumers ( it doesnt reuse the other stream)

And finally if I configure the only stream with backchannel=0, it seems to apply that same config when I access the camera through frigrate UI

Basiaclly I believe you solved my first original issue by providing information, but I need to reach out to frigate team to find out ow I can only enable two way for live view