Reset FfmpegOutput incase of Broken Pipe?

[kevinjacb]

Hello folks,
Lately, I've been streaming video using ffmpegOutput with RTSP protocol, but if there happens to be slight network issue, it results in a broken pipe. Now this issue as far as I know, can be resolved by stopping the stream and the camera object, but that messes with the recording that happens in the background, is there any way to handle this error and restart streaming without interfering with the overall camera process? Thank you.

[davidplowman]

Would it help if the "broken" output could be abandoned and the application could start a replacement output?

[kevinjacb]

Absolutely, that should work. I am trying to avoid restarting the entire script as a whole when this happens, but the only way I've been able to do this is by restarting the picam process which interferes with the recording process.

[kevinjacb]

Hello again @davidplowman , to solve the issue of having to restart the picam module, I suggest we wrap

def outputframe(self, frame, keyframe=True, timestamp=None):
        if self.recording:
            self.ffmpeg.stdin.write(frame)
            self.ffmpeg.stdin.flush()  # forces every frame to get timestamped individually
            self.outputtimestamp(timestamp)

(in picamera2/outputs/ffmpegoutput.py) in a try-catch block, this prevents the error from causing further issues and the broken pipe can be detected like so in the main loop:

# HERE stream_ouput = FfmpegOutput('-f rtsp rtsp://127.0.0.1:8554/test') or something similar

while True:
        try:
            if stream_output.ffmpeg == None or stream_output.ffmpeg.poll() != None:
                print('stream output not running')
                stream_output.start()
                print('stream output restarted')
        except Exception as e:
            print(traceback.format_exc())
            time.sleep(0.5)

what do you think? This works for me.

[davidplowman]

Hi, I think that sounds like a good idea. I wonder if we should do a little more, for example, allowing users to attach a callback to their output object which can get invoked if there's an error. This might even avoid the need for polling - would that be an improvement?

[kevinjacb]

That sounds great! Maybe even add a retry option. May I make the necessary changes and open an PR?

[kevinjacb]

Hello @davidplowman, would this fix suffice? Is there something else you'd like to add?

    def outputframe(self, frame, keyframe=True, timestamp=None):
        try:
            if self.recording:
                self.ffmpeg.stdin.write(frame)
                self.ffmpeg.stdin.flush()  # forces every frame to get timestamped individually
                self.outputtimestamp(timestamp)
        except Exception as exception:
            self.write_error = True
            if self.err_callback is not None:
                self.err_callback(exception)
            else:
                raise exception

init function:

   def __init__(self, output_filename, audio=False, audio_device="default", audio_sync=-0.3,
                 audio_samplerate=48000, audio_codec="aac", audio_bitrate=128000, pts=None, err_callback = None):
        super().__init__(pts=pts)
        self.ffmpeg = None
        self.output_filename = output_filename
        self.audio = audio
        self.audio_device = audio_device
        self.audio_sync = audio_sync
        self.audio_samplerate = audio_samplerate
        self.audio_codec = audio_codec
        self.audio_bitrate = audio_bitrate

        # error section
        self.err_callback = err_callback

[davidplowman]

Hi, yes I think that looks good. I'd be against adding automatic retries, TBH.

You could submit a PR if you wanted, but check the contribution guidelines. Alternatively, as it's a very small change I'd be happy to do it if you could wait a week or so for the update to appear. There's some other work going on at the moment so there should be a Picamera2 update anyway in that kind of timeframe.

[kevinjacb]

Hello again @davidplowman, I forgot to mention that the ffmpeg processes aren't killed, I think, when the error occurs but new ones are spawned when restarted which leads to memory leaks. I suppose self.ffmpeg.terminate() does the job?

[davidplowman]

Probably, try it and see!

Obviously you'll have seen in the press why we've been quite so busy of late. There will be updates to all our camera software soon so that they support both older Pis and the new Pi 5. Once we're through that I'll make those changes, if you can test stuff and let me know exactly what the necessary changes are, that would be helpful. Thanks!

[kevinjacb]

Sure, will do, waiting for rpi 5 :)