How to seek a specific frame quickly?

[Akascape]

I want to get a particular video frame using pyav. I found that container.seek method only seeks to time(int) offset, so it will not help in finding the required frame.

Perhaps, I used this method to get the required frame:

container.seek(0, whence='time', backward=True) # seek to the starting
frame_num = 30  # the frame I want to seek
time.sleep(0.2) # limit cpu usage
for _ in range(frame_num): 
    frame = next(container.decode(video=0))
return frame

This method works properly but it is too slow while seeking the last frames of any long video file. (with frames>1000)

I tried to solve this issue by modifying the seek method :

frame_num = 30
framerate = container.streams.video[0].average_rate # get the frame rate
sec = int(frame_num/framerate) # timestamp (sec) for that frame_num
container.seek(sec*1000000, whence='time', backward=True)  # seek to that nearest timestamp
sec_frame = sec * framerate # get the key frame number of that timestamp
       
for _ in range(sec_frame, frame_num):
    frame = next(container.decode(video=0))
return frame

This reduces the time while seeking any frame of the video, but it doesn't return the proper frame because container.seek method doesn't seek to the proper timestamp. I found that container.seek is giving the same key frame for consecutive 3 seconds. I don't know if I am missing something...

Is there any other way to get the required frame with pyav?

[Akascape]

So, I somehow managed to fix the lag issue:

frame_num = 1000 # the frame I want
framerate = container.streams.video[0].average_rate # get the frame rate
time_base = container.streams.video[0].time_base # get the time base
sec = int(frame_num/framerate) # timestamp for that frame_num
container.seek(sec*1000000, whence='time', backward=True)  # seek to that nearest timestamp
frame = next(container.decode(video=0)) # get the next available frame
sec_frame = int(frame.pts * time_base * framerate) # get the proper key frame number of that timestamp

for _ in range(sec_frame, frame_num):
    frame = next(container.decode(video=0))
return frame

Now it returns the required frame in less than a second.

[Scr44gr]

@pauljurczak @haoxurt I guess is the version, which version did u use? I test with 11.0.0 and it works

[pauljurczak]

pyav 12.0.5. I'm curious why backward=True? Shouldn't seek search forward to find frame 622 after opening the video file?

[robonrrd]

I am using 13.1.0 and have the same issue with only getting the first frame

[Max-Thuc]

the constant 1000000 is different for each video

[FirefoxMetzger]

@Max-Thuc is correct, you need to account for the timebase of the video stream when converting your time-offset from seconds to stream time.

Even then, you also have the issue that some containers (most actually) don't space out frames at regular intervals. Instead they dynamically create frames when enough change in the image has occurred so that a new frame is needed to keep the video quality within the specified range. In other words, stepping forward N times after checking where the seek got you may not put you where you think you end up neither.

[robinechuca]

A more reliable but more complex way consists in extracting the position of the keyframes:

def _extract_key_frames(av_stream: av.video.stream.VideoStream):
    av_stream.container.seek(0, backward=True, any_frame=False, stream=av_stream)
    av_stream.codec_context.skip_frame = "NONKEY"
    yield from av_stream.container.decode(av_stream)
    av_stream.container.seek(0, backward=True, any_frame=False, stream=av_stream)
    av_stream.codec_context.skip_frame = "DEFAULT"

key_times: np.ndarray[Fraction] = np.fromiter(
    (frame_dates(frame)[0] for frame in _extract_key_frames(stream)),
    dtype=object,
)

You seek to the previous keyframe (accurate), then you decode each frame until reaching the write timestamp.
Here an example of the full implementation of this process (tested on different video formats):

from cutcutcodec.core.io import read
with read("my_video_file.mp4") as container:
    stream = container.out_select("video")[0]
    frame = stream.snapshot(120, (stream.height, stream.width))   # seek and get frame at time 2 min

In addition to seeingker directly on keyframes, the proposed code performs verifications to ensure that the position after seek is correct. If it isn't, it seeks further back until it does.

It's up to you to take inspiration from the source code and adapt it to your own more specific project.