FFmpegFrameGrabber only grabs keyframes, skips frames in between

[yervantk]

I'm running a basic grab on OSX. It's a Kotlin project, not on Android. Here's the code:

    private val converter = Java2DFrameConverter()
    private val frameGrabber = FFmpegFrameGrabber(mp4Path)
    private val frameRate: Double

    init {
        frameGrabber.start()
        frameRate = frameGrabber.frameRate
        println("Video has " + frameGrabber.lengthInFrames + " frames and has frame rate of " + frameRate)
    }

    fun grabFrame(frameNum: Int): BufferedImage? {
        try {
            frameGrabber.frameNumber = frameNum

            val frame = frameGrabber.grabImage()
            return converter.convert(frame)

        } catch (e: Exception) {
            e.printStackTrace()
        }

        return null
    }

It captures the same frame 6 or 7 times in a row, then repeats. When I debugged the grabber, it seems it only sends back keyframes. All frames had keyFrame = true.

The line where it may be messing up is 1276:

javacv/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java

Line 1276 in fe48427

if (doVideo && video_st != null && pkt.stream_index() == video_st.index()

When setting the frameNumber, it calls grabFrame(), and the value of this.pkt.stream_index() is not the same as this.video_st.index().

This means that it enters that condition, and not the next closure where it would have called
this.frame.image = this.image_buf.

Because of this, the this.frame.image is null by the time it gets to actually grabbing the frame, i.e. not while incrementing the timestamp. It falls back to returning the most recent keyframe. That's my guess.

I've looked through all the other bugs and haven't found anyone having a similar situation.

[saudet]

It sounds like your file has multiple video streams? That's not currently supported. Check with ffprobe to make sure that's not the case.

[yervantk]

This is the probe

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/abc/Downloads/Movie.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2014-07-21T18:50:13.000000Z
  Duration: 00:22:58.00, start: 0.000000, bitrate: 1757 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 1625 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
    Metadata:
      creation_time   : 2014-07-21T18:50:13.000000Z
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      creation_time   : 2014-07-21T18:24:37.000000Z

Looks like there is only one video stream.

[saudet]

But you're saying the value of pkt.stream_index changes? What does it change to?

[yervantk]

this.pkt.stream_index = 1
this.video_st.index = 0

[saudet]

Those are audio packets...

[yervantk]

I can't explain why it is trying to access audio packets when I try to grabImage(). The code is exactly as listed above.

Now that I know what's happening, I'm going to dig into it and try to see why/when it's getting the audio stream instead of video.

Edit: this happened on more than one video, from two different sources.

[saudet]

Any updates on that?

[saudet]

Since you're the only person that is having this issue, I'm assuming that this isn't a problem with JavaCV, but simply something that you're doing incorrectly. If you have some sample source code with some sample data that fails, please share it so that someone can help. Thanks!