Tutorial for loading frames

[codeAC29]

I have created a simple repo for loading frames using either camera or video. You can also control the resolution of the grabbed frame. This repository can be found here.

• I think it will be helpful for anyone who wants to start using this cv package. Let me know if you all think it is worth adding my repository's link or the whole repository here.
• Capturing frames from camera works fine but loading each frame of a video takes non-uniform time. I have not been able to figure out the reason behind it. Let me know if anyone finds it. (You can see the timing values showing loading and displaying time for each frame using -v option)

[Edelbert]

Capturing frames from camera works fine but loading each frame of a video takes non-uniform time. I have not been able to figure out the reason behind it. Let me know if anyone finds it. (You can see the timing values showing loading and displaying time for each frame using -v option)

For video capturing, it should be something like this:

while _ do
_, frame = capture:read{frame}
end

[shrubb]

Hi @codeAC29 ,
Awesome! Thanks for this helpful tool, and sorry for a huge delay. See my pull request. Let's keep it separate under your ownership, I'll include links here!

[codeAC29]

Thanks @shrubb. Did you see notice that loading individual frame does not take uniform time? Do you see any reason behind it?
@Edelbert Does it stabilize loading of each frame from video file, because it didn't for me. Moreover, _, x = capture:read{x} and _, x = capture:read{} are only different in terms of the usage of self, which I don't think matters here.

[shrubb]

@codeAC29
Actually, one should use capture:read{x} whenever possible. _, x = capture:read{} allocates a new Tensor (obviously), while using the former line even lets you not to collectgarbage() every K frames.

As for loading times, the fluctuations are normal when working with camera, especially with a simple webcam. In particular, FPS rate often drops in low light. It should be stable with video files (tested myself).

[codeAC29]

Thats weird, because in my case (repo which I have added) fps is fluctuating for videos and not camera and is somewhat periodic; thats why I was wondering if I made any fundamental mistake in my repo.

[shrubb]

Hmm, strange indeed, because I've tested just your script, and didn't have any problems with it. Also, your code seems to be OK. Probably a video file or an FFMPEG issue 😕

[Edelbert]

@codeAC29 it stabilized loading frames from video for me, because I use _ as condition for the loop. It returns false when the end of the video is reached, however x returns the last frame. You can check it by saving frames to the hard-drive. But I don't have any problems with loading individual frames.
@shrubb oh, thanx. I will use the amount of frames as terminal condition. I tried to fix this strange behaviour of 'last frame problem', but the code seems OK for me. I have this issues for any video (I think so, I cannot check it right know).

[shrubb]

@Edelbert
Do you mean the problem of knowing that you've reached the end of a video? If yes, then check out the following snippet. Should be a good place to start:

local cv = require 'cv'
require 'cv.videoio'
require 'cv.highgui'

cap = cv.VideoCapture{'/some/movie'}
local ok, frame = cap:read{}

while ok do

    -- do something with frame, for example, show it
    cv.imshow{'Window by OpenCV', frame}
    cv.waitKey{20}

    ok = cap:read{frame}
end

[codeAC29]

I am using win:valid() here because in case user closes the window in which the frames are being displayed (while grabbing frame from camera or before a video is complete), I want the code to exit.
Anyways what I wanted to ask was, when I run my script with -v option, for videos I get the following result:

Reading:    364.8 ms
Displaying: 18.7 ms
fps:        2.61
Reading:    314.8 ms
Displaying: 18.1 ms
fps:        3.00
Reading:    85.2 ms
Displaying: 16.5 ms
fps:        9.83
Reading:    407.3 ms
Displaying: 21.8 ms
fps:        2.33
Reading:    371.4 ms
Displaying: 15.7 ms
fps:        2.58
Reading:    69.7 ms
Displaying: 15.7 ms
fps:        11.70
Reading:    66.5 ms
Displaying: 15.4 ms
fps:        12.21
Reading:    452.0 ms
Displaying: 15.5 ms
fps:        2.14
Reading:    60.0 ms
Displaying: 18.3 ms
fps:        12.76
Reading:    435.1 ms
Displaying: 16.0 ms
fps:        2.22
Reading:    75.8 ms
Displaying: 15.1 ms
fps:        10.99
Reading:    44.9 ms
Displaying: 18.9 ms
fps:        15.67
Reading:    70.0 ms
Displaying: 17.1 ms
fps:        11.47
Reading:    491.4 ms
Displaying: 15.0 ms
fps:        1.97

If you will notice, fps keeps on fluctuating which is mainly because of fluctuation during reading of frames. While loading frames from camera there are no such spikes in fps.

@shrubb did you try to load a video, and did it get displayed smoothly for you?

[shrubb]

Yes, as I've said above, I can play videos smoothly with just your script.
Can you try the same in C++ and with different files?

[ShuangjunLiu]

My torch cv is not functioning well.
opencv is from pip install.
then
luarock install cv.

The cap can't open the video file even avi.

Can you let me know your torch cv install process exactly?
So I can try your way.

Thanks.

[shrubb]

@junjun16
Looks like the trouble is in pip OpenCV install, not luarocks cv. While your OpenCV was installing, it seems to haven't picked up ffmpeg/gstreamer. Perharps you don't have them installed.

[ShuangjunLiu]

@shrubb I have fixed it. Solution is as follows:

1. Build opencv from source.
2. You also have to inform luarocks explicitly where opencv is during luarocks install.

Thanks.