RFC: Options to maximize field of view for video calls

[ASerbinski]

There are three aspects of camera configuration that result in cropped video output and a substantially reduced field of view;

1. Use of 16:9 aspect ratio instead of the sensor native 4:3

   talk-android/app/src/main/java/com/nextcloud/talk/activities/CallActivity.java

   Line 1745 in a43a51b

   videoCapturer.startCapture(1280, 720, 30);

2. Use of Electronic Image Stabilization (EIS)
3. Inability to control zoom ratio results in fixed ratio of 1:1.

The aspect ratio will affect devices universally, since as far as I can tell, all camera sensors seem to have a 4:3 native aspect ratio. The other two will apply on a sensor by sensor basis. Some phones have a mixture of available zoom ratios, and support for EIS or OIS (where OIS is available, EIS is disabled).

Naturally, there will be tradeoffs for each of the three causes of cropping;

1. User may have a preference for a 16:9 ratio.
2. Holding the device in user's hand will result in a jittery video without stabilization.
3. Changing the zoom ratio could cause the HAL to select a different sensor with different characteristics, for example, the main camera on a Pixel 6 [/Pro] has OIS, but reducing the zoom level to lower than 1:1 will cause it to switch to the wide angle lens, which does not have OIS.

Consequently, each of these configuration changes should be user-controlled.

Access to the camera on this software is through webrtc, which unfortunately, does not provide means of user control for EIS or zoom ratio, however, others have coerced it into allowing settings; https://stackoverflow.com/questions/48598752/android-how-to-turn-on-flashlight-using-webrtc-android-io-pristine-libjingle1

[ASerbinski]

I've implemented a proof of concept https://github.com/nextcloud/talk-android/tree/feature/2052/video-fov

Sets 4:3 aspect ratio (1280x960) at activities/CallActivity.java:1745
Adds custom implementations of webrtc classes Camera2Enumerator, CameraSession, and Camera2Capturer where in ExtCameraSession (CameraSession), zoom ratio is set to 0.5 at line 186, and EIS is switched off at line 210.

With this proof of concept, instead of user's face filling up the entire preview screen, now fills about 1/9th of the preview screen.

This is pretty hacky since it copy/pastes the entirety of org/webrtc/Camera2Session.java

[ASerbinski]

@AndyScherzinger : I'm not seeming to get any action on this RFC that's been up since May. Care to comment on where I'm going with this or details of how it should eventually be implemented?

[AndyScherzinger]

Argh, I'm sorry @ASerbinski this one totally slipped through :/ Sorry for that. I'll check with @AlvaroBrey and @timkrueger tomorrow, while from a high-level view I personally am fine with your approach. One thing that came to my mind, was that the branch is there but no PR yet (could be opened as draft) and these usually catch my eye easier - not an excuse or a requirement, just an explanation for not reacting to this one.

[timkrueger]

Let me also say sorry for that long delay @ASerbinski 😕

We took a look into your proof of concept and want to have this integrated. But we want more a detection of the parameters instead a configuration by the user.

Would you be willing to implement a real PR with needed guidance from our site? That would be the fastest path to get in Talk.

[ASerbinski]

@timkrueger - absolutely! The reason I haven't continued to work on this is because I don't want to waste a lot of time implementing something that is fundamentally different than what is required.

[timkrueger]

@timkrueger - absolutely! The reason I haven't continued to work on this is because I don't want to waste a lot of time implementing something that is fundamentally different than what is required.

That's really nice 😃

Please feel free to reach out when ever you need help or something to discuss. Here or in the Talk team public conversation.

[ASerbinski]

@timkrueger -- when you say "But we want more a detection of the parameters instead a configuration by the user", I'm not sure how that can be accomplished, since each of the changes really comes down to user preference.

1. does the user prefer to capture the full field of view (4x3) or to fit closer to the aspect ratio of a phone/tv (16x9)?
2. which does the user value more? Video stability or field of view? This could be situation dependent, if the user is walking around holding a phone in their hand, stabilization may be more valuable, but if the user has a phone in a desk cradle, field of view may be more valuable.
3. this can probably just be set to maximum zoomed out, but there is still a matter of preference if the user would like to adjust the zoom ratio.

[timkrueger]

At Nextcloud we try to avoid new settings where it's possible. We have many not so experienced users. And adding possibilities like choosing the aspect ration or en-/disable stabilization will results in questions.

The vision is that the best configuration is detected on the base of available device/sensors/camera information. What the best configuration is must be discussed and decided by us.
But I am aware that it can be very challenging. So maybe a good first step is to make a short list of devices of which we now the specifications and select good configurations for them. For all other devices we select a good-as-possible default. In a second step we replace the list by a auto-configuration feature.

[ASerbinski]

Hmm, I understand where you're coming from, but I don't understand how to take that and make something out of it. I don't believe that its a question of different configurations for different devices at all since it doesn't seem like there is anything to adjust to on different devices.

If you are strictly against user settings, then the patch, in my opinion, should be good as it is for the purpose of maximizing field of view, however, with the following compromises;

1. Video aspect ratio will be less conforming to most displays,
2. There will be NO image stabilization unless the camera supports OIS.

[timkrueger]

OK. Let me have a look tomorrow again.

You wrote for your solution:

This is pretty hacky since it copy/pastes the entirety of org/webrtc/Camera2Session.java

Maybe there is a chance to get rid of all that copy/pastes?

[ASerbinski]

Maybe there is a chance to get rid of all that copy/pastes?

I'll put some more time into trying to figure that out. This is a big part of what I didn't want to waste too much time on if it appeared that these changes wouldn't be accepted.

[timkrueger]

... then the patch, in my opinion, should be good as it is for the purpose of maximizing field of view, however, with the following compromises ...

I agree.

Then it would be great if you try to get rid of that copy/pastes. After that we can merge that in my opinion.

[ghost]

Hi everyone, just wanted to see if there's any updates on this? It sure would be nice to have this :) thank you all for the hard work.

[ASerbinski]

@andyxheli ; I'm a bit stumped on how to clear the code copy/paste, hence no real progress in a little while.

[ChildLearningClub]

Would this just maximize field of view, or allow for pinching to zoom in and out?

[ASerbinski]

@ChildLearningClub : pinch to zoom would make absolutely no sense in this context. It controls what the OTHER PARTY can see.

[ASerbinski]

@timkrueger ; I don't think it is possible to eliminate the copy/pastes. The code that has to be affected in order to set the zoom and video stabilization mode is deep in a quagmire of "private".

https://webrtc.googlesource.com/src/+/refs/heads/main/sdk/android/src/java/org/webrtc/Camera2Session.java#214

[ghost]

Hey all any updates on this ? I think where till waiting on @timkrueger