Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manually calculate x-offset #75

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Manually calculate x-offset #75

wants to merge 1 commit into from

Conversation

taichino
Copy link
Collaborator

@taichino taichino commented Jan 28, 2021

Fix for #33

This PR fixes an issue that the rendered result is off-centered when using the Metal rendering pipeline.
The root problem is the frame we receive from AVCaptureVideoDataOutput has a different resolution from our output frame.

The frame's resolution from AVCaptureVideoDataOutput is 1080 x 1920 while the output frame size is 887 x 1920 on iPhone 11 Pro Max, which is bound to the device's screen size. (We're calculating the output size so it has the same height as input.)

We create textures for both input and output, then send both textures to shaders to let shaders apply filters to the input texture and write the result to the output texture.

What we want to do essentially is to crop the input texture then apply filters. In the iPhone 11 Pro Max's case, we want to crop 96.5 pixels from each side.

explanation

The first approach I looked for was to change the input resolution. If we can receive a frame with the same dimension as the output, we don't have to deal with this problem at all. But looks like AVFoundation doesn't allow us to do it. (We can actually choose resolutions from pre-defined options, but we cannot set exact size)

The second approach I looked for was to find a metal shader function that sets certain offsets to textures. With OpenGL, we can configure to make input texture centered. But with Metal's kernel shaders, I was not able to find an easy way to do the same.

So we need to crop the input frame somehow somewhere in the pipeline and turned out it's not easy(In terms of processing power) to crop CVPixelBuffer. Essentially we need to create a new CVPixelBuffer instance with an output size then re-render the input image onto the new CVPixelBuffer instance. (Let me know if I missed something)
And I didn't want to perform that process in the CPU land as it is executed in every frame.

Long story short, I decided to implement a pseudo cropping logic (getAdjustedPosition) in shaders. Yes, it looks messy as every shader function needs to use it. But at least it is very light and it seems to be working well.

I still feel there should be a better way, but this is the best I can do at least for now.

How to test

Use filters with Metal feature flags on and make sure all perform as expected.

@taichino
Copy link
Collaborator Author

Hey @bjtitus , any chance you can review this PR?

Copy link
Collaborator

@bjtitus bjtitus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. I believe I saw this on an iPhone 11 but my test device doesn't exhibit the problem. All of the filters look good, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants