Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix input queueing when some of the inputs do not produce frames/samples #625

Merged
merged 1 commit into from
Jul 19, 2024

Conversation

wkozyra95
Copy link
Member

@wkozyra95 wkozyra95 commented Jul 16, 2024

Steps to reproduce:

  • start more than one stream when one of them is shorter than the others
  • when the first stream is finished other streams might disappear

What happens:

  • normally queue is
    • checking if all inputs are ready (and pulling from channel if necessary)
    • for each input:
      • dropping old frames
      • taking first frame (first not dropped should be the one we want to use)
  • if one input is done without sending EOS, but just stops sending data
    • checking if all inputs are ready is lazy, so if it checks that one specific input that ended it does not check any further
    • for each input:
      • dropping old frames will work, but because previous step did not enqueue new frames from the channel we are left with one outdated frame
      • taking first frame return old frame (latter renderer can see that frame is older by more than 500ms and drops it, replacing with a black screen)

Alternative approach could be to replace call to any https://github.com/membraneframework/live_compositor/blob/master/compositor_pipeline/src/queue/video_queue.rs#L103 with non-lazy alternative, but I think this is cleaner.

@wkozyra95 wkozyra95 force-pushed the @wkozyra95/fix-queue-scheduling-on-input-end branch from 2f0ecbb to 61c5021 Compare July 16, 2024 10:33
@wkozyra95 wkozyra95 changed the title Fix frame/samples scheduling when one of inputs does not produce frames Fix input queueing when some of the inputs do not produce frames/samples Jul 16, 2024
@wkozyra95 wkozyra95 force-pushed the @wkozyra95/fix-queue-scheduling-on-input-end branch from 61c5021 to ca75417 Compare July 16, 2024 10:34
@wkozyra95 wkozyra95 force-pushed the @wkozyra95/fix-queue-scheduling-on-input-end branch from ca75417 to fac9f7f Compare July 16, 2024 10:40
@wkozyra95 wkozyra95 marked this pull request as ready for review July 16, 2024 10:42
Comment on lines +156 to +157
// ignore result, we only need to ensure samples are enqueued
self.check_ready_for_pts(pts_range, queue_start);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's awful. I hate it. Over and over we rely on some internal behavior of this function (enqueuing frames / reading them from the channel).

I'll try try to simplify this logic, cause reading all the queue code slowly feels like reading brainfuck.

Copy link
Member Author

@wkozyra95 wkozyra95 Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a tradeoff we can either:

  • check_ready_for_pts even though we don't need return value
  • write function that is almost identical, or inline that logic inside this function
  • ensure that when we check all inputs that check_ready_for_pts is always called for all inputs

My reasoning here is that:

  • approach 1 is very safe, because it's "public` method that can be used atomically. Safe here means that it does not break anything, but still does not grantee that enqueue will happen inside.
  • approach 2 would be more readable, but generated more code, or required risky refactor. It might be good idea for follow up, I'm open if you want to try to improve that, but note that this is more tricky than it seems and big rewrite is more likely to introduce more bugs.
  • approach 3, Similar to approach 1, but worse because we need to be careful that checking port happens before get_frames

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer having a function like get_frames and calling it here and in the check_ready_for_pts function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that would not work with current architecture:

  • get frames needs to return function whether it's ready or not
  • currently get frames relies that returned frame will be used (it's needed for example to check whether EOS was sent)

And if you are suggesting to change the above, then your suggestion boils down to almost full rewrite of queuing code.

We can talk about it more at the office, topic is to complex for discussion via comments.

@wkozyra95 wkozyra95 self-assigned this Jul 19, 2024
@wkozyra95 wkozyra95 merged commit 4af2185 into master Jul 19, 2024
1 check passed
@wkozyra95 wkozyra95 deleted the @wkozyra95/fix-queue-scheduling-on-input-end branch July 19, 2024 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants