What browsers are supported?

[trackedsupport]

Right now I'm trying to support every browser except Safari. It looks like Firefox needs mozCaptureStream or else this breaks. If I change the code to use that then it breaks const stream = blackSilence(); trying to get the stream with a TypeError.

I'll keep seeing if I can get it to work but do you know of any other issues with Firefox or other browsers?

[QVDev]

No I am not aware of any other issues on other browsers. We probably need some logic to support other browser along with chrome.

We can check the browsee upfront and than decide what we need to use.

blackSilence() 

Can be adjustes/extendes to support mozilla browser

[trackedsupport]

I can take a look to see if I can figure it out.

Do you know if this method with the caprureStream has any degradation in audio or video quality?

[QVDev]

No quality etc is handled by your constraints so I dont expect lower quality.

[trackedsupport]

Have you seen this before? https://github.com/muaz-khan/MultiStreamsMixer

Not sure how the implementation is different.

[QVDev]

I know it and its widely used. But we solve different issues.

The multistreammixer uses canvas to layout multiple streams into on stream by placing them on a canvas. So you can have a sharescreen with a small webcam on the bottom right corner for instance.

What StreamSwitcher solves is switching full streams without need to pause video and keep recording.

If you are looking for something to mix two streams into one have a look at

https://github.com/bbc/VideoContext

That is wat more performant as it uses webgl I believe. So if you do heavy lifting and want to mix streams you best go is to use bbc VideoContext.

See her for my repo demo to add lots of video into one stream using videocontext

https://qvdev.github.io/video_mux_rec/all.html

And here is the repo

https://github.com/QVDev/video_mux_rec

What wont work with the one you mentioned due to canvas

[QVDev]

@trackedsupport what are you trying to solve / trying to do?

[QVDev]

@trackedsupport also see https://qvdev.github.io/video_mux_rec/

that is screen share with smaller webcam stream at right bottom corner

And that you can record as well and has good performance

[trackedsupport]

oh dang, I thought the stream mixer would allow me to switch streams on the mixer while keeping the same stream with the recorder. All this stuff is new to me.

I want the ability to switch the video stream to a different camera or to a screen share while still recording locally in high quality. Right now with Iris there's an issue that if you switch your camera or mic during the recording it stops and I want to be able to add screen share soon too.

[QVDev]

@trackedsupport than StreamSwitcher is what you want. It avoids the mediarecorder to stop

[trackedsupport]

Alrright thanks for the help. Ill focus on getting this to work on other browsers.

First thing I noticed was that the supported method checks HTMLVideoElement.prototype.captureStream but the captureStream used in the code is on a canvas element so it should be HTMLCanvasElement.prototype.captureStream right?

[QVDev]

StreamSwitcher does not use canvas so can directly do capture stream on the videobject as in the demo

[trackedsupport]

oh okay - I guess I just don't understand enough yet! I only said that because I saw this.

StreamSwitcher/js/ReplaceableMediaStream.js

Line 31 in 5e5ab14

let stream = canvas.captureStream();

[trackedsupport]

Right now im not directly supporting Safari but I would like to since its so close to working.

If I get this to work with Firefox, which I believe it can, it still wont work with Safari right?

https://caniuse.com/?search=captureStream

[QVDev]

oh okay - I guess I just don't understand enough yet! I only said that because I saw this.

StreamSwitcher/js/ReplaceableMediaStream.js

Line 31 in 5e5ab14

let stream = canvas.captureStream();

That canvas is the blackstream empty one not from the videos. The empty / silence stream is for initial.setup

[QVDev]

Right now im not directly supporting Safari but I would like to since its so close to working.

If I get this to work with Firefox, which I believe it can, it still wont work with Safari right?

https://caniuse.com/?search=captureStream

Indeed capturestream is not supported by safari. Question if it ever will be

[trackedsupport]

oh okay - I guess I just don't understand enough yet! I only said that because I saw this.

StreamSwitcher/js/ReplaceableMediaStream.js

Line 31 in 5e5ab14

let stream = canvas.captureStream();

That canvas is the blackstream empty one not from the videos. The empty / silence stream is for initial.setup

So then it has to support HTMLCanvasElement.prototype.captureStream as well right?

[QVDev]

Yes or you can change that code to a short black video or even a black gif I think that it wont need the canvas

[QVDev]

But you need capture stream for the video object as well so eliminating this canvas capturestream wont help on the long run

[trackedsupport]

Yea I see your point.

Just to let you know I did play around with the MultiStreamMixer and it did work. The height and width require some setting and I'm not sure if the quality is as good. Also when you reset the video there's no longer any audio even if I add the stream.

Is there a downside in replicating this way of updating the streams? It's like what you said, it's built for another purpose but seems like it could be good with it working on all browsers.

[trackedsupport]

So I had a working solution with the MultiStreamMixer through RecordRTC but that thing eats up so much memory. I have an older macbook pro and it crashes it after a few minutes eating gigs of memory on Chrome.

Going to trash it since it won't work. I guess I'll circle back around to this and see what I can learn and improve.

[trackedsupport]

I'm trying to circle back to this. Altho im not sure what I am doing wrong now.

var local_media_stream = new MediaStream();

await navigator.mediaDevices.getUserMedia({ audio: this.audio_options, video: this.exact_video_options })
          .then(function(camera) {
            local_media_stream.replaceVideoTrack(camera.getVideoTracks()[0]);
            local_media_stream.replaceAudioTrack(camera.getAudioTracks()[0]);
           })


local_media_stream.getTracks() #returns []

but if I inspect the local_media_stream with local_media_stream.videoSender.track that is correct. The problem is that im trying to record with the MediaRecorder.

recorder = new MediaRecorder(local_media_stream)

[QVDev]

Will have to need a look into your example and I come back to you.

[QVDev]

What about local_media_stream.remoteStream

[QVDev]

Does that work for you?

[trackedsupport]

No that didn't work. I think before I used a hidden video element on the page to copy your example. But when I record local_media_stream.remoteStream then an empty file is created.

[trackedsupport]

I added this to get it to work on Firefox. I also got this to work when I used a video html object and just hid it from the webpage. Im going to test for Safari to run our old way and use this for any browsers that support capture stream. Thanks for your work!

if (HTMLVideoElement.prototype.mozCaptureStream){ HTMLVideoElement.prototype.captureStream = HTMLVideoElement.prototype.mozCaptureStream; }

[QVDev]

Glad you found the issue feel free to open a pull request so other developers can benefit from it as well. Great work 👍

[trackedsupport]

Im still messing around. I got it working on Safari. It appears you can save local_media_stream.remoteStream with the media recorder as long as you do local_video.srcObject = local_media_stream.remoteStream; first. Not sure why but I'm still looking. If you do it this way, you don't need the HTML element capture stream which safari doesn't have.

[trackedsupport]

I did have to add this also btw to get it to work with Safari.

    var AudioContext = window.AudioContext;
    if (typeof AudioContext === 'undefined') {
        if (typeof webkitAudioContext !== 'undefined') { AudioContext = webkitAudioContext; }
        if (typeof mozAudioContext !== 'undefined') { AudioContext = mozAudioContext; }
    }

[trackedsupport]

Am I correct that replaceVideoTrack does not use the fps, width, or height from the stream? I have a 1280x720 24fps stream but for some reason its 320x180 30fps.

Edit: It appears to slowly grow. First 320, then 540, then 720. Weird?

example

:

[trackedsupport]

Do you know what causes these sizing issues? Or do you know how to speed it up? I'm okay if it takes a second or two to adjust but this takes something like 30 seconds.

[trackedsupport]

Hey I wanted to follow up, do you know what causes these sizing issues when changing the camera? Or do you know how to speed it up? I'm okay if it takes a second or two to adjust but this takes something like 30 seconds.

[trackedsupport]

Safari has some wierd crashing bug with the black canvas video and addTrack on RTCPeerConnection. I cannot figure out how to create a mediastreamtrack of type video without the canvas or html element captureStream. That is the thing holding me back from deploying this right now.

[trackedsupport]

Never mind I just realized the frame rate is constantly changing. I guess this is just a pipe dream of mine!

[trackedsupport]

Looks like a similar solution. One file looks almost exactly the same.

https://github.com/Choi-Heesu/replaceable-media-recorder