Add audio visualization

[shakeelmohamed]

First of all, I'm not sure if this is even possible since we don't have direct access to audio files (for YouTube or Soundcloud), but it would be a cool option to add.

Of course this needs to be off by default otherwise it defeats the purpose of this project 😃

One possible viz library we can use is Wayou/HTML5_Audio_Visualizer as long as we leave a reference to his work. We may want to fork it to customize it to our liking.

If we can't do this directly, there may be some APIs in Chrome (and other browsers) that we can access with an extension.

edit: just use some of @MrAnyone's code from https://github.com/MrAnyone/YouTube-Audio-Visualizer as it's MIT licensed 🎉

[shakeelmohamed]

A blog post on creating a custom viz: https://www.patrick-wied.at/blog/how-to-create-audio-visualizations-with-javascript-html

[MatMercer]

Using a WebGL library is the best way to do that, because audio processing is in real time (can be at 60FPS) and the WebGL have the power to render at 60FPS, making the visualizer veeery smooth.

Currently I use Pixi.js

[timchoh585]

To add on, i think the hardest thing would be just to get the mp3 file from YouTube or Soundcloud. Not too sure how to do that myself, but that would be the biggest challenge.

I would go as far to say that even creating something with D3 could be a good solution as well. Not too familiar with how this would be compared to Pixi.js I would like to try and implement a version with D3 just because i am more familiar with it. But will definitely look into Pixi.js because it looks pretty sleek.

[shakeelmohamed]

I would like to try and implement a version with D3 just because i am more familiar with it.

@timchoh585 go for it! It looks like @MrAnyone is using audioContext to get audio data, more on there here

[timchoh585]

@shakeelmohamed thanks! I will definitely be working on this throughout the week! I'll try and keep this as updated as possible

[shakeelmohamed]

@timchoh585 excellent, please ping me if you need anything

[timchoh585]

@shakeelmohamed sorry, got pretty busy with school and work. should have something to show within the week.

[andreasvirkus]

@timchoh585 @shakeelmohamed Would it be ok for me to take a crack at this?
Any specific design(colour palette)/positioning you had in mind?

[shakeelmohamed]

@andreasvirkus go for it, maybe go with the blue we use for links? Let's get something working then we can tweak the colors later

[andreasvirkus]

After two nights of tinkering, I'm calling out for some help (if a solution exists at all).
All of the AV libs rely on AudioContext, which can be accessed through an HTML5 <video> or <audio> element. Since youtube doesn't allow CORS, we can't access the video element in the iFrame. So we'd need the url for YT's source files to create our own tag (which we can scrape). But YouTube's blocked access to those URLs so you get a 403..

I'm not aware of any way to access the audio stream through the iFrame. Any thoughts/suggestions on how to proceed/what to try?

[andreasvirkus]

I stumbled upon youtube-audio-stream and videojs-youtube, which I can try out over the weekend.

[MatMercer]

Hey there @andreasvirkus, I know what problem you are having. I made the Youtube Audio Visualizer in the way it works because when you get the video inside the same page of youtube you bypass the CORS restriction, and I was very bad at programming 👎.

Well, time passed and last year @hoboman313 implementing huegasm as a chrome extension contacted me to solve this problem.

The problem is that you can't access the source of the audio without bypassing CORS restriction (with a custom Chrome flag, witch is not enabled by default).

After some time @hoboman313 managed to fix this issue in the best way possible I can see 🎉. He used the tab capture from google chrome to capture the audio data directly from the tab. You can't do that without a chrome extension, or Firefox whatever.

Huegasm Chrome Extension

But how was it implemented?
@hoboman313 emailed me this implementation:

chrome.runtime.onMessage.addListener(function (request, sender, sendResponse) {
    if (request.action === 'start-listening') {
        chrome.tabCapture.capture({
            audio: true,
            video: false
        }, function (_stream) {
            let error = null;

            if (!_stream) {
                error = chrome.runtime.lastError.message;
            } else {
//do stuff with valid stream here

So, the implementation of the audio visualization relies in the Chrome extension, in my view it should work this way:

1. Chrome extension listen to the audio from the tab.
2. Chrome extension interacts with the page.
3. 💥, you can make even the background of the page beat with css.

[andreasvirkus]

Hi @MtMercer! Awesome to know about the tab capture. If the audio streaming libraries don't work out, then it's nice to have a working back-up and I guess that's the way we'll have to go. Just seems that making a solution Chrome-only and requiring an extension seems very limiting though.

Edit: Talked with @shakeelmohamed and we'll add the solution you propsed to the existing extension. Thanks again for sharing the solution!

[MatMercer]

@andreasvirkus

Just seems that making a solution Chrome-only and requiring an extension seems very limiting though.

IKR, sadly the web audio API is not so 'open' for development.

Edit: Talked with @shakeelmohamed and we'll add the solution you propsed to the existing extension. Thanks again for sharing the solution!

You are welcome! Any questions about how to capture message me. In my vacations I implemented a very small web audio visualization library (very small) and I hope to continue developing it. I got the same effect of this video that introduces Oscilloscope Music.
Visualadio

Thanks for the fast feedback.

[sleepingdude]

Hi guys! Accidentally peeped your conversation) I created chrome extension "Music Storm", it have similar idea) if you have any questions I can help you)
https://chrome.google.com/webstore/detail/music-storm-visualize-any/egmilngkgddnnmkpkonkpkjkipiihmkh

[shakeelmohamed]

Thanks @Hadaev-Ivan - any chance you can open source your extension? Or maybe you'd like to submit a PR for this 🎉

[Ayushbajpai19]

I think we can extract only audio from youtube and then add https://github.com/MrAnyone/YouTube-Audio-Visualizer and then display it, also I would really like to contribute to the issues mentioned here, can u guide me through the process?

[shakeelmohamed]

@Ayushbajpai19 what guidance are you looking for?

[etroo44]

Hello, is this issue still open?

[shakeelmohamed]

@etroo44 hello, yes it is

[jstjnsn]

I'm trying to implement the visualization and I can't seem to find any audio element to read data from. I noticed that you use a very old version of plyr (you use 1.6.x, the newest version is 3.7.8). Also, playing audio of a youtube video without showing the video alongside it breaks the terms of youtube apparently. How do you want to deal with this? It might be doable as long as the site does not get much traffic, but if youtube notices it you might get taken down.

[shakeelmohamed]

@jstjnsn hello, @Ayushbajpai19 shared a project we can draw some inspiration from.

I created this project as a proof of concept 8 years ago and am not concerned about breaking YouTube’s ToS. If requested to take down the site, I am happy to do so.