Automation best practices

[abayochocoball]

Write about how to automate archiving process.

Starting point:
Task scheduler/cron

[GoldElysium]

Automating it consistently is a horrible thing to do, YT will detect these patterns and block you from downloading. (Speaking from experience). Possibly immediately download when a stream starts could be a work around?

[abayochocoball]

When you were running a schedule, how often were you polling for updates?

[GoldElysium]

Weekly in my case

[SusuTawar]

immidiate download is also detectable pattern, though i'm not sure if youtube can tell the difference between watch and download.

i have a few things in mind using PubSubHubbub its server to server push notification, not sure how it behave on live schedule i can test it may took 24hours or so

[GoldElysium]

It seems YT can detect it, other people noticed that it blocked ytdl but they could watch videos fine

[SusuTawar]

update, i did an oppsie by using wrong parser. but the notification are coming.

so automation "may" be is possible. by utilizing server that recieve notification as middleware to broadcast youtube link to automated script that call youtube-dl in the client computer.

i can write the server files and maybe the client script
my only concern is youtube ToS, if its not allowed then i wont continue.

[abayochocoball]

https://www.youtube.com/static?template=terms

access the Service using any automated means (such as robots, botnets or scrapers) except (a) in the case of public search engines, in accordance with YouTube’s robots.txt file; or (b) with YouTube’s prior written permission;

That isn't going to stop me from writing about it though.

[GoldElysium]

If you're interested in the PubHubSubBub thing, I've got some code. It's for a different purpose(notifs in a Discord server), but can be used. It's still in a PR and needs to be tested, but here is the branch: link. If that link doesn't work, here is the master branch: link. Let me know if you have any questions.

[SusuTawar]

the pubsubhubbub (let's call it websub) is working perfectly, i tested it for about 12-17 hours. i get all kind of notification privated video, stream schedule, stream start, stream/video name change. which can be filtered later using youtube-dl

also its totally possible to create a server to forward the notification to python script (where youtube-dl reside) to automaticly start the download.

maybe we can get around the "automated means" by asking for user action before start downloading ?

[SusuTawar]

server are done, havent tested it.
i'll try work on the download script, dont put much hope it wont have any gui.
client script

on that note can anyone help to fill the data in channels directory ? still missing out a lot.

[GoldElysium]

@SusuTawar You get stream start notifications? You sure? I'm pretty sure you don't, at least I don't get any and I can't find anything about it either. I've been writing code to work around it.
Also, I've tried using your HMAC verification code but it fails for me.

[abayochocoball]

I'll start looking into this soon. Currently trying to create a very simple timer based solution and detect for collabs based on video description.

[SusuTawar]

@GoldElysium i dont. but client script check if its live/downloadable or not. the error handling pretty straightforward with youtube-dl. the only thing that i havent sure yet is video name change and video upload. as it is now it'll find live metadata from youtube-dl before start downloading, so no haachama cooking.

HMAC ? maybe the secret ? i didnt include the secret in the code. but i make sure its working before working on anything else, havent touch it since then.
EDIT: yeah i made sure secret is required when subscribing, just in case.

i cant really continue for maybe a week, got my laptop broke and my phone dead..

[GoldElysium]

i dont. but client script check if its live/downloadable or not. the error handling pretty straightforward with youtube-dl. the only thing that i havent sure yet is video name change and video upload. as it is now it'll find live metadata from youtube-dl before start downloading, so no haachama cooking.

So you can't handle planned livestreams?

HMAC ? maybe the secret ? i didnt include the secret in the code. but i make sure its working before working on anything else, havent touch it since then.

Weird, I'm sure I include the secret but it doesn't work in my code, will look into it a bit more.

i cant really continue for maybe a week, got my laptop broke and my phone dead..

F

[SusuTawar]

i just get my phone back, i planned livestream work its usually has 3 or 4 notification depend on the channel but only 1 that has is_live metadata.
and that is when the download start