Download indexed videos #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adding the metadata model made me realize that, in many cases, yt-dlp returns undesired input in stdout, breaking parsing. In order to get the metadata model working, I had to change the way in which the app interacts with yt-dlp. Now, output is written as a file to disk which is immediately re-read and returned.
kieraneglin
added
bug
kieraneglin
added a commit
that referenced
this pull request
Jan 31, 2024
* Updated package.json (and made an excuse to make a branch) * Video filepath parser (#6) * Restructured files; Added parser placeholder * More restructuring * Added basic parser for hydrating template strings * Improved docs * More docs * Initial implementation of media profiles (#7) * [WIP] Added basic video download method * [WIP] Very-WIP first steps at parsing options and downloading * Made my options safe by default and removed special safe versions * Ran html generator for mediaprofile model - leaving as-is for now * Addressed a bunch of TODO comments * Add "channel" type Media Source (#8) * [WIP] Working on fetching channel metadata in yt-dlp backend * Finished first draft of methods to do with querying channels * Renamed CommandRunnerMock to have a more descriptive name * Ran the phx generator for the channel model * Renamed Downloader namespace to MediaClient * [WIP] saving before attempting LiveView * LiveView did not work out but here's a working controller how about * Index a channel (#9) * Ran a MediaItem generator; Reformatted to my liking * [WIP] added basic index function * setup oban * Added basic Oban job for indexing * Added in workers for indexing; hooked them into record creation flow * Added a task model with a phx generator * Tied together tasks with jobs and channels * Download indexed videos (#10) * Clarified documentation * more comments * [WIP] hooked up basic video downloading; starting work on metadata * Added metadata model and parsing Adding the metadata model made me realize that, in many cases, yt-dlp returns undesired input in stdout, breaking parsing. In order to get the metadata model working, I had to change the way in which the app interacts with yt-dlp. Now, output is written as a file to disk which is immediately re-read and returned. * Added tests for video download worker * Hooked up video downloading to the channel indexing pipeline * Adds tasks for media items * Updated video metadata parser to extract the title * Ran linting
kieraneglin
added a commit
that referenced
this pull request
Mar 29, 2024
* Updated package.json (and made an excuse to make a branch) * Video filepath parser (#6) * Restructured files; Added parser placeholder * More restructuring * Added basic parser for hydrating template strings * Improved docs * More docs * Initial implementation of media profiles (#7) * [WIP] Added basic video download method * [WIP] Very-WIP first steps at parsing options and downloading * Made my options safe by default and removed special safe versions * Ran html generator for mediaprofile model - leaving as-is for now * Addressed a bunch of TODO comments * Add "channel" type Media Source (#8) * [WIP] Working on fetching channel metadata in yt-dlp backend * Finished first draft of methods to do with querying channels * Renamed CommandRunnerMock to have a more descriptive name * Ran the phx generator for the channel model * Renamed Downloader namespace to MediaClient * [WIP] saving before attempting LiveView * LiveView did not work out but here's a working controller how about * Index a channel (#9) * Ran a MediaItem generator; Reformatted to my liking * [WIP] added basic index function * setup oban * Added basic Oban job for indexing * Added in workers for indexing; hooked them into record creation flow * Added a task model with a phx generator * Tied together tasks with jobs and channels * Download indexed videos (#10) * Clarified documentation * more comments * [WIP] hooked up basic video downloading; starting work on metadata * Added metadata model and parsing Adding the metadata model made me realize that, in many cases, yt-dlp returns undesired input in stdout, breaking parsing. In order to get the metadata model working, I had to change the way in which the app interacts with yt-dlp. Now, output is written as a file to disk which is immediately re-read and returned. * Added tests for video download worker * Hooked up video downloading to the channel indexing pipeline * Adds tasks for media items * Updated video metadata parser to extract the title * Ran linting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's new?
A lot.
MediaMetadatamodel for storing things like fullyt-dlpresponsesMetadataParsermodule foryt-dlpto interpret the JSON responsemedia_item_idto theTasksmodelMediaItems for processingrandom_stringhelper methodRepomodule to help withObanjob insertion - thisinsert_unique_jobmethod returns an error tuple if a job would've created a duplicateWhat's changed?
create_task,delete_tasks_for, anddelete_pending_tasks_forto work withMediaItemsdownload_for_media_profileintodownload_for_media_itemsince I had no use for the former. Now, it takes a givenMediaItemand downloads its media based on the settings from theMediaProfileyt-dlprunner module to take a third argument of its output template. More on that Below.What's fixed?
yt-dlps stdout to parse its output, but that's no good. It would emit warnings sometimes and would attempt to parse those. Surprise, didn't work! Now, output is written to a tmpfile (on disk) and that file is immediately re-read and its contents parsed. The biggest practical change is that the runner now takes a 3rd argument that specifies the output template (reference) which is what's returned from the runneryt-dlprunner args parser so that it appends flags instead of preprending them. This old behaviour brokeyt-dlps multi-part args by adding them out of orderAny other comments?
N/A