Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redo indexing mechanism #16

Merged
merged 3 commits into from
Feb 9, 2024
Merged

Redo indexing mechanism #16

merged 3 commits into from
Feb 9, 2024

Conversation

kieraneglin
Copy link
Owner

What's new?

  • adds .credo.exs file because it was yelling at me about cyclomatic complexity
  • adds list_downloaded_media_items_for to Media context for future use
  • adds original_url (string) and livestream (boolean) fields to MediaItem model
  • adds new FunctionUtils module with methods to help fulfill contracts

What's changed?

  • moves index_media_items into SourceTasks
  • get_video_ids has been renamed to get_media_attributes
  • get_media_attributes has been updated to pre-fetch more details about media such as the title, its original URL, etc.
  • generally, the indexing flow has been updated to index ALL content and use in-app logic to determine what should actually be downloaded. This makes a lot of things easier like downloading content as a one-off, being able to act on profile changes more quickly, and supporting different indexing backends/strategies in the future

What's fixed?

N/A

Any other comments?

N/A

@kieraneglin
Bumped up the line length because I fear no man
a330f54
@kieraneglin
Refactored indexing
da06881 
Previously, indexing worked by collecting the video IDs of only videos
that matched indexing criteria. This new model instead stores ALL videos
for a given source, but will only _download_ videos that meet that criteria.
This lets us backfill without indexing, makes it easier to add in other
backends, lets us download one-off videos for a source that don't quite
meet criteria, you name it.
@kieraneglin
Updated media finders to respect format filters; Added credo file
ce42f8a
@kieraneglin kieraneglin self-assigned this Feb 9, 2024
@kieraneglin kieraneglin merged commit dabec62 into master Feb 9, 2024
1 check passed
@kieraneglin kieraneglin deleted the ke/redo-indexing-mechanism branch February 9, 2024 21:23
kieraneglin added a commit that referenced this pull request Mar 29, 2024
@kieraneglin
Redo indexing mechanism (#16)
e063733 
* Bumped up the line length because I fear no man

* Refactored indexing

Previously, indexing worked by collecting the video IDs of only videos
that matched indexing criteria. This new model instead stores ALL videos
for a given source, but will only _download_ videos that meet that criteria.
This lets us backfill without indexing, makes it easier to add in other
backends, lets us download one-off videos for a source that don't quite
meet criteria, you name it.

* Updated media finders to respect format filters; Added credo file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@kieraneglin kieraneglin

Labels
enhancement New feature or request media categorization media downloads media indexing refactor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kieraneglin