[Triage] Missing a few minutes of live streams

[viktorrauch]

I'm subscribed to some playlists with live streams. Sadly most of the time when pinchflat auto downloads those live streams the first few minutes are missing. I have to delete the file and manually redownload it via pinchflat to get the full video.

I'm subscribed to this playlist for example: https://youtube.com/playlist?list=PLQYAznaaoaja35mxluifFNvGOFPD0uKKG&si=xz91rltzov7Mod_H

Is there a way to tell pinchflat to download a live stream after it's over and grab the full video on demand?
Or is there something I'm just misunderstanding or missing in the settings?

[kieraneglin]

Hey there! Thanks for the report (:

That's strange! I'll look into this, but I suspect it may be an upstream issue with yt-dlp. In any case, you can set a redownload delay on the source to have it automatically redownload the file after a day or two.

To help with troubleshooting, could you also please open that source, click Actions > Copy JSON, and post that here?

[2IkHLHB]

This has happened to me too, but on videos and subscriptions I use as audio podcasts. And yes, those videos / audio are or were live streams.

[kieraneglin]

Does this happen when you're downloading livestreams that have been over for days/weeks or does this only happen when you're downloading a livestream that has very recently completed?

[2IkHLHB]

Does this happen when you're downloading livestreams that have been over for days/weeks or does this only happen when you're downloading a livestream that has very recently completed?

In my case, because I have Fast Indexing enabled, I believe Pinchflat gets livestreams that have recently completed.

[viktorrauch]

Sorry for the delay,
here the json of a playlist in which the error happend a couple of days ago:

{
"id": 43,
"description": null,
"uuid": "f3e28242-4051-479b-922d-4f39344f3c45",
"inserted_at": "2024-09-23T08:22:16Z",
"updated_at": "2024-10-04T08:59:05Z",
"media_profile": {
"id": 1,
"name": "Viktor",
"inserted_at": "2024-04-21T11:59:36Z",
"updated_at": "2024-04-24T08:08:30Z",
"output_path_template": "/Viktor/{{ source_custom_name }}/{{ season_episode_from_date }} - {{ title }}.{{ ext }}",
"media_container": null,
"preferred_resolution": "2160p",
"sponsorblock_categories": [],
"sponsorblock_behaviour": "disabled",
"download_thumbnail": true,
"embed_thumbnail": true,
"download_auto_subs": false,
"download_subs": true,
"embed_subs": true,
"sub_langs": "de,en",
"download_metadata": true,
"embed_metadata": true,
"download_nfo": true,
"download_source_images": true,
"livestream_behaviour": "include",
"marked_for_deletion_at": null,
"redownload_delay_days": 1,
"shorts_behaviour": "exclude"
},
"nfo_filepath": null,
"original_url": "https://youtube.com/playlist?list=PLQYAznaaoajbxR1friAusCMKI136LEQvo&si=vlotP0IsF03CrJW2",
"media_profile_id": 1,
"custom_name": "Jorgensons Reaktionen",
"collection_id": "PLQYAznaaoajbxR1friAusCMKI136LEQvo",
"collection_name": "Reaktionen",
"collection_type": "playlist",
"download_media": true,
"use_cookies": false,
"fast_index": false,
"download_cutoff_date": null,
"min_duration_seconds": null,
"max_duration_seconds": null,
"title_filter_regex": null,
"banner_filepath": null,
"fanart_filepath": null,
"poster_filepath": null,
"marked_for_deletion_at": null,
"series_directory": null,
"last_indexed_at": "2024-10-04T08:59:05Z",
"index_frequency_minutes": 60,
"output_path_template_override": null,
"retention_period_days": null
}

[kieraneglin]

Hey there! Can anyone confirm whether this is still happening after the latest release? That release included a new version of yt-dlp which may have included some bug fixes that relate to this

[viktorrauch]

Sadly yes. I'm on Pinchflat 2024.10.2 and yt-dlp 2024.09.27 and just had the problem yesterday. The first 3 minutes of a live stream were missing.

I subscribed to the playlist as video feed and as audio only feed and the problem was on both files. After I hit "force download" in pinchflat on the episode, the full video / audio file was downloaded.

Here are the jsons, if it helps:

JSON #1:

{
"id": 1423684,
"description": ""gequatscht wird viel" - LIVE
Stream unterstützen: https://ko-fi.com/jorgenson
Zur Playlist: https://www.youtube.com/playlist?list=PLQYAznaaoaja35mxluifFNvGOFPD0uKKG

Gast: @SkaiLW

#Jorgenson #Gothic #GothicRemake #gequatschtwirdviel

---

• Discord: https://discord.com/invite/Agk8K56KHd

• Soundcloud: https://soundcloud.com/user-168650688

• Merchandise: https://jorgensons-warenhaus.myspreadshop.de/

• Spenden: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BZNFU3PSQPNHN&source=url

• Ko-fi: https://ko-fi.com/jorgenson

• Patreon: https://www.patreon.com/Jorgenson

• Kanalmitgliedschaft: https://www.youtube.com/channel/UCqO_TLwH0FrjlIkZW9xkQ6A/join",
"title": "Gothic, Mods und Remake │ "gequatscht wird viel" - LIVE",
"source": {
"id": 46,
"description": null,
"uuid": "48be7da3-4e66-4c4e-a419-86f6b580a581",
"inserted_at": "2024-09-23T11:44:30Z",
"updated_at": "2024-10-11T06:27:40Z",
"media_profile": {
"id": 1,
"name": "Viktor",
"inserted_at": "2024-04-21T11:59:36Z",
"updated_at": "2024-04-24T08:08:30Z",
"output_path_template": "/Viktor/{{ source_custom_name }}/{{ season_episode_from_date }} - {{ title }}.{{ ext }}",
"media_container": null,
"preferred_resolution": "2160p",
"sponsorblock_categories": [],
"sponsorblock_behaviour": "disabled",
"download_thumbnail": true,
"embed_thumbnail": true,
"download_auto_subs": false,
"download_subs": true,
"embed_subs": true,
"sub_langs": "de,en",
"download_metadata": true,
"embed_metadata": true,
"download_nfo": true,
"download_source_images": true,
"livestream_behaviour": "include",
"marked_for_deletion_at": null,
"redownload_delay_days": 1,
"shorts_behaviour": "exclude"
},
"nfo_filepath": null,
"original_url": "https://www.youtube.com/playlist?list=PLQYAznaaoaja35mxluifFNvGOFPD0uKKG",
"media_profile_id": 1,
"custom_name": "Gequatscht wird viel (live)",
"collection_id": "PLQYAznaaoaja35mxluifFNvGOFPD0uKKG",
"collection_name": ""gequatscht wird viel" - LIVE",
"collection_type": "playlist",
"download_media": true,
"use_cookies": false,
"fast_index": false,
"download_cutoff_date": null,
"min_duration_seconds": null,
"max_duration_seconds": null,
"title_filter_regex": null,
"banner_filepath": null,
"fanart_filepath": null,
"poster_filepath": null,
"marked_for_deletion_at": null,
"series_directory": null,
"last_indexed_at": "2024-10-11T06:27:40Z",
"index_frequency_minutes": 60,
"output_path_template_override": "/Viktor/Gequatscht wird viel/{{ season_episode_from_date }} - {{ title }}.{{ ext }}",
"retention_period_days": null
},
"uuid": "aa513826-7949-4640-bc74-8586eb856e54",
"inserted_at": "2024-10-10T16:22:25Z",
"updated_at": "2024-10-10T19:57:22Z",
"culled_at": null,
"duration_seconds": 10858,
"livestream": true,
"matching_search_term": null,
"media_downloaded_at": "2024-10-10T19:57:18Z",
"media_filepath": "/downloads/Viktor/Gequatscht wird viel/s2024e1010 - Gothic, Mods und Remake │ ＂gequatscht wird viel＂ - LIVE.mp4",
"media_id": "-IVEnzHNWZU",
"media_redownloaded_at": null,
"media_size_bytes": 5046699059,
"metadata_filepath": "/downloads/Viktor/Gequatscht wird viel/s2024e1010 - Gothic, Mods und Remake │ ＂gequatscht wird viel＂ - LIVE.info.json",
"nfo_filepath": "/downloads/Viktor/Gequatscht wird viel/s2024e1010 - Gothic, Mods und Remake │ ＂gequatscht wird viel＂ - LIVE.nfo",
"original_url": "https://www.youtube.com/watch?v=-IVEnzHNWZU",
"playlist_index": 0,
"prevent_culling": false,
"prevent_download": false,
"short_form_content": false,
"source_id": 46,
"subtitle_filepaths": [],
"thumbnail_filepath": "/downloads/Viktor/Gequatscht wird viel/s2024e1010 - Gothic, Mods und Remake │ ＂gequatscht wird viel＂ - LIVE-thumb.jpg",
"upload_date_index": 0,
"uploaded_at": "2024-10-10T19:11:10Z"
}

JSON #2:

{
"id": 1423661,
"description": ""gequatscht wird viel" - LIVE
Stream unterstützen: https://ko-fi.com/jorgenson
Zur Playlist: https://www.youtube.com/playlist?list=PLQYAznaaoaja35mxluifFNvGOFPD0uKKG

Gast: @SkaiLW

#Jorgenson #Gothic #GothicRemake #gequatschtwirdviel

---

• Discord: https://discord.com/invite/Agk8K56KHd

• Soundcloud: https://soundcloud.com/user-168650688

• Merchandise: https://jorgensons-warenhaus.myspreadshop.de/

• Spenden: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BZNFU3PSQPNHN&source=url

• Ko-fi: https://ko-fi.com/jorgenson

• Patreon: https://www.patreon.com/Jorgenson

• Kanalmitgliedschaft: https://www.youtube.com/channel/UCqO_TLwH0FrjlIkZW9xkQ6A/join",
"title": "Gothic, Mods und Remake │ "gequatscht wird viel" - LIVE",
"source": {
"id": 51,
"description": null,
"uuid": "03c4a890-bdd9-4de7-9066-e96f6562d216",
"inserted_at": "2024-09-25T09:21:57Z",
"updated_at": "2024-09-25T09:22:36Z",
"media_profile": {
"id": 7,
"name": "Podcast",
"inserted_at": "2024-09-25T09:12:24Z",
"updated_at": "2024-09-25T10:14:47Z",
"output_path_template": "/Podcast/{{ title }}.{{ ext }}",
"media_container": null,
"preferred_resolution": "audio",
"sponsorblock_categories": [],
"sponsorblock_behaviour": "disabled",
"download_thumbnail": false,
"embed_thumbnail": true,
"download_auto_subs": false,
"download_subs": false,
"embed_subs": false,
"sub_langs": "en",
"download_metadata": false,
"embed_metadata": true,
"download_nfo": false,
"download_source_images": false,
"livestream_behaviour": "include",
"marked_for_deletion_at": null,
"redownload_delay_days": null,
"shorts_behaviour": "exclude"
},
"nfo_filepath": null,
"original_url": "https://www.youtube.com/playlist?list=PLQYAznaaoaja35mxluifFNvGOFPD0uKKG",
"media_profile_id": 7,
"custom_name": "Gequatscht wird viel (live)",
"collection_id": "PLQYAznaaoaja35mxluifFNvGOFPD0uKKG",
"collection_name": ""gequatscht wird viel" - LIVE",
"collection_type": "playlist",
"download_media": true,
"use_cookies": false,
"fast_index": true,
"download_cutoff_date": null,
"min_duration_seconds": null,
"max_duration_seconds": null,
"title_filter_regex": null,
"banner_filepath": null,
"fanart_filepath": null,
"poster_filepath": null,
"marked_for_deletion_at": null,
"series_directory": null,
"last_indexed_at": "2024-09-25T09:22:36Z",
"index_frequency_minutes": 43200,
"output_path_template_override": null,
"retention_period_days": null
},
"uuid": "668e0a95-2e16-41d8-9900-d42b2c032116",
"inserted_at": "2024-10-10T16:02:48Z",
"updated_at": "2024-10-10T19:44:46Z",
"culled_at": null,
"duration_seconds": 10858,
"livestream": true,
"matching_search_term": null,
"media_downloaded_at": "2024-10-10T19:44:42Z",
"media_filepath": "/downloads/Podcast/Gothic, Mods und Remake │ ＂gequatscht wird viel＂ - LIVE.m4a",
"media_id": "-IVEnzHNWZU",
"media_redownloaded_at": null,
"media_size_bytes": 175853684,
"metadata_filepath": null,
"nfo_filepath": null,
"original_url": "https://www.youtube.com/watch?v=-IVEnzHNWZU",
"playlist_index": 0,
"prevent_culling": false,
"prevent_download": false,
"short_form_content": false,
"source_id": 51,
"subtitle_filepaths": [],
"thumbnail_filepath": null,
"upload_date_index": 0,
"uploaded_at": "2024-10-10T19:11:10Z"
}

[nosoop]

If --live-from-start isn't passed in then yt-dlp will only get the contents of the stream starting from when the command is run. That might be what's happening here, but I'm not familiar with Elixir to verify it.

(Probably should also poll in the post-live state to redownload once it's fully processed.)

[kieraneglin]

That would fix this issue, but the problem there is that I want to ignore currently-live videos entirely. The app only allows 2 concurrent download operations by default and it's super inefficient if one download gets tied up waiting for a livestream to end.

That said, something mentioned in #480 gave me a huge hint as to what's happening here and I'm currently looking into it! Fingers crossed I figure it out today 🤞

[kieraneglin]

I think I got it with #485! That'll be going out sometime this week (I want to squeeze one more change into the next release if I can)

Thank you all for the help! If the issue still isn't fixed post-release, please re-open this issue