Re-opening an old scrape session

[DerAndereJohannes]

Hi ytfzf community,
If this idea already exists in the current version, I am struggling to find it but here goes:

Motivation: I like watching videos spread out throughout the day rather than all in one go. I have a subscription list which gets refreshed every time I want to watch a new video. Of course scraping all these channels takes some time, and honestly, I would rather minimize my bandwidth use for links I already have in the history (in ~/.cache/ytfzf) and not wanting to update the video list as I would perhaps feel inclined to watch more.

Therefore if there was a flag to search through old scrapes which would open the old scrape result, it would open fast and without and updates.

Proposal: A fzf/dmenu to see all old searches (maybe a separate flag to select the most recent one), which then leads to the old scrape in the fzf/dmenu like normal where you can then open the video.

I feel like this would be a worthy addition to the program. Let me know how you feel (or if this exists already). Thanks.

[Euro20179]

This is possible in the development version using the --keep-cache option, --keep-cache exists in the stable version but doesn't work.

What you do is ytfzf --keep-cache -c SI to scrape subscriptions and keep cache.

Then when you want to reopen the session you can do ytfzf -c p ~/.cache/ytfzf/SCRAPE-SI-<some-pid-number>/videos_json

<some-pid-number> will be different every time, so i would check exactly which folder it saved in.

I might add a scraper called from-cache so it would be possible to do ytfzf -c from-cache to make this easier.

edit:
this will also redownload thumbnails every time, for now you can use the smart-thumb-download extention by doing -e smart-thumb-download (always use it)

Essentially all this does is download thumbnails to the config folder (~/.config/ytfzf/thumbnails) where ytfzf always looks for thumbnails.

I plan on making it so that -c from-cache copies the saved thumbnails to where ytfzf will read them.

edit 2:
in the development version, you can now scrape something using --keep-cache, then scrape it again from cache using -c from-cache
eg:

ytfzf -cSI --keep-cache

ytfzf -c from-cache

[DerAndereJohannes]

Holy smokes! You are very fast to respond and deliver! I have tried the dev branch and it is exactly what I imagined this feature to be. Thank you so much!

Just a couple of considerations:

1. It seems like the from-cache scrape creates another copy of the scrape in the cache folder (if I have have the keep_cache variable set to 1 in the conf.sh) maybe that can be mitigated.
2. Possibility of having a special flag to get the last cached scrape e.g., -c from-last-cache or -c from-cache <number> where <number> is how far back it should go, so 0 would be the latest.
3. Would it be feasible to add a timestamp to the cache folder name? Like add an ISO date at the front so you can see the caches in order in the new command? e.g., 2023-08-01-20-30-SCRAPE-SI-2938 or similar

If this is too complex or would break other code, then don't worry too much about it. I believe that this would definitely enhance this feature though.

Please let me know if I can contribute somehow too!

[Euro20179]

It seems like the from-cache scrape creates another copy of the scrape in the cache folder (if I have have the keep_cache variable set to 1 in the conf.sh) maybe that can be mitigated.

I'll look into it.

Would it be feasible to add a timestamp to the cache folder name? Like add an ISO date at the front so you can see the caches in order in the new command? e.g., 2023-08-01-20-30-SCRAPE-SI-2938 or similar

Not going to lie, this would probably break a bunch of stuff lol, but I think I could have a workaround, where there's just a file in the cache folder that contains the timestamp of when it was created, or i could see if i could use stat to get when the folder is created.

Possibility of having a special flag to get the last cached scrape e.g., -c from-last-cache or -c from-cache where is how far back it should go, so 0 would be the latest.

The best way to implement this would be by keeping track of when the caches were created, so if i implement your 3rd suggestion this could probably easily get implemented.

Please let me know if I can contribute somehow too!

Your feedback is very helpful, I'll let you know of anything else.