diff --git a/README.md b/README.md index 4bb8dc8..6ecc86b 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,7 @@ - Customize the **structure** of downloaded post/creator **directories** - Search for creators/artists and posts, and **export the results** - **Cross-platform** support & **iOS shortcuts** available +- For Coomer.su / Coomer.party support, check document [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/) for more. ## Dev Plan diff --git a/README_zh-CN.md b/README_zh-CN.md index 873a3df..5ce6efc 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -53,6 +53,7 @@ - 可自定义下载的作品/画师 **目录结构** - 可搜索画师和作品，并 **导出结果** - 支持全平台，并提供 **iOS 快捷指令** +- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)。 ## 开发计划 diff --git a/docs/en/commands/guide.md b/docs/en/commands/guide.md new file mode 100644 index 0000000..18e8610 --- /dev/null +++ b/docs/en/commands/guide.md @@ -0,0 +1,79 @@ +# Guide + +Check [Reference](reference.md) for all commands and their flags / parameters. + +## Get general help + +- `--help`, `-h` + +```bash +ktoolbox -h +``` + +## Get help of a command + +- `--help`, `-h` + +```bash +ktoolbox download-post -h +``` + +## Download a specific post + +`download-post` + +```bash +ktoolbox download-post https://kemono.su/fanbox/user/49494721/post/6608808 +``` +??? info "If some files failed to download" + If some files failed to download, you can try to execute the command line again, + the downloaded files will be **skipped**. + +## Download all posts from a creator + +`sync-creator` + +```bash +ktoolbox sync-creator https://kemono.su/fanbox/user/9016 +``` +??? info "Output" + By default, you will get a `creator-indices.ktoolbox` file in the creator directory, + you can use it to update the directory anytime. + + +## Update a downloaded creator directory + +`sync-creator` + +- `--update-with` + +```bash +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --update-with=./xxx/creator-indices.ktoolbox +``` +??? info "About `creator-indices.ktoolbox` file" + The `creator-indices.ktoolbox` file contains the information and filepath of posts inside the directory. + +## Download posts that published within the specified time range + +`sync-creator` + +- `--start-time` +- `--end-time` + +```bash +# From 2023-8-5 to 2023-12-6 +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --start-time=2023-8-5 --end-time=2023-12-6 + +# From 2023-8-5 to now +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --start-time=2023-8-5 + +# Before 2023-8-5 +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --end-time=2023-8-5 +``` + +### Time Format + +The time value should match `%Y-%m-%d`, for example: +- `2023-12-7` +- `2023-12-07` +- `2023-12-31` diff --git a/docs/en/commands.md b/docs/en/commands/reference.md similarity index 51% rename from docs/en/commands.md rename to docs/en/commands/reference.md index 6fd76e4..52e4b6d 100644 --- a/docs/en/commands.md +++ b/docs/en/commands/reference.md @@ -1,7 +1,4 @@ -# Commands - -!!! info "Usage" - Visit [Command](index.md#command) +# Reference ::: ktoolbox.cli.KToolBoxCli options: diff --git a/docs/en/coomer.md b/docs/en/coomer.md new file mode 100644 index 0000000..9a9ffe3 --- /dev/null +++ b/docs/en/coomer.md @@ -0,0 +1,23 @@ +# Coomer + +KToolBox support downloading from Coomer.su / Coomer.party, but currently need to **configure** before it can use. + +You need to set the configuration by `prod.env` dotenv file or system environment variables: +```dotenv +# For Coomer API +KTOOLBOX_API__NETLOC=coomer.su + +# For downloading files from Coomer server +KTOOLBOX_API__FILES_NETLOC=coomer.su +``` + +## About Coomer + +Description from [https://coomer.su](https://coomer.su) : + +> Coomer is a public archiver for: +> +> - OnlyFans +> - Fansly +> +> Contributors here upload content and share it here for easy searching and organization. To get started viewing content, either search for creators on the creators page, or search for content on the posts page. If you want to contribute content, head over to the import page. diff --git a/docs/en/index.md b/docs/en/index.md index c93071d..e95c144 100644 --- a/docs/en/index.md +++ b/docs/en/index.md @@ -68,7 +68,7 @@ ### Command -For more information, use the help command or goto [Commands](commands.md) page. +For more information, use the help command or goto [Commands](commands/guide.md) page. #### ❓ Get general help ```bash diff --git a/docs/zh/commands/guide.md b/docs/zh/commands/guide.md new file mode 100644 index 0000000..f0f8529 --- /dev/null +++ b/docs/zh/commands/guide.md @@ -0,0 +1,77 @@ +# 向导 + +前往 [参考](reference.md) 所有命令以及它们的标志/参数. + +## 获取帮助总览 + +- `--help`, `-h` + +```bash +ktoolbox -h +``` + +## 获取某个命令的帮助信息 + +- `--help`, `-h` + +```bash +ktoolbox download-post -h +``` + +## 下载指定的作品 + +`download-post` + +```bash +ktoolbox download-post https://kemono.su/fanbox/user/49494721/post/6608808 +``` +??? info "如果部分文件下载失败" + 如果部分文件下载失败，你可以尝试重新运行命令，已下载完成的文件会被 **跳过**。 + +## 下载作者的所有作品 + +`sync-creator` + +```bash +ktoolbox sync-creator https://kemono.su/fanbox/user/9016 +``` +??? info "输出" + 默认情况下你会在作者目录下得到一个 `creator-indices.ktoolbox` 文件，你可以用它来更新目录。 + + +## 更新一个作者目录 + +`sync-creator` + +- `--update-with` + +```bash +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --update-with=./xxx/creator-indices.ktoolbox +``` +??? info "关于 `creator-indices.ktoolbox` 文件" + `creator-indices.ktoolbox` 包含目录下的所有作品的信息和路径。 + +## 下载在指定时间范围内发布的作品 + +`sync-creator` + +- `--start-time` +- `--end-time` + +```bash +# 从 2023-8-5 到 2023-12-6 +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --start-time=2023-8-5 --end-time=2023-12-6 + +# 从 2023-8-5 到 现在 +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --start-time=2023-8-5 + +# 2023-8-5 之前 +ktoolbox sync-creator https://kemono.su/fanbox/user/641955 --end-time=2023-8-5 +``` + +### 时间格式 + +时间值应当符合 `%Y-%m-%d` 格式，例如： +- `2023-12-7` +- `2023-12-07` +- `2023-12-31` diff --git a/docs/zh/commands/reference.md b/docs/zh/commands/reference.md new file mode 100644 index 0000000..1455e5b --- /dev/null +++ b/docs/zh/commands/reference.md @@ -0,0 +1,5 @@ +# 参考 + +::: ktoolbox.cli.KToolBoxCli + options: + show_signature: false \ No newline at end of file diff --git a/docs/zh/coomer.md b/docs/zh/coomer.md new file mode 100644 index 0000000..baead25 --- /dev/null +++ b/docs/zh/coomer.md @@ -0,0 +1,23 @@ +# Coomer + +KToolBox 支持从 Coomer.su / Coomer.party 下载，但目前在它能够使用之前需要进行 **配置**。 + +你需要通过 dotenv文件 `prod.env` 或系统环境变量来设置配置： +```dotenv +# Coomer API +KTOOLBOX_API__NETLOC=coomer.su + +# 用于从 Coomer 服务器下载文件 +KTOOLBOX_API__FILES_NETLOC=coomer.su +``` + +## 关于 Coomer + +官网 [https://coomer.su](https://coomer.su) 的介绍： + +> Coomer is a public archiver for: +> +> - OnlyFans +> - Fansly +> +> Contributors here upload content and share it here for easy searching and organization. To get started viewing content, either search for creators on the creators page, or search for content on the posts page. If you want to contribute content, head over to the import page. diff --git a/docs/zh/index.md b/docs/zh/index.md index 0f5d34d..32b6dc1 100644 --- a/docs/zh/index.md +++ b/docs/zh/index.md @@ -68,7 +68,7 @@ ### 命令 -使用帮助命令或前往 [命令](./commands.md) 页面查看更多帮助。 +使用帮助命令或前往 [命令](commands/guide.md) 页面查看更多帮助。 #### ❓ 获取帮助总览 ```bash diff --git a/ktoolbox/__init__.py b/ktoolbox/__init__.py index ac75f85..6a31ea3 100644 --- a/ktoolbox/__init__.py +++ b/ktoolbox/__init__.py @@ -1,4 +1,4 @@ __title__ = "KToolBox" # noinspection SpellCheckingInspection __description__ = "A useful CLI tool for downloading posts in Kemono.party / .su" -__version__ = "0.3.1" +__version__ = "0.3.2" diff --git a/ktoolbox/action/job.py b/ktoolbox/action/job.py index 1b59b72..a515298 100644 --- a/ktoolbox/action/job.py +++ b/ktoolbox/action/job.py @@ -1,19 +1,20 @@ +from datetime import datetime from pathlib import Path -from typing import List, Union, Tuple +from typing import List, Union, Optional import aiofiles from loguru import logger from pathvalidate import sanitize_filename from ktoolbox.action import ActionRet, fetch_all_creator_posts, FetchInterruptError -from ktoolbox.action.utils import generate_post_path_name +from ktoolbox.action.utils import generate_post_path_name, filter_posts_by_time, filter_posts_by_indices from ktoolbox.api.model import Post from ktoolbox.api.posts import get_creator_post from ktoolbox.configuration import config, PostStructureConfiguration from ktoolbox.enum import PostFileTypeEnum, DataStorageNameEnum from ktoolbox.job import Job, CreatorIndices -__all__ = ["create_job_from_post", "filter_posts_with_indices", "create_job_from_creator"] +__all__ = ["create_job_from_post", "create_job_from_creator"] async def create_job_from_post( @@ -82,27 +83,6 @@ async def create_job_from_post( return jobs -def filter_posts_with_indices(posts: List[Post], indices: CreatorIndices) -> Tuple[List[Post], CreatorIndices]: - """ - Compare and filter posts by ``CreatorIndices`` data - - Only keep posts that was edited after last download. - - :param posts: Posts to filter - :param indices: ``CreatorIndices`` data to use - :return: A updated ``List[Post]`` and updated **new** ``CreatorIndices`` instance - """ - new_list = list( - filter( - lambda x: x.id not in indices.posts or x.edited > indices.posts[x.id].edited, posts - ) - ) - new_indices = indices.model_copy(deep=True) - for post in new_list: - new_indices.posts[post.id] = post - return new_list, new_indices - - async def create_job_from_creator( service: str, creator_id: str, @@ -112,7 +92,9 @@ async def create_job_from_creator( all_pages: bool = False, o: int = None, save_creator_indices: bool = True, - mix_posts: bool = None + mix_posts: bool = None, + start_time: Optional[datetime], + end_time: Optional[datetime] ) -> ActionRet[List[Job]]: """ Create a list of download job from a creator @@ -127,6 +109,8 @@ async def create_job_from_creator( :param save_creator_indices: Record ``CreatorIndices`` data for update posts from current creator directory :param mix_posts: Save all files from different posts at same path, \ ``update_from``, ``save_creator_indices`` will be ignored if enabled + :param start_time: Start time of the time range + :param end_time: End time of the time range """ mix_posts = config.job.mix_posts if mix_posts is None else mix_posts @@ -145,13 +129,17 @@ async def create_job_from_creator( post_list = ret.data else: return ActionRet(**ret.model_dump(mode="python")) + + # Filter posts by publish time + if start_time or end_time: + post_list = list(filter_posts_by_time(post_list, start_time, end_time)) logger.info(f"Get {len(post_list)} posts, start creating jobs") # Filter posts and generate ``CreatorIndices`` if not mix_posts: indices = None if update_from: - post_list, indices = filter_posts_with_indices(post_list, update_from) + post_list, indices = filter_posts_by_indices(post_list, update_from) logger.info(f"{len(post_list)} posts will be downloaded") elif save_creator_indices: # It's unnecessary to create indices again when ``update_from`` was provided indices = CreatorIndices( diff --git a/ktoolbox/action/utils.py b/ktoolbox/action/utils.py index 1cd30e0..f7b062a 100644 --- a/ktoolbox/action/utils.py +++ b/ktoolbox/action/utils.py @@ -1,9 +1,13 @@ +from datetime import datetime +from typing import Optional, List, Generator, Any, Tuple + from pathvalidate import sanitize_filename from ktoolbox.api.model import Post from ktoolbox.configuration import config +from ktoolbox.job import CreatorIndices -__all__ = ["generate_post_path_name"] +__all__ = ["generate_post_path_name", "filter_posts_by_time", "filter_posts_by_indices"] def generate_post_path_name(post: Post) -> str: @@ -12,3 +16,60 @@ def generate_post_path_name(post: Post) -> str: return post.id else: return sanitize_filename(post.title) + + +def _match_post_time( + post: Post, + start_time: Optional[datetime], + end_time: Optional[datetime] +) -> bool: + """ + Check if the post publish date match the time range. + + :param post: Target post object + :param start_time: Start time of the time range + :param end_time: End time of the time range + :return: Whether if the post publish date match the time range + """ + if start_time and post.published < start_time: + return False + if end_time and post.published > end_time: + return False + return True + + +def filter_posts_by_time( + post_list: List[Post], + start_time: Optional[datetime], + end_time: Optional[datetime] +) -> Generator[Post, Any, Any]: + """ + Filter posts by publish time range + + :param post_list: List of posts + :param start_time: Start time of the time range + :param end_time: End time of the time range + """ + post_filter = filter(lambda x: _match_post_time(x, start_time, end_time), post_list) + yield from post_filter + + +def filter_posts_by_indices(posts: List[Post], indices: CreatorIndices) -> Tuple[List[Post], CreatorIndices]: + """ + Compare and filter posts by ``CreatorIndices`` data + + Only keep posts that was edited after last download. + + :param posts: Posts to filter + :param indices: ``CreatorIndices`` data to use + :return: A updated ``List[Post]`` and updated **new** ``CreatorIndices`` instance + """ + new_list = list( + filter( + lambda x: x.id not in indices.posts or x.edited > indices.posts[x.id].edited, posts + ) + ) + new_indices = indices.model_copy(deep=True) + for post in new_list: + new_indices.posts[post.id] = post + return new_list, new_indices diff --git a/ktoolbox/cli.py b/ktoolbox/cli.py index f807079..251946a 100644 --- a/ktoolbox/cli.py +++ b/ktoolbox/cli.py @@ -1,5 +1,6 @@ +from datetime import datetime from pathlib import Path -from typing import Union, overload +from typing import Union, overload, Tuple import aiofiles from loguru import logger @@ -190,7 +191,8 @@ async def sync_creator( *, update_from: Path = None, save_creator_indices: bool = True, - mix_posts: bool = None + mix_posts: bool = None, + time_range: Tuple[str, str] = None ): ... @@ -203,7 +205,8 @@ async def sync_creator( *, update_from: Path = None, save_creator_indices: bool = True, - mix_posts: bool = None + mix_posts: bool = None, + time_range: Tuple[str, str] = None ): ... @@ -216,7 +219,9 @@ async def sync_creator( *, update_from: Union[Path, str] = None, save_creator_indices: bool = True, - mix_posts: bool = None + mix_posts: bool = None, + start_time: str = None, + end_time: str = None, ): """ Sync all posts from a creator @@ -224,17 +229,24 @@ async def sync_creator( You can update the directory anytime after download finished, \ such as to update after creator published new posts. - * If ``update_from`` was provided, it should be located **inside the creator directory**. + * If ``update_from`` was provided, the file should be located **inside the creator directory**. + * ``start_time`` & ``end_time`` example: ``2023-12-7``, ``2023-12-07`` :param url: The post URL :param service: The service where the post is located :param creator_id: The ID of the creator :param path: Download path, default is current directory :param update_from: ``CreatorIndices`` data path for update posts from current creator directory, \ - ``save_creator_indices`` will be enabled if this provided + ``save_creator_indices`` will be enabled if this provided :param save_creator_indices: Record ``CreatorIndices`` data for update posts from current creator directory :param mix_posts: Save all files from different posts at same path, \ - ``update_from``, ``save_creator_indices`` will be ignored if enabled + ``update_from``, ``save_creator_indices`` will be ignored if enabled + :param start_time: Start time of the published time range for posts downloading. \ + Set to ``0`` if ``None`` was given. \ + Time format: ``%Y-%m-%d`` + :param end_time: End time of the published time range for posts downloading. \ + Set to latest time (infinity) if ``None`` was given. \ + Time format: ``%Y-%m-%d`` """ # Get service, creator_id if url: @@ -291,7 +303,9 @@ async def sync_creator( update_from=indices, all_pages=True, save_creator_indices=save_creator_indices, - mix_posts=mix_posts + mix_posts=mix_posts, + start_time=datetime.strptime(start_time, "%Y-%m-%d") if start_time else None, + end_time=datetime.strptime(end_time, "%Y-%m-%d") if end_time else None ) if ret: job_runner = JobRunner(job_list=ret.data) diff --git a/ktoolbox/downloader/downloader.py b/ktoolbox/downloader/downloader.py index 2a73c42..59dfc1b 100644 --- a/ktoolbox/downloader/downloader.py +++ b/ktoolbox/downloader/downloader.py @@ -111,6 +111,7 @@ def cancel(self): generate_msg( f"Retrying ({x.attempt_number})", message=x.outcome.result().message if not x.outcome.failed else None, + exception=x.outcome.exception() ) ), reraise=True diff --git a/mkdocs.yml b/mkdocs.yml index 6cf5835..1570aba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,11 +14,14 @@ nav: - Welcome to KToolBox: index.md - About Kemono: about-kemono.md - More: more.md - - Commands: commands.md + - Commands: + - Guide: commands/guide.md + - Reference: commands/reference.md - Configuration: - - Guide: configuration/guide.md - - Reference: configuration/reference.md + - Guide: configuration/guide.md + - Reference: configuration/reference.md - iOS Shortcuts: shortcut.md + - Coomer: coomer.md - FAQ: faq.md - API Documentation: api.md diff --git a/poetry.lock b/poetry.lock index 967743a..143ffe4 100644 --- a/poetry.lock +++ b/poetry.lock @@ -915,13 +915,13 @@ mkdocs = ">=1.1" [[package]] name = "mkdocs-material" -version = "9.4.14" +version = "9.5.0" description = "Documentation that simply works" optional = false python-versions = ">=3.8" files = [ - {file = "mkdocs_material-9.4.14-py3-none-any.whl", hash = "sha256:dbc78a4fea97b74319a6aa9a2f0be575a6028be6958f813ba367188f7b8428f6"}, - {file = "mkdocs_material-9.4.14.tar.gz", hash = "sha256:a511d3ff48fa8718b033e7e37d17abd9cc1de0fdf0244a625ca2ae2387e2416d"}, + {file = "mkdocs_material-9.5.0-py3-none-any.whl", hash = "sha256:47323c9943756ced14487d0607caa283e350e8fd4c724b224bd69ccb0b0731f2"}, + {file = "mkdocs_material-9.5.0.tar.gz", hash = "sha256:f8f9ccb526bd8a8ac2e5ec2e64e1a167cfb53cf6a84c8924dbb607f0a70e74ae"}, ] [package.dependencies] @@ -1667,13 +1667,13 @@ pyyaml = "*" [[package]] name = "referencing" -version = "0.31.1" +version = "0.32.0" description = "JSON Referencing + Python" optional = false python-versions = ">=3.8" files = [ - {file = "referencing-0.31.1-py3-none-any.whl", hash = "sha256:c19c4d006f1757e3dd75c4f784d38f8698d87b649c54f9ace14e5e8c9667c01d"}, - {file = "referencing-0.31.1.tar.gz", hash = "sha256:81a1471c68c9d5e3831c30ad1dd9815c45b558e596653db751a2bfdd17b3b9ec"}, + {file = "referencing-0.32.0-py3-none-any.whl", hash = "sha256:bdcd3efb936f82ff86f993093f6da7435c7de69a3b3a5a06678a6050184bee99"}, + {file = "referencing-0.32.0.tar.gz", hash = "sha256:689e64fe121843dcfd57b71933318ef1f91188ffb45367332700a86ac8fd6161"}, ] [package.dependencies] diff --git a/pyproject.toml b/pyproject.toml index b1a60c4..a0d6bd0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "ktoolbox" -version = "0.3.1" +version = "0.3.2" description = "A useful CLI tool for downloading posts in Kemono.party / .su" authors = ["Ljzd-PRO "] readme = "README.md"