Skip to content

Commit

Permalink
Merge branch 'devel' into pure-py
Browse files Browse the repository at this point in the history
# Conflicts:
#	poetry.lock
#	pyproject.toml
  • Loading branch information
Ljzd-PRO committed Nov 18, 2024
2 parents f6e535f + fc37cf0 commit a1c33e6
Show file tree
Hide file tree
Showing 12 changed files with 170 additions and 133 deletions.
33 changes: 26 additions & 7 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,40 @@ assignees: ''

---

**Describe the bug**
## Describe the bug

A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
## Command & Configuration

The command you execute and the configuration you use.

For example:

### Command

```shell
ktoolbox sync-creator https://kemono.su/fanbox/user/9016 --length=10
```

**Expected behavior**
A clear and concise description of what you expected to happen.
### Configuration

```dotenv
KTOOLBOX_API__NETLOC=coomer.su
KTOOLBOX_API__FILES_NETLOC=coomer.su
```

## Screenshots

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Platform(please complete the following information):**
## Platform

Please complete the following information

- OS: [e.g. Windows]
- Python Version [e.g. 3.11]
- KToolBox Version [e.g. 0.10.0]

**Additional context**
Add any other context about the problem here.
19 changes: 8 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,21 @@

### 💡 Feature

- Added a **graphical configuration editor**, making configuration editing simple and convenient.
- Run `ktoolbox config-editor`
- Added a command to generate a complete sample configuration file.
- Run `ktoolbox example-env`
- **Python** versions below 3.8.1 (**<3.8.1**) are no longer supported
- The `search-creator` command will include search results with similar names.
- For example, the search parameter `--name abc` might return author information such as: `abc, abcd, hi-abc`
- Share an HTTPX client to reuse underlying TCP connections through an HTTP connection pool when calling APIs and downloading,
**significantly improving query and download speeds as well as connection stability**

[//]: # (### 🪲 Fix)

- - -

### 💡 新特性

- 新增 **图形化配置编辑器**,配置编辑将变得简单方便
- 运行 `ktoolbox config-editor`
- 新增命令可生成完整的配置文件样例
- 运行 `ktoolbox example-env`
- **Python** 3.8.1 以下 (**<3.8.1**) 的版本不再受支持
- search-creator 搜索作者的命令将包含那些名字相近的搜索结果
- 如搜索参数 `--name abc` 可能得到如下作者信息:`abc, abcd, hi-abc`
- 共享 HTTPX 客户端,调用 API 和下载时将通过 HTTP 连接池重用底层 TCP 连接,**显著提升查询、下载速度和连接稳定性**

[//]: # (### 🪲 修复)

**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.9.0...v0.10.0
**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.10.0...v0.11.0
31 changes: 19 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,22 @@

<p align="center">
KToolBox is a useful CLI tool for downloading posts content in
<a href="https://kemono.su/">Kemono.party / Kemono.su</a>
<a href="https://kemono.su/">Kemono.su / Kemono.party</a>
</p>

<p align="center">
<a href="https://pypi.org/project/ktoolbox" target="_blank">
<img src="https://img.shields.io/github/v/release/Ljzd-PRO/KToolBox?logo=python" alt="Version">
</a>

<a href="https://pypi.org/project/ktoolbox" target="_blank">
<img src="https://img.shields.io/pypi/dm/ktoolbox?label=pypi-downloads" alt="PyPI Downloads">
</a>

<a href="https://github.com/Ljzd-PRO/KToolBox/releases" target="_blank">
<img src="https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/total?label=github-downloads" alt="GitHub Release Downloads">
</a>

<a href="./LICENSE">
<img src="https://img.shields.io/github/license/Ljzd-PRO/KToolBox" alt="BSD 3-Clause"/>
</a>
Expand All @@ -41,23 +49,22 @@
<a href="./README.md">English</a> | <a href="./README_zh-CN.md">中文</a>
</p>

<img src="https://cdn.jsdelivr.net/gh/Ljzd-PRO/KToolBox@latest/static/preview-1.png" alt="Preview">

## Features

- Support for **multi-thread** downloads (technically, coroutine)
- **Retry** after download failed
- Ability to download individual post as well as **all post** by a specified creator/artist
- **Update downloaded** creator/artist directories to the latest status
- Customize the **structure** of downloaded post/creator **directories**
- Search for creators/artists and posts, and **export the results**
- **Cross-platform** support & **iOS shortcuts** available
- For Coomer.su / Coomer.party support, check document [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/) for more.
- Supports concurrent downloads
- Automatically retries API calls and downloads after failures
- Allows downloading individual posts or **all posts** of a specified artist
- Can **update downloaded** artist directories to the latest state
- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists
- Enables excluding **specified file formats** or downloading only specified formats
- Allows searching for artists and posts, with options to export results
- Compatible with all platforms, with iOS shortcuts provided
- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)

## Dev Plan

- [ ] GUI
- [x] Add uvloop support for Unix platform
- [ ] Discord support

## Tutorial

Expand Down
27 changes: 17 additions & 10 deletions README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<p align="center">
KToolBox 是一个用于下载
<a href="https://kemono.su/">Kemono.party / Kemono.su</a>
<a href="https://kemono.su/">Kemono.su / Kemono.party</a>
中作品内容的实用命令行工具
</p>

Expand All @@ -17,6 +17,14 @@
<img src="https://img.shields.io/github/v/release/Ljzd-PRO/KToolBox?logo=python" alt="Version">
</a>

<a href="https://pypi.org/project/ktoolbox" target="_blank">
<img src="https://img.shields.io/pypi/dm/ktoolbox?label=pypi-downloads" alt="PyPI Downloads">
</a>

<a href="https://github.com/Ljzd-PRO/KToolBox/releases" target="_blank">
<img src="https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/total?label=github-downloads" alt="GitHub Release Downloads">
</a>

<a href="./LICENSE">
<img src="https://img.shields.io/github/license/Ljzd-PRO/KToolBox" alt="BSD 3-Clause"/>
</a>
Expand All @@ -42,23 +50,22 @@
<a href="./README.md">English</a> | <a href="./README_zh-CN.md">中文</a>
</p>

<img src="https://cdn.jsdelivr.net/gh/Ljzd-PRO/KToolBox@latest/static/preview-1.png" alt="Preview">

## 功能

- 支持 **多线程** 下载(技术上是协程)
- 下载失败后进行 **重试**
- 支持并发下载
- API 调用和下载失败后 **自动重试**
- 支持下载单个作品以及指定的画师的 **所有作品**
-**更新已下载** 的画师目录至最新状态
- 可自定义下载的作品/画师 **目录结构**
- 可搜索画师和作品,并 **导出结果**
- 支持全平台,并提供 **iOS 快捷指令**
- 对于 Coomer.su / Coomer.party 的支持,请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)
- 支持自定义下载的作品/画师 **文件和目录名格式****目录结构**
- 支持排除 **指定格式** 的文件或仅下载指定格式的文件
- 可搜索画师和作品,并导出结果
- 支持全平台,并提供 iOS 快捷指令
- 对于 Coomer.su / Coomer.party 的支持,请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)

## 开发计划

- [ ] GUI
- [x] 对 Unix 平台增加 uvloop 支持
- [ ] Discord 下载支持

## 使用方法

Expand Down
16 changes: 9 additions & 7 deletions docs/en/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,15 @@

## Features

- Support for **multi-thread** downloads (technically, coroutine)
- **Retry** after download failed
- Ability to download individual post as well as **all post** by a specified creator/artist
- **Update downloaded** creator/artist directories to the latest status
- Customize the **structure** of downloaded post/creator **directories**
- Search for creators/artists and posts, and **export the results**
- **Cross-platform** support & **iOS shortcuts** available
- Supports concurrent downloads
- Automatically retries API calls and downloads after failures
- Allows downloading individual posts or **all posts** of a specified artist
- Can **update downloaded** artist directories to the latest state
- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists
- Enables excluding **specified file formats** or downloading only specified formats
- Allows searching for artists and posts, with options to export results
- Compatible with all platforms, with iOS shortcuts provided
- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)

## Tutorial

Expand Down
14 changes: 8 additions & 6 deletions docs/zh/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<p style="text-align: center">
KToolBox 是一个用于下载
<a href="https://kemono.su/">Kemono.party / Kemono.su</a>
<a href="https://kemono.su/">Kemono.su / Kemono.party</a>
中作品内容的实用命令行工具
</p>

Expand Down Expand Up @@ -40,13 +40,15 @@

## 功能

- 支持 **多线程** 下载(技术上是协程)
- 下载失败后进行 **重试**
- 支持并发下载
- API 调用和下载失败后 **自动重试**
- 支持下载单个作品以及指定的画师的 **所有作品**
-**更新已下载** 的画师目录至最新状态
- 可自定义下载的作品/画师 **目录结构**
- 可搜索画师和作品,并 **导出结果**
- 支持全平台,并提供 **iOS 快捷指令**
- 支持自定义下载的作品/画师 **文件和目录名格式****目录结构**
- 支持排除 **指定格式** 的文件或仅下载指定格式的文件
- 可搜索画师和作品,并导出结果
- 支持全平台,并提供 iOS 快捷指令
- 对于 Coomer.su / Coomer.party 的支持,请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)

## 使用方法

Expand Down
2 changes: 1 addition & 1 deletion ktoolbox/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
__title__ = "KToolBox"
# noinspection SpellCheckingInspection
__description__ = "A useful CLI tool for downloading posts in Kemono.party / .su"
__version__ = "v0.10.0"
__version__ = "v0.11.0"
32 changes: 16 additions & 16 deletions ktoolbox/action/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,23 @@ async def search_creator(id: str = None, name: str = None, service: str = None)
:param service: The service for the creator
"""

async def inner(**kwargs):
def filter_func(creator: Creator):
"""Filter creators with attributes"""
for key, value in kwargs.items():
if value is None:
continue
elif creator.__getattribute__(key) != value:
return False
return True

ret = await get_creators()
if not ret:
return ret
creators = ret.data
return ActionRet(data=iter(filter(filter_func, creators)))
def filter_func(creator: Creator):
"""Filter creators with attributes"""
if id is not None and creator.id != id:
return False
if name is not None and name not in creator.name:
return False
if service is not None and creator.service != service:
return False
return True

return await inner(id=id, name=name, service=service)
ret = await get_creators()
if not ret:
base_ret = BaseRet.model_validate(ret.model_dump())
base_ret.data = iter([])
return base_ret
creators = ret.data
return ActionRet(data=iter(filter(filter_func, creators)))


# noinspection PyShadowingBuiltins
Expand Down
16 changes: 8 additions & 8 deletions ktoolbox/api/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ class BaseAPI(ABC, Generic[_T]):
path: str = "/"
method: Literal["get", "post"]
extra_validator: Optional[Callable[[str], BaseModel]] = None
client = httpx.AsyncClient(verify=config.ssl_verify)

Response = BaseModel
"""API response model"""
Expand Down Expand Up @@ -111,14 +112,13 @@ async def request(cls, path: str = None, **kwargs) -> APIRet[_T]:
url_parts = [config.api.scheme, config.api.netloc, f"{config.api.path}{path}", '', '', '']
url = str(urlunparse(url_parts))
try:
async with httpx.AsyncClient(verify=config.ssl_verify) as client:
res = await client.request(
method=cls.method,
url=url,
timeout=config.api.timeout,
follow_redirects=True,
**kwargs
)
res = await cls.client.request(
method=cls.method,
url=url,
timeout=config.api.timeout,
follow_redirects=True,
**kwargs
)
except Exception as e:
return APIRet(
code=RetCodeEnum.NetWorkError,
Expand Down
6 changes: 4 additions & 2 deletions ktoolbox/api/model/creator.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
from datetime import datetime

from pydantic import BaseModel

__all__ = ["Creator"]
Expand All @@ -10,11 +12,11 @@ class Creator(BaseModel):
"""The number of times this creator has been favorited"""
id: str
"""The ID of the creator"""
indexed: float
indexed: datetime
"""Timestamp when the creator was indexed, Unix time as integer"""
name: str
"""The name of the creator"""
service: str
"""The service for the creator"""
updated: float
updated: datetime
"""Timestamp when the creator was last updated, Unix time as integer"""
Loading

0 comments on commit a1c33e6

Please sign in to comment.