From dc227b2d0010dd66c7564ef5d88cc3b006cd07a0 Mon Sep 17 00:00:00 2001 From: N/Ame <173015200+grqz@users.noreply.github.com> Date: Sat, 14 Sep 2024 15:21:41 +1200 Subject: [PATCH 1/8] Create a simple CONTRIBUTING.md --- CONTRIBUTING.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..a6eef48 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,18 @@ +### Coding conventions +Since the provider comes in two parts(the **Provider**(coded in typescript) and the **Provider plugin**(coded in python)), we have different code formatting standards for them. +Please format your code by running this script below before you push a commit to your pull request: +```shell +# Make sure you have ruff, autopep8 and prettier installed already +ruff check --fix plugin/ +autopep8 --in-place plugin/ + +cd server +npx prettier --check --write 'src/**/*.{js,ts}' +cd .. +``` + +#### **Provider**(typescript): + + +#### **Provider plugin**(python): +We follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). From 18dd0c3d2cfe2df5f07b3221f8f268c3663d11d0 Mon Sep 17 00:00:00 2001 From: N/Ame <173015200+grqz@users.noreply.github.com> Date: Sat, 14 Sep 2024 15:22:33 +1200 Subject: [PATCH 2/8] add a newline --- CONTRIBUTING.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a6eef48..ebe3f76 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,7 @@ ### Coding conventions Since the provider comes in two parts(the **Provider**(coded in typescript) and the **Provider plugin**(coded in python)), we have different code formatting standards for them. Please format your code by running this script below before you push a commit to your pull request: + ```shell # Make sure you have ruff, autopep8 and prettier installed already ruff check --fix plugin/ From 2ed66bc3dd816cca1071d8ee7e181ff17066adf4 Mon Sep 17 00:00:00 2001 From: N/Ame <173015200+grqz@users.noreply.github.com> Date: Mon, 16 Sep 2024 16:58:31 +1200 Subject: [PATCH 3/8] More details --- CONTRIBUTING.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ebe3f76..02bface 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,7 @@ ### Coding conventions + Since the provider comes in two parts(the **Provider**(coded in typescript) and the **Provider plugin**(coded in python)), we have different code formatting standards for them. -Please format your code by running this script below before you push a commit to your pull request: +Please format your code by running this script below before you push a commit: ```shell # Make sure you have ruff, autopep8 and prettier installed already @@ -12,8 +13,21 @@ npx prettier --check --write 'src/**/*.{js,ts}' cd .. ``` +If you don't want the code formatter to change your code, you may check if there's any problem with your code with this script: + +```shell + +# Make sure you have ruff, autopep8 and prettier installed already +ruff check plugin/ +autopep8 plugin/ + +cd server +npx prettier --check 'src/**/*.{js,ts}' +cd .. +``` + #### **Provider**(typescript): #### **Provider plugin**(python): -We follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). +As a yt-dlp plugin, we follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). From c26c657f533b30a8957340f548e3b1a7f82dbf9b Mon Sep 17 00:00:00 2001 From: N/Ame <173015200+grqz@users.noreply.github.com> Date: Tue, 17 Sep 2024 13:56:27 +1200 Subject: [PATCH 4/8] add info about typescript --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 02bface..a4fdb4e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,7 +27,7 @@ cd .. ``` #### **Provider**(typescript): - +Please make sure your code formatting passes eslint and prettier checks by running the script above. #### **Provider plugin**(python): -As a yt-dlp plugin, we follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). +As a yt-dlp plugin, we follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). You can use ruff and autopep8 to format your code, as shown in the script above. From 6c11edf9018629c475a80c90546408497448aab1 Mon Sep 17 00:00:00 2001 From: N/Ame <173015200+grqz@users.noreply.github.com> Date: Sat, 21 Sep 2024 13:19:52 +1200 Subject: [PATCH 5/8] Update README.md --- README.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 6f06010..358c025 100644 --- a/README.md +++ b/README.md @@ -13,17 +13,17 @@ The provider comes in two parts: 1. **Provider**: Two options - - (a) An HTTP server that generates the POT, and has interfaces for the plugin to retrieve data from (easy setup + docker image provided) - - (b) A POT generation script supplied via extractor arguments -2. **Provider plugin**: uses POT plugin framework to retrieve data from the provider, allowing yt-dlp to simulate having passed the 'bot check' + - (b) A POT generation script, and has command line options for the plugin to invoke (needs to transpile the script) +2. **Provider plugin**: uses POT plugin framework to retrieve data from the provider, allowing yt-dlp to simulate having passed the 'bot check'. ## Installation ### Base Requirements -1. **Requires yt-dlp `NIGHTLY 2024.09.13.232912` or above.** +1. Requires yt-dlp `NIGHTLY 2024.09.13.232912` or above. 2. If using Docker image for option (a) for the provider, the Docker runtime is required. - Otherwise, Node.js (>= 18) and Yarn are required. You will also need to clone the repository. + Otherwise, Node.js (>= 18) and Yarn are required. You will also need git to clone the repository. ### 1. Set up the provider @@ -31,7 +31,7 @@ There are two options for the provider, an always running POT generation HTTP se #### (a) HTTP Server Option -The provider is a Node.js HTTP server. You have two options of running it: as a prebuilt docker image, or manually as a node application. +The provider is a Node.js HTTP server. You have two options for running it: as a prebuilt docker image, or manually as a node application. **Docker:** @@ -42,7 +42,7 @@ docker run --name bgutil-provider -d -p 4416:4416 brainicism/bgutil-ytdlp-pot-pr **Native:** ```shell -# replace 0.6.0 with the latest version, or a matching plugin +# Replace 0.6.0 with the latest version or the one that matches the plugin git clone --single-branch --branch 0.6.0 https://github.com/Brainicism/bgutil-ytdlp-pot-provider.git cd bgutil-ytdlp-pot-provider/server/ yarn install --frozen-lockfile @@ -61,6 +61,11 @@ node build/main.js - **POST /get_pot**: Accepts a `visitor_data` (unauthenticated), `data_sync_id` (authenticated) or an empty body in the request body. If no identifier is passed, a new unauthenticated `visitor_data` will be generated. Returns `po_token` and the associated identifier `visit_identifier`. - **POST /invalidate_caches**: Resets the PO token cache, forcing new tokens to be generated on next fetch. +- **GET /ping**: Ping the server. The response includes: + * `logging`: Logging verbosity(`normal` or `verbose`) + * `token_ttl_hours`: The current applied `TOKEN_TTL` value, defaults to 6. + * `server_uptime`: Uptime of the server process. + * `version`: Current server version. #### (b) Generation Script Option @@ -68,10 +73,11 @@ node build/main.js 1. Transpile the generation script to Javascript: ```shell -# Clone/extract the contents into your home directory (`~/` on Unix-based systems, `%USERPROFILE%` for Windows) -# if you want to use this method without needing to specify `getpot_bgutil_script` extractor argument on each yt-dlp invocation. +# If you want to use this method without specifying `getpot_bgutil_script` extractor argument +# on each yt-dlp invocation, clone/extract the source code into your home directory. +# Replace `~` with `%USERPROFILE%` if using Windows cd ~ -# replace 0.6.0 with the latest version, or a matching plugin +# Replace 0.6.0 with the latest version or the one that matches the plugin git clone --single-branch --branch 0.6.0 https://github.com/Brainicism/bgutil-ytdlp-pot-provider.git cd bgutil-ytdlp-pot-provider/server/ yarn install --frozen-lockfile @@ -113,7 +119,7 @@ If changing the port or IP used for the provider server, pass it to yt-dlp via ` --- -If using option (b) script for the provider, with the default script location in your home directory (i.e: `~/bgutil-ytdlp-pot-provider`), you can also use yt-dlp like normal. +If using option (b) script for the provider, with the default script location in your home directory (i.e: `~/bgutil-ytdlp-pot-provider` or `%USERPROFILE%\bgutil-ytdlp-pot-provider`), you can also use yt-dlp like normal. If you installed the script in a different location, pass it as the extractor argument `getpot_bgutil_script` to `youtube` for each yt-dlp call. From c5f1f96ae2cfe6d786cadf96c060730bdb03c07f Mon Sep 17 00:00:00 2001 From: grqx_wsl <173253225+grqx@users.noreply.github.com> Date: Sat, 21 Sep 2024 03:54:23 +0000 Subject: [PATCH 6/8] simplify script --- CONTRIBUTING.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a4fdb4e..217efba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,26 +8,21 @@ Please format your code by running this script below before you push a commit: ruff check --fix plugin/ autopep8 --in-place plugin/ -cd server -npx prettier --check --write 'src/**/*.{js,ts}' -cd .. +npx prettier --check --write "server/src/**/*.{js,ts}" ``` If you don't want the code formatter to change your code, you may check if there's any problem with your code with this script: ```shell - # Make sure you have ruff, autopep8 and prettier installed already ruff check plugin/ autopep8 plugin/ -cd server -npx prettier --check 'src/**/*.{js,ts}' -cd .. +npx prettier --check --write "server/src/**/*.{js,ts}" ``` #### **Provider**(typescript): -Please make sure your code formatting passes eslint and prettier checks by running the script above. +Please make sure your code passes eslint and prettier checks by running the script above. #### **Provider plugin**(python): As a yt-dlp plugin, we follow the [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions). You can use ruff and autopep8 to format your code, as shown in the script above. From 88d8b8c733d04e928951425807b62e455f98b250 Mon Sep 17 00:00:00 2001 From: grqx_wsl <173253225+grqx@users.noreply.github.com> Date: Sat, 21 Sep 2024 04:07:38 +0000 Subject: [PATCH 7/8] add eslint --- CONTRIBUTING.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 217efba..c3c3417 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,21 +4,27 @@ Since the provider comes in two parts(the **Provider**(coded in typescript) and Please format your code by running this script below before you push a commit: ```shell -# Make sure you have ruff, autopep8 and prettier installed already +# Make sure you have ruff, autopep8, eslint and prettier installed already ruff check --fix plugin/ autopep8 --in-place plugin/ -npx prettier --check --write "server/src/**/*.{js,ts}" +cd server +npx eslint --fix --max-warnings=0 src/ +npx prettier --check --write "src/**/*.{js,ts}" +cd .. ``` -If you don't want the code formatter to change your code, you may check if there's any problem with your code with this script: +If you don't want the code formatter to fix your code automatically, you may check if there's any problem with your code with this script: ```shell -# Make sure you have ruff, autopep8 and prettier installed already +# Make sure you have ruff, autopep8, eslint and prettier installed already ruff check plugin/ autopep8 plugin/ -npx prettier --check --write "server/src/**/*.{js,ts}" +cd server +npx eslint --max-warnings=0 src/ +npx prettier --check "src/**/*.{js,ts}" +cd .. ``` #### **Provider**(typescript): From 700e3c634a8e1fe30943ffdeffd0ad32324d5389 Mon Sep 17 00:00:00 2001 From: grqx_wsl <173253225+grqx@users.noreply.github.com> Date: Sat, 21 Sep 2024 12:03:33 +0000 Subject: [PATCH 8/8] Update CONTRIBUTING.md --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c3c3417..80134dc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ ### Coding conventions -Since the provider comes in two parts(the **Provider**(coded in typescript) and the **Provider plugin**(coded in python)), we have different code formatting standards for them. +Since the provider consists of two parts(the **Provider**(coded in typescript) and the **Provider plugin**(coded in python)), we have different code formatting standards for them. Please format your code by running this script below before you push a commit: ```shell