[reject_files_outside_pct_range_of_original] initial version

source/reject_files_outside_pct_range_of_original/.gitignore

@@ -0,0 +1,4 @@
+**/__pycache__
+*.py[cod]
+**/site-packages
+settings.json

source/reject_files_outside_pct_range_of_original/changelog.md

@@ -0,0 +1,5 @@
+
+**<span style="color:#56adda">0.0.1</span>**
+- Initial version
+- Based on reject_files_larger_than_original v0.0.4
+- adds a configurable range for file size expressed as percentages of original size

source/reject_files_outside_pct_range_of_original/description.md

@@ -0,0 +1,91 @@
+
+:::warning
+Be sure to set the Plugin flow!
+See examples below.
+:::
+
+---
+
+### Config description:
+
+#### <span style="color:blue">Mark the task as failed</span>
+During the worker processing, if this plugin determines that a file is larger than the original file, it will fail the task and no other plugins will be run.
+
+Tasks that fail this way will be marked as failed in the Completed Tasks list.
+
+
+#### <span style="color:blue">Ignore files in future scans if end result is larger than source</span>
+If during post-processing, the final file is determined to be larger than the original file, the source file will be flagged to be ignored for all future library file tests.
+
+If during the worker processing, the file is determined to be larger than the original file and the Mark the task as failed plugin option is enabled, the source file will be flagged to be ignored for all future library file tests.
+
+If during a library scan or file event, a file is found that would otherwise meet the criteria to be added as a new pending task, if that file has been previously flagged to be ignored, then it shall be ignored regardless of the file's status in the Completed Tasks list.
+
+
+---
+
+#### Examples:
+
+###### <span style="color:magenta">Transcoding to H265 with the NVIDIA NVENC Plugin:</span>
+If you are trying to reduce your video library size by converting files to H265, you may want to first attempt to transcode 
+a file with the NVENC HEVC encoder to reduce its size.
+However, the results of such a conversion may be larger than the source depending on your hardware and configuration.
+
+If you end up with a larger file than the source after using this Plugin, you can use this Plugin to reject the encoded file 
+in order to keep the original.
+
+To do this, configure your Plugin Flow as follows:
+
+1. Video Encoder H265/HEVC - hevc_nvenc (NVIDIA GPU)
+2. Reject File if Larger than Original (This Plugin)
+3. Any other Plugins you wish to run against this library.
+
+**<span style="color:blue">Mark the task as failed</span>**
+> *(UNSELECTED)*
+
+**<span style="color:blue">Ignore files in future scans if end result is larger than source</span>**
+> *(SELECTED)*
+
+###### <span style="color:magenta">Transcoding to H265 with the NVIDIA NVENC Plugin - Fallback to CPU libx265 if larger:</span>
+Almost exactly the same flow as the previous example...
+But this time with a re-attempt to transcode using the CPU. 
+
+To do this, set the flow like this:
+
+1. Video Encoder H265/HEVC - hevc_nvenc (NVIDIA GPU)
+2. Reject File if Larger than Original (This Plugin)
+3. Video Encoder H265/HEVC - libx265 (CPU)
+4. Any other Plugins you wish to run against this library.
+
+In this case, the CPU HEVC encoder will re-attempt to transcode the file to HEVC/H265. 
+You should get much better results with this encoder, however the time to transcode will be much longer.
+
+**<span style="color:blue">Mark the task as failed</span>**
+> *(UNSELECTED)*
+
+**<span style="color:blue">Ignore files in future scans if end result is larger than source</span>**
+> *(SELECTED)*
+
+###### <span style="color:magenta">Reject file as the last in the flow (carry out post-processing file movements):</span>
+Placing this plugin as the last option in the plugin flow will cause it to simply reset the current working file.
+This will not fail the task process.
+
+During post-processing, the original file will be copied to the cache directory and any post-processing tasks can be carried out on that file.
+This allows other post-processing plugins to carry out tasks as they otherwise normally would, but on the original file.
+
+**<span style="color:blue">Mark the task as failed</span>**
+> *(UNSELECTED)*
+
+**<span style="color:blue">Ignore files in future scans if end result is larger than source</span>**
+> *(SELECTED)*
+
+###### <span style="color:magenta">Reject file at any time during worker processing (fail the task - skip post-processing):</span>
+Regardless of where this plugin is in the flow, if you select **Mark the task as failed**, it will fail the task and no post-processing file movements will be carried out.
+
+Future library scans will ignore this file, even after you have removed it from the Completed Task list.
+
+**<span style="color:blue">Mark the task as failed</span>**
+> *(SELECTED)*
+
+**<span style="color:blue">Ignore files in future scans if end result is larger than source</span>**
+> *(SELECTED)*

source/reject_files_outside_pct_range_of_original/info.json

@@ -0,0 +1,20 @@
+{
+    "author": "Josh.5, yajrendrag",
+    "compatibility": [
+        2
+    ],
+    "description": "This plugin will reset the current working file to the original source if it is outside the configure size range of the original source.",
+    "icon": "https://raw.githubusercontent.com/Josh5/unmanic.plugin.reject_files_larger_than_original/master/icon.png",
+    "id": "reject_files_outside_pct_range_of_original",
+    "name": "Reject files if outside percentage range of original",
+    "platform": [
+        "all"
+    ],
+    "priorities": {
+        "on_library_management_file_test": 2,
+        "on_worker_process": 2,
+        "on_postprocessor_file_movement": 5
+    },
+    "tags": "library file test",
+    "version": "0.0.1"
+}

source/reject_files_outside_pct_range_of_original/plugin.py

@@ -0,0 +1,270 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""
+    unmanic-plugins.plugin.py
+
+    Written by:               Josh.5 <[email protected]>
+    Date:                     31 Aug 2021, (12:11 PM)
+
+    Copyright:
+        Copyright (C) 2021 Josh Sunnex
+
+        This program is free software: you can redistribute it and/or modify it under the terms of the GNU General
+        Public License as published by the Free Software Foundation, version 3.
+
+        This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
+        implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+        for more details.
+
+        You should have received a copy of the GNU General Public License along with this program.
+        If not, see <https://www.gnu.org/licenses/>.
+
+"""
+import filecmp
+import logging
+import os
+import shutil
+from configparser import NoSectionError, NoOptionError
+
+from unmanic.libs.directoryinfo import UnmanicDirectoryInfo
+from unmanic.libs.unplugins.settings import PluginSettings
+
+# Configure plugin logger
+logger = logging.getLogger("Unmanic.Plugin.reject_files_outside_pct_range_of_original")
+
+
+# TODO: Write config options in description
+class Settings(PluginSettings):
+    settings = {
+        'fail_task_if_file_detected_outside':                 False,
+        'if_end_result_file_is_still_outside_mark_as_ignore': False,
+        'min_percentage_size':                               '',
+        'max_percentage_size':                               '',
+    }
+    def __init__(self, *args, **kwargs):
+        super(Settings, self).__init__(*args, **kwargs)
+        self.form_settings = {
+            "fail_task_if_file_detected_outside":                 {
+            "label": "Mark the task as failed",
+            },
+            "if_end_result_file_is_still_outside_mark_as_ignore": {
+            "label": "Ignore files in future scans if end result is outside the configured size range of the source (regardless of task history)",
+            },
+            "min_percentage_size": {
+            "label": "enter a percentage (as an integer) of original file size to use as a lower bound for acceptable file size",
+            },
+            "max_percentage_size": {
+            "label": "enter a percentage (as an integer) of original file size to use as an upper bound for acceptable file size",
+            },
+    }
+
+
+def file_marked_as_failed(settings, path):
+    """Read directory info to check if file was previously marked as failed"""
+    if settings.get_setting('if_end_result_file_is_still_outside_mark_as_ignore'):
+        directory_info = UnmanicDirectoryInfo(os.path.dirname(path))
+
+        try:
+            previously_failed = directory_info.get('reject_files_ouside_pct_range_of_original', os.path.basename(path))
+        except NoSectionError as e:
+            previously_failed = ''
+        except NoOptionError as e:
+            previously_failed = ''
+        except Exception as e:
+            logger.debug("Unknown exception {}.".format(e))
+            previously_failed = ''
+
+        if previously_failed:
+            # This stream already has been attempted and failed
+            return True
+
+    # Default to...
+    return False
+
+
+def write_file_marked_as_failed(path):
+    """Write entry to directory infor to mark this file as failed"""
+    directory_info = UnmanicDirectoryInfo(os.path.dirname(path))
+    directory_info.set('reject_files_ouside_pct_range_of_original', os.path.basename(path), 'Ignoring')
+    directory_info.save()
+    logger.debug("Ignore on next scan written for '{}'.".format(path))
+
+
+def on_library_management_file_test(data):
+    """
+    Runner function - enables additional actions during the library management file tests.
+
+    The 'data' object argument includes:
+        path                            - String containing the full path to the file being tested.
+        issues                          - List of currently found issues for not processing the file.
+        add_file_to_pending_tasks       - Boolean, is the file currently marked to be added to the queue for processing.
+
+    :param data:
+    :return:
+
+    """
+    # Get the path to the file
+    abspath = data.get('path')
+
+    # Configure settings object
+    settings = Settings(library_id=data.get('library_id'))
+
+    if file_marked_as_failed(settings, abspath):
+        # Ensure this file is not added to the pending tasks
+        data['add_file_to_pending_tasks'] = False
+        logger.debug("File '{}' has been previously marked as failed.".format(abspath))
+
+
+def on_worker_process(data):
+    """
+    Runner function - enables additional configured processing jobs during the worker stages of a task.
+
+    The 'data' object argument includes:
+        library_id              - The library that the current task is associated with.
+        exec_command            - A command that Unmanic should execute. Can be empty.
+        command_progress_parser - A function that Unmanic can use to parse the STDOUT of the command to collect progress stats. Can be empty.
+        file_in                 - The source file to be processed by the command.
+        file_out                - The destination that the command should output (may be the same as the file_in if necessary).
+        original_file_path      - The absolute path to the original file.
+        repeat                  - Boolean, should this runner be executed again once completed with the same variables.
+
+    :param data:
+    :return:
+    
+    """
+    # Default to no command required.
+    data['exec_command'] = []
+    data['repeat'] = False
+
+    # Get the path to the file
+    abspath = data.get('file_in')
+    original_file_path = data.get('original_file_path')
+    if not os.path.exists(abspath):
+        logger.debug("File in '{}' does not exist.".format(abspath))
+
+    if not os.path.exists(original_file_path):
+        logger.debug("Original file '{}' does not exist.".format(original_file_path))
+
+    # Configure settings object
+    settings = Settings(library_id=data.get('library_id'))
+
+    # Current cache file stats
+    current_file_stats = os.stat(os.path.join(abspath))
+    # Get the original file stats
+    original_file_stats = os.stat(os.path.join(original_file_path))
+
+    # Calculate minimum file size
+    min_pct_size = int(settings.get_setting('min_percentage_size'))/100
+    min_file_size = min_pct_size * int(original_file_stats.st_size)
+    # Calculate maximum file size
+    max_pct_size = int(settings.get_setting('max_percentage_size'))/100
+    max_file_size = max_pct_size * int(original_file_stats.st_size)
+
+    # Debug Logging
+    logger.debug("original file: '{}'".format(original_file_path))
+    logger.debug("current file: '{}'".format(abspath))
+    logger.debug("min_file_size: '{}', max_file_size: '{}'".format(min_file_size, max_file_size))
+    logger.debug("current_file_size: '{}', original_file_size: '{}'".format(current_file_stats.st_size, original_file_stats.st_size))
+
+    # Test that the source file is not outside the configured size range of the new file
+    if int(current_file_stats.st_size) > max_file_size or int(current_file_stats.st_size) < min_file_size:
+        if settings.get_setting('fail_task_if_file_detected_outside'):
+            # Add some worker logs to be transparent as to what is happening
+            if data.get('worker_log'):
+                data['worker_log'].append("\nFailing task as current cache file is outside the configured size range of the original file:")
+                data['worker_log'].append(
+                    "\n  - Original File: {} bytes '<em>{}</em>'".format(original_file_stats.st_size,
+                                                                         original_file_path))
+                data['worker_log'].append(
+                    "\n  - Cache File: {} bytes '<em>{}</em>'".format(current_file_stats.st_size, abspath))
+            # Create a job that will exit false
+            data['exec_command'] = ['false']
+            if settings.get_setting('if_end_result_file_is_still_outside_mark_as_ignore'):
+                # Write the failure file here because the post-processor file movement runner will not be executed
+                # if the task fails
+                write_file_marked_as_failed(original_file_path)
+            # Return here because the rest does not matter
+            return
+
+        # Add some more worker logs...
+        if data.get('worker_log'):
+            data['worker_log'].append(
+                "\nResetting task file back to original source as current cache file is outside the configured size range of the original file:")
+            data['worker_log'].append(
+                "\n  - Original File: {} bytes '<em>{}</em>'".format(original_file_stats.st_size,
+                                                                     original_file_path))
+            data['worker_log'].append(
+                "\n  - Cache File: {} bytes '<em>{}</em>'".format(current_file_stats.st_size, abspath))
+
+        # The current file is outside the configured size range of the original. Reset the cache file to the 'file_in'
+        data['file_in'] = original_file_path
+        logger.debug(
+            "Rejecting processed file as it is outside the configured size range of the original: '{}' > '{}'.".format(abspath,
+                                                                                              original_file_path))
+    else:
+        logger.debug("Keeping the processed file as it is inside the configured size range of the original.")
+
+
+def on_postprocessor_file_movement(data):
+    """
+    Runner function - configures additional postprocessor file movements during the postprocessor stage of a task.
+
+    The 'data' object argument includes:
+        source_data             - Dictionary containing data pertaining to the original source file.
+        remove_source_file      - Boolean, should Unmanic remove the original source file after all copy operations are complete.
+        copy_file               - Boolean, should Unmanic run a copy operation with the returned data variables.
+        file_in                 - The converted cache file to be copied by the postprocessor.
+        file_out                - The destination file that the file will be copied to.
+
+    :param data:
+    :return:
+
+    """
+    # Configure settings object
+    settings = Settings(library_id=data.get('library_id'))
+
+    if settings.get_setting('if_end_result_file_is_still_outside_mark_as_ignore'):
+        # Get the original file's absolute path
+        original_source_path = data.get('source_data', {}).get('abspath')
+        if not original_source_path:
+            logger.error("Provided 'source_data' is missing the source file abspath data.")
+            return
+        if not os.path.exists(original_source_path):
+            logger.error("Original source path could not be found.")
+            return
+
+        abspath = data.get('file_in')
+
+        # Check if the file in and the original source file are the same
+        if filecmp.cmp(original_source_path, abspath, shallow=True):
+            logger.debug("Original file was unchanged.")
+            # Mark it as to be ignored on the next scan
+            write_file_marked_as_failed(original_source_path)
+            return
+
+        # Current cache file stats
+        current_file_stats = os.stat(os.path.join(abspath))
+        # Get the original file stats
+        original_file_stats = os.stat(os.path.join(original_source_path))
+
+        # Calculate minimum file size
+        min_pct_size = int(settings.get_setting('min_percentage_size'))/100
+        min_file_size = min_pct_size * int(original_file_stats.st_size)
+        # Calculate maximum file size
+        max_pct_size = int(settings.get_setting('max_percentage_size'))/100
+        max_file_size = max_pct_size * int(original_file_stats.st_size)
+
+        # Test that the source file is not outside the configured size range of the new file
+        if int(current_file_stats.st_size) > max_file_size or int(current_file_stats.st_size) < min_file_size:
+            # The current file is outside the configured size range of the original.
+            # Mark it as failed
+            write_file_marked_as_failed(original_source_path)
+            if not filecmp.cmp(original_source_path, data.get('file_in'), shallow=True):
+                # Copy the original file to the cache file
+                # Do this rather than letting the Unmanic post-processor handle the copy.
+                # This prevents a destination file from being recorded in history which would skew any metrics if
+                # they were being recorded.
+                shutil.copyfile(original_source_path, data.get('file_in'))
+                logger.debug("Original file copied to replace cache file.")
+