Merge pull request #63 from BitMind-AI/camo_neurons_refactor

Modularized detectors and neuron with enhanced UCF training flow

.gitignore

@@ -163,6 +163,7 @@ testing/
 data/
 checkpoints/
 .requirements_installed
+base_miner/NPR/weights/*
 base_miner/UCF/weights/*
 base_miner/UCF/logs/*
 miner_eval.py

base_miner/NPR/config/constants.py

@@ -0,0 +1,9 @@
+import os
+
+# Path to the directory containing the constants.py file
+CONFIGS_DIR = os.path.dirname(os.path.abspath(__file__))
+
+# The base directory for NPR-related files, i.e., NPR directory
+NPR_BASE_PATH = os.path.abspath(os.path.join(CONFIGS_DIR, ".."))  # Points to bitmind-subnet/base_miner/UCF/
+# Absolute paths for the required files and directories
+WEIGHTS_DIR = os.path.join(NPR_BASE_PATH, "weights/") # Path to pretrained weights directory

base_miner/README.md

@@ -1,5 +1,43 @@
 ## Base Miners
 
-This directory contains subfolders with model architectures and training loops for base miners.
+The `base_miner/` directory facilitates the training, orchestration, and deployment of modular and highly customizable deepfake detectors.
+We broadly define **detector** as an algorithm that either employs a single model or orchestrates multiple models to perform the binary real-or-AI inference task. These **models** can be any algorithm that processes an image to determine its classification. This includes not only pretrained machine learning architectures, but also heuristic and statistical modeling frameworks.
 
-Read about [CAMO (Content Aware Model Orchestration)](https://bitmindlabs.notion.site/CAMO-Content-Aware-Model-Orchestration-CAMO-Framework-for-Deepfake-Detection-43ef46a0f9de403abec7a577a45cd075), our generalized framework for creating “hard mixture of expert” models for deepfake detection. The latest and most performant iteration of our CAMO miner neuron uses finetuned expert and generalist UCF models.
+## Our Base Miner Detector: Content-Aware Model Orchestration (CAMO)
+
+Read about [CAMO (Content Aware Model Orchestration)](https://bitmindlabs.notion.site/CAMO-Content-Aware-Model-Orchestration-CAMO-Framework-for-Deepfake-Detection-43ef46a0f9de403abec7a577a45cd075), our generalized framework for creating “hard mixture of expert” detectors.
+
+- **Latest Iteration**: The most performant iteration of `class CAMODetector(DeepfakeDetector)` used in our base miner `neurons/miner.py` incorporates a `GatingMechanism(Gate)` that routes to a fine-tuned face expert model and generalist model with the `UCF` architecture.
+
+## Directory Structure
+
+### 1. Architectures and Training
+- **UCF/** and **NPR/**
+
+These folders contain model architectures and training loops for `UCF (ICCV 2023)` and `NPR (CVPR 2024)`, adapted to use curated and preprocessed training datasets on our [BitMind Huggingface](https://huggingface.co/bitmind).
+
+### 2. deepfake_detectors/
+The modular structure for detectors used in the miner neuron is defined here, through `DeepfakeDetector` abstract base class and subclass implementations.
+
+- **deepfake_detectors/** contains:
+  - **configs/**: YAML configuration files to load detector instance attributes, including any pretrained model weights.
+  - **Abstract Base Class**: A foundational class that outlines the standard structure for implementing detectors.
+  - **Detector Subclasses**: Specialized detector implementations that can be dynamically loaded and managed based on configuration.
+
+The `DeepfakeDetector design` allows for high configurability and extension.
+
+### 3. gating_mechanisms/
+Similar to `deepfake_detectors/`, this folder contains abstract base classes and implementations of `Gate`s that are used to handle content-aware preprocessing and routing. This is especially useful for multi-agent detection systems, such as the `DeepfakeDetector` subclass `CAMODetector` in `deepfake_detectors/camo_detector.py`.
+
+- **Abstract Gate Class**: A base class for implementing image content gating.
+- **Gate Subclasses**: These subclasses define specific gating mechanisms responsible for routing inputs to appropriate expert detectors or preprocessing steps based on content characteristics. This is useful for multi-detector or mixture-of-expert detector setups.
+
+### 4. registry.py
+The `registry.py` file is responsible for managing the creation of detectors and gates using a **Factory Method** design pattern. It auto-registers all `DeepfakeDetector` and `Gate` subclasses from their subfolders to respective `Registry` constants, making it simple to instantiate detectors and gates dynamically based on predefined constants.
+
+- **Factory Pattern**: Ensures a clean, maintainable, and scalable method for creating instances of detectors and gating mechanisms.
+- **Auto-Registration**: Automatically registers all available detector and gate subclasses, enabling a flexible and extensible system.
+
+## Integration with `miner.py`
+
+- **Modular Initialization**: The miner neuron in `bitmind-subnet/neurons/miner.py` leverages the registry system to dynamically initialize the detector used for the forward function, facilitating a highly modular design. The detector module used is determined by neuron config args, defaulting to `"CAMO"`.

base_miner/UCF/config/constants.py

@@ -1,19 +1,15 @@
 import os
 
 # Path to the directory containing the constants.py file
-UCF_CONFIGS_DIR = os.path.dirname(os.path.abspath(__file__))
+CONFIGS_DIR = os.path.dirname(os.path.abspath(__file__))
 
 # The base directory for UCF-related files, i.e., UCF directory
-UCF_BASE_PATH = os.path.abspath(os.path.join(UCF_CONFIGS_DIR, ".."))  # Points to bitmind-subnet/base_miner/UCF/
+UCF_BASE_PATH = os.path.abspath(os.path.join(CONFIGS_DIR, ".."))  # Points to bitmind-subnet/base_miner/UCF/
 # Absolute paths for the required files and directories
-CONFIG_PATH = os.path.join(UCF_BASE_PATH, "config/ucf.yaml")  # Path to the ucf.yaml file
-WEIGHTS_PATH = os.path.join(UCF_BASE_PATH, "weights/")        # Path to pretrained weights directory
+CONFIG_PATH = os.path.join(CONFIGS_DIR, "ucf.yaml")  # Path to the ucf.yaml file
+WEIGHTS_DIR = os.path.join(UCF_BASE_PATH, "weights/") # Path to pretrained weights directory
 
-WEIGHTS_HF_PATH = "bitmind/ucf"
-DFB_CKPT = "ucf_best.pth"
-BM_CKPT = "ucf_bitmind_best.pth"
+HF_REPO = "bitmind/ucf"
 BACKBONE_CKPT = "xception_best.pth"
-BM_FACE_CKPT = "ucf_bitmind_face.pth"
-BM_18K_CKPT = "ucf-bitmind-18k.pth"
 
 DLIB_FACE_PREDICTOR_PATH = os.path.abspath(os.path.join(UCF_BASE_PATH, "../../bitmind/dataset_processing/dlib_tools/shape_predictor_81_face_landmarks.dat"))