Merge pull request #58 from RigvedManoj/test_readme_fixes

Updated readme to include details for evaluation

README.md

@@ -106,6 +106,16 @@ python -m src.facematch.face_match_server
 ---
 
 # Metrics and Testing
+
+We conduct a series of experiments to evaluate the performance and accuracy of FaceMatch. The experiments are conducted on a dataset of 1,680 images of people from the Labelled Faces in the Wild (LFW) dataset. 
+
+We evaluate FaceMatch using the following metrics:
+
+1. **Upload Time**: The time taken to upload images to the database.
+2. **Search Time per Image**: The time taken to find matches for a single image.
+3. **Faiss Accuracy (Top-n)**: The percentage of images in the database that have at least one match within the top-n results using FAISS search algorithm.  Atleast one match in top-n is considered a positive match, meaning that the person of interest is identified within the image.
+
+
 ## Metrics
 | Face Recognition Model | Number of Faces in Database | Upload Time (in seconds) | Search Time per Image (in seconds) | Top-n | Faiss Accuracy |
 |-------------------------|-----------------------------|---------------------------|------------------------------------|-------|----------------|
@@ -114,10 +124,18 @@ python -m src.facematch.face_match_server
 | VGGFace                | 1680                        | 650                       | < 3                                | 10    | 90             |
 
 
+The above metrics were calculated using the following system configuration:
+
+- **OS**: Windows 11 Pro 64-bit
+- **Processor**: AMD Ryzen 7 7735HS with Radeon Graphics (16 CPUs), ~3.2 GHz
+- **RAM**: 32 GB
+
 ## Testing
 
 Check out [Testing README](./benchmark_testing/README.md) for complete details on dataset and testing.
 
+Check out this [document](https://docs.google.com/document/d/1CpN__oPgmAvY65s-tWg4X-pZPCPwNEAU-ULrkdKWES4/edit?usp=sharing) for more details on the face detection and recognition models, datasets and testing.
+
 ---
 
 # For Developers

benchmark_testing/README.md

@@ -6,44 +6,83 @@ This repository contains scripts and tools to benchmark the performance and accu
 
 ## Dataset
 
-The dataset for testing can be found at [Dataset](add link here).
+The dataset for testing can be found at [Dataset](https://drive.google.com/file/d/1dpfTxxAbM-3StVNCsA6qQyYNhhlsfPOu/view?usp=sharing).
 
 ---
 
 ## Folder Structure
 - **`test_data_setup.py`**  
   Prepares the test dataset by randomly selecting one image per person for upload and one image for testing. The input directory should be a recursive directory such that each directory contains different images of the same person. It outputs two directories:
-    - `upload directory`: Contains images to be uploaded to the database.
-    - `test directory`: Contains images used for face recognition accuracy testing.
+    - `sample_database directory`: Contains images to be uploaded to the database.
+    - `sample_queries directory`: Contains query images used for face recognition accuracy testing.
+
+Note: One such pair of sample database and queries directories have already been created for testing (available in the dataset download mentioned above).
 
 - **`run_bulk_upload.sh`**  
-  Starts the server and uploads images from the `upload directory` to the database, measuring the time taken for the operation.
+  Starts the server and uploads images from the `sample_database directory` to a database named `sample_db`, measuring the time taken for the operation.
 
 - **`run_face_find_time.sh`**  
   Starts the server and tests the face recognition function by running a single query image against the database, measuring the time taken to return results.
 
 - **`run_face_find_accuracy.sh`**  
-  Starts the server and evaluates the face recognition system. It tests the face recognition function on all images in the `test directory` one by one, saving the results to a CSV file.
+  Starts the server and evaluates the face recognition system. It tests the face recognition function on all images in the `sample_queries directory` one by one, saving the results to a CSV file.
 
 - **`test_data_metrics.py`**  
   Analyzes the output CSV file generated by `run_face_find_accuracy.sh` to calculate and return the accuracy metrics for the face recognition system.
 
+- **`edgecase_testing.py`**
+  A script to test edge cases of a face recognition system allowing us to find reasons for failure. It visualizes detected faces by drawing bounding boxes on images and verifies the similarity between two images, providing the distance and the metric used for comparison. 
 ---
 
 ## Instructions to Run Testing
 1. **Prepare Dataset**  
-   - Download the dataset containing multiple directories (each directory represents a person and contains their images) from Google Drive Link.
-2. **Run Test Data Setup**
+   - Download the dataset containing multiple directories (each directory represents a person and contains their images) from the link under the [Dataset](#dataset) section.
+
+    > [!NOTE]\
+    > *All the code below should be executed from the `benchmark_testing` directory.*
+
+2. **Run Test Data Setup (Optional)**
    - Replace the directory names in the `test_data_setup.py` file with appropriate directory locations.
-   - Execute `test_data_setup.py` to create the `upload directory` and `test directory`.
+   - Execute `test_data_setup.py` to create the `sample_database directory` and `sample_queries directory`.
+
+    ```
+    python test_data_setup.py
+    ```
+
 3. **Run Bulk Upload and Measure Upload Time**
    - Replace the directory names in the `run_bulk_upload.sh` file with appropriate directory locations.
    - Execute `run_bulk_upload.sh` to upload images to the database.
+
+    ```
+    ./run_bulk_upload.sh
+    ```
+
 4. **Run Face Find and Measure Search Time**
    - Replace the directory names in the `run_face_find_time.sh` file with appropriate directory locations.
    - Execute `run_face_find_time.sh` to measure the response time of the face recognition system for a single query.
+
+    ```
+    ./run_face_find_time.sh
+    ```
+
 5. **Run Face Find and Measure Accuracy**  
    - Replace the directory names in the `run_face_find_accuracy.sh` file with appropriate directory locations.
    - Execute `run_face_find_accuracy.sh` to create output csv.
+
+   ```
+   ./run_face_find_accuracy.sh
+   ```
+
    - Use `test_data_metrics.py` to analyze the accuracy of the face recognition system.
+  
+    ```
+    python test_data_metrics.py
+    ```
+
+6. **Visualize face recognition results**
+  - Replace the directory names in the `edgecase_testing.py` file with appropriate image paths. 
+  - Execute `edgecase_testing.py` to visualize the results of the face recognition system.
 
+    ```
+    python edgecase_testing.py
+    ```

benchmark_testing/edgecase_testing.py

@@ -8,7 +8,7 @@ def check_face_recognition_edge_cases(img1_path, img2_path):
         img2_path=img2_path,
         model_name="ArcFace",
         enforce_detection=True,
-        detector_backend="mtcnn"
+        detector_backend="yolov8",
     )
     print(result)
 
@@ -17,23 +17,29 @@ def check_face_detection_edge_cases(image_path):
     img = cv2.imread(image_path)
     results = DeepFace.represent(
         image_path,
-        model_name='ArcFace',
-        detector_backend='mtcnn',
-        enforce_detection=True
+        model_name="ArcFace",
+        detector_backend="yolov8",
+        enforce_detection=True,
     )
 
     # Iterate over the results
     for i, result in enumerate(results):
         # Get face region for drawing bounding box
-        x, y, width, height = result['facial_area']['x'], result['facial_area']['y'], result['facial_area']['w'], \
-            result['facial_area']['h']
+        x, y, width, height = (
+            result["facial_area"]["x"],
+            result["facial_area"]["y"],
+            result["facial_area"]["w"],
+            result["facial_area"]["h"],
+        )
 
         if result["face_confidence"] > 0.7:
             key = f"face_{i + 1}"  # Creating a unique identifier for each face
 
             # Draw a rectangle around the detected face
             cv2.rectangle(img, (x, y), (x + width, y + height), (255, 0, 0), 2)
-            cv2.putText(img, key, (x, y - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 0), 2)
+            cv2.putText(
+                img, key, (x, y - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 0), 2
+            )
 
     # Display the image with bounding boxes
     cv2.imshow("Detected Faces", img)