Liveness Timeout Images (#253)

* decouple selfieviewmodel from livenesscheckmanager

* improve object references so to prevent retain cycles.

* write custom encoding function for failure reason, replace forced failure with failure reason enum. append failure reason data to multipart form request body.

* check that submission task is nil before assigning it.

* remove unnecessary comment

* feat: update changelog (#254)

* feat: update changelog

* chore: lint fix

* fix wrong version set for fingerprintjs package and dependency name causing spm not to resolve (#257)

* added autoassign to workflow (#259)

* added autoassign to workflow

* added autoassign to workflow/fix

* pod install

* add beta tag to strict mode products.
add a cancel toolbar button to all product screens.
remove cancel button from liveness instructions screen.

* use a different multiplier for checking face bounds for selfie and liveness capture
hide liveness progress if face is not valid.

* bump build number and add haptic feedback to selfie capture.

* reduce luminance threshold lowerbound

* reduce luminace threshold lower bound

* introduce a function to flip the selfie image for preview during submission.

* reduce head turn thresholds

* remove brightness check from selfie capture v2

* adjust screen brightness for selfie capture screen v2

* Chore(deps): bump rexml from 3.3.6 to 3.3.9 (#249)

Bumps [rexml](https://github.com/ruby/rexml) from 3.3.6 to 3.3.9.
- [Release notes](https://github.com/ruby/rexml/releases)
- [Changelog](https://github.com/ruby/rexml/blob/master/NEWS.md)
- [Commits](ruby/rexml@v3.3.6...v3.3.9)

---
updated-dependencies:
- dependency-name: rexml
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JNdhlovu <[email protected]>
Co-authored-by: Tobi Omotayo <[email protected]>

* bump version number.

* replace selfie quality check with vision face quality in face detector and face validator.

* reset the submission task when there is a response or error.

* improve publisher reference for legacy selfie view model.

* use weak reference in legacy selfie view model to prevent retain cycle.

* introduce a backport of StateObject into sdk helpers

* disable idle timer when capturing selfie and enable it back when dismissed.

* use a backport of stateobject in selfie capture screen.

* bump build number.

* restore brightness check

* check orientation before analyzing camera frames.

* add a did cancel delegate method to selfie result delegate.

* Chore(deps): bump slackapi/slack-github-action (#260)

* add property to set camera name in camera manager. implement fallback for pre-ios 15 devices. add didCancel endpoint to smart selfie result delegate.

* use uniqueID for cameraname. add a delay when capturing random liveness images during timeout.

* import new typeface DMSans and define a model to manage its fonts and styles.

* add face not within frame case to face bounds state.

* remove overlay from face bounding area. introduce new version of animations.

* code formatting.

* correct instructions typo.

* change new smart selfie products name. disable lint checks for backport files. code formatting.

* rename smart selfie v2 to enhanced

* pod install.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: JNdhlovu <[email protected]>
Co-authored-by: Davina Anthony <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Example/SmileID/Home/HomeView.swift

@@ -54,7 +54,7 @@ struct HomeView: View {
                         )
                         ProductCell(
                             image: "smart_selfie_enroll",
-                            name: "SmartSelfie™ Enrollment (Strict Mode)",
+                            name: "SmartSelfie™ Enrollment (Enhanced)",
                             onClick: {
                                 viewModel.onProductClicked()
                             },
@@ -74,7 +74,7 @@ struct HomeView: View {
                         )
                         ProductCell(
                             image: "smart_selfie_authentication",
-                            name: "SmartSelfie™ Authentication (Strict Mode)",
+                            name: "SmartSelfie™ Authentication (Enhanced)",
                             onClick: {
                                 viewModel.onProductClicked()
                             },

Example/SmileID/Home/ProductCell.swift

@@ -46,6 +46,16 @@ struct ProductCell<Content: View>: View {
                     content: {
                         NavigationView {
                             content()
+                                .toolbar {
+                                    ToolbarItem(placement: .cancellationAction) {
+                                        Button {
+                                            isPresented = false
+                                        } label: {
+                                            Text(SmileIDResourcesHelper.localizedString(for: "Action.Cancel"))
+                                                .foregroundColor(SmileID.theme.accent)
+                                        }
+                                    }
+                                }
                         }
                         .environment(\.modalMode, $isPresented)
                     }

Example/SmileID/WelcomeScreen.swift

@@ -24,7 +24,7 @@ struct WelcomeScreen: View {
                 .padding(.vertical)
 
             Text("To begin testing, you need to add a configuration from the Smile Portal")
-                .font(EpilogueFont.regular(with: 16))
+                .font(DMSansFont.regular(with: 16))
                 .foregroundColor(SmileID.theme.onLight)
                 .padding(.vertical)
 

Example/Tests/FaceValidatorTests.swift

@@ -24,7 +24,7 @@ class FaceValidatorTests: XCTestCase {
     func testValidateWithValidFace() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 190, height: 190),
-            selfieQualityData: SelfieQualityData(failed: 0.1, passed: 0.9),
+            faceQuality: 0.5,
             brighness: 100
         )
 
@@ -36,7 +36,7 @@ class FaceValidatorTests: XCTestCase {
     func testValidateWithFaceTooSmall() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 100, height: 100),
-            selfieQualityData: SelfieQualityData(failed: 0.1, passed: 0.9),
+            faceQuality: 0.5,
             brighness: 100
         )
 
@@ -48,7 +48,7 @@ class FaceValidatorTests: XCTestCase {
     func testValidateWithFaceTooLarge() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 250, height: 250),
-            selfieQualityData: SelfieQualityData(failed: 0.1, passed: 0.9),
+            faceQuality: 0.5,
             brighness: 100
         )
 
@@ -60,7 +60,7 @@ class FaceValidatorTests: XCTestCase {
     func testValidWithFaceOffCentre() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 125, y: 164, width: 190, height: 190),
-            selfieQualityData: SelfieQualityData(failed: 0.1, passed: 0.9),
+            faceQuality: 0.5,
             brighness: 100
         )
 
@@ -72,19 +72,19 @@ class FaceValidatorTests: XCTestCase {
     func testValidateWithPoorBrightness() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 190, height: 190),
-            selfieQualityData: SelfieQualityData(failed: 0.1, passed: 0.9),
-            brighness: 70
+            faceQuality: 0.5,
+            brighness: 35
         )
 
         XCTAssertTrue(result.faceInBounds)
         XCTAssertFalse(result.hasDetectedValidFace)
         XCTAssertEqual(result.userInstruction, .goodLight)
     }
 
-    func testValidateWithPoorSelfieQuality() {
+    func testValidateWithPoorFaceQuality() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 190, height: 190),
-            selfieQualityData: SelfieQualityData(failed: 0.6, passed: 0.4),
+            faceQuality: 0.2,
             brighness: 70
         )
 
@@ -96,7 +96,7 @@ class FaceValidatorTests: XCTestCase {
     func testValidateWithLivenessTask() {
         let result = performValidation(
             faceBoundingBox: CGRect(x: 65, y: 164, width: 190, height: 190),
-            selfieQualityData: SelfieQualityData(failed: 0.3, passed: 0.7),
+            faceQuality: 0.3,
             brighness: 100,
             livenessTask: .lookLeft
         )
@@ -111,7 +111,7 @@ class FaceValidatorTests: XCTestCase {
 extension FaceValidatorTests {
     func performValidation(
         faceBoundingBox: CGRect,
-        selfieQualityData: SelfieQualityData,
+        faceQuality: Float,
         brighness: Int,
         livenessTask: LivenessTask? = nil
     ) -> FaceValidationResult {
@@ -124,7 +124,7 @@ extension FaceValidatorTests {
         )
         faceValidator.validate(
             faceGeometry: faceGeometry,
-            selfieQuality: selfieQualityData,
+            faceQuality: faceQuality,
             brightness: brighness,
             currentLivenessTask: livenessTask
         )

Sources/SmileID/Classes/Camera/CameraManager.swift

@@ -35,6 +35,8 @@ class CameraManager: NSObject, ObservableObject {
         (session.inputs.first as? AVCaptureDeviceInput)?.device.position
     }
 
+    private(set) var cameraName: String?
+
     // Used to queue and then resume tasks while waiting for Camera permissions
     private let sessionQueue = DispatchQueue(label: "com.smileidentity.ios")
     private let videoOutput = AVCaptureVideoDataOutput()
@@ -90,6 +92,7 @@ class CameraManager: NSObject, ObservableObject {
             status = .failed
             return
         }
+        cameraName = camera.uniqueID
 
         do {
             let cameraInput = try AVCaptureDeviceInput(device: camera)

Sources/SmileID/Classes/Camera/CameraViewController.swift

@@ -3,7 +3,7 @@ import Vision
 import AVFoundation
 
 class CameraViewController: UIViewController {
-    var faceDetector: FaceDetectorV2?
+    var faceDetector: EnhancedFaceDetector?
 
     var previewLayer: AVCaptureVideoPreviewLayer?
     private weak var cameraManager: CameraManager?

Sources/SmileID/Classes/FaceDetector/FaceDetectorV2.swift → Sources/SmileID/Classes/FaceDetector/EnhancedFaceDetector.swift

@@ -16,16 +16,15 @@ protocol FaceDetectorViewDelegate: NSObjectProtocol {
 
 protocol FaceDetectorResultDelegate: AnyObject {
     func faceDetector(
-        _ detector: FaceDetectorV2,
+        _ detector: EnhancedFaceDetector,
         didDetectFace faceGeometry: FaceGeometryData,
         withFaceQuality faceQuality: Float,
-        selfieQuality: SelfieQualityData,
         brightness: Int
     )
-    func faceDetector(_ detector: FaceDetectorV2, didFailWithError error: Error)
+    func faceDetector(_ detector: EnhancedFaceDetector, didFailWithError error: Error)
 }
 
-class FaceDetectorV2: NSObject {
+class EnhancedFaceDetector: NSObject {
     private var selfieQualityModel: SelfieQualityDetector?
 
     private let cropSize = (width: 120, height: 120)
@@ -78,29 +77,32 @@ class FaceDetectorV2: NSObject {
 
             let uiImage = UIImage(pixelBuffer: imageBuffer)
             let brightness = self.calculateBrightness(uiImage)
-            let croppedImage = try self.cropImageToFace(uiImage)
-
-            let selfieQualityData = try self.selfieQualityRequest(imageBuffer: croppedImage)
 
+            let faceGeometryData: FaceGeometryData
             if #available(iOS 15.0, *) {
-                let faceGeometryData = FaceGeometryData(
+                faceGeometryData = FaceGeometryData(
                     boundingBox: convertedBoundingBox,
                     roll: faceObservation.roll ?? 0.0,
                     yaw: faceObservation.yaw ?? 0.0,
                     pitch: faceObservation.pitch ?? 0.0,
                     direction: faceDirection(faceObservation: faceObservation)
                 )
-                self.resultDelegate?
-                    .faceDetector(
-                        self,
-                        didDetectFace: faceGeometryData,
-                        withFaceQuality: faceQualityObservation.faceCaptureQuality ?? 0.0,
-                        selfieQuality: selfieQualityData,
-                        brightness: brightness
-                    )
-            } else {
-                // Fallback on earlier versions
+            } else { // Fallback on earlier versions
+                faceGeometryData = FaceGeometryData(
+                    boundingBox: convertedBoundingBox,
+                    roll: faceObservation.roll ?? 0.0,
+                    yaw: faceObservation.yaw ?? 0.0,
+                    pitch: 0.0,
+                    direction: faceDirection(faceObservation: faceObservation)
+                )
             }
+            self.resultDelegate?
+                .faceDetector(
+                    self,
+                    didDetectFace: faceGeometryData,
+                    withFaceQuality: faceQualityObservation.faceCaptureQuality ?? 0.0,
+                    brightness: brightness
+                )
         } catch {
             self.resultDelegate?.faceDetector(self, didFailWithError: error)
         }
@@ -180,8 +182,8 @@ class FaceDetectorV2: NSObject {
 
     private func calculateBrightness(_ image: UIImage?) -> Int {
         guard let image, let cgImage = image.cgImage,
-            let imageData = cgImage.dataProvider?.data,
-            let dataPointer = CFDataGetBytePtr(imageData)
+              let imageData = cgImage.dataProvider?.data,
+              let dataPointer = CFDataGetBytePtr(imageData)
         else {
             return 0
         }

Sources/SmileID/Classes/FaceDetector/FaceValidator.swift

@@ -15,9 +15,10 @@ final class FaceValidator {
     private var faceLayoutGuideFrame: CGRect = .zero
 
     // MARK: Constants
-    private let selfieQualityThreshold: Float = 0.5
-    private let luminanceThreshold: ClosedRange<Int> = 80...200
-    private let faceBoundsMultiplier: CGFloat = 1.5
+    private let faceQualityThreshold: Float = 0.25
+    private let luminanceThreshold: ClosedRange<Int> = 40...200
+    private let selfiefaceBoundsMultiplier: CGFloat = 1.5
+    private let livenessfaceBoundsMultiplier: CGFloat = 2.2
     private let faceBoundsThreshold: CGFloat = 50
 
     init() {}
@@ -28,7 +29,7 @@ final class FaceValidator {
 
     func validate(
         faceGeometry: FaceGeometryData,
-        selfieQuality: SelfieQualityData,
+        faceQuality: Float,
         brightness: Int,
         currentLivenessTask: LivenessTask?
     ) {
@@ -42,22 +43,22 @@ final class FaceValidator {
         // check brightness
         let isAcceptableBrightness = luminanceThreshold.contains(brightness)
 
-        // check selfie quality
-        let isAcceptableSelfieQuality = checkSelfieQuality(selfieQuality)
+        // check face quality
+        let isAcceptableFaceQuality = checkFaceQuality(faceQuality)
 
         // check that face is ready for capture
         let hasDetectedValidFace = checkValidFace(
             isAcceptableBounds,
             isAcceptableBrightness,
-            isAcceptableSelfieQuality
+            isAcceptableFaceQuality
         )
 
         // determine what instruction/animation to display to users
         let userInstruction = userInstruction(
             from: faceBoundsState,
             detectedValidFace: hasDetectedValidFace,
             isAcceptableBrightness: isAcceptableBrightness,
-            isAcceptableSelfieQuality: isAcceptableSelfieQuality,
+            isAcceptableFaceQuality: isAcceptableFaceQuality,
             livenessTask: currentLivenessTask
         )
 
@@ -73,7 +74,7 @@ final class FaceValidator {
         from faceBoundsState: FaceBoundsState,
         detectedValidFace: Bool,
         isAcceptableBrightness: Bool,
-        isAcceptableSelfieQuality: Bool,
+        isAcceptableFaceQuality: Bool,
         livenessTask: LivenessTask?
     ) -> SelfieCaptureInstruction? {
         if detectedValidFace {
@@ -88,29 +89,36 @@ final class FaceValidator {
                 }
             }
             return nil
-        } else if faceBoundsState == .detectedFaceOffCentre {
+        } else if faceBoundsState == .detectedFaceOffCentre
+                    || faceBoundsState == .detectedFaceNotWithinFrame {
             return .headInFrame
         } else if faceBoundsState == .detectedFaceTooSmall {
             return .moveCloser
         } else if faceBoundsState == .detectedFaceTooLarge {
             return .moveBack
-        } else if !isAcceptableSelfieQuality || !isAcceptableBrightness {
+        } else if !isAcceptableFaceQuality || !isAcceptableBrightness {
             return .goodLight
         }
         return nil
     }
 
     // MARK: Validation Checks
-    private func checkFaceSizeAndPosition(using boundingBox: CGRect, shouldCheckCentering: Bool) -> FaceBoundsState {
+    private func checkFaceSizeAndPosition(
+        using boundingBox: CGRect,
+        shouldCheckCentering: Bool
+    ) -> FaceBoundsState {
         let maxFaceWidth = faceLayoutGuideFrame.width - 20
+        let faceBoundsMultiplier = shouldCheckCentering ? selfiefaceBoundsMultiplier : livenessfaceBoundsMultiplier
         let minFaceWidth = faceLayoutGuideFrame.width / faceBoundsMultiplier
 
+        // check how far/close face is
         if boundingBox.width > maxFaceWidth {
             return .detectedFaceTooLarge
         } else if boundingBox.width < minFaceWidth {
             return .detectedFaceTooSmall
         }
 
+        // check that face is centered for selfie capture only
         if shouldCheckCentering {
             let horizontalOffset = abs(boundingBox.midX - faceLayoutGuideFrame.midX)
             let verticalOffset = abs(boundingBox.midY - faceLayoutGuideFrame.midY)
@@ -123,15 +131,15 @@ final class FaceValidator {
         return .detectedFaceAppropriateSizeAndPosition
     }
 
-    private func checkSelfieQuality(_ value: SelfieQualityData) -> Bool {
-        return value.passed >= selfieQualityThreshold
+    private func checkFaceQuality(_ value: Float) -> Bool {
+        return value >= faceQualityThreshold
     }
 
     private func checkValidFace(
         _ isAcceptableBounds: Bool,
         _ isAcceptableBrightness: Bool,
-        _ isAcceptableSelfieQuality: Bool
+        _ isAcceptableFaceQuality: Bool
     ) -> Bool {
-        return isAcceptableBounds && isAcceptableBrightness && isAcceptableSelfieQuality
+        return isAcceptableBounds && isAcceptableBrightness && isAcceptableFaceQuality
     }
 }