VR Face Tracking Pipeline

Table of Contents

• Requirements
• Configuration Files
  • Main
  • Calibration Mode
  • Transformation Mode
  • Device
  • Pattern Detector
  • Landmark Detector

Requirements

• Ubuntu >= 20.04
• Python 3.9
• Install dependencies:

  pip install -r requirements.txt
  

• An Azure Kinect and/or a fisheye (mouth) camera device (connected over USB). Note: The fisheye camera applied in this prototype is a 'MACROSIL AV TO USB2.0' device.

Configuration Files

Main Configuration

• in config/main_config.py

  Setting	Value options	Description
  ACTIVE_MODE	enum ModeType	The active mode to run.
  	.CALIBRATION_IMAGESET	Calibration over existing imageset. A connected device is not required.
  	.CALIBRATION_REALTIME	Captures images and calibrates afterwards. One device has to be connected.
  	.TRANSFORMATION_REALTIME	Transformation of detected features from source to target device. Both devices have to be connected.
  SHOW_CAPTURE	enum DisplayType	The camera image to display in a window.
  	.BOTH	Display captured images of both devices. Both devices have to be connected.
  	.KINECT	Display captured images of a Kinect device. A Kinect device has to be connected.
  	.FISHEYE	Display captured images of a Fisheye device. A Fisheye device has to be connected.

Calibration Mode Configuration

• in config/modes/calibration_config.py:

  Setting	Value options	Description
  DEVICE_NAME	string	Device identifier. This setting will name the output .xml file. The output file will have this format '<timestamp>_<DEVICE_NAME>_<SENSOR_TYPE>.xml'
  	e.g. 'kinect'	Example output file: '2023-01-06_17-10-35_fisheye_COLOR.xml'
  OUTPUT_PATH	string
  	e.g. '_data/calibration/xml/'
  SENSOR_TYPE	enum SensorType
  	.COLOR
  	.DEPTH
  IMAGE_RESOLUTION	dataclass KinectColorResolution	Pre-defined resolutions for the Kinect color camera.
  	.COLOR_1280x720
  	.COLOR_1920x1080
  	.COLOR_2048x1536
  	.COLOR_2560x1440
  	.COLOR_3840x2160
  	dataclass KinectDepthResolution	Pre-defined resolutions for the Kinect depth camera.
  	.DEPTH_NFOV_2X2BINNED	320 x 288
  	.DEPTH_WFOV_2X2BINNED	512 x 512
  	.DEPTH_NFOV_UNBINNED	640 x 576
  	.DEPTH_PASSIVE_IR	1024 x 1024
  	dataclass FisheyeColorResolution	Pre-defined resolutions for the applied Fisheye camera 'MACROSIL AV TO USB2.0'.
  	.COLOR_640x480
  	.COLOR_320x240
  	.COLOR_160x120
  	.COLOR_720x480
  IMAGE_PATH	string	Path to an pre-existing imageset or output path for newly captured images.
  	e.g. '_data/calibration/imageset/charuco/kinect/color/'
  IMAGE_EXT	string	The image format.
  	e.g. '.png'
  MIN_IMAGE_COUNT	int	The minimum of images for calibration.
  	e.g. 50
  PATTERN_DETECTOR	enum PatternDetectorType	The desired pattern to detect for calibration.
  	.CHARUCO
  	.CHESSBOARD

Transformation Mode Configuration

• in config/modes/transformation_config.py:

  Setting	Value options	Description
  DETECTOR	enum DetectorType	The desired feature detector.
  	.FAN	The Facial Alignment Network by Adrian Bulat et al.
  	.DLIB	Dlib's Shape Predictor based on the Ensemble of Regression Trees by Vahid Kazemi et al.
  	.CHARUCO	OpenCV's Charuco Detector.
  	.CHESSBOARD	OpenCV's Chessboard Detector.

Device Configuration

• in config/devices/device_config.py:

  KinectDevice Configuration

  Setting	Value options	Description
  KINECT_ID_MODEL	string	The udev ID_MODEL of the Kinect device. *
  	e.g. 'Azure_Kinect_4K_Camera'
  KINECT_COLOR_RESOLUTION	dataclass KinectColorResolution
  	.COLOR_1280x720
  	.COLOR_1920x1080
  	.COLOR_2048x1536
  	.COLOR_2560x1440
  	.COLOR_3840x2160
  KINECT_DEPTH_RESOLUTION	dataclass KinectDepthResolution
  	.DEPTH_NFOV_2X2BINNED
  	.DEPTH_WFOV_2X2BINNED
  	.DEPTH_NFOV_UNBINNED
  	.DEPTH_PASSIVE_IR
  KINECT_FRAMERATE	dataclass KinectFramerate
  	.FPS_5
  	.FPS_15
  	.FPS_30
  KINECT_DISABLE_STREAMING_LED	bool
  	True or False
  KINECT_USE_INTERNAL_CALIBRATION	bool
  	True or False
  KINECT_COLOR_CALIBRATION	string	Path to a custom color camera calibration .xml file. Leave empty when none available.
  	e.g. '_data/calibration/xml/2022-12-23_00-38-14_kinect.xml'
  KINECT_DEPTH_CALIBRATION	string	Path to a custom depth camera calibration .xml file. Leave empty when none available.
  	e.g. ''

  FisheyeDevice Configuration

  Setting	Value options	Description
  FISHEYE_ID_MODEL	string	The udev ID_MODEL of the Fisheye device. *
  	e.g. 'AV_TO_USB2.0'
  FISHEYE_COLOR_RESOLUTION	dataclass FisheyeColorResolution
  	.COLOR_640x480
  	.COLOR_320x240
  	.COLOR_160x120
  	.COLOR_720x480
  FISHEYE_FRAMERATE	dataclass FisheyeFramerate
  	.FPS_30
  FISHEYE_COLOR_CALIBRATION	string	Path to a color camera calibration .xml file. Leave empty when none available.
  	e.g. '_data/calibration/xml/fisheye.xml'

  * On Ubuntu: Find the ID_MODEL of video devices using udev

  sudo apt-get install udev
  udevadm info --query=all /dev/video* | grep 'ID_MODEL'
  

  

2.3 Pattern Detector Configuration

• in config/detectors/pattern_detector_config.py:

  ChAruco Configuration

  Setting	Value options	Description
  CHARUCO_SHAPE	tuple
  	e.g. (6, 6)
  SQUARE_LENGTH_MM	float
  	e.g. 21.0
  MARKER_LENGTH_MM	float
  	e.g. 16.0
  MIN_MARKERS_TO_DETECT	int
  	e.g. 6
  DICTIONARY	OpenCV PredefinedDictionaryType
  	e.g. cv.aruco.DICT_4X4_50
  POSE_ESTIMATION	enum PoseEstimationType
  	.SOLVEPNP
  	.CHARUCO
  	.BOARD
  USE_CUSTOM_PARAMETERS	bool
  	True or False
  CORNER_REFINEMENT_METHOD	OpenCV CornerRefineMethod
  	e.g. cv.aruco.CORNER_REFINE_CONTOUR
  ADAPTIVE_TRESH_CONSTANT	int
  	e.g. 7
  ADAPTIVE_TRESH_WIN_SIZE_MIN	int
  	e.g. 5
  ADAPTIVE_TRESH_WIN_SIZE_MAX	int
  	e.g. 21
  ADAPTIVE_TRESH_WIN_SIZE_STEP	int
  	e.g. 4
  MIN_MARKER_PERIMETER_RATE	float
  	e.g. 0.01
  MIN_MARKER_DISTANCE_RATE	float
  	e.g. 0.01
  PERSPECTIVE_REMOVE_PIXEL_PER_CELL	int
  	e.g. 10
  PERSPECTIVE_REMOVE_IGNORED_MARGIN_PER_CELL	float
  	e.g. 0.2

  Chessboard Configuration

  Setting	Value options	Description
  CHESSBOARD_SHAPE	tuple
  	e.g. (9, 6)
  TERMINATION_CRITERIA	OpenCV TermCriteria
  	e.g. (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.1)
  WINDOW_SIZE	tuple
  	e.g. (5, 5)
  ZERO_ZONE	tuple
  	e.g. (-1, -1)

2.4 Landmark Detector Configuration

• in config/detectors/landmark_detector_config.py:

  Facial Alignment Network (FAN) Configuration

  Setting	Value options	Description
  FAN_PROCESSING_DEVICE	dataclass FANProcessingDevice
  	.CUDA
  	.CPU
  FAN_FLIP_IMAGES	bool
  	True or False
  FAN_LANDMARK_TYPE	enum LandmarksType
  	._2D
  	._2halfD
  	._3D
  FAN_NETWORK_SIZE	enum NetworkSize
  	.LARGE
  FAN_VERBOSE	bool
  	True or False
  FAN_FACE_DETECTOR	dataclass FANFaceDetector
  	.SFD
  	.DLIB
  FAN_SFD_FILTER_TRESHOLD	float
  	e.g. 0.8
  FAN_DLIB_MODEL_PATH	string
  	e.g. '_data/models/mmod_human_face_detector.dat'

  Dlib's Shape Predictor (Ensemble of Regression Trees (ERT)) Configuration

  Setting	Value options	Description
  DLIB_MODEL_PATH	string
  	'_data/models/shape_predictor_68_face_landmarks.dat'