This C++ application recognizes a person from a database of more than 2000 faces. It is built for a Jetson Nano, but can easily be ported to other platforms.
First, the faces and their landmarks are detected by RetinaFace or MTCNN. The next stage checks if the person is wearing a mask. If not, the database is scanned with Arcface for the matching face. In the end, Face Anti Spoofing tests whether the person in front of the camera is real and not a mask or a cardboard photo.
If the face is not found in the database, it will be added automatically. A blur filter ensures only sharp faces in the database. One photo per person is sufficient, although more does not hurt.
Special made for a Jetson Nano see Q-engineering deep learning examples
| Model | Jetson Nano 2015 MHz | Jetson Nano 1479 MHz | RPi 4 64-OS 1950 MHz | RPi 4 64-OS 1500 MHz |
|---|---|---|---|---|
| MTCNN | 11 mS | 14 mS | 22 mS | 25 mS |
| RetinaFace | 15 mS | 19 mS | 35 mS | 37 mS |
| ArcFace | +17 mS | +21 mS | +36 mS | +40 mS |
| Mask | +8 mS | +10 mS | +10 mS | +12 mS |
| Spoofing | +25 mS | +37 mS | +37 mS | +45 mS |
To run the application, you have to:
- The Paddle Lite framework installed. Install Paddle
- The Tencent ncnn framework installed. Install ncnn
- Code::Blocks installed. (
$ sudo apt-get install codeblocks)
To extract and run the application in Code::Blocks
$ mkdir MyDir
$ cd MyDir
$ wget https://github.com/Qengineering/Face-Recognition-with-Mask-Jetson-Nano/archive/refs/heads/main.zip
$ unzip -j master.zip
Remove master.zip and README.md as they are no longer needed.
$ rm master.zip
$ rm README.md
Your MyDir folder must now look like this:
Graham Norton.jpg (example image)
FaceRecognition.cbp (code::blocks project file)
Norton_A.mp4 (movie with faces to load)
Norton_M.mp4 (movie to check)
img (database folder)
models (folder with used ncnn deep learning models)
src (C++ source files)
include (the C++ headers)
To run the application load the project file FaceRecognition.cbp in Code::Blocks.
First, we are going to fill the database with new faces. The database img initial holds one face, Graham.jpg.
Check in main.cpp line 253. It must be cv::VideoCapture cap("Norton_A.mp4");
Compile and run the app. Movie Norton_A.mp4 will be played and new faces are stored in the database. In the end, you have the database filled as below.
Next, alter the name of the movie in line 253 of main.cpp to Norton_2.mpg.
Compile and run the application again. You will see that all the faces are correctly recognized. It can still happen that faces are added to the database due to strange angles or grimaces.
The application can easily contain more than 2000 faces. There are reports that ArcFace works flawlessly with over 5000 faces. With large databases, it is important to keep your face "natural". It means a front view photo with eyes open and mouth closed without a smile or other funny faces.
The database is filled "on the fly", as you have seen above. It is also possible to manually add a face to the databases. To do this, run the application from the command-line and enter the name of the image as an argument. For example ./FaceRecognition "Graham Norton.jpg" Note the quotation marks around the name if it has a space.
You can give the faces a corresponding name. By using a hash, you can associate multiple pictures with the same name.
By the way, note the warp perspective of Graham Norton's face that we added via a command-line argument and the crop of the same photo already saved in the database. This is done by the ArcFace algorithms.
The blur filter prevents vague or imprecise faces from being added to the database. Below you see a few examples of faces we encounter in the database when de blur filter was switched off.
Another safety measure is the orientation of the face. Only faces in front of the camera are added to the database. Faces "in profile" are often inaccurate in large databases.
We use Paddle-Lite mask detection for face mask detection. We have previously applied this network in the Face-Mask-Detection-Jetson-Nano app. It's lightweight and fast. However, you will need to install Paddle-Lite on your Jetson Nano as an additional framework in addition to ncnn. Please note that PaddlePaddle has only trained its face mask detection network with white (or light-coloured) masks. Nowadays you see a lot of black masks or masks with fancy prints. Unfortunately, they are not recognized by this network
The application is written in C ++. The setup is flexible and easy to adapt to your own needs. See it as a skeleton which you can expand yourself. Some hints. In main .cpp at line 21 you see a few defines.
#define RETINA //comment if you want to use MtCNN landmark detection instead
#define RECOGNIZE_FACE
#define TEST_LIVING
#define TEST_MASK
#define AUTO_FILL_DATABASE
#define BLUR_FILTER_STRANGER
// some diagnostics
#define SHOW_LEGEND
#define SHOW_LANDMARKS
By commenting the line the define is switched off. For instance, if you do not want to incorporate the anti-spoofing test (saves you 37 mS), comment this line. The MtCNN face detection is switched on by turning RETINA off.
Another important point is that only one face is labelled. It is no problem to loop through all faces. However, they are usually too small to be recognized with great accuracy. Besides, your FPS will drop also.
Note, the input image for the RetinaFace is 324 x 240 pixels. Larger pictures are resized to that format. ArcFace works with an input of 112 x 112 pixels.
If you have a large input format, you could extract the faces at a larger scale from this image, once you have the coordinates from the RetinaFace network. Now, faces are to be recognized with much greater accuracy. Of course, there will be not much of an FPS left.
If you want to use a camera please alter line 253 in main.cpp to
cv::VideoCapture cap(0); //WebCam
If you want to run a movie please alter line 253 in main.cpp to
cv::VideoCapture cap("Norton_M.mp4"); //Movie
MTCNN
RetinaFace
ArcFace
Anti spoofing
https://github.com/Tencent/ncnn
https://github.com/PaddlePaddle/PaddleHub
https://github.com/nihui
https://github.com/LicheeX/MTCNN-NCNN
https://github.com/XinghaoChen9/LiveFaceReco_RaspberryPi
https://github.com/deepinsight/insightface
https://github.com/minivision-ai/Silent-Face-Anti-Spoofing
https://github.com/Qengineering/Blur-detection-with-FFT-in-C