Skip to content

My personal project that reconstructs a 3D face model from a single image. Used C++, Qt, OpenCV, OpenGL with the help of Surrey Face Model.

Notifications You must be signed in to change notification settings

JerryWu96/3D_Face_Reconstrction

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

84 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

3D Face Reconstrction - Personal Project

I built this out of my personal interest back to 2018 when I was a senior undergraduate in Compute Science. The total time that i spent on this project was a week.

The main goal of the program is to reconstruct a 3D face model from a single front-face image with the help of C++, QT, OpenGL, OpenCV, eos/dlib libraries and surrey face modesl. it extracts face texture, constructs a face model in.obj format, and reconstructs the full 3D model.

The interactive UI was implemented using C++ Qt.

Following is a screenshot of the model viewer UI. You could adjust the model angles with sliders or drag with your mouse to adjust the pose of the model at east.

Dependencies

I used Visual Studio 2018 on Windows to do my project.

  • In order to compile the program, you need to configure the following dependencies in order:

OpenCV -> OpenGL -> dlib -> eos -> Qt

  • you could find the following two files in dependencies folder:
    • dlib_to_sfm.txt: mapping bettween dlib and sfm model
    • expression_blendshapes_3448.bin: basic sfm model.
  • Some useful links:

https://github.com/patrikhuber/eos

http://dlib.net

The Surrey Face Model

I used a shape-only version of the Surrey Morphable Face Model as the skeleton of the model.

Surrey Face Model shape picture

The full models with higher resolution are available at http://www.cvssp.org/facemodel.

Face Landmark Detection

Here is the face landmark detection part. By selecting 68 landmarks, a face could be identified and located in the image.

Given the input sample photo:

By running face detection module, my program identified the front face and its outline and some interest points, e.g. nose and eyes.

Texture Extraction

By implementing the texture extraction API, combined with Dlib face landmark detection data, the texture can be extracted from the original image.

The face texture could be presented in this sturcture:

Face Reconstruction

Here comes the most challanging part.

  • First, the program fits the original model to different face image, adjusts its surface parameters in order to present more natural and realistic outputs.

  • Secondly, given an extracted texture of the face and the surrey face model, the program calculates a mapping between the coordinates from a 2D image to a 3D model.

Some final outputs:

Special Notice

You may have noticed that in Reconstruct.cpp, code that reads higher-resolution sfm model has been commented out, since these models are not publicly available, and i am not permitted to upload them here.

Back to 2019, the research team behind Surrey face model updated their models with new formats, as well as new python code to handle them, which i haven't looked into yet. My program uses a deprecated version which was updated in 2018. Please email them if you still want to get versions that are compatible with my program.

I have received more than 10 emails asking whether i will add support to the new Surrey face model. Note that currently I do not have the plan to use their new versions since it will add some extra overhead to the code, and i haven't found real benefits in doing so. Please understand that this is a project i spent a week working on, and refactoring could take efforts since i no longer have any Windows laptop with me.

However, i am happy to discuss with you the questions about the current code. It is up to you to decide if you would like to make it better:)

Please submit PRs if you decide to have a major update, thanks!

About

My personal project that reconstructs a 3D face model from a single image. Used C++, Qt, OpenCV, OpenGL with the help of Surrey Face Model.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages