Facial Photo-Guided Head Anatomy Modeling Based on Deep Learning and 2D/3D Shape Prior Model Registration

Abstract

Three-dimensional (3D) models of individualized head anatomy are often used in the preoperative planning of plastic surgery. Most current methods use Computed Tomography (CT) to acquire the head anatomy, but CT scan exposes the patients to X-ray radiation. Alternatively, artificial intelligence-based estimation of 3D head anatomy from the facial photo is a safer and more efficient choice, especially for scenarios that do not require high modeling accuracy of internal structures (e.g., doctor-patient communication and surgery procedure demonstration). We develop a method for constructing a personalized 3D model of complete head anatomy (including skin and internal structures) from the front and side view photos. We first detect key facial landmarks from the photos with a convolution neural network (CNN), then perform 2D/3D registration to morph a statistical shape model (SSM) of complete head anatomy to match the dual-view photos. Our method uses deep learning to achieve accurate facial landmark detection and employ the anatomy shape prior to yielding reasonable internal structure estimation. We evaluated the method based on 12 subjects (including 7 males and 5 females). The facial surface reconstruction error was assessed using the 3D surface scan of the subjects as the ground truth. Our method’s root-mean-square-error (RMSE) was 3.60±0.49 mm, which was 1.06±0.39 mm lower than the state-of-the-art (SOTA) CNN-based face reconstruction method. Our method also predicts all the internal head structures (bones, muscles, vessels, nerve fibers, fat, glands, and brain structures) which are not provided by the SOTA method. To the best of our knowledge, this is the first study modeling complete head anatomy from facial photos toward clinical applications.