Image and Video Processing Lab, The Chinese University of Hong Kong

1. Introduction

As illustrated in Figure 1 and the demo video, the system can track the 3D head position and pose in real-time. As using 3D face template, eye positions are also provided by the system. Therefore, the system can be used as a natural human computer interface in many attractive applications. For example, it can be applied in gaming or free-viewpoint video to provide user viewpoints; it can be used to control cursor with head motion, which will facilitate people with disabled arms; it can serve as a preprocessing step for free-head-movements gaze estimation, and so on.

Figure 1. Exemplar tracking results.

2. Framework

Using a RGBD camera, our head pose tracking system can reconstruct a user-specific face template on-the-fly. During the real-time tracking, up to nine video frames are automatically selected for the reconstruction. The BFM face model is fitted to each frame, which is formulated as an iterative optimization process. The optimization process consists of three processing steps, each of which has a time-efficient solution.

Figure 2. General system framework.

The head pose tracking is implemented using the ICP algorithm which registers the user-specific face template to the input depth video. This head pose tracking system has a low complexity and can achieve the state-of-the-art performances on two public head pose datasets.

3. Additional Videos

− Laboratory test

− Field test

− Robustness test

− Results on public dataset 1 (BIWI)

− Results on public dataset 2 (ICT-3DHP)

− Effectiveness of the hair removal method

− App. 1: free-viewpoint display

− App. 2: cursor movement control

− App. 3: face shape reconstruction

4. Aknowledgement

This work was supported by the Hong Kong Innovation and Technology Commission (Project ITS/070/13).

5. Publications

[1] Songnan Li, King Ngi Ngan, Raveendran Paramesran, Lu Sheng, ''Real-time head pose tracking with online face template reconstruction'', TPAMI, 2015 [PDF] [Supplementary Note]

[2] Songnan Li, King Ngi Ngan, Lu Sheng, ''A Head Pose Tracking System using RGB-D Camera'', ICVS2013 [PDF]