3D Hand Pose Estimation in Point Cloud Using 3D Convolutional Neural Network on Egocentric Datasets

  • Van Hung Le MICA Institute, HUST - CNRS/UMI-2954 - GRENOBLE INP, Vietnam
  • Hai Yen Tran Vietnam Academy of Dance
Keywords: 3D Hand Pose Estimation, Point Cloud Data, 3D Convolutional Neural Networks, Egocentric Vision Dataset


3D hand pose estimation from egocentric vision is an important study in the construction of assistance systems and modeling of robot hand in robotics. In this paper, we propose a complete method for estimating 3D hand pose
from the complex scene data obtained from the egocentric sensor. In which we propose a simple yet highly efficient pre-processing step for hand segmentation. In the estimation process, we used the Hand PointNet (HPN), V2V-PoseNet
(V2V), Point-to-Point Regression PointNet (PtoP) for finetuning to estimate the 3D hand pose from the collected data obtained from the egocentric sensor, such as CVRA, FPHA (First-Person Hand Action) datasets. HPN, V2V, PtoP are the
deep networks/Convolutional Neural Networks (CNNs) for estimating 3D hand pose that uses the point cloud data of the hand. We evaluate the estimation results using the preprocessing step and do not use the pre-processing step to see the effectiveness of the proposed method. The results show that 3D distance error is increased many times compared to estimates on the hand datasets are not obstructed (the hand data obtained from surveillance cameras, are viewed from top view, front view, sides view) such as MSRA, NYU, ICVL datasets. The results are quantified, analyzed, shown on the point cloud data of CVAR dataset and projected on the color image of FPHA dataset.

Author Biography

Hai Yen Tran, Vietnam Academy of Dance

Hai-Yen Tran received Bachelor degree at Faculty Information Technology National Economics University (2009). She received M.Sc. degree at Faculty Information Technology Hanoi National University of Education (2013).
Currently, she is a lecture of Vietnam Academy of Dance. Her research interests include computer science, deep learning.
Email: yenth@vnad.edu.vn