R-CNNs for Pose Estimation and Action Detection

March, 2015

Abstract

We present convolutional neural networks for the tasks of keypoint (pose) prediction and action classification of people in unconstrained images. Our approach involves training an R-CNN detector with loss functions depending on the task being tackled. We evaluate our method on the challenging PASCAL VOC dataset and compare it to previous leading approaches. Our method gives state-of-theart results for keypoint and action prediction. Additionally, we introduce a new dataset for action detection, the task of simultaneously localizing people and classifying their actions, and present results using our approach.

paper

Deploy Code

Before using the available source code, you need to install Caffe.

Action Classification

• pascal_finetune_fc7.prototxt prototxt to extract fc7 features
• pascal_finetune_HumanNet2_iter_10000 model weights trained on PASCAL VOC 2012 Action train set

Keypoint Prediction

• kp_reg_deploy.prototxt prototxt to extract fc8 features
• pascal_finetune_HumanNet3_flip_iter_90000 model weights trained on PASCAL VOC 2012 Main train set
• fc8_to_coords.m transforms fc8 features to keypoint coordinates (in matlab)
• kps_names.txt list of names of the keypoints

Person Detection+Action+Pose

• pascal_finetune_fc7.prototxt prototxt to extract fc7 features
• pascal_finetune_HumanNet123_iter_100000 model weights trained on PASCAL VOC 2012 Main+Action train set jointly for the three tasks

Dataset

You can download the action dataset, as used in the paper. The dataset contains the PASCAL VOC Action 2012 images, with complete annotations of all the people and their action labels.

Dataset download: action_dataset.tar.gz

How to cite

When citing our system, please cite this work. The bibtex entry is provided below for your convenience.

@inproceedings{poseactionrcnn,
 Author = {G. Gkioxari and B. Hariharan and R. Girshick and J. Malik},
 Title = {R-CNNs for Pose Estimation and Action Detection},
 ArchivePrefix = {arXiv},
 Eprint = {1406.5212},
 PrimaryClass = {cs.CV},
 Year = {2014}}

Contact

For any questions regarding the work or the implementation, contact the author at gkioxari@eecs.berkeley.edu