Learning Social Affordance for Human-Robot Interaction

Tianmin Shu1, Michael S. Ryoo2 and Song-Chun Zhu1

Center for Vision, Cognition, Learning, and Autonomy, UCLA1

School of Informatics and Computing, Indiana University Bloomington2

Introduction

Abstract

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions.

We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful social affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.

Paper and Demo

Paper

Tianmin Shu, M. S. Ryoo and Song-Chun Zhu. Learning Social Affordance for Human-Robot Interaction. International Joint Conference on Artificial Intelligence (IJCAI), 2016. [arXiv] [slides] [poster] 

@inproceedings{ShuIJCAI16,
  title     = {Learning Social Affordance for Human-Robot Interaction},
  author    = {Tianmin Shu and M. S. Ryoo and Song-Chun Zhu},
  year      = {2016},
  booktitle = {International Joint Conference on Artificial Intelligence (IJCAI)}
}

UCLA Human-Human-Object Interaction (HHOI) Dataset

Download

Skeleton + Annotation (69.7 MB)

Skeleton + RGB Images + Depth Images + Depth2RGB Mapper + Annotation (55.2 GB)

The dataset is available for free to researchers from academic institutions (e.g., universities, government research labs, etc.) for non-commercial purposes.

We greatly appreciate emails about bugs or suggestions.

Please cite this paper if you use the dataset:

Tianmin Shu, M. S. Ryoo and Song-Chun Zhu. Learning Social Affordance for Human-Robot Interaction. International Joint Conference on Artificial Intelligence (IJCAI), 2016.

Contact

Any question? Please contact Tianmin Shu (tianmin.shu [at] ucla.edu)