GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment

Lance Ying, Kunal Jha, Shivam Aarya, Joshua B. Tenenbaum, Antonio Torralba, Tianmin Shu.

International Conference on Intelligent Robots and Systems (IROS), 2024.

Paper   Project   Code

MMToM-QA: Multimodal Theory of Mind Question Answering

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer D. Ullman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu.

62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

(Outstanding Paper Award)

Paper   Project   Code   Data

Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input

Andi Peng, Yuying Sun, Tianmin Shu, David Abel.

41st International Conference on Machine Learning (ICML), 2024

Paper   Code

COMBO: Compositional World Models for Embodied Multi-Agent Cooperation

Hongxin Zhang*, Zeyuan Wang*, Qiushi Lyu*, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Yilun Du, Chuang Gan.

arXiv preprint, 2024.

Paper   Project   Code

Building Cooperative Embodied Agents Modularly with Large Language Models

Hongxin Zhang*, Weihua Du*, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan.

12th International Conference on Learning Representations (ICLR), 2024.

Paper   Project   Code

Neural Amortized Inference for Nested Multi-agent Reasoning

Kunal Jha, Tuan Anh Le, Chuanyang Jin, Yen-Ling Kuo, Joshua B. Tenenbaum, Tianmin Shu.

38th AAAI Conference on Artificial Intelligence (AAAI), 2024.

Paper   Project   Code   Supp

Risk-Bounded Online Team Interventions via Theory of Mind

Yuening Zhang, Paul Robertson, Tianmin Shu, Sungkweon Hong, Brian C. Williams.

IEEE International Conference on Robotics and Automation (ICRA), 2024.

Paper

An asynchronous, automated workflow for looking time experiments with infants

Gal Raz, Sabrina Piccolo, Janine Medrano, Shari Liu, Kirsten Lydic, Catherine Mei, Victoria Nguyen, Tianmin Shu, Rebecca Saxe.

Developmental Psychology, 2024.

Paper

Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning

Zhiting Hu* and Tianmin Shu*.

arXiv preprint, 2023.

Paper   NeurIPS 2023 Tutorial

Language Models Meet World Models: Embodied Experiences Enhance Language Models

Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu.

37th Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.

Paper   Code

NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants

Xavier Puig*, Tianmin Shu*, Joshua B. Tenenbaum, and Antonio Torralba.

IEEE International Conference on Robotics and Automation (ICRA), 2023.

Paper   Project   Code   TechXplore

Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptation

Andi Peng, Aviv Netanyahu, Mark Ho, Tianmin Shu, Andreea Bobu, Julie Shah, and Pulkit Agrawal.

40th International Conference on Machine Learning (ICML), 2023.

Paper   Project

Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning

Dianbo Liu*, Vedant Shah*, Oussama Boussif*, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Mozer, Nicolas Heess, and Yoshua Bengio.

11th International Conference on Learning Representations (ICLR), 2023.

Paper   Code

Zero-shot linear combinations of grounded social interactions with Linear Social MDPs.

Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum, Boris Katz, and Andrei Barbu.

37th AAAI Conference on Artificial Intelligence (AAAI), 2023.

Paper

Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning

Aviv Netanyahu*, Tianmin Shu*, Joshua B. Tenenbaum, and Pulkit Agrawal.

39th International Conference on Machine Learning (ICML), 2022.

Paper   Project   Code

RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu.

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.

Paper   Code   Blog

Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration

Xiaofeng Gao, Luyao Yuan, Tianmin Shu, Hongjing Lu, and Song-Chun Zhu.

IEEE Robotics and Automation Letters (RA-L), 2022.

Paper   Project   TechXplore

Incorporating Rich Social Interactions Into MDPs.

Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Bennett Stankovits, Dan Gutfreund, Joshua B. Tenenbaum, Boris Katz, and Andrei Barbu.

IEEE International Conference on Robotics and Automation (ICRA), 2022.

(Excellent Paper Award at IROS Cognitive and Social Aspects of Human Multi-Robot Interaction Workshop, 2022)

Paper   Project

Social Interactions as Recursive MDPs

Ravi Tejwani*, Yen-Ling Kuo*, Tianmin Shu, Boris Katz, and Andrei Barbu.

5th Conference on Robot Learning (CoRL), 2021.

Paper   Project   MIT News   Discover Magazine

AGENT: A Benchmark for Core Psychological Reasoning

Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin A. Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua B. Tenenbaum, and Tomer D. Ullman.

38th International Conference on Machine Learning (ICML), 2021.

Paper   Supp   Project   Code (Data Generation)   ScienceNews for Students   VentureBeat   Blog

Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba.

9th International Conference on Learning Representations (ICLR), 2021. (Spotlight presentation)

(Best Paper Award at NeurIPS Cooperative AI Workshop, 2020)

Paper   Project   Code   VentureBeat

PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception

Aviv Netanyahu*, Tianmin Shu*, Boris Katz, Andrei Barbu, and Joshua B. Tenenbaum.

35th AAAI Conference on Artificial Intelligence (AAAI), 2021.

(Best Paper Award at NeurIPS Shared Visual Representations in Human and Machine Intelligence Workshop, 2020)

Paper   Supp   Project   Code (Data Generation)

A Unified Psychological Space for Human Perception of Physical and Social Events

Tianmin Shu, Yujia Peng, Song-Chun Zhu, and Hongjing Lu.

Cognitive Psychology, 128: 101398, 2021.

Paper   Code (Stimuli Generation)   Code (Model)   Data

Exploring Biological Motion Perception in Two-stream Convolutional Neural Networks

Yujia Peng, Hannah Lee, Tianmin Shu, and Hongjing Lu

Vision Research, 178: 28-40, 2021.

Paper

Adventures in Flatland: Perceiving Social Interactions Under Physical Dynamics

Tianmin Shu, Marta Kryven, Tomer D. Ullman, and Joshua B. Tenenbaum.

42nd Annual Meeting of the Cognitive Science Society (CogSci), 2020.

Paper   Bibtex

Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks

Xiaofeng Gao*, Ran Gong*, Yizhou Zhao, Shu Wang, Tianmin Shu, and Song-Chun Zhu.

International Conference on Robot & Human Interactive Communication (RO-MAN), 2020.

Paper   Project   Code

Active Visual Information Gathering for Vision-Language Navigation

Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen.

European Conference on Computer Vision (ECCV), 2020.

Paper   Code

Learning to Infer Human Attention in Daily Activities

Zhixiong Nan, Tianmin Shu, Ran Gong, Shu Wang, Ping Wei, Song-Chun Zhu, and Nanning Zheng.

Pattern Recognition, 103: 107314, 2020.

Paper

M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

Tianmin Shu and Yuandong Tian.

7th International Conference on Learning Representations (ICLR), 2019.

Paper   Code   Bibtex

Partitioning the Perception of Physical and Social Events Within a Unified Psychological Space

Tianmin Shu, Yujia Peng, Hongjing Lu, and Song-Chun Zhu.

41st Annual Meeting of the Cognitive Science Society (CogSci), 2019.

(Oral presentation)

Paper   Project   Bibtex

VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks

Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, and Song-Chun Zhu.

ICML workshop on Reinforcement Learning for Real Life, 2019.

Paper   Project   Code   Tech Xplore   Bibtex

Social Scene Understanding: Group Activity Parsing, Human-Robot Interactions, and Perception of Animacy

Tianmin Shu.

PhD Thesis, 2019.

Paper

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

Tianmin Shu, Caiming Xiong, and Richard Socher.

6th International Conference on Learning Representations (ICLR), 2018.

Paper   ScienceNews   Bibtex

Learning and Inferring "Dark Matter"
and Predicting Human Intents and Trajectories in Videos

Dan Xie, Tianmin Shu, Sinisa Todorovic, and Song-Chun Zhu.

IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 40(7): 1639-1652, 2018.

Paper   Project   Bibtex

Interactive agent modeling by learning to probe

Tianmin Shu, Caiming Xiong, Ying Nian Wu, and Song-Chun Zhu.

NeurIPS Deep Reinforcement Learning Workshop, 2018.

Paper

Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks

Ping Wei, Yang Liu, Tianmin Shu, Nanning Zheng, and Song-Chun Zhu.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

Paper   Bibtex

Perception of Human Interaction Based on Motion Trajectories: from Aerial Videos to Decontextualized Animations

Tianmin Shu*, Yujia Peng*, Lifeng Fan, Hongjing Lu, and Song-Chun Zhu.

Topics in Cognitive Science, 10(1): 225 - 241, 2018.

Paper   Bibtex

Inferring Human Interaction from Motion Trajectories in Aerial Videos

Tianmin Shu*, Yujia Peng*, Lifeng Fan, Hongjing Lu, and Song-Chun Zhu.

39th Annual Meeting of the Cognitive Science Society (CogSci), 2017.

(Computational Modeling Prize)

Paper   Project   Bibtex

CERN: Confidence-Energy Recurrent Network for Group Activity Recognition

Tianmin Shu, Sinisa Todorovic, and Song-Chun Zhu.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

Paper   Supp   Bibtex

Learning Social Affordance Grammar from Videos:
Transferring Human Interactions to Human-Robot Interactions

Tianmin Shu, Xiaofeng Gao, Michael S. Ryoo, and Song-Chun Zhu.

IEEE International Conference on Robotics and Automation (ICRA), 2017.

Paper   Project   Slides   New Scientist   Bibtex

Critical Features of Joint Actions that Signal Human Interaction

Tianmin Shu*, Steven Thurman*, Dawn Chen, Song-Chun Zhu, and Hongjing Lu.

38th Annual Meeting of the Cognitive Science Society (CogSci), 2016.

Paper   Bibtex

Learning Social Affordance for Human-Robot Interaction

Tianmin Shu, M. S. Ryoo, and Song-Chun Zhu.

25th International Joint Conference on Artificial Intelligence (IJCAI), 2016.

Paper   Project   Slides   Bibtex

Joint Inference of Groups, Events and Human Roles in Aerial Videos

Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, and Song-Chun Zhu.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

(Oral presentation, acceptance rate: 71/2123 = 3.3%)

Paper   Project   Supp  Slides   Bibtex