Duolando: Follower GPT with Off-Policy Reinforcement Learning to Dance Accompaniment

ICLR 2024

1S-Lab, Nanyang Technological University 2Lexica 3SenseTime 4Shanghai AI Laboratory
  • ✉corresponding author


We introduce a novel task within the field of 3D dance generation, termed dance accompaniment, which necessitates the generation of responsive movements from a dance partner, the "follower", synchronized with the lead dancer's movements and the underlying musical rhythm. Unlike existing solo or group dance generation tasks, a duet dance scenario entails a heightened degree of interaction between the two participants, requiring delicate coordination in both pose and position. To sup- port this task, we first build a large-scale and diverse duet interactive dance dataset, DD100, by recording about 117 minutes of professional dancers' performances. To address the challenges inherent in this task, we propose a GPT-based model, Duolando, which autoregressively predicts the subsequent tokenized motion condi- tioned on the coordinated information of the music, the leader's and the follower's movements. To further enhance the GPT's capabilities of generating stable results on unseen conditions (music and leader motions), we devise an off-policy rein- forcement learning strategy that allows the model to explore viable trajectories from out-of-distribution samplings, guided by human-defined rewards. Based on the collected dataset and proposed method, we establish a benchmark with several carefully designed metrics.


AR Demo (OOD)

Data Examples (Quick Scan)

This section showcases clips from 10 dance genres in the DD100 dataset.
Cha Cha


  author    = {Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, Chen Change Loy},
  title     = {Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment},
  booktitle = {ICLR},
  year      = {2024},


Dataset and Code are released under NTU S-Lab 1.0 License.