InfCoiL: Coordinated Planner-Controller Learning for Closed-Loop Physics-Based Human-Object Interaction

Yude Zou1,2,*, Yifei Yao2,*, Yiwei Hao1,*, Hanqing Wang2, Xing Gao2,†

1Shanghai Jiao Tong University 2Shanghai Artificial Intelligence Laboratory

*Equal contribution †Corresponding author

Table
Tripod

Abstract

Text- and goal-conditioned human-object interaction (HOI) provides an intuitive interface for specifying high-level intents in physics-based character animation. Mapping such intents to physically plausible, contact-rich behaviors naturally calls for a generative planner and a physics-based controller, yet existing planner-controller systems either execute plans open-loop or close the loop while keeping the planner fixed, so execution feedback never reshapes what is planned. We present InfCoiL, a closed-loop framework for physics-based HOI from coordinated planner-controller learning. InfCoiL couples a morphology-aware flow-matching planner with a unified multi-morphology controller through a shared morphology interface in an autoregressive plan-and-execute loop, and alternately fine-tunes both components with reinforcement learning from the same physical rollouts. To optimize the flow-matching planner online, we introduce Semantic Flow Policy Optimization (SFPO), which stochasticizes selected denoising transitions with channel-wise semantic covariance over heterogeneous HOI features and assigns group-relative, sequence-level rewards from complete closed-loop executions. On held-out OMOMO sequences, InfCoiL outperforms open-loop hierarchical baselines in goal-reaching task success and physical tracking accuracy.

Methodology

Overview of InfCoiL.
Figure 1. Overview of InfCoiL. (a) Closed-loop framework: the flow-matching planner generates an Nf-frame reference window, the controller executes its first Nr frames in Isaac Gym, and the realized human, object, and contact states are fed back for receding-horizon replanning. (b) Coordinated learning: controller (PPO) and planner (SFPO) training alternate to narrow the gap between planner-generated references and what the controller can execute. (c) SFPO: for each task condition, a group of plans is sampled with semantic noise injected during denoising and executed in closed loop. Their sequence rewards are normalized within the group into relative advantages that update the planner. (d) Reward design: the controller is trained on stepwise transition and task rewards, and each plan is scored by the time-averaged sequence reward. The interaction term is used only in open-loop controller training.

Results and Analysis

We compare InfCoiL with two open-loop baselines and four internal variants.

Table 1. Task-oriented evaluation on the re-split OMOMO test set. Each sequence is evaluated three times and metrics are averaged. C and P denote controller (PPO) and planner (SFPO) fine-tuning phases: InfCoiL (C→P) uses 10k controller iterations followed by 500 planner iterations, and InfCoiL (C→P→C) adds 4k further controller iterations.

Method SR@0.3↑ Eh↓ Eo↓ Pelvis Goal↓ Object Goal↓
InterMimic student (open-loop) 24.62 9.23 16.89 90.39 111.89
SimGenHOI* (open-loop + SFT) 52.14 9.69 18.00 50.12 68.15
InfCoiL open-loop 49.23 9.40 18.08 57.15 75.51
InfCoiL w/o FT P & C 56.09 7.25 16.10 46.89 69.83
InfCoiL w/o FT C 56.98 7.75 16.36 47.02 70.60
InfCoiL w/o FT P 73.37 8.44 14.67 35.72 49.45
InfCoiL (C→P) 77.24 8.90 14.76 32.90 45.37
InfCoiL (C→P→C) 81.19 8.87 14.21 31.31 41.51

BibTeX

@inproceedings{zou2026infcoil,
  title     = {{InfCoiL}: Coordinated Planner-Controller Learning for Closed-Loop Physics-Based Human-Object Interaction},
  author    = {Zou, Yude and Yao, Yifei and Hao, Yiwei and Wang, Hanqing and Gao, Xing},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}