TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interaction

Authors: Yiyao Huang, Zhedong Zheng, Ziwei Yu, Yaxiong Wang, Tze Tse, Angela Yao

Published in IEEE International Conference on Robotics and Automation (ICRA), 2026

Recommended citation: Yiyao Huang, Zhedong Zheng, Ziwei Yu, Yaxiong Wang, Tze Tse, Angela Yao, "TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interaction." IEEE International Conference on Robotics and Automation (ICRA), 2026.
Download PDF: https://zdzheng.xyz/files/2026/Huang_Tiger.pdf

Code is available at: https://github.com/huangyiyNUS/TIGeR/

Abstract: Pre-defined 3D object templates are widely used in 3D reconstruction of hand-object interactions. However, they often require substantial manual efforts to capture or source, and inherently restrict the adaptability of models to unconstrained interaction scenarios, e.g., heavily-occluded objects. To overcome this bottleneck, we propose a new Text-Instructed Generation and Refinement (TIGeR) framework, harnessing the power of intuitive text-driven priors to steer the object shape refinement and pose estimation. We use a two-stage framework: a text-instructed prior generation and vision-guided refinement. As the name implies, we first leverage off-the-shelf models to generate shape priors according to the text description without tedious 3D crafting. Considering the geometric gap between the synthesized prototype and the real object interacted with the hand, we further calibrate the synthesized prototype via 2D-3D collaborative attention. TIGeR achieves competitive performance, i.e., 1.979 and 5.468 object Chamfer distance on the widely-used Dex-YCB and Obman datasets, respectively, surpassing existing template-free methods. Notably, the proposed framework shows robustness to occlusion, while maintaining compatibility with heterogeneous prior sources, e.g., retrieved hand-crafted prototypes, in practical deployment scenarios.

@inproceedings{huang2026tiger,
author = "Huang, Yiyao and Zheng, Zhedong and Yu, Ziwei and Wang, Yaxiong and Tse, Tze Ho Elden and Yao, Angela",
title = "TIGeR: Text-Instructed Generation and Refinement for Template-Free Hand-Object Interaction",
abstract = "Pre-defined 3D object templates are widely used in 3D reconstruction of hand-object interactions. However, they often require substantial manual efforts to capture or source, and inherently restrict the adaptability of models to unconstrained interaction scenarios, e.g., heavily-occluded objects. To overcome this bottleneck, we propose a new Text-Instructed Generation and Refinement (TIGeR) framework, harnessing the power of intuitive text-driven priors to steer the object shape refinement and pose estimation. We use a two-stage framework: a text-instructed prior generation and vision-guided refinement. As the name implies, we first leverage off-the-shelf models to generate shape priors according to the text description without tedious 3D crafting. Considering the geometric gap between the synthesized prototype and the real object interacted with the hand, we further calibrate the synthesized prototype via 2D-3D collaborative attention. TIGeR achieves competitive performance, i.e., 1.979 and 5.468 object Chamfer distance on the widely-used Dex-YCB and Obman datasets, respectively, surpassing existing template-free methods. Notably, the proposed framework shows robustness to occlusion, while maintaining compatibility with heterogeneous prior sources, e.g., retrieved hand-crafted prototypes, in practical deployment scenarios.",
booktitle = "IEEE International Conference on Robotics and Automation (ICRA)",
url = "https://zdzheng.xyz/files/2026/Huang\_Tiger.pdf",
code = "https://github.com/huangyiyNUS/TIGeR/",
year = "2026" }