AdaRoboVLG
This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands. Unlike existing VLG methods that tightly couple foundation models with end-to-end grasp policies, AdaRoboVLG learns an efficient generalizable base policy that generates and evaluates physically feasible grasp candidates through explicit kinematic mapping and force-closure-based stability estimation, while offloading task-dependent understanding to specialized foundation-model modules. These modules provide composable priors that are integrated into the grasp synthesis process, enabling contextually adaptive grasp synthesis without retraining the underlying grasp policy. Through extensive simulation and real-world experiments, we demonstrate that (i) the base policy exhibits efficient learning and strong cross-hand generalization, (ii) the framework effectively incorporates spatial, cognitive, and temporal priors to address three representative grasping challenges without compromising grasp synthesis performance compared to state-of-the-art methods, and (iii) these priors can operate jointly to enable functional grasping in cluttered and dynamic environments. These results indicate that decoupling physical grasp synthesis from task-dependent understanding provides a scalable paradigm for robotic grasping, allowing future advances in foundation models to be directly translated into improved grasp capabilities without redesigning or retraining the underlying grasp policy.
The key idea of AdaRoboVLG is to decouple task-driven understanding from physical grasp synthesis by leveraging diverse foundation-model priors. It first composes diverse foundation-model priors to understand task contexts and organizes their outputs into a structured grasp interface. Based on this interface, a shared base policy then generates executable grasp poses and enables generalization across different robotic hands.
(1) Structured Grasp Interface. The interface organizes task-dependent information into a unified representation consisting of object geometry, Contact Grasp Representations (CGRs), and hand-compatible grasp types.
(2) Base Grasp Policy. The policy first converts this interface into executable grasp poses through hand-specific grasp mapping. It evaluates candidate grasps using a hand-object interaction (HOI) representation and a hand-agnostic grasp decision model, enabling generalizable grasp synthesis across different robotic hands.
(3) Composable Foundation Priors. The spatial prior module extracts scene-level CGRs for clutter-aware grasping; the cognitive prior module infers task-consistent grasp intents to support functional grasp synthesis; and the temporal prior module estimates target motion and continuously updates the structured grasp interface for dynamic grasping.
To validate the effectiveness of AdaRoboVLG, we organize the experiments around six key questions. At the component level, we evaluate the cross-hand generalization capability of the base grasp policy and the contribution of individual spatial, cognitive, and temporal priors in simulation. At the system level, we further evaluate the composability of these priors in complex real-world tasks and the extensibility of the framework toward new capabilities.
Base Policy
Complementary Priors
System Evaluation
The HOI representation (a) is inherently compact and hand-agnostic. It not only (b) improves the learning efficiency of the grasp decision model, achieving faster convergence and lower training loss, but also (c) enables joint training across different robotic hands, facilitating cross-hand generalization of the base grasp policy.
To validate the spatial prior for clutter-aware grasping, we evaluate AdaRoboVLG on the DexGraspNet 2.0 benchmark across three robotic hands and diverse clutter levels.
Quantitative results demonstrate that AdaRoboVLG consistently generates spatially feasible grasps across diverse clutter levels. This robustness comes from the compact CGRs and human-inspired grasp taxonomy, which reduce the grasp search space, as well as the effective compatibility between the spatial prior and the base policy.
| Methods | Hands | Success Rate (%) | ||
|---|---|---|---|---|
| Dense | Random | Loose | ||
| AnyDexGrasp | DH3 | 91.2 | 85.5 | 78.2 |
| Allegro | 90.5 | 86.6 | 79.4 | |
| Inspire | 92.7 | 85.0 | 76.1 | |
| AnyDexGrasp* | DH3 | 87.8 | 86.4 | 85.7 |
| Allegro | 88.2 | 87.2 | 83.8 | |
| Inspire | 86.1 | 85.4 | 83.9 | |
| AdaRoboVLG (ours) | DH3 | 90.3 | 89.5 | 88.6 |
| Allegro | 89.5 | 88.2 | 85.9 | |
| Inspire | 88.1 | 88.5 | 86.7 | |
AnyDexGrasp* is a variant of AnyDexGrasp equipped with our base policy.
To validate the cognitive prior for functional grasping, we conduct two complementary evaluation tasks. The first evaluates grasp functionality inference on open-set instructions, while the second assesses language-guided target grasping in simulation.
Results from Task A demonstrate that equipping the LLM with RAG and CoT techniques significantly improves grasp functionality inference accuracy.
| Method | Functional Part Accuracy โ |
Grasp Type Accuracy โ |
|---|---|---|
| Base LLM | 0.82 | 0.63 |
| Base LLM + CoT | 0.91 | 0.66 |
| Base LLM + RAG | 0.85 | 0.78 |
| AdaRoboVLG (ours) | 0.94 | 0.87 |
125 instructions | 57 manipulation tasks | 80 object categories
Results from Task B verify that LLM-based grounding and SAM3-based segmentation provide accurate target constraints for generating executable and semantically consistent grasp poses across different robotic hands.
| Methods | Success Rate (%) | |||||
|---|---|---|---|---|---|---|
| GraspNet-1Billion (90 scenes) | GraspClutter6D (235 scenes) | |||||
| DH3 | Allegro | Inspire | DH3 | Allegro | Inspire | |
| ๐(โ, ๐ช)-Grasp | 66.2 | 53.3 | 57.3 | 64.7 | 52.1 | 55.9 |
| AnyDexGrasp | 85.2 | 80.8 | 84.4 | 82.5 | 74.1 | 79.3 |
| AnyDexGrasp* | 86.6 | 81.2 | 85.0 | 82.9 | 75.9 | 81.4 |
| AdaRoboVLG (ours) | 88.9 | 82.2 | 86.9 | 84.9 | 76.4 | 82.6 |
All methods use the same cognitive reasoning module.
To validate the temporal prior for dynamic grasping, we evaluate AdaRoboVLG on the GraspNet-1Billion benchmark. It achieves perfect association accuracy, the lowest translation error, and competitive rotation error over consecutive tracking frames.
| Methods | Seen | Similar | Novel | ||||||
|---|---|---|---|---|---|---|---|---|---|
| MGTA โ | etrans (cm) โ | erot (ยฐ) โ | MGTA โ | etrans (cm) โ | erot (ยฐ) โ | MGTA โ | etrans (cm) โ | erot (ยฐ) โ | |
| Nearest | -0.81 | 15.2 | 95.98 | -0.80 | 15.4 | 97.61 | -0.83 | 16.6 | 96.90 |
| AnyGrasp | -0.32 | 2.31 | 53.24 | -0.30 | 2.68 | 57.59 | -0.39 | 2.93 | 58.49 |
| BundleTrack | 0.74 | 1.52 | 15.8 | 0.72 | 1.47 | 16.37 | 0.72 | 1.93 | 15.41 |
| Target-referenced | 0.84 | 1.34 | 10.15 | 0.85 | 1.52 | 9.84 | 0.78 | 1.9 | 12.96 |
| MotionGrasp | 0.99 | 1.1 | 0.86 | 0.99 | 1.1 | 0.84 | 0.99 | 1.07 | 1.12 |
| AdaRoboVLG (ours) | 1.0 | 0.22 | 3.57 | 1.0 | 0.23 | 3.68 | 1.0 | 0.28 | 3.28 |
We further present long-horizon tracking visualization. Notably, we update grasps using the estimated rigid transformations of the target object. Based on these transformations, we project the initial 3D semantic features of objects into the current image space for visualizations. Better alignment between the projected features and target objects indicates more accurate tracking.
To validate the composability of these priors for complex real-world tasks, we conduct two experiments. Task A combines the spatial and cognitive priors for functional grasping in static clutter, while Task B further incorporates the temporal prior for functional grasping in dynamic clutter.
We test 102 everyday objects over 510 real-world grasping trials. For each trial, the spatial prior constrains grasp generation to avoid collisions with nearby objects, while the cognitive prior infers task-consistent functional grasp intents. By composing these priors, AdaRoboVLG generates spatially feasible and functionally appropriate grasp poses in cluttered scenes.
We present representative results of functional grasping in dynamic clutter on a conveyor belt moving at 2โ5 cm/s with 16 target objects and multiple distractors. These results demonstrate the effective synergy among the spatial, cognitive, and temporal priors, enabling spatially feasible, functionally appropriate, and temporally consistent grasping in real-world dynamic scenes.
We further evaluate grasp tracking robustness under four human-induced disturbances across 10 objects with five trials per object, achieving an average success rate of 89.7%.
“Use the white gauze for bandaging.”
“Pick up the sponge for dishwashing.”
“Hand the red toy car to the child for play.”
“Squeeze the shampoo bottle to release product.”
Finally, we demonstrate the extensibility of AdaRoboVLG. It enables new capabilities through additional modules without retraining the entire policy, such as bimanual sorting with task allocation and transparent-object grasping with depth completion.
The component study asks four separate questions: whether the shared policy generalizes across hands, and whether each spatial, cognitive, and temporal prior contributes its intended capability.
The HOI representation reduces learning difficulty, while joint training across three robotic hands improves overall performance and enables cross-hand generalization.
On DexGraspNet 2.0, the spatial prior produces feasible grasps across three robotic hands and varying scene complexity, with less degradation than the baselines.
RAG and structured chain-of-thought improve open-set functionality inference; grounding and segmentation then provide executable semantic constraints to the base policy.
The temporal prior achieves perfect association accuracy, the lowest translation error, and competitive rotation error while estimating target motion online.
After independent validation, the system study evaluates whether the spatial, cognitive, and temporal priors work together on more complex functional grasping tasks in static and dynamic clutter.
Because task understanding is decoupled from grasp synthesis, new capabilities can be introduced as modules while the shared policy and structured interface remain unchanged.
