1) Learning affordances from human videos
We learn a prior affordance model on where (wrist location + rotation) and how (finger joint angles) to grasp objects from human videos. This model initializes the real-world policy.
Dexterity is often seen as a cornerstone of complex manipulation. Humans are able to perform a host of skills with their hands, from making food to operating tools. In this paper, we investigate these challenges, especially in the case of soft, deformable objects as well as complex, relatively long-horizon tasks. Although, learning such behaviors from scratch can be data inefficient. To circumvent this, we propose a novel approach, DEFT (DExterous Fine-Tuning for Hand Policies), that leverages human-driven priors, which are executed directly in the real world. In order to improve upon these priors, DEFT involves an efficient online optimization procedure. With the integration of human-based learning and online fine-tuning, coupled with a soft robotic hand, DEFT demonstrates success across various tasks, establishing a robust, data-efficient pathway toward general dexterous manipulation.
We learn a prior affordance model on where (wrist location + rotation) and how (finger joint angles) to grasp objects from human videos. This model initializes the real-world policy.
We improve upon the affordance by fine-tuning in the real world. We sample from the affordance and fine-tune the policy by learning a residual policy on the trajectory and grasp parameters. Across all tasks, experiments, and ablations, we spend 100+ hours collecting over 3000 rollouts in the real world.
To ensure safe learning, we use the DASH hand that we introduced in DASH: A Framework for Designing Anthropomorphic Soft Hands through Interaction. This soft hand is very durable to bumps and does not damage the environment.
We show results on several kitchen tasks, including tasks that manipulate liquids and soft objects. All videos are in real time.
We also attempt harder tasks that involve manipulating tools like spoons. We show early results in the videos below. A limitation of our hand is that its fingers cannot curl in fully, which makes it difficult to grasp thin objects like spoon/spatula handles.
One failure case is the hand does not curl enough to maintain a strong grasp of the object. While the hand has human-like form, it does not have human-like strength. Another drawback is that the arm has restrictions on its strength because the arm uses inverse kinematics and does not have a force-based controller.
@article{kannan2023deft,
author = {Kannan, Aditya and Shaw, Kenneth and Bahl, Shikhar and Mannam, Pragna and Pathak, Deepak},
title = {DEFT: Dexterous Fine-Tuning for Real-World Hand Policies},
journal = {CoRL},
year = {2023},
}