Robot Learning with Prior Knowledge: Leveraging Small Network Modules and Large Foundation Models

Abstract

Teaching robots versatile skills often consume much training time if the robots learn every skill from scratch. Moreover, current robot learning approaches, such as reinforcement learning (RL) and imitation learning (IL), require extensive human labor to collect expert data or design reward functions for various tasks. Despite this, some challenging tasks remain unsolved. To bridge these gaps, a promising solution is to leverage prior knowledge to facilitate the training of the current tasks. There are many sources of prior knowledge, such as policies trained in other tasks, foundation models trained with internet scale data, human preference feedback, and pre-defined kinematics and dynamics models of robots. Upon the fundamental mechanism, robot learning with prior knowledge achieves various advantages to training from scratch because we can make use of more structured information to optimize the robot policies other than optimizing them with conventional data-collection protocols.This thesis studies leveraging two types of prior knowledge sources, foundation models and pre-trained policies, improving robot learning efficiency, achieving higher performance, and enabling more versatile robot behaviors. In Chapter 1, we start by introducing the background knowledge and motivation of robot learning with prior knowledge. Then in Chapter 2, we focus on reusing the prior knowledge of the robot and the task learned in the pre-trained robot modules and task modules of policy networks. In Chapter 3, we generalize the previous results from low-dimensional state observations to high-dimensional image observations and enable the reusing of the visual configuration knowledge stored in the visual encoders. The previous two chapters focus on reusing the specific knowledge stored in the pre-trained network modules, while the next Chapter 4 shifts attention to leveraging common sense knowledge stored in the foundation models. Finally, we reach the conclusions of this thesis in Chapter 5.