Transfer and Curriculum Learning
Introduction
Human’s are intrinsically able to transfer the knowledge from one scenario to another task. With the inception of deep reinforcement learning algorithms, we are able to see
super human performance in many domains. These methods often spend a huge amount of time and resources to train a deep model to perform a single task.
When the problem is extend to a multi-agent setting, the system gets exponentially difficult to solve. Quite often when we include decision making model
in belief state, the system becomes even more complicated to run. Currently applying knowledge gained from performing a single task to performing an other related task remains a challenging problem. This is the problem Transfer
learning is trying to solve (Taylor Stone, 2009).
A curriculum is an efficient tool for humans to progressively learn from simple concepts to hard problems.
It breaks down complex knowledge by providing a sequence of learning steps of increasing difficulty.
While curriculum learning has been presented in a wide range of human robot intersection and application, an excellent blog to understand
Curriculum learning for Reinforcement Learning is here
Sebastian Ruder, posted a blog in titled “Transfer Learning- Machine Learning’s Next Frontier”. The article presents the idea of transfer learning, applications, scenarios and methods and related work in the research area. The article was presented in 2017 but is a beginner’s guide to transfer learning. In sections below, we will be presenting some of the state of the art algorithms in transfer learning and focus on avenues of research at RISE-Lab.
Multi Task Learning and Transfer
There exist various techniques that incorporate tasks directly into the definition of the value function for multitask learning (Kaelbling, 1993; Ashar, 1994; Sutton et al., 2011). UVFAs have been used for zero-shot generalisation to combinations of tasks (Mankowitz et al., 2018; Hermann et al., 2017), or to learn a set of fictitious goals previously encountered by the agent (Andrychowicz et al., 2017). One of the key aspects of learning can be through MetaLearning where the agents are taught different task and the genralization of the task to a new task is tested. Thus signfying true transfer. \
When we think of value functions or universal value functions for the function approximation across tasks, we can look into how transfer learning can be approached using Reinforcement Learning. One of the key aspects of transfer in RL to do multi task is through UVFAs. Another aspect for transfer in RL is done through the idea of Successor Features, where the reward of the RL algorithm is divided into weights and features associated to the environment. Deepmind, has been working on this idea for years now. The associated research for this can be found in here
Based on the application, the transfer learning algorithms vary to a certain degree. Here we provide some of the sample projects on transfer learning on different applicaitons.
Sample Projects
Transfer Learning in Computer Vision
Transfer learning is a popular method in computer vision because it allows us to build accurate models in a timesaving way (Rawat & Wang 2017). With transfer learning, instead of starting the learning process from scratch, you start from patterns that have been learned when solving a different problem here.
The website provides sample code on using deep learning method to develop classifier for identifying cats from re-purposing a pre trainined model.
The algorithms are not limited to image classifier, the research has been used in identifying cancer cells in absence of large dataset|
Transfer Learning in Human-Robot Interaction
The authors of Conventions introduce the idea of conventions to differentiate the knowledge that may be specific to the task and to the partners.
Multi-level Perceptrons are used to encode the behavior of the agents.
Spotlight Talk about the framework is available here
The code for the paper is available here
Simulation Software for Transfer and Generalization in Autonomous Vehicles
To generate a generalizable policy for the autonomous vehicles in real world scenario, the authors have developedMetaDrive. The system currently supports RL algorithms and MARL algorithms with varying driving scenarios, vehicle dynamics and real data set inclusion.
Posing the same property, CommonRoad is a collection of composable benchmarks for motion planning on roads, which provides researchers with a means of evaluating and comparing their motion planners.
RL framework has been integrated in the CommonRoad framework. There are available blogs and forums for support.