Approximate Dynamic Programming and Reinforcement Learning for Control

Lecturer: Lucian Busoniu

This course provides methods for controlling systems that are too complex or insufficiently known to apply classical control design techniques. Classical foundations are connected to recent developments. The focus is placed on learning algorithms for control, in particular reinforcement learning (RL). Special attention is also paid to model-based techniques related to RL, as they can be very useful in controlling complex systems even when a model is known. After introducing the RL problem, the dynamic programming algorithms that sit at the foundation of RL are described. Then, classical, discrete-variable RL algorithms are introduced. In the second part of the course, the dynamical programming and RL algorithms are extended with approximation techniques, in order to make them applicable to continuous-variable control, as well as to large-scale discrete-variable problems. Several online planning techniques are discussed.

Lecture slides

• Part 1: Problem definition. Discrete case.
• Part 2: Continuous case.
• Part 3: Optimistic planning.

The demos supporting the course can be downloaded below:

• Demos for part 1.
• Demos for parts and 3.

These depend on the Approximate DP and RL toolbox, and you can download the latest snapshot here. Unzip the toolbox, point Matlab to its root directory, and run startupapproxrl. Then you can run the demos in cell-mode (CTRL+ENTER in each cell separated by %% comments).

Contact

Comments, suggestions, questions etc. related to this course; please contact the lecturer.