artificial intelligence

Module 9: reinforcement learning

executive Summary

  • Topics: 
    • We will do a very broad overview of reinforcement learning in this module
    • If you love the topic, the ML class does a lot more detail on RL (at least when I teach it, you should as Dr Diochnos for his syllabus if you are interested in taking the ML class)!  This module is designed to get you excited about the topics and familiar with the ideas.  
  • Length: This module will take one week to complete
  • Assigned chapters: Chapter 22
    • Optional reading: If you really like reinforcement learning, the full book on reinforcement learning is online for free here.  Highly recommended!
  • Project: Project 5 was assigned last module but you should be working on it during this module as well!  It is due during the next module.

Reinforcement Learning

Reinforcement learning (RL) is a different kind of machine learning than what we have been studying so far.  In the introduction to ML module, we talked about the different types of ML and we mostly have been studying supervised learning methods with a handful of unsupervised methods.  RL is very different because the external teacher provides evaluative feedback but does not provide the right answer.  This is perfect for situations such as games (where the goal is usually to win but the path to winning is not known) and robotics (where the goal is usually able to be specified but the exact sequence of motors to turn on is difficult to pre-specify).  

This module will give a quick overview of the main idea behind RL along with an overview of some of the popular RL learning methods.  If you are interested in learning more, I linked to both the RL book (available online for free) and/or you could take the ML class which covers RL in more depth.

Deep RL is a relatively new area where deep learning methods are combined with RL, typically for playing complicated games.  Deep RL is used in Alpha Zero and Alpha Star.  It has also been used successfully for robotics.  Although the full extent of deep RL is beyond the scope of the class, we will have a brief topic below explaining how Alpha Zero ties together adversarial search (minimax and monte carlo tree search) with RL and deep learning.

An RL agent learning to ride a Bicycle 

RL bicycle path (and book cover)

The cover of the RL book shows the tracks of an RL agent learning to ride a bicycle

Assignments for Module 9

Topic 1: Introduction to Reinforcement learning 

For the first topic, we will learn about the foundations of RL and Markov Decision Processes.

  • (30 min) Reading
    • Read Section 22.1-22.2 (Introduction through Passive RL)
  • (30 min) 
    • Watch the video on an overview of RL
    • Watch the video on notation and how the RL problem is setup
    • Watch the video on Markov Decision Processes
    • Watch the video on how to choose an action
    • Watch the video on value functions
  • Complete the exercise on the introduction to RL
    •  

Topic 2: TD learning and Q-learning

In this module, we will discuss several specific learning methods for RL.

  • (30 min) Reading
    • Read Section 22.3 (Active RL)
    •  

Topic 3: Deep RL and games

This final topic is super quick since we have discussed alpha go before.  It is just an overview of how RL, minimax, MCTS, and deep learning can be combined for a super powerful gaming agent.

  • Watch my video on Alpha Go Zero

project for module 9

Project

suggested schedule for module 9

week 1

  • Work on Project 5 all week
  • Complete Topic 1 by Tuesday
  • Complete Topic 2 by Thursday
  • Complete Topic 3 by Sunday