1. Reinforcement Learning

In reinforcement learning, the program learns from the consequences of its actions (reward or punishment), rather than being explicitly taught and selects its actions on basis of its past experiences (exploitation) but also by new choices (exploration).

Basic Concept

The agent takes actions in an environment by observing the current state. Execution leads to a transformation of the state and a reward is fed back into the agent.

So the reinforcement learning problem has a goal of selecting actions that maximise the future reward but actions can have long term consequences and the reward may be delayed. Sometimes immediate reward inhibits long-term reward.

We need to establish a balance between exploration and exploitation, and learn through trial-and-error.

Definitions

• Environment (AI)

  Environment is the representation of the world, establishing the states of the problem, the available actions and their impact.

  Link to original

• Model (AI)

  Model is the agents’ representation of the state. Predicting what the environment will do next.

  Link to original

• Reward (AI)

  Reward is indicative of the quality of the action after execution of a given action (including non-action). Agent should maximise cumulative reward.

  Link to original

• Timestep (AI)

  Timestep: a time point $t$ within the problem. States, actions, and rewards are recorded with specific timestep, $S_v,A_v,A_r$.

  Link to original

• State Value Function

  State Value Function $V^{\ast }(s)$: the total amount of reward an agent can expect to accumulate in the future from a given state. The long term viability of a given state.

  Link to original

• State-Action Value Function

  State-Action Value Function $Q^{\ast }(s,a)$: the total amount of reward an agent can expect to accumulate in the future from a given state when taking a given action.

  Link to original

• Policy (AI)

  Policy: a representation that encodes what actions should be selected when we are in a given state.

  Link to original

• Exploration (AI)

  Exploration: explores environment with intent to learn

  Link to original

• Exploitation (AI)

  Exploitation: capitalises on existing knowledge of the environment to maximise reward

  Link to original

Reinforcement Learning vs AI Planning

Reinforcement Learning	AI Planning
Environment is unknown, agent must explore it	We build the PDDL model of the problem
Agent must interact with model to improve	Not concerned with rewards; heuristics define relevant of states/actions to goals

[gridworld example]