Approximate Policy Evaluation

Run a simplified value iteration. The policy is fixed, so we know what action to do in each state. Repeat the following a fixed number of times:

$$U_{i+1}(s)\leftarrow R(s)+\gamma \sum _{s^{\prime }}P(s^{\prime }|s,{\pi }_i(s))U_i(s^{\prime })$$