A proper policy always ends up in a terminal state eventually, thus have a finite expected utility.