Supplementary InformationReinforcement Learning Under Moral Uncertainty

It may be objected that some ethical theories (e.g. variants of deontology such as the categorical imperative (Kant & Paton, 1964)) appear to be better represented ordinally rather than cardinally. MacAskill (2014) proposes that the Borda count (Pacuit, 2019) is a principled way of obtaining a cardinal utility function from a purely ordinal theory under circumstances of moral uncertainty. Thus, for simplicity and because it is often possible to convert ordinal theories to a cardinal representation, our work focuses on cardinal utility functions only. However, handling these seemingly ordinal theories more directly is an interesting avenue for future work, for which work on ordinal RL (Wirth et al., 2017; Zap et al., 2019) could serve as a starting point. It has also been argued that many seemingly ordinal theories are in fact better represented lexicographically (MacAskill, 2014) (a combination of ordinal and cardinal representation), suggesting lexicographic RL (Gábor et al., 1998) as an alternative starting point.