Urban heritage conservation has often been portrayed as a practice shaped by ‘authorised discourses’ which are produced by powerful actors including the state, international organisations and experts. But the literature has also paid attention to non-governmental actors who produce ‘unauthorised heritage discourses’ by calling for broader and more diversified heritage interpretations and practices. Using the case of Phnom Penh, the capital of Cambodia, we argue that the dichotomy between authorised and unauthorised heritage has produced artificial boundaries between those legacies which have been (or should be) identified as heritage and the multiple remains of the past which nobody has ever attempted to define as such.