Predictions of future biological invasions often rely on the assumption that introduced species establish only under climatic conditions similar to those in their native range. To date, 135 studies have tested this assumption of 'niche conservatism', yielding contradictory results. Here we revisit this literature, consider the evidence for niche shifts, critically assess the methods used, and discuss the authors' interpretations of niche shifts. We find that the true frequency of niche shifts remains unknown because of diverging interpretations of similar metrics, conceptual issues biasing conclusions towards niche conservatism, and the use of climatic data that may not be biologically meaningful. We argue that these issues could be largely addressed by focussing on trends or relative degrees of niche change instead of dichotomous classifications (shift versus no shift), consistently and transparently including non-analogous climates, and conducting experimental studies on mismatches between macroclimates and microclimates experienced by the study organism. Furthermore, an observed niche shift may result either from species filling a greater part of their fundamental niche during the invasion (a 'realised niche shift') or from rapid evolution of traits adapting species to novel climates in the introduced range (a 'fundamental niche shift'). Currently, there is no conclusive evidence distinguishing between these potential mechanisms of niche shifts. We outline how these questions may be addressed by combining computational analyses and experimental evidence.