Popular comparative phylogenetic models such as Brownian Motion, Ornstein-Ulhenbeck, and their extensions, assume that, at speciation, a trait value is inherited identically by two descendant species. This assumption contrasts with models of speciation at a micro-evolutionary scale where descendants' phenotypic distributions are sub-samples of the ancestral distribution. Different speciation mechanisms can lead to a displacement of the ancestral phenotypic mean among descendants and an asymmetric inheritance of the ancestral phenotypic variance. In contrast, even macro-evolutionary models that account for intraspecific variance assume symmetrically conserved inheritance of ancestral phenotypic distribution at speciation. Here we develop an Asymmetric Brownian Motion model (ABM) that relaxes the assumption of symmetric and conserved inheritance of the ancestral distribution at the time of speciation. The ABM jointly models the evolution of both intra- and inter-specific phenotypic variation. It also infers the mode of phenotypic inheritance at speciation, which can range from a symmetric and conserved inheritance, where descendants inherit the ancestral distribution, to an asymmetric and displaced inheritance, where descendants inherit divergent phenotypic means and variances. To demonstrate this model, we analyze the evolution of beak morphology in Darwin finches, finding evidence of displacement at speciation. The ABM model helps to bridge micro- and macro-evolutionary models of trait evolution by providing a more robust framework for testing the effects of ecological speciation, character displacement, and niche partitioning on trait evolution at the macro-evolutionary scale.