In this paper, we provide a historical survey of the most significant results concerning the computational power of neural models. We distinguish three important periods: first, the early works from McCulloch and Pitts, Kleene, and Minky, where the computational equivalence between Boolean recurrent neural networks and finite state automata is established. Secondly, the two breakthroughs by Siegelmann and Sontag showing the Turing universality of rational-weighted neural networks, and the super-Turing capabilities of analog recurrent neural networks. Thirdly, the recent results by Cabessa, Siegelmann and Villa revealing the super-Turing computational potentialities of interactive and evolving recurrent neural networks.