The lactose uptake pathway of E. coli is a paradigmatic example of multistability in gene regulatory circuits. In the induced state of the lac pathway, the genes comprising the lac operon are transcribed, leading to the production of proteins that import and metabolize lactose. In the uninduced state, a stable repressor-DNA loop frequently blocks the transcription of the lac genes. Transitions from one phenotypic state to the other are driven by fluctuations, which arise from the random timing of the binding of ligands and proteins. This stochasticity affects transcription and translation, and ultimately molecular copy numbers. Our aim is to understand the transition from the induced to the uninduced state of the lac operon. We use a detailed computational model to show that repressor-operator binding and unbinding, fluctuations in the total number of repressors, and inducer-repressor binding and unbinding all play a role in this transition. Based on the timescales on which these processes operate, we construct a minimal model of the transition to the uninduced state and compare the results with simulations and experimental observations. The induced state turns out to be very stable, with a transition rate to the uninduced state lower than 2×10 ^-9 per minute. In contrast to the transition to the induced state, the transition to the uninduced state is well described in terms of a 2D diffusive system crossing a barrier, with the diffusion rates emerging from a model of repressor unbinding.