The purpose of this article is to propose a global discrete-time modeling of the term structure of interest rates which is able to capture simultaneously the following important features: (i) a historical dynamics of the factor driving term structure shapes involving several lagged values, and switching regimes; (ii) a specification of the stochastic discount factor (SDF) with time-varying and regime-dependent risk-premia; (iii) explicit or quasi explicit formulas for zero-coupon bond (ZCB) and interest rate derivative prices. We develop the switching autoregressive normal (SARN) and the switching vector autoregressive normal (SVARN) Factor-Based Term Structure Models of order p. The factor is considered as a latent variable or an observable variable: in the second case the factor is a vector of several yields. Regime shifts are described by a Markov chain with (historical) nonhomogeneous transition probabilities. An empirical analysis of bivariate VAR(p) and SVARN(p) Factor-Based Term Structure Models, using monthly observations of the U.S. term structure of interest rates, and a goodness-of-fit and expectation hypothesis puzzle comparison with competing models in the literature, shows the determinant role played by the observable nature of the factor, lags, and switching regimes in the term structure modeling.