Bayesian methods are extensively used to analyse geophysical data sets. A critical and somewhat overlooked component of high-dimensional Bayesian inversion is the definition of the prior probability density function that describes the joint probability of model parameters before considering available data sets. If insufficient prior information is available about model parameter correlations, then it is tempting to assume that model parameters are uncorrelated. When working with a spatially gridded model representation, this overparametrization leads to posterior realizations with far too much variability to be deemed realistic from a geological perspective. In this study, we introduce a new approach for structure-based prior sampling with Markov chain Monte Carlo that is suitable when only limited prior information is available. We evaluate our method using model structure measures related to standard roughness and damping metrics for l1- and l2-norms. We show that our structure-based prior approach is able to adequately sample the chosen prior distribution of model structure. The usefulness and applicability of the methodology is demonstrated on synthetic and field-based crosshole ground penetrating radar data. We find that our method provides posterior model realizations and statistics that are significantly more satisfactory than those based on underlying assumptions of uncorrelated model parameters or on explicit penalties on model structure within an empirical Bayes framework.