Surface-based common-offset ground-penetrating radar (GPR) reflection profiling is a popular geophysical exploration technique for obtaining high-resolution images of the shallow subsurface in a cost-effective manner. One drawback of this technique is that, without complementary borehole information in form of dielectric permittivity and/or porosity logs along the profile, it is currently not possible to obtain reliable estimates of the high-frequency electromagnetic velocity distribution of the probed subsurface region. This is problematic because adequate knowledge of the velocity is needed for accurate imaging and depth conversion of the data, as well as for quantifying the distribution of soil water content. To overcome this issue, we have developed a novel methodology for estimating the detailed subsurface velocity structure from common-offset GPR reflection measurements, which does not require additional conditioning information. The proposed approach combines two key components: Diffraction analysis is used to infer the smooth, large-scale component of the velocity distribution, whereas the superimposed small-scale fluctuations are inferred via inversion of the reflected wavefield. We test and validate our method on two synthetic datasets having increasing degrees of complexity and realism before applying it to a field example from the Boise Hydrogeophysical Research Site (BHRS), where independent control data in the form of neutron-neutron porosity logs are available for validation. The results obtained demonstrate the viability and robustness of the proposed approach. Further, due to its efficiency, both in terms of field effort and computational cost, the method can be readily extended to 3D, which further enhances its attractiveness compared to multi-offset-based GPR velocity estimation techniques.