Magneto-active polymers are a class of smart materials commonly manufactured by mixing micron-sized iron particles in a rubber-like matrix. When cured in the presence of an externally applied magnetic field, the iron particles arrange themselves into chain-like structures that lend an overall anisotropy to the material. It has been observed through electron micrographs and X-ray tomographs that these chains are not always perfect in structure, and may have dispersion due to the conditions present during manufacturing or some undesirable material properties. We model the response of these materials to coupled magneto-mechanical loading in this paper using a probability based structure tensor that accounts for this imperfect anisotropy. The response of the matrix material is decoupled from the chain phase, though still being connected through kinematic constraints. The latter is based on the definition of a 'chain deformation gradient' and a 'chain magnetic field'. We conclude with numerical examples that demonstrate the effect of chain dispersion on the response of the material to magnetoelastic loading.