Objective.Determining the detectability of targets for the different imaging modalities in mammography in the presence of anatomical background noise is challenging. This work proposes a method to compare the image quality and detectability of targets in digital mammography (DM), digital breast tomosynthesis (DBT) and synthetic mammography.Approach. The low-frequency structured noise produced by a water phantom with acrylic spheres was used to simulate anatomical background noise for the different types of images. A method was developed to apply the non-prewhitening observer model with eye filter (NPWE) in these conditions. A homogeneous poly(methyl) methacrylate phantom with a 0.2 mm thick aluminium disc was used to calculate 2D in-plane modulation transfer function (MTF), noise power spectrum (NPS), noise equivalent quanta, and system detective quantum efficiency for 30, 50 and 70 mm thicknesses. The in-depth MTFs of DBT volumes were determined using a thin tungsten wire. The MTF, system NPS and anatomical NPS were used in the NPWE model to calculate the threshold gold thickness of the gold discs contained in the CDMAM phantom, which was taken as reference. Main results.The correspondence between the NPWE model and the CDMAM phantom (linear Pearson correlation 0.980) yielded a threshold detectability index that was used to determine the threshold diameter of spherical microcalcifications and masses. DBT imaging improved the detection of masses, which depended mostly on the reduction of anatomical background noise. Conversely, DM images yielded the best detection of microcalcifications.Significance.The method presented in this study was able to quantify image quality and object detectability for the different imaging modalities and levels of anatomical background noise.