Memory self-ratings are often used in primary care practice and social surveys due to its relative ease of administration. Yet their usefulness to accurately measure individuals’ memory-related cognitive function is largely unknown. This article assesses the construct validity of self-rated memory for measuring memory-related cognitive function among middle-aged and older adults (45+ years) in China using a national sample (N = 13,690) from the China Health and Retirement Longitudinal Study. We first compare self-rated memory with a number of well-established memory tests and then investigate the role of sociodemographic factors in determining self-reports conditional on memory test performance. We find that self-rated memory is only weakly correlated with memory test performance and yields low sensitivity and specificity in detecting individuals with relatively poor memory test performance. We also find evidence for substantial differential item functioning across sociodemographic groups. Our results, therefore, caution against the use of simple self-rated memory measures for memory assessments in primary care or survey research.