The rapid development of ancient DNA analysis in the last decades has induced a paradigm shift in ecology and evolution. Driven by a combination of breakthroughs in DNA isolation techniques, high-throughput sequencing, and bioinformatics, ancient genome-scale data for a rapidly growing variety of taxa are now available, allowing researchers to directly observe demographic and evolutionary processes over time. However, the vast majority of paleogenomic studies still focus on human or animal remains. In this article, we make the case for a vast untapped resource of ancient plant material that is ideally suited for paleogenomic analyses: plant remains, such as needles, leaves, wood, seeds, or fruits, that are deposited in natural archives, such as lake sediments, permafrost, or even ice caves. Such plant remains are commonly found in large numbers and in stratigraphic sequence through time and have so far been used primarily to reconstruct past local species presences and abundances. However, they are also unique repositories of genetic information with the potential to revolutionize the fields of ecology and evolution by directly studying microevolutionary processes over time. Here, we give an overview of the current state-of-the-art, address important challenges, and highlight new research avenues to inspire future research.