Mitochondria are central regulators of cellular metabolism, most known for their role in energy production. They can be "enhanced" by physical activity (including exercise), which increases their integrity, efficiency and dynamic adaptation to stressors, in short "mitochondrial fitness". Mitochondrial fitness is closely associated with cardiorespiratory fitness and physical activity. Given the importance of mitochondria in immune functions, it is thus not surprising that cardiorespiratory fitness is also an integral determinant of the antiviral host defense and vulnerability to infection. Here, we first briefly review the role of physical activity in viral infections. We then summarize mitochondrial functions that are relevant for the antiviral immune response with a particular focus on the current Coronavirus Disease (COVID-19) pandemic and on innate immune function. Finally, the modulation of mitochondrial and cardiorespiratory fitness by physical activity, aging and the chronic diseases that represent the most common comorbidities of COVID-19 is discussed. We conclude that a high mitochondrial - and related cardiorespiratory - fitness should be considered as protective factors for viral infections, including COVID-19. This assumption is corroborated by reduced mitochondrial fitness in many established risk factors of COVID-19, like age, various chronic diseases or obesity. We argue for regular analysis of the cardiorespiratory fitness of COVID-19 patients and the promotion of physical activity - with all its associated health benefits - as preventive measures against viral infection.