Objectives: The clinical and social burden of the COVID-19 pandemic were high among asylum seekers (ASs). We aimed to understand better ASs' experiences of the pandemic and their sources of worries. Methods: Participants (n = 203) completed a survey about their worries, sleep disorders, and fear of dying. We also conducted semi-structured interviews with ASs living in a community center (n = 15), focusing on how social and living conditions affected their experiences and worries. Results: ASs in community centers experienced more sleep disorders related to the COVID-19 pandemic than those living in private apartments (aOR 2.01, p = 0.045). Similarly, those with lower education had greater fear for their life due to the COVID-19 pandemic (aOR 2.31, p = 0.015). Qualitative findings showed that sharing living spaces was an important source of worries for ASs and that protective measures were perceived to increase social isolation. Conclusion: Our study highlighted the impact of the COVID-19 pandemic for ASs and the importance of tailoring public health measures to their needs and living conditions.