With the recent approval of highly effective coronavirus disease 2019 (COVID-19) vaccines, functional and lasting immunity to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is currently under investigation as antibody levels in plasma were shown to decline during convalescence. Since the absence of antibodies does not equate to absence of immune memory, we evaluate the presence of SARS-CoV-2-specific memory B cells in convalescent individuals. Here, we report a longitudinal assessment of humoral immune responses on 32 donors up to 8 months post-symptom onset. Our observations indicate that anti-Spike and anti-receptor binding domain (RBD) immunoglobulin M (IgM) in plasma decay rapidly, whereas the reduction of IgG is less prominent. Neutralizing activity also declines rapidly when compared to Fc-effector functions. Concomitantly, the frequencies of RBD-specific IgM+ B cells wane significantly when compared to RBD-specific IgG+ B cells, which remain stable. Our results add to the current understanding of immune memory following SARS-CoV-2 infection, which is critical for secondary infection prevention and vaccine efficacy.