There is evidence that gray matter networks are disrupted in Mild Cognitive Impairment (MCI) and associated with cognitive impairment and faster disease progression. However, it remains unknown how these alterations are related to the presence of Apolipoprotein E isoform E4 (ApoE4), the most prominent genetic risk factor for late-onset Alzheimer's disease (AD). To investigate this topic at the individual level, we explore the impact of ApoE4 and the disease progression on the Single-Subject Gray Matter Networks (SSGMNets) using the graph theory approach. Our data sample comprised 200 MCI patients selected from the ADNI database, classified as non-Converters and Converters (will progress into AD). Each group included 50 ApoE4-positive ('Carriers', ApoE4 + ) and 50 ApoE4-negative ('non-Carriers', ApoE4-). The SSGMNets were estimated from structural MRIs at two-time points: baseline and conversion. We investigated whether altered network topological measures at baseline and their rate of change (RoC) between baseline and conversion time points were associated with ApoE4 and disease progression. We also explored the correlation of SSGMNets attributes with general cognition score (MMSE), memory (ADNI-MEM), and CSF-derived biomarkers of AD (Aβ42, T-tau, and P-tau). Our results showed that ApoE4 and the disease progression modulated the global topological network properties independently but not in their RoC. MCI converters showed a lower clustering index in several regions associated with neurodegeneration in AD. The SSGMNets' topological organization was revealed to be able to predict cognitive and memory measures. The findings presented here suggest that SSGMNets could indeed be used to identify MCI ApoE4 Carriers with a high risk for AD progression.