Changes of functional connectivity in prodromal and early Alzheimer's disease can arise from compensatory and/or pathological processes. We hypothesized that i) there is impairment of effective inhibition associated with early Alzheimer's disease that may lead to ii) a paradoxical increase of functional connectivity. To this end we analyzed effective connectivity in 14 patients and 16 matched controls using dynamic causal modeling of functional MRI time series recorded during a visual inter-hemispheric integration task. By contrasting co-linear with non co-linear bilateral gratings, we estimated inhibitory top-down effects within the visual areas. The anatomical areas constituting the functional network of interest were identified with categorical functional MRI contrasts (Stimuli>Baseline and Co-linear gratings>Non co-linear gratings), which implicated V1 and V3v in both hemispheres. A model with reciprocal excitatory intrinsic connections linking these four regions and modulatory inhibitory effects exerted by V3v on V1 optimally explained the functional MRI time series in both subject groups. However, Alzheimer's disease was associated with significantly weakened intrinsic and modulatory connections. Top-down inhibitory effects, previously detected as relative deactivations of V1 in young adults, were observed neither in our aged controls nor in patients. We conclude that effective inhibition weakens with age and more so in early Alzheimer's disease.