When navigating the environment, animals need to prioritize responses to the most relevant stimuli. Although a theoretical framework for selective visual attention exists, its circuit implementation has remained obscure. Here we investigated how larval zebrafish select between simultaneously presented visual stimuli. We found that a mix of winner-take-all (WTA) and averaging strategies best simulates behavioral responses. We identified two circuits whose activity patterns predict the relative saliencies of competing visual objects. Stimuli presented to only one eye are selected by WTA computation in the inner retina. Binocularly presented stimuli, on the other hand, are processed by reciprocal, bilateral connections between the nucleus isthmi (NI) and the tectum. This interhemispheric computation leads to WTA or averaging responses. Optogenetic stimulation and laser ablation of NI neurons disrupt stimulus selection and behavioral action selection. Thus, depending on the relative locations of competing stimuli, a combination of retinotectal and isthmotectal circuits enables selective visual attention.