Both rapid eye movement sleep and cataplexy in the narcoleptic canine have been shown to increase after both systemic and local administration of cholinergic agonists in the pontine reticular formation. Furthermore, binding studies indicate an increase in the number of M2 muscarinic receptors in the pontine reticular formation of narcoleptic canines. In the present study we have investigated the receptor subtypes involved in mediating the cholinergic stimulation of cataplexy, as defined by brief periods of hypotonia induced by emotions, within the pontine reticular formation of narcoleptic canines. Specific cholinergic and monoaminergic agonists and antagonists, and excitatory or inhibitory amino-acid neurotransmitter receptor agonists, were perfused through microdialysis probes implanted bilaterally in the pontine reticular formation of narcoleptic canines, and cataplexy was monitored using the Food-Elicited Cataplexy Test and recordings of electroencephalogram, electrooculogram and electromyogram. In narcoleptic canines, bilateral perfusion with oxotremorine (M2 muscarinic) (10(-5)-10(-3) M) in the pontine reticular formation produced a dose-dependent increase in cataplexy, which reached complete muscle atonia (status cataplecticus) during the highest concentration. In control canines bilateral perfusion with oxotremorine (10(-5)-10(-3) M) did not produce any cataplectic attacks, but did produce muscle atonia after the highest concentration. Bilateral perfusion with either McN-A-343 (M1 muscarinic) or nicotine (both 10(-5)-10(-3) M) did not have any effect on cataplexy in either narcoleptic or control canines. The increase in cataplexy in narcoleptic canines produced by local perfusion with carbachol (10(-4) M) followed by equimolar perfusion with a muscarinic antagonist was rapidly reversed by atropine (muscarinic) and gallamine (M2 muscarinic), partially reversed by 4-DAMP (M3/M1 muscarinic) and completely unaffected by pirenzepine (M1 muscarinic). Bilateral perfusion with excitatory, glutamatergic receptor agonists N-methyl-D-aspartate, AMPA (both at 10(-4)-10(-3) M) and kainic acid (10(-5)-10(-4) M) did not have any effect on cataplexy, whereas bilateral perfusion with the inhibitory GABAergic receptor agonist muscimol (10(-4)-10(-3) M) produced a moderate increase in cataplexy in the narcoleptic canines. Bilateral perfusion with numerous monoaminergic compounds, BHT-920 (alpha-2 agonist), yohimbine (alpha-2 antagonist), propranolol (beta antagonist) and prazosin (alpha-1 antagonist), did not have any effect on cataplexy. These findings demonstrate that cholinergic regulation of cataplexy in the narcoleptic canine at the level of the pontine reticular formation is mediated by M2, and possibly M3, muscarinic receptors. The effects of muscimol indicate that the stimulation of cataplexy might be elicited by local neuronal inhibition.