The role of guanine nucleotide-binding proteins (G proteins) in the cAMP-dependent action of serotonin (5-HT) and the antagonistic action of the neuropeptide Phe-Met-Arg-Phe-NH2 (FMRF-amide), mediated by the lipoxygenase metabolites of arachidonic acid, was investigated in Aplysia sensory neurons. Intracellular injection of guanosine 5'-[gamma-thio]triphosphate (GTP[gamma-S]) mimics the hyperpolarizing action of FMRF-amide due to activation of the S K+ current and alters the transient response to FMRF-amide into an irreversible (or only partially reversible) response. At higher concentrations, GTP[gamma-S] occludes the response to FMRF-amide. Injection of activated pertussis toxin inhibits the response to FMRF-amide but not to 5-HT. Injection of guanosine 5'-[beta-thio]diphosphate inhibits the response to FMRF-amide by approximately equal to 50% and completely blocks the response to 5-HT. Three lines of evidence suggest that the FMRF-amide-activated G protein is involved at an early stage of the arachidonic acid cascade, prior to the release of arachidonate. (i) Pertussis toxin injection blocks the hyperpolarizing response to FMRF-amide but not to exogenously applied arachidonic acid. (ii) Two blockers of the arachidonic acid cascade inhibit the hyperpolarizing responses to both FMRF-amide and GTP[gamma-S] (and unmask a 5-HT-like depolarizing response to the nucleotide). (iii) Concentrations of GTP[gamma-S] that alter the kinetics of the FMRF-amide response have no effect on the hyperpolarizing response to arachidonic acid. We conclude that a pertussis toxin-sensitive G protein most likely acts to couple the FMRF-amide receptor to phospholipase activation and arachidonic acid release, whereas a pertussis toxin-insensitive G protein couples the 5-HT receptor to adenylate cyclase.