The present study is devoted to the problem of automatic sorting of extracellularly recorded action potentials of neurons. The classification of spike waveform is considered as a pattern recognition problem of segments of signal that corresponds to the appearance of spikes. Nonlinear oscillating model with perturbation is used to describe the waveforms of spikes. It allows characterizing the signal distortions in both amplitude and phase. The spikes generated by one neuron assumed to be described by the same equation and should be recognized as one class. The problem of spike recognition is reduced to the separation of mixture of normal distributions in the transformed feature space. An unsupervised iteration-learning algorithm that estimates the number of classes and their centers is developed. It scans the learning set in order to evaluate spikes trajectories in phase space with maximal probability density in their neighborhood. To estimate the trajectories the integral operators with piece-wise polynomial kernels were used that provides computational efficiency. The new algorithm was tested on simulated and real data sets.