Natural selection may favor two very different types of social behaviors that have costs in vital rates (fecundity and/or survival) to the actor: helping behaviors, which increase the vital rates of recipients, and harming behaviors, which reduce the vital rates of recipients. Although social evolutionary theory has mainly dealt with helping behaviors, competition for limited resources creates ecological conditions in which an actor may benefit from expressing behaviors that reduce the vital rates of neighbors. This may occur if the reduction in vital rates decreases the intensity of competition experienced by the actor or that experienced by its offspring. Here, we explore the joint evolution of neutral recognition markers and marker-based costly conditional harming whereby actors express harming, conditional on actor and recipient bearing different conspicuous markers. We do so for two complementary demographic scenarios: finite panmictic and infinite structured populations. We find that marker-based conditional harming can evolve under a large range of recombination rates and group sizes under both finite panmictic and infinite structured populations. A direct comparison with results for the evolution of marker-based conditional helping reveals that, if everything else is equal, marker-based conditional harming is often more likely to evolve than marker-based conditional helping.