Recrystallization texture is determined by the competition among various texture components during nucleation and grain growth. The stored energy and orientation gradient depend on the grain orientation in the deformed microstructure. Texture components, nucleating at positions with high stored energy and a sharp orientation gradient have kinetic advantages, can consume the nucleation sites and potential growth space of recrystallized grains in adjacent deformed grains. Segregation elements can hinder nucleation and growth of recrystallization grains by reducing grain boundary mobility, and thus prevent texture components with kinetic advantages from invading adjacent deformed grains. It is valuable to provide a basis for precise recrystallization texture design and control by investigating the competitive relations among recrystallization texture components under the intervention of segregation elements. The recrystallization texture competition in a body-centered cubic Fe-3%Si alloy containing Sb was studied through experiment and simulation. It was found that the segregation element can weaken the γ (<111>//ND, ND—normal direction) and strengthen the α (<110>//RD, RD—rolling direction), as well as other recrystallization texture components with low stored energy, by inhibiting the invasion of γ-recrystallized grains into adjacent deformed grains. The two dominant factors for segregation effects are deformation texture and critical invasion radius. A quantitative model, based on nucleation and growth kinetics, was proposed to explore the effect of critical invasion radius and deformation texture on recrystallization texture competition mediated by segregation elements. It was found that segregation elements can prolong the invasion incubation period and reduce the invasion rate to inhibit the consumption of α-deformed grains by γ-recrystallized grains. The inhibition effect initially strengthened and then weakened with the increasing γ deformation texture.