Quenching is a key process in cold-rolled high strength steel manufacturing for the improvement of the material strength and plasticity. The quenching, however, may bring initial flatness defects of the steel strips, which causes problems for subsequent production process. It is thus necessary to study the flatness defects evolution during the quenching process. Using the secondary development of ABAQUS subroutine UMAT, this work establishes a temperature-microstructure-stress coupling finite element modeling (FEM) model to simulate the quenching process of the high strength steel with initial buckling defects. Thermal simulation experiments are further conducted to verify the present FEM model. Then, the elastic-plastic deformation behavior of the steel plates and its effects on flatness buckling during the quenching process is investigated using the FEM model. As a consequence, the buckling defect evolution mechanism in heat treatment process is obtained for the cold-rolled high strength steel. The flatness change or the forming of new flatness defect is mainly caused by the longitudinal extension arising from temperature gradient and the sequential phase transformation different in width and transverse directions. Change rates of the wave height, width, and length are used to describe the flatness change degree, quantifying the influence of the tension and initial transverse temperature difference on flatness change. The simulation shows that the tension has a positive correlation with the improvement of initial bucking defects. The initial edge waves become more severe after quenching along with the appearance of the new quarter waves, when the initial temperature of strip center is higher than that of the edge. On the contrary, the initial central waves become more serious when the initial temperature of strip center is lower. Meanwhile, joint impact of the tension and the initial transverse temperature difference on wave height is revealed for the application of industrial practice. Furthermore, quenching experiments of the high strength steel plates with initial single edge wave buckling defects are carried out using the experiment system in lab. Different sides of the plates quench into the water tank to reproduce the sequence of the phase change. The simulation and experiments produce consistent results qualitatively. This work makes connections between technological parameters and flatness change during quenching process, which can provide support to industrial heat treatment of high strength steel.