Surface nanocrystallization (SNC) can effectively enhance the surface and global properties of the metallic materials, such as microhardness, intensity, fatigue, wear and corrosion resistances, therefore provides more promising practical industrial applicability. Up to now, several SNC treatment methods were developed based either on the principles of ball impactions or friction sliding, however, difficulty still exists for the surface treatment of large-dimensional samples with high efficiency. Recently, more attentions were focused on the asymmetric rolling, of which upper and lower rolls rotate with different circumferential speeds, and then an extra shear strain was applied to metal sheet in addition to compression strain. The shear strain could refine the grains into micro- or submicro-scales. In order to investigate the possibility to realize the SNC for metal sheet in the rolling process and examine the effects of rolling parameters, silicon steel sheet was rolled by means of asymmetric rolling and conventional rolling respectively, the microstructural evolution in the top surface layer was observed for the samples rolled for different parameters including mismatch speed ratio, rolling reduction and rolling pass. Experimental results show that after the asymmetric rolling, nanocrystallines about 10~50 nm in size with nearly random orientations form in the top-surface layer of sheet. Meanwhile, dislocation cells can be observed after conventional rolling, which indicates that the asymmetric rolling can be utilized for the surface nanocrystallization of the cubic metal sheets. The surface nanocrystallization mechanism induced by asymmetric rolling was summarized as follows: (1) upon the application of repeated shear force, submicro-grains/dislocation cells form through formations, slips, annihilations and recombinations of high density of dislocations; (2) with a further increment of rolling reduction and rolling pass, high density of dislocations in the refined cells/grains developing in above route lead to reduction of grain size and increment of misorientations between the refined grains; (3) nanocrystallines with nearly random orientations form. Larger reduction and multi-passes are necessary for the surface nanocrystallization induced by asymmetric rolling, and the increment of mismatch speed ratio can accelerate the grain refinement process.