Due to the extensive applications in automotive and aerospace industries of Al-7Si-Mg casting alloys, its understanding of the dendrite microstructural formation is of great importance to control the desirable microstructure and thereby to modify the performance of castings. In this work, through analyzing the measured cooling curves in different cooling conditions of Al-7Si-0.36Mg ternary alloy during sand casting, a theoretical nucleation model correlated maximum nucleation undercooling with the nucleation density is proposed. Besides, a 2D and 3D cellular automaton (CA) model allowing for the quantitatively predicting dendrite growth of ternary alloys is presented. This model introduces a new tracking neighboring rule algorithm to eliminate the effect of mesh dependency on dendrite growth. The thermodynamic and kinetic data needed in the simulations is obtained by coupling with Pandat software package in combination with thermodynamic/kinetic/equilibrium phase diagram calculation databases. This model has also taken account the multi-component diffusion, constitutional undercooling, curvature undercooling, dendrite preferential growth angles as well as the effect of interactions between the alloying elements etc. This model is applied to quantitatively simulate the dendrite growth with various crystallographic orientations of Al-7Si-0.36Mg ternary alloy in 2D and 3D during polycrystalline solidification, and the predicted secondary dendrite arm spacing (SDAS) shows a reasonable agreement with the experimental results. The experimental observed complicated and diverse dendrite microstructure occurring in solidification process can be well reproduced by this 3D-CA model which has considered the effects of various preferred growth orientations, the interactions of adjacent dendrites as well as the influence of S/L interface anisotropies. The simulated results effectively demonstrated the abilities of this model in prediction of dendrite microstructure in ternary alloys.