Aluminum alloys are widely used in the automobile, rail transportation, and aerospace industries owing to their excellent properties such as low density, high thermal conductivity, and high specific strength. Metal additive manufacturing (MAM) enables the high-quality integrated forming of aluminum alloy components. Among the MAM techniques, droplet and arc additive manufacturing (DAAM) is a newly proposed method that offers advantages, such as high efficiency and low cost. In DAAM process, a droplet generation system is designed above the substrate fixed on a three-dimensional motion platform. Below the droplet generation system, an arc heat source with variable polarity is tilted. During the DAAM process, the metal droplets drop vertically and sequentially into the molten pool generated by the arc heat source to realize metallurgical bonding. Layer-by-layer deposition of aluminum alloy components is achieved by moving the substrate. This study focuses on the DAAM process for 2319 aluminum alloy. The temperature field distribution, microstructure evolution, and epitaxial growth characteristics were investigated. First, the temperature field distribution during the deposition process was calculated using the finite element method combined with element birth and death techniques. Based on the temperature field analysis, the solidification parameters at different positions of the molten pool were calculated. These parameters were then substituted into a phase field (PF) model to determine the growth and evolution of the microstructure at different positions in the molten pool. Columnar crystal structures were formed in the bottom and middle regions of the molten pool. From the bottom to the upper part of the molten pool, the temperature gradient decreased and the solidification speed increased. Therefore, columnar crystals to equiaxed transition occurred in the middle and upper regions. Additionally, misorientation angles were introduced in the PF model to investigate the epitaxial growth characteristics of the solidification process. Larger misorientation angles had a more obvious influence on dendrite morphology and were more likely to be eliminated during competitive growth. Finally, the metallographic analysis showed that from the bottom to the upper part of the deposition layer, the microstructure changed from columnar to equiaxed crystals, and the presence of columnar crystal epitaxial growth agreed well with the simulation results.