Austenite stainless steels such as SUS304, owing to their good combination of mechanical properties, corrosion resistance and weldability, are widely used in a variety of industries. In the simulation of welding residual stress of an austenite stainless steel joint, because of the high strain hardening rate and the heating-cooling thermal cycles, both the work hardening phenomenon and the annealing effect have to be taken into account in the material constitutive relations. Though a number of numerical models have included the work hardening by using isotropic rule, kinematic rule or mixed rule, limited models have dealt with the annealing effect. For the steels or alloys with high strain hardening coefficient, neglecting the annealing effect will overestimate the welding residual stresses to a large extent. In this study, the thermal elastic plastic finite element method (T-E-P FEM) was used to simulate welding temperature and residual stresses in a SUS304 steel bead-on joint. In the computational approach based on the T-E-P FEM, a moving heat source with uniform density distribution was used to model the heat input, and a simple model was proposed to consider the annealing effect. Using the developed computational approach, the influences of work hardening and annealing effect on the welding residual stress were clarified. In addition, the effect of annealing temperature on the distribution and magnitude of welding residual stress in the weld zone and its vicinity was examined. The simulated results show that annealing effect has a significant influence on the longitudinal residual stress, and the peak value of longitudinal tensile stress increases with annealing temperature. The longitudinal tensile stresses in the fusion zone and its vicinity also increase with annealing temperature. It seems that the annealing temperature has insignificant influence on the transverse residual stresses. Comparing the simulated results and the measured data, it was found that when the annealing temperature was assumed to be 1000 ℃ for SUS304 steel, the longitudinal residual stresses predicted by the T-E-P FEM generally match the measurements. The present work is helpful for developing more advanced materials model to calculate welding residual stress with high accuracy.