Stress corrosion cracking (SCC) is a major problem in the welded components of austenitic stainless steel in nuclear power plants. High tensile residual stress is an important factor resulting in the SCC of materials. Austenitic stainless steel has a strong tendency for work hardening owing to its fcc crystal structure and low stacking-fault energy. High plastic strain can accumulate during a multipass welding process. On the other hand, accumulated strain hardening can be reduced or even eliminated during the welding thermal cycles owing to dynamic recovery, recrystallization, and grain growth below the melting point, which is called the annealing effect. Influence of strain hardening and annealing effect needs to be investigated to predict the welding-induced residual stresses accurately in austenitic stainless steel joints. In this study, a new time-temperature-dependent annealing model was proposed based on the Johnson-Mehl-Avrami equation. Numerical Satoh tests were performed to clarify the influence of strain-hardening models (i.e., the isotropic strain-hardening model and Chaboche mixed isotropic-kinematic strain-hardening model) and annealing models (i.e., the single-stage annealing model and new time-temperature-dependent annealing model) on the formation of residual stresses and the accumulated plastic strain during multiple thermal cycles. Thermoelastic-plastic finite element (FE) analyses were carried out to predict the welding residual stresses and accumulated plastic strain in a thick-wall 316 stainless steel butt-welded pipe joint with 85 welding passes. The residual stresses of the welded joint were measured by the sectioning method, inherent strain method, and deep-hole drilling method. The simulations of welding residual stresses were compared with the measurements. Annealing effect significantly influences the formation of accumulated plastic strain and welding residual stresses, neglecting which will result in a significant overestimation of FE results. The proposed annealing model showed an excellent match to the experimental data. With the consideration of the annealing effect, the isotropic strain-hardening model overestimated the welding residual stresses slightly, while the FE results of welding residual stresses using the Chaboche mixed strain-hardening model showed better agreement with the measurements. The single-stage annealing model revealed a recommended annealing temperature of 900-1000°C for austenitic stainless steel such as 316 stainless steel.