A reliability generalization meta-analysis of foreign language anxiety measurements

Reliability refers to the consistency of test scores across occasions, instruments or raters. High reliability can increase statistical power and reduce the risk of Type I and Type II errors. Reliability is not the property of tests but the function of test scores that can change with characteristics of the test, the conditions of administration as well as the group of examinees. However, research shows that many previous studies did not measure reliability but induced them instead, which would cast doubt on the accuracy of their statistical analysis conducted. Reliability Generalization (RG) meta-analysis is a useful method to investigate the extent to which the scores obtained from various
measurements are reliable and what factors may cause the variance in reliability. This study utilizes RG meta-analysis to investigate the reliability of instruments measuring foreign language anxiety, the most-often investigated affect variable in L2 research.

This study meta-analyzed 204 Cronbach’s alpha coefficients from 197 studies and 48 foreign language anxiety instruments to aggregate the overall reliability obtained from foreign language anxiety scores. It also investigated the possible variables contributing to the variability of anxiety scores. The pooling of effect sizes yielded an average reliability of 0.8717 and 95% confidence interval (0.8629-0.8806). A large amount of heterogeneity was detected. Therefore, a comprehensive moderator analysis of the study, instrument and population characteristics was carried out to explore possible sources of this variability. Features such as standard deviation of test scores, number of items, number of negative wording items, number of factors in factor analysis, administration methods, etc. were found to significantly affect the reliability of anxiety scores. A sensitivity analysis was also performed to test the robustness of the pooled reliability and diagnose the potential publication bias.

This study also extended the scope of conventional RG meta-analysis by investigating the impact of reliability of language anxiety scores on its predictive validity (anxiety-language proficiency correlations). Linear mixed-effect modeling was adopted to examine the effect of moderators on obtained anxiety-proficiency correlations. Based on the final model, the reliability of anxiety scores was found to significantly moderate the anxiety-proficiency correlations. Specifically, higher anxiety reliability coefficients were often associated with larger anxiety-language proficiency effect sizes, indicating that more reliable anxiety scores could strengthen its extrapolation to proficiency test scores. Additionally, as an exploratory study, the p-values around the anxiety-proficiency correlations were also coded and entered into correlation analysis. Higher reliability was found to significantly correlate with lower p-values, indicating that the stronger reliability could decrease the risk of Type I error. Implications of the findings were also discussed.