Reliability generalization meta-analysis of L2 listening tests

URI
Thumbnail Image
Type
Thesis
Files
Author
Shang, Yuxin
Supervisor
Aryadoust, Vahid
Abstract
Second language (L2) listening has been widely examined for decades, and L2 listening assessments have received much attention. A reliable L2 listening test is recognized as a valuable tool for researchers and teachers to better assess learners’ listening ability and draw valid inferences. However, there is a lack of a synthesized view of reliability of L2 listening tests. To investigate the reliability of L2 listening tests and explore potential factors affecting the reliability, a reliability generalization (RG) meta-analysis was conducted in the present study. 167 out of 260 journal articles were not included in the analysis due to the absence of reported reliability coefficients. Thus, a total number of 122 reliability coefficients of L2 listening tests from 92 published articles were collected and submitted to a linear mixed effects RG analysis. The papers were coded based on a coding scheme consisting of 16 variables classified into three categories: study descriptions, test features, and statistical results. The results showed an average reliability of 0.8181 (CI= 0.8034 - 0.8329), with 40% of reliability estimates falling below the lower bound of CI. The presence of publication bias and heterogeneity was found in the reliability of L2 listening tests, indicating that low reliability coefficients were likely omitted from some published studies. In addition, two factors predicting the reliability of L2 listening tests were the number of items and test type (standardized vs and researcher-/teacher-designed tests). The study also found that reliability is not a moderator of the relationship between L2 listening scores and theoretically relevant constructs. Reliability induction, referencing reliability coefficients from previous studies, was identified in reporting the reliability of L2 listening tests. Implications for researchers and teachers were discussed.
Date Issued
2024
Call Number
P118.2 Sha