Examining the factor structure and its replicability across multiple listening test forms: Validity evidence for the Michigan English Test

Abstract

This study evaluated the validity of the Michigan English Test (MET) Listening Section by investigating its underlying factor structure and the replicability of its factor structure across multiple test forms. Data from 3255 test takers across four forms of the MET Listening Section were used. To investigate the factor structure, each form was fitted with four Bayesian confirmatory factor analysis (CFA) models: (1) a three correlated-factor model, (2) a bi-factor model, (3) a higher-order factor model, and (4) a single general-factor model. In addition, a four-pronged heuristic comprising construct delineation, construct operationalization, factor structure analysis, and congruence coefficient was developed to examine the replicability of factor structures across the test forms. Results from the CFA models showed that the test forms were unidimensional and the four pronged heuristic indicated that the test construct was consistently operationalized across forms. Furthermore, the congruence coefficient indicated that the factor structure representing listening was highly similar and replicable across test forms. In sum, the construct of the MET Listening Section did not comprise divisible subskills. Yet, the unidimensional factor structure of the test was replicable across the test forms.