Options
A corpus-based frame semantic analysis of commercialized listening tests : implications for content validity
Commercialized listening tests can significantly impact test-takers’ lives, as they are often required for purposes such as immigration, employment opportunities, and university admissions. However, there is a noticeable research gap regarding the content validity of these tests. To address the gap, this study aims to examine the semantic features of the simulated mini-lectures in the listening sections of the Test of English as a Foreign Language (TOEFL) and the International English Language Testing System (IELTS) to explore the content validity of the two tests.
This study utilized two study corpora, the IELTS corpus with 68 mini-lectures (46,823 words) and the TOEFL corpus with 285 mini-lectures (207,296 words). The reference corpus comprised 59 lectures from the Michigan Corpus of Academic Spoken English (MICASE), totaling 571,354 words. The theoretical framework employed in the study is frame semantics that asserts words should be understood within cognitive frames. The data was submitted to Wmatrix5 for automatized semantic tagging, which generated 488 semantic frames. Three comparisons were conducted: IELTS vs. TOEFL, IELTS vs. MICASE lectures, and TOEFL vs. MICASE lectures.
The results suggest that the mini-lectures of IELTS listening tests cover fewer academic discourse fields than TOEFL mini-lectures. Therefore, it is suggested that IELTS test developers prioritize materials resembling genuine academic lectures over non-specialist texts. TOEFL test developers should extend the coverage of the test content and continue to mirror the academic discourse.
Furthermore, IELTS and TOEFL mini-lectures reflected the similarity of 78% and 64% of the examined semantic frames respectively, underlining their relative authenticity. Similarly, a pervasive ‘objectivity’ was evident across all three corpora, with emotion-related categories being sparse. Nevertheless, specific topics, such as politics, war, and intimate and sexual relationships, were notably absent from the test corpora, even though they appeared in the academic lecture corpus.
Finally, as the simulated mini-lectures in IELTS and TOEFL are significantly shorter than authentic lectures, the positive results supporting the authenticity of the simulated lectures are attenuated. It is necessary to confirm whether these mini-lectures in the listening tests can engage test takers in the same cognitive processes as authentic academic lectures.