Aryadoust, Vahid

This study employed eye-tracking and mouse click frequency analysis to investigate the predictive power of gaze behaviors, mouse-clicking, and their interactive effects with linguistic backgrounds on the IELTS (International English Language Testing System) listening test scores. A total of 77 test takers (45 with English as their first language (E-L1) and 32 with English as their second language (E-L2)) participated in this study. Their eye movements and mouse click frequencies were recorded as they took a computer-based IELTS listening test. The subsequent data analysis, utilizing linear mixed models, showed that gaze patterns, mouse actions, and language background significantly predicted listening test outcomes across four listening test sections and between E-L1 and E-L2 candidates, accounting for 33.2% of the variance observed in test scores. These results indicate the effect of potential sources of construct-irrelevant variance on test scores, which are not predicted in the available construct definitions of the test used in the study. Implications for the listening construct and test validity are discussed.

The aim of this study was to investigate how test methods affect listening test takers' performance and cognitive load. Test methods were defined and operationalized as while-listening performance (WLP) and post-listening performance (PLP) formats. To achieve the goal of the study, we examined test takers' (N = 80) brain activity patterns (measured by functional near-infrared spectroscopy (fNIRS)), gaze behaviors (measured by eye-tracking), and listening performance (measured by test scores) across the two test methods. We found that the test takers displayed lower activity levels across brain regions supporting comprehension during the WLP tests relative to the PLP tests. Additionally, the gaze behavioral patterns exhibited during the WLP tests suggested that the test takers adopted keyword matching and "shallow listening." Together, the neuroimaging and gaze behavioral data indicated that the WLP tests imposed a lower cognitive load on the test takers than the PLP tests. However, the test takers performed better with higher test scores for one of two WLP tests compared with the PLP tests. By incorporating eye-tracking and neuroimaging in this exploration, this study has advanced the current knowledge on cognitive load and the impact imposed by different listening test methods. To advance our knowledge of test validity, other researchers could adopt our research protocol and focus on extending the test method framework used in this study.

Differential item functioning (DIF) analysis is a way of determining whether test items function differently across subgroups of test takers after controlling for ability level. DIF results are used to evaluate tests' validity arguments. This study uses Rasch measurement to examine the Michigan English Language Assessment Battery listening test for DIF across gender subgroups. After establishing the unidimensionality and local independence of the data, the authors used two methods to test for DIF: (a) a t-test uniform DIF analysis, which showed that two test items displayed substantive DIF, and favored different gender subgroups; and (b) nonuniform DIF analysis, which revealed several test items with significant DIF, many of which favored low-ability male test takers. A possible explanation for gender-ability DIF is that lower ability male test takers are more likely to attempt lucky guesses, particularly on multiple-choice items with unattractive distracters, and that having only two distracters makes this strategy likely to succeed.

This chapter introduces evolutionary algorithm-based (EA-based) symbolic regression, which is an optimization model inspired by nature. EA-based symbolic regression is used to predict reading comprehension proficiency by using English learners' vocabulary and grammatical knowledge. EA-based symbolic regression draws on the fundamental concepts of Darwinian evolution, such as breeding and variety, and applies modeling to assess the accuracy and relevance of the prediction models. In this technique, multiple models are generated among which the one with the optimal fit is chosen as the “parent” and the basis for “breeding” further models, called offspring, for the following generations. The present study finds a significant nonlinear relationship between lexicogrammatical knowledge and reading comprehension proficiency (R2 = .520). Details and computational requirements are discussed and implications for language assessment are explored.

Studies have shown that test-takers tend to use keyword-matching strategies when taking listening tests. Keyword-matching involves matching content words in the written modality (test items) against those heard in the audio text. However, no research has investigated the effect of such keywords in listening tests, or the impact of gazing upon these keywords on listening test scores. Thus, this study examined whether test-takers’ performance on a listening test can be explained by their gaze behaviors across three types of content words in the written modality: nouns, verbs, and adjectives. Using eye-tracking technology, this study measured the gaze behavior of 66 listening test-takers during reading content words in test item stems. Using linear mixed effect model, binary probit regression, and multinomial logistic regression, we found that test-takers’ performance was predicted by gaze behavioral measures on content words. Among the content words, fixating on nouns in written test items had the most significant role in predicting test performance, followed by adjectives and verbs. By shedding light on how keywords in test items are attended to by test-takers and the relationship between keyword-matching and listening test performance, this study has provided significant evidence for the overwhelming role of reading in listening tests. Implications for test score interpretation are discussed.

This is the first study to investigate the effects of test methods (while-listening performance and post-listening performance) and gender on measured listening ability and brain activation under test conditions. Functional near-infrared spectroscopy (fNIRS) was used to examine three brain regions associated with listening comprehension: the inferior frontal gyrus and posterior middle temporal gyrus, which subserve bottom-up processing in comprehension, and the dorsomedial prefrontal cortex, which mediates top-down processing. A Rasch model reliability analysis showed that listeners were homogeneous in their listening ability. Additionally, there were no significant differences in test scores across test methods and genders. The fNIRS data, however, revealed significantly different activation of the investigated brain regions across test methods, genders, and listening abilities. Together, these findings indicated that the listening test was not sensitive to differences in the neurocognitive processes underlying listening comprehension under test conditions. The implications of these findings for assessing listening and suggestions for future research are discussed.

Structural equation modeling (SEM) has been shown to be an effective and powerful tool in language testing and assessment as it offers advantages over fundamental statistical techniques such as regression and correlation analysis. Specifically, SEM is capable of confirming multiple hypothesized direct and indirect relationships among different variables in one analysis. For example, it can be used to identify the relationship of task characteristics with test takers' performance or to explore the relationship of a test and test takers. It can also be used to confirm the factor structure of tests. This chapter provides an overview of the key concepts in SEM analysis, identifies five stages in SEM analysis, and provides an overview of language assessment studies that have utilized this method. An application of this method is shown through a sample study that aims to determine whether test scores on an English diagnostic test of receptive skills can predict performance in an English proficiency test of receptive and productive skills and whether academic background impacts these test scores. The chapter concludes with a discussion on the benefits and limitations of SEM in language testing and assessment research.

To date many test designers have relied heavily on either the interpretation or uses of scores in high-stakes tests as evidence of validity. As a case in point, in the listening section of International English Language Testing System (IELTS), the consequences and the correlation of the scores with other measures such as academic performance of students have been extensively researched in pursuit of consequential and criterion validity. While these research inquiries are valuable to especially research test usefulness, the test should be validated for its main objectives. Consequential and criterion validity studies into the IELTS listening module have proposed different and controversial evidence. We argue if the construct validity is not established, supportive evidence of usefulness is either very difficult or impossible to find. The main purpose of this study is to investigate the construct validity of the IELTS listening test. We will employ a two-parameter logistic Item Response Theory (IRT) model to investigate the construct representation and irrelevant factor (Messick, 1988, 1989).

Although second language (L2) listening assessment has been the subject of much research interest in the past few decades, there remain a multitude of challenges facing the definition and operationalization of the L2 listening construct(s). Notably, the majority of L2 listening assessment studies are based upon the (implicit) assumption that listening is reducible to cognition and metacognition. This approach ignores emotional, neurophysiological, and sociocultural mechanisms underlying L2 listening. In this paper, the role of these mechanisms in L2 listening assessment is discussed and four gaps in understanding are explored: the nature of L2 listening, the interaction between listeners and the stimuli, the role of visuals, and authenticity in L2 listening assessments. Finally, a review of the papers published in the special issue is presented and recommendations for further research on L2 listening assessments are provided.