Options
Investigating test method effects in listening assessment : application of gaze behaviors, brain activation, and metacognitive strategies
Author
Zhai, Jiayu
Supervisor
Aryadoust, Vahid
Abstract
Listening is a complex process and a multidimensional construct by nature involving test takers’ behavioral, cognitive, and physiological processes. The listening assessment literature has identified several indicators of test performance which consist of test- and listener-specific factors. Test-specific factors refer to test method effects that impact test takers’ listening performance at the behavioral and neurocognitive levels. Listener-specific factors refer to the effect of listeners’ features and characteristics on their test performance as indicated by gaze behavior, brain activation, and metacognitive awareness. However, most of the previous studies only examined these factors separately. To my knowledge, there is no study in the listening assessment literature integrating the listener-related factors (i.e., gaze behavior, brain activation, and metacognitive awareness) and their interactions with the test-specific factor (i.e., listening test methods). This collective investigation of gaze behavior, brain activation, metacognitive awareness, and behavioral performance (i.e., test scores) in one study constitutes the first study of its kind in listening assessment and contributes to the exploration of the nature of listening construct to advance the understanding of listening construct under test conditions. To address the research gap, this study aims to explore whether test takers’ behavioral performance measured by test scores can be predicted by test takers’ neurophysiological process (measured by gaze behavior and brain activation) and self-appraisal of metacognitive awareness under different listening test methods (i.e., while-listening performance (WLP) and post-listening performance (PLP) tests).
Eighty university students completed two listening tests under WLP and PLP conditions, respectively. During the tests, their gaze behavior and brain activation data were simultaneously collected by an eye-tracker and functional near-infrared spectroscopy (fNIRS). Their metacognitive strategy use was measured using the Metacognitive Awareness Listening Questionnaire (MALQ) after the tests. Automatic linear modeling was used as the data analysis method. The results showed that WLP and PLP test performances were significantly predicted by different listener factors. To be specific, 25.9% of the variance in WLP test scores was accounted for by three factors: i) person knowledge and ii) mental translation in metacognitive awareness, and iii) average fixation duration in gaze behaviors. For PLP tests, 32.4% of the variance in PLP test scores were explained by six factors: i) the dmPFC measure in PLP-Question, ii) the IFG measure in PLP-Audio, and iii) the IFG measure in PLP-Question of brain activation, iv) mental translation and v) directed attention of metacognitive awareness, and vi) visit counts in PLP-Question of gaze behaviors.
This study proposed an extended approach to construct validity: neurocognitive validity, which conceptualizes listening test performance as a function of test takers’ behavioral, neurophysiological, and psychological mechanisms. Implications for listening assessment and listening pedagogy are also discussed.
Eighty university students completed two listening tests under WLP and PLP conditions, respectively. During the tests, their gaze behavior and brain activation data were simultaneously collected by an eye-tracker and functional near-infrared spectroscopy (fNIRS). Their metacognitive strategy use was measured using the Metacognitive Awareness Listening Questionnaire (MALQ) after the tests. Automatic linear modeling was used as the data analysis method. The results showed that WLP and PLP test performances were significantly predicted by different listener factors. To be specific, 25.9% of the variance in WLP test scores was accounted for by three factors: i) person knowledge and ii) mental translation in metacognitive awareness, and iii) average fixation duration in gaze behaviors. For PLP tests, 32.4% of the variance in PLP test scores were explained by six factors: i) the dmPFC measure in PLP-Question, ii) the IFG measure in PLP-Audio, and iii) the IFG measure in PLP-Question of brain activation, iv) mental translation and v) directed attention of metacognitive awareness, and vi) visit counts in PLP-Question of gaze behaviors.
This study proposed an extended approach to construct validity: neurocognitive validity, which conceptualizes listening test performance as a function of test takers’ behavioral, neurophysiological, and psychological mechanisms. Implications for listening assessment and listening pedagogy are also discussed.
Date Issued
2021
Call Number
P53.47 Zha
Date Submitted
2021