Options
Aryadoust, Vahid
- PublicationOpen AccessA functional analysis of the dialogues in the new interchange intro textbookThis study investigates language functions in the New Interchange Intro textbook. It is observed that more difficult grammar structures, e.g., wh-questions, have a gradual increase throughout the dialogues, but simple yes/no questions and statements are more pronounced in the opening lessons. Also, declarative sentences outnumber other grammar functions, and wh-questions rank second. Grammar structures establish three major macro-pragmatic functions: representative, directive, and expressive. However, commissive and declaration functions are not observed in the dialogues, which can be due to their difficulty for new learners to communicate these language functions. Among micros, stating greetings, requesting, and expressing gratitude have the least frequency whereas exchanging information has the greatest frequency. Naturally, research on language functions in English textbooks helps teachers in selecting the best materials.
209 292 - PublicationOpen AccessA systematic review of digital storytelling in language learning in adolescents and adultsDigital storytelling (DST) is a novel approach that uses modern computer technology to amplify language learning and teaching. The present study aims to review how the published DST research utilizes visuals and audio to influence the learning environment and engage adolescent and adult language learners. This was measured through their improvement in the four main language skills: reading, writing, listening, and speaking. A total 71 journal papers were identified using the Scopus database. The papers studied both first and second language learning and were coded in full-text screening for the research topics and methods adopted, theories or frameworks adopted, outcomes across the language skills, and reliability investigation of the studies. The results showed a range of research used in the studies, with 39.7% of the total studies using a mixed number of methods. The theories adopted in these studies were limited to components of DST, age group, and the type of study. Most studies neither tested nor mentioned the use of the three theoretical variables mentioned above. Notably, a majority of the studies reported positive outcomes when DST was used in the learning environment. However, not all claims were supported with evidence. Lastly, only a handful of the studies reviewed reported reliability, highlighting a lack of verification of the precision of the measurement instruments used. Implications of these findings and recommendations for designing DST and language learning research in the future will be discussed.
WOS© Citations 4Scopus© Citations 11 212 812 - PublicationOpen AccessCORE 2018: A quantitative study of teaching and learning in Singapore classrooms(National Institute of Education (Singapore), 2022)
; ; 184 281 - PublicationMetadata onlyInvestigating differential item functioning across interaction variables in listening comprehension assessment
Differential item functioning (DIF) analysis is essential to ensuring the equity of measurement for different subgroups at the item level and is an integral part of validity. However, existing DIF research often overlooks within-group heterogeneity, commonly assuming that test takers from different subgroups comprise a homogeneous population. This study investigated DIF across gender, academic background, and their interaction in listening comprehension assessment using Rasch measurement. It found that ignoring within-group heterogeneity would lead to the under-detection of DIF, likely due to the cancellation of DIF at broader group levels. In addition, the study is the first to investigate DIF in a linked test, a scenario more prevalent in practical testing. The findings of the study highlight the importance of accounting for within-group heterogeneity in test fairness investigations in language assessment research and point to the potential effect of test linking and equating on DIF analysis and interpretation.
Scopus© Citations 1 48 - PublicationOpen AccessUsing recursive partitioning Rasch trees to investigate differential item functioning in second language reading testsThe present study applied recursive partitioning Rasch trees to a large-scale reading comprehension test (n=1550) to identify sources of DIF. Rasch trees divide the sample by subjecting the data to recursive non-linear partitioning and estimate item difficulty per partition. The variables used in the recursive partitioning of the data were vocabulary and grammar knowledge and gender of the test takers. This generated 11 non-prespecified DIF groups, for which the item difficulty parameters varied significantly. This is grounded within the third generation of DIF analysis and it is argued that DIF induced by the readers’ vocabulary and grammar knowledge is not construct-irrelevant. In addition, only 204 (13.16%) test takers who had significantly high grammar scores were affected by gender DIF. This suggests that DIF caused by manifest variables only influences certain subgroups of test takers with specific ability profiles, thus creating a complex network of relationships between construct-relevant and -irrelevant variables.
WOS© Citations 6Scopus© Citations 8 169 463 - PublicationOpen AccessThe Metacognitive and neurocognitive signatures of test methods in academic listeningThis study aims to investigate whether and how test takers’ academic listening test performance is predicted by their metacognitive and neurocognitive process under different test methods conditions. Eighty test takers completed two tests consisting of while-listening performance (WLP) and post-listening performance (PLP) test methods. Their metacognitive awareness was measured by the Metacognitive Awareness Listening Questionnaire (MALQ), and gaze behavior and brain activation were measured by an eye-tracker and functional near-infrared spectroscopy (fNIRS), respectively. The results of automatic linear modeling indicated that WLP and PLP test performances were predicted by different factors. The predictors of WLP test performance included two metacognitive awareness measures (i.e., person knowledge and mental translation) and fixation duration. In contrast, the predictors of the PLP performance comprised two metacognitive awareness measures (i.e., mental translation and directed attention), visit counts, and importantly, three brain activity measures: the dmPFC measure in the answering phase, IFG measure in the listening phase, and IFG measure in the answering phase. Implications of these findings for language assessment are discussed.
WOS© Citations 2Scopus© Citations 3 62 194 - PublicationMetadata onlyExploring the relative merits of cognitive diagnostic models and confirmatory factor analysis for assessing listening comprehensionA number of scaling models – developed originally for psychological and educational studies – have been adapted into language assessment. Although their application has been promising, they have not yet been validated in language assessment contexts. This study discusses the relative merits of two such models in the context of second language (L2) listening comprehension tests: confi rmatory factor analysis (CFA) and cognitive diagnostic models (CDMs). Both CFA and CDMs can model multidimensionality in assessment tools, whereas other models force the data to be statistically unidimensional. The two models were applied to the listening test of the Michigan English Language Assessment Battery (MELAB). CFA was found to impose more restrictions on the data than CDMs. It is suggested that CFA might not be suitable for modelling dichotomously scored data of L2 listening tests, whereas the CDM used in the study (the Fusion Model) appeared to successfully portray the listening sub- skills tapped by the MELAB listening test. The paper concludes with recommendations about how to use each of these models in modelling L2 listening.
43 - PublicationOpen AccessOrchestrating teacher, peer, and self-feedback to enhance learners’ cognitive, behavioral, and emotional engagement and public speaking competence
Previous research on providing feedback on public speaking has investigated the effectiveness of feedback sources, namely teacher feedback, peer feedback, and self-feedback, in enhancing public speaking competence, predominantly individually. However, how these sources of feedback can be collectively harnessed to optimize learner engagement and public speaking performance still warrants further investigation. Adopting a pre- and post-test quasi-experimental design, this study randomly assigned four classes to four feedback conditions: Group 1 received teacher feedback, Group 2 self-feedback and teacher feedback, Group 3 peer and teacher feedback, and Group 4 feedback from all three sources. Both student engagement, measured using the Public Speaking Feedback Engagement Scale (PSFES), and their public speaking performance ratings, assessed using the Public Speaking Competency Instrument (PSCI), were validated using Rasch analysis. The inferential statistics revealed that Group 3 showed significant improvements across nearly all three dimensions of engagement, whereas Group 2 experienced significant declines in all dimensions of engagement except behavioral engagement. Group 3 demonstrated significantly greater engagement gain compared to Groups 2 and 4, indicating the synergistic effect of peer and teacher feedback in contrast to the limited impact of self-feedback. Additionally, all groups demonstrated significant improvements except for Group 2, which showed significantly lower improvement compared to Group 4. The following correlation analysis identified a significant correlation between the gain of students’ behavioral engagement and the gain of public speaking performance, whereas such association was absent between cognitive or emotional engagement and public speaking competence. This study suggests that peer feedback should be preceded by group discussion and supplemented with teacher feedback in classes for enhancing the teacher–student dialog, while self-feedback should be conducted after class to improve student engagement and public speaking performance.
40 291 - PublicationMetadata onlyThe predictive value of gaze behavior and mouse-clicking in testing listening proficiency: A sensor technology studyThis study employed eye-tracking and mouse click frequency analysis to investigate the predictive power of gaze behaviors, mouse-clicking, and their interactive effects with linguistic backgrounds on the IELTS (International English Language Testing System) listening test scores. A total of 77 test takers (45 with English as their first language (E-L1) and 32 with English as their second language (E-L2)) participated in this study. Their eye movements and mouse click frequencies were recorded as they took a computer-based IELTS listening test. The subsequent data analysis, utilizing linear mixed models, showed that gaze patterns, mouse actions, and language background significantly predicted listening test outcomes across four listening test sections and between E-L1 and E-L2 candidates, accounting for 33.2% of the variance observed in test scores. These results indicate the effect of potential sources of construct-irrelevant variance on test scores, which are not predicted in the available construct definitions of the test used in the study. Implications for the listening construct and test validity are discussed.
36 - PublicationOpen AccessA Rasch analysis of an international English language testing system listening sample test(2009-06)
; This study reports on an investigation of the construct validity of an International English Language Testing System (IELTS) Listening sample test. The test was administered to 148 multinational participants. The Rasch modeling of data was used to fulfill the research objectives. Four major conclusions were made: 1) the Rasch differential item functioning analysis revealed that limited production items behave differently across different test taker groups suggesting the presence of construct-irrelevancies, 2) multiple choice questions (MCQ) do not cause construct-irrelevancies unless testees need to make ‘close paraphrases’ to comprehend the item stem or the question demands more than one answer; this nominates short MCQ as a best item format in listening tests, 3) evidence was found for ‘lexical processing’ which is different from top-down/bottom-up processing, and 4) the Wright map provided evidence for construct under-representation of the test. Findings from this study provide different sorts of evidence supporting and disproving the claim of the construct validity of the test, although they should be further investigated in future studies with different samples. Implications of the findings for IELTS and item writers are also discussed.251 589