Aryadoust, Vahid

This article evaluates the construct validity of the Michigan English Language Assessment Battery (MELAB) listening test by investigating the underpinning structure of the test (or construct map), possible construct under representation and construct-irrelevant threats. Data for the study, from the administration of a form of the MELAB listening test to 916 international test takers, were provided by the English Language Institute of the University of Michigan. The researchers sought evidence of construct validity primarily through correlated uniqueness models (CUM) and the Rasch model. A five factor CUM was fitted into the data but did not display acceptable measurement properties. The researchers then evaluated a three-traits1 confirmatory factor analysis (CFA) that fitted the data sufficiently. This fitting model was further evaluated with parcel items, which supported the proposed CFA model. Accordingly, the underlying structure of the test was mapped out as three factors: ability to understand minimal context stimuli, short interactions, and long-stretch discourse. The researchers propose this model as the tentative construct map of this form of the test. To investigate construct under representation and construct-irrelevant threats, the Rasch model was used. This analysis showed that the test was relatively easy for the sample and the listening ability of several higher ability test takers were sufficiently tested by the items. This is interpreted to be a sign of test ceiling effects and minor construct-underrepresentation, although the researchers argue that the test is intended to distinguish among the students who have a minimum listening ability to enter a program from those who don’t. The Rasch model provided support of the absence of construct-irrelevant threats by showing the adherence of data to uni dimensionality and local independence, and good measurement properties of items. The final assessment of the observed results showed that the generated evidence supported the construct validity of the test.

The testing and teaching of listening has been partially guided by the notion of sub-skills, or a set of listening abilities that are needed for achieving successful comprehension and utilisation of the information from listening texts. Although this notion came about mainly through applications of theoretical perspectives from psychology and communication studies, the actual divisibility of the sub-skills has rarely been examined. This paper reports an attempt to do so by using data from the answers of 916 test takers of a retired version of the Michigan English Language Assessment Battery (MELAB) listening test. First, an iterative content analysis of items was carried out, identifying five key sub-skills. Next, the discriminability of sub-skills was examined through confirmatory factor analysis (CFA). Five independent measurement models representing the sub-skills were evaluated. The overall CFA model comprising the measurement models showed excessively high correlations between factors. Further tests through CFA resolved the inadmissible correlations, though the high correlations persisted. Finally, we made 23 aggregate-level items which were used in a higher-order model, which induced best fit indices and resolved the inadmissible estimates. The results show that the sub-skills in the test were empirically divisible, hence lending support to scholarly attempts in discussing components in the listening construct for the purpose of teaching and assessment.

The field of Translation Studies has expanded rapidly in the twenty-first century, largely due to the growing demand for translation and interpreting professionals. This study provides a scientometric review of Translation Studies to identify its developmental trends and patterns over the past two decades. Document co-citation analysis was conducted on 6007 journal articles published in the fifteen translation studies journals indexed in the Web of Science between January 2001 and December 2020. Twelve document co-citation analysis networks were generated and compared. Quantitative analyses, including temporal and structural metrics, confirmed the robustness and reliability of a network comprising ten discrete research clusters. A timeline view was generated to visualize how these clusters have evolved over time. Ten clusters were identified as major research subdomains in Translation Studies, namely translation competence, translation in conflict zones, translator training, collaborative translation, translation and society, language policy, post-editing and revision, media translation, the translation profession, and web localization. In addition, burst detection analysis identified the twenty most influential publications in this sample. Based on these findings, we discuss how the observed trends in each cluster contribute to further developments in Translation Studies. The implications for teaching, research, and theory are discussed and some methodological guidelines are proposed for future research.

Digital storytelling (DST) is a novel approach that uses modern computer technology to amplify language learning and teaching. The present study aims to review how the published DST research utilizes visuals and audio to influence the learning environment and engage adolescent and adult language learners. This was measured through their improvement in the four main language skills: reading, writing, listening, and speaking. A total 71 journal papers were identified using the Scopus database. The papers studied both first and second language learning and were coded in full-text screening for the research topics and methods adopted, theories or frameworks adopted, outcomes across the language skills, and reliability investigation of the studies. The results showed a range of research used in the studies, with 39.7% of the total studies using a mixed number of methods. The theories adopted in these studies were limited to components of DST, age group, and the type of study. Most studies neither tested nor mentioned the use of the three theoretical variables mentioned above. Notably, a majority of the studies reported positive outcomes when DST was used in the learning environment. However, not all claims were supported with evidence. Lastly, only a handful of the studies reviewed reported reliability, highlighting a lack of verification of the precision of the measurement instruments used. Implications of these findings and recommendations for designing DST and language learning research in the future will be discussed.

To date many test designers have relied heavily on either the interpretation or uses of scores in high-stakes tests as evidence of validity. As a case in point, in the listening section of International English Language Testing System (IELTS), the consequences and the correlation of the scores with other measures such as academic performance of students have been extensively researched in pursuit of consequential and criterion validity. While these research inquiries are valuable to especially research test usefulness, the test should be validated for its main objectives. Consequential and criterion validity studies into the IELTS listening module have proposed different and controversial evidence. We argue if the construct validity is not established, supportive evidence of usefulness is either very difficult or impossible to find. The main purpose of this study is to investigate the construct validity of the IELTS listening test. We will employ a two-parameter logistic Item Response Theory (IRT) model to investigate the construct representation and irrelevant factor (Messick, 1988, 1989).

We compared the extraneous cognitive load imposed by two wording conditions: negatively worded and non-negated items in an L2 questionnaire. The questionnaire is the metacognitive awareness listening questionnaire (MALQ), a widely used instrument for assessing metacognitive awareness strategies. Respondents' (N = 109) eye movements measured by an eye-tracker and brain activation levels measured by functional near-infrared spectroscopy (fNIRS) were obtained to examine their extraneous cognitive load in responding to MALQ. Using MANOVA, we identified distinct gaze behavior associated with negatively worded items, indicating that the presence of such items may increase extraneous cognitive load. In addition, English-as-a-second-language participants generally exhibited higher extraneous cognitive load than their L1 counterparts. The results of the neuroimaging investigation further indicated that negatively worded items require more visual attention and cognitive effort, while positively worded items may engage higher-order reflective processing in the brain, highlighting a complex interaction between attention and extraneous cognitive load. Additionally, linear mixed effects models showed that although the models under the two-wording conditions explained a significant amount of variation in respondents’ MALQ scores, they had relatively lower explanatory power (R2) though better global fit compared to the models based on the five constructs that MALQ measures (planning-evaluation, directed attention, person knowledge, mental translation, and problem-solving). We suggest that a balanced approach that considers both the target constructs and the negeative wording effect might be the most effective strategy in questionnaire design and validation. Further implications of these findings are discussed.

The present study aims to meta-analyze the reliability of second language (L2) reading assessments and identify the potential moderators of reliability in L2 reading comprehension tests. We examined 3,247 individual studies for possible inclusion and assessed 353 studies as eligible for the inclusion criteria. Of these, we extracted 150 Cronbach’s alpha estimates from 113 eligible studies (years 1998–2024) that reported Cronbach’s alpha coefficients properly and coded 27 potential predictors comprising of the characteristics of the study, the test, and test takers. We subsequently conducted a reliability generalization (RG) meta-analysis to compute the average reliability coefficient of L2 reading comprehension tests and identify potential moderators from 27 coded predictor variables. The RG meta-analysis found an average reliability of 0.79 (95% CI [0.78, 0.81]). The number of test items, test piloting, test takers’ educational institution, study design, and testing mode were found to respectively explain 16.76%, 5.92%, 4.91%, 2.58%, and 1.36% of variance in reliability coefficients. The implications of this study and future directions are further discussed.

The purpose of the present study was twofold: (a) it examined the relationship between peer-rated likeability and peer-rated oral presentation skills of 96 student presenters enrolled in a science communication course, and (b) it investigated the relationship between student raters’ severity in rating presenters’ likeability and their severity in evaluating presenters’ skills. Students delivered an academic presentation and then changed roles to rate their peers’ performance and likeability, using an 18-item oral presentation scale and a 10-item likeability questionnaire, respectively. Many-facet Rasch measurement was used to validate the data, and structural equation modeling (SEM) was used to examine the research questions. At an aggregate level, likeability explained 19.5% of the variance of the oral presentation ratings and 8.4% of rater severity. At an item-level, multiple cause-effect relationships were detected, with the likeability items explaining 6–30% of the variance in the oral presentation items. Implications of the study are discussed.

The present study explored the potential of a new neurocognitive approach to test equity which integrates evidence from eye-tracking and functional near-infrared spectroscopy with conventional test content analysis and psychometric analysis. The participants of the study (η = 29) were neurotypical university students who took two tests of English lecture comprehension. Test equity was examined in this study at four levels: the linguistic level (content evidence) and the test scores level which are conventional levels in test equity; and gaze behavior level and neurocognitive level which are novel to this study. It was found that the linguistic features of the two test forms being equated were similar and that there was no significant difference at neurocognitive and behavioral levels. However, there was a significant difference in gaze behaviors, measured by fixation counts and visit counts, although fixation duration and visit duration did not vary across the two tests. Overall, test equity was supported, despite partial counterevidence from the gaze data. We discuss the implication of this approach for future equity research and response process in language assessment.