Options
A systematic review of automated writing evaluation within the argument-based validation framework
Author
Shi, Huawei
Supervisor
Aryadoust, Vahid
Abstract
Automated writing evaluation (AWE) or automated essay scoring (AES) systems have been adopted in language assessment and learning since at least five decades ago. Despite their wide application in various disciplines in recent years, controversies have always surrounded the use of AWE systems, and there have been limited retrospective reviews of the extant research and development in this field to inform the research in this area.
This dissertation provides a systematic review of empirical research on AES/AWE from 2010 to 2020. A total of 134 published articles on AWE were selected based on a set of inclusion criteria. All articles were analyzed within an argument-based validation framework (Kane, 1992, 2006, 2013; Chapelle et al. 2008; Fan & Yan, 2020). The papers were coded based on a coding scheme consisting of 22 variables classified into six categories: administrative information, AWE system, study context, inferences of validity, research design, and results.
The major findings are: 1. The AWE research in the past 10 years was heterogeneous in terms of the number of AWE systems being studied, the location of studies, the AWE types, the language environments, ecological settings, and educational context; 2. A disproportional ratio of studies was found to have been done on each validity inference, with the domain description inference being the neglected one, and the evaluation and utilization inferences receiving the most attention; 3. Regarding methodologies, a dominance of quantitative methods was noticed among all studies: while the use of quantitative methods was predominant in the evaluation, generalization, explanation, and extrapolation inferences, most qualitative studies, as well as studies with mixed or eclectic methods, were used to examine the utilization inference; and 4. Both backing and rebuttals were found for each validity inference across AWE types except the domain description inference. Still, the lack of research on the domain description inference indicated that construct representation suffers from a lack of enough research. In the end, implications and directions for future research are also discussed.
This dissertation provides a systematic review of empirical research on AES/AWE from 2010 to 2020. A total of 134 published articles on AWE were selected based on a set of inclusion criteria. All articles were analyzed within an argument-based validation framework (Kane, 1992, 2006, 2013; Chapelle et al. 2008; Fan & Yan, 2020). The papers were coded based on a coding scheme consisting of 22 variables classified into six categories: administrative information, AWE system, study context, inferences of validity, research design, and results.
The major findings are: 1. The AWE research in the past 10 years was heterogeneous in terms of the number of AWE systems being studied, the location of studies, the AWE types, the language environments, ecological settings, and educational context; 2. A disproportional ratio of studies was found to have been done on each validity inference, with the domain description inference being the neglected one, and the evaluation and utilization inferences receiving the most attention; 3. Regarding methodologies, a dominance of quantitative methods was noticed among all studies: while the use of quantitative methods was predominant in the evaluation, generalization, explanation, and extrapolation inferences, most qualitative studies, as well as studies with mixed or eclectic methods, were used to examine the utilization inference; and 4. Both backing and rebuttals were found for each validity inference across AWE types except the domain description inference. Still, the lack of research on the domain description inference indicated that construct representation suffers from a lack of enough research. In the end, implications and directions for future research are also discussed.
Date Issued
2020
Call Number
P53.27 Shi
Date Submitted
2020