Options
Investigating the quality of quantitative mobile assisted language learning research: a systematic review
Author
Hou, Zhuohan
Supervisor
Aryadoust, Vahid
Abstract
Mobile assisted language learning (MALL) is a novel approach to language learning and teaching. Studies that attempted to review the research trends in MALL research have shown the effectiveness of MALL in motivating and enhancing language learning. However, to date, no reviews focusing on the methodological quality of MALL studies have been conducted. The present study aims to review the statistical techniques that are used in quantitative MALL research. Statistical methods within General Linear Model (GLM) are specifically underinvestigated from the viewpoint of report-writing and the satisfaction of pertinent statistical assumptions. The present research purports to address this gap and investigates the reliability of the instruments and coding practices in MALL research.
To conduct a comprehensive and systematic review of MALL research, 41 journals that are related to language learning or educational technology were identified using the Scopus database. Among these, 77 quantitative MALL studies that investigated English as second or foreign language, which were eligible for the proposed inclusion and exclusion criteria, were selected for review. In full-text screening, the researcher coded each study for the statistical techniques applied, the basic statistical output, assumptions reported, and the reliability investigation of the instruments and the coding practice involved. The reliability of the overall coding process of the present study was inspected using indices of agreement percentage and Cohen’s Kappa.
Results show the ubiquity of GLM (i.e., mean-based data analysis such as t-test, univariate analysis of variances (ANOVA), multivariate analysis of variances (MANOVA) for data analysis in 62.50% of the overall statistical methods used in MALL research. It was found that for all the tests within the GLM configuration, the underlying and essential assumptions (i.e., normality, homogeneity of variance, and linearity) were generally neglected. Meanwhile, researchers commonly reported the mean and standard deviation of the groups accompanied with p values. On the other hand, effect size that reveals the practical significance of the investigated effect or relations and the confidence intervals that show the precision of the analysis were found to be overlooked. In addition, less than half of MALL research reported reliability, indicating a lack of evidence of the precision of the measurement instruments used in more than half of MALL studies.
The researcher discusses implications of the study for methodology in MALL research. Suggestions and guidelines for the future research are also provided.
To conduct a comprehensive and systematic review of MALL research, 41 journals that are related to language learning or educational technology were identified using the Scopus database. Among these, 77 quantitative MALL studies that investigated English as second or foreign language, which were eligible for the proposed inclusion and exclusion criteria, were selected for review. In full-text screening, the researcher coded each study for the statistical techniques applied, the basic statistical output, assumptions reported, and the reliability investigation of the instruments and the coding practice involved. The reliability of the overall coding process of the present study was inspected using indices of agreement percentage and Cohen’s Kappa.
Results show the ubiquity of GLM (i.e., mean-based data analysis such as t-test, univariate analysis of variances (ANOVA), multivariate analysis of variances (MANOVA) for data analysis in 62.50% of the overall statistical methods used in MALL research. It was found that for all the tests within the GLM configuration, the underlying and essential assumptions (i.e., normality, homogeneity of variance, and linearity) were generally neglected. Meanwhile, researchers commonly reported the mean and standard deviation of the groups accompanied with p values. On the other hand, effect size that reveals the practical significance of the investigated effect or relations and the confidence intervals that show the precision of the analysis were found to be overlooked. In addition, less than half of MALL research reported reliability, indicating a lack of evidence of the precision of the measurement instruments used in more than half of MALL studies.
The researcher discusses implications of the study for methodology in MALL research. Suggestions and guidelines for the future research are also provided.
Date Issued
2020
Call Number
P120.I6 Hou
Date Submitted
2020