Options
Measuring undergraduate students' reliance on generative AI during problem-solving: Scale development and validation
Citation
Hou, C., Zhu, G., Vidya Sudarshan, Lim, F. S., & Ong, Y. S. (2025). Measuring undergraduate students' reliance on generative AI during problem-solving: Scale development and validation. Computers & Education, 234, Article 105329. https://doi.org/10.1016/j.compedu.2025.105329
Abstract
Reliance on AI describes the behavioral patterns of when and how individuals depend on AI suggestions, and appropriate reliance patterns are necessary to achieve effective human-AI collaboration. Traditional measures often link reliance to decision-making outcomes, which may not be suitable for complex problem-solving tasks where outcomes are not binary (i.e., correct or incorrect) or immediately clear. Therefore, this study aims to develop a scale to measure undergraduate students' behaviors of using Generative AI during problem-solving tasks without directly linking them to specific outcomes. We conducted an exploratory factor analysis on 800 responses collected after students finished one problem-solving activity, which revealed four distinct factors: reflective use, cautious use, thoughtless use, and collaborative use. The overall scale has reached sufficient internal reliability (Cronbach's alpha = .84). Two confirmatory factor analyses (CFAs) were conducted to validate the factors using the remaining 730 responses from this activity and 1173 responses from another problem-solving activity. CFA indices showed adequate model fit for data from both problem-solving tasks, suggesting that the scale can be applied to various human-AI problem-solving tasks. This study offers a validated scale to measure students' reliance behaviors in different human-AI problem-solving activities and provides implications for educators to responsively integrate Generative AI in higher education.
Publisher
Elsevier
Journal
Computers & Education
Project
RG 133/24
ARC 1/24 ZG
Funding Agency
Ministry of Education, Singapore