Development and validation of computerised adaptive tests with and without performance diagnosis
This research has focused on an application, in actual classroom practice, of computerised adaptive testing (CAT) in assessing learning achievement. The concept of CAT, an application of item response theory, has become feasible with the advent of high-speed computers. Basically, the testing in CAT is tailored to an examinee's ability level: the computer first obtains an estimate of the examinee's ability level based on his or her responses to initial test items, and then makes subsequent item-selection decisions that are most appropriate for measuring his/her performance level.
Based on the literature review, investigative directions for this study were conceptualised in terms of the development and validation of computerised adaptive tests with and without the provision of performance diagnosis. 'Item response theory' and the unitary validity concept formed the theoretical basis of the research.
This research on CAT was thus partitioned into three closely related studies. Study 1 focused on the calibration of a test-items bank on biology under the item response theory (IRT). It also examined relevant practical issues encountered in the development of a computerised item bank. In Study 2, an automated construction of tests using the calibrated pool of test items was performed. Whilst the IRT-calibrated item bank was used to construct both conventional fixed-length and adaptive variable-length tests, the functioning of the item pool was particularly investigated for computerised adaptive testing (CAT). Study 3 validated the computerised adaptive tests using students in a Singapore school. The effects of student factors - i.e. gender, ability, computer familiarity (viz. computer ownership, computing experience and frequency of computer use), and affective characteristics (viz. attitude towards computers and to science learning) - on three measures of CAT outcomes, namely, test performance, attitude towards the test administration and benefits derived from the testing, were investigated.
Results of this study indicated comparability of test scores obtained with the computerised adaptive tests and those obtained with a parallel paper-and-pencil test. In brief, ability was the best predictor of test performance in biology using CAT. Overall the students were positive towards the CAT administration; but they were most bothered by their inability to review test items during the testing. Some differences were found in the reactions to certain aspects of CAT of students in different subgroups defined by gender, computer familiarity, and affective characteristics. However, these differences did not appear to affect their test performance on CAT. Satisfactory benefits were derived from the use of CAT. These included an average of 10% reduction in test length and 50% reduction in test time, and the testing showed adequate adaptability to individual students' ability level. Students who were treated with CAT with performance diagnosis indicated the usefulness of the feedback and diagnosis facility.
With these insights into the use of CAT in a secondary school in Singapore, implications for future implementation and use of computerised adaptive tests in school-based testing are discussed. The limitations of the research are also explained and suggestions for further research indicated.
Using computer-based modelling for primary science learning and assessment
Computer-based modeling is not just a means for students to learn important scientific knowledge and skills, but also a technique to assess student understandings of science. A software tool called Model-It allows young students to create their own models so that their learning becomes more interactive and engaged. However, there is a mismatch between how students learn and how they are assessed if conventional paper-administered tests are used. This paper argues for alternative assessments to be better aligned with curriculum and instruction. Forty 4th grade students in a local Singapore school participated in a science inquiry activity that involved learning with modeling as an alternative assessment. The students individually created models of food webs to illustrate their understanding of energy flows and photosynthesis. A scoring rubric based on four criteria (“focus and structure”, "accuracy", "completeness" and "functionality") was used to evaluate the models, with the modeling scores being compared to student scores of the school’s paper-based assessments of science learning. In addition, 18 students were interviewed about their understanding of models and modeling. The data is currently being analyzed and the findings of this study and potential implications for educational assessments will be presented in this paper.
Using a Rasch model to analyse parallel forms of two achievement tests on biology
Achievement tests on biology were constructed as part of an item pool development for a larger study on computerised adaptive testing. A total of 220 four-alternative multiple-choice test items were targeted. A test specification, modelled after the 1995 GCE ‘O’ level syllabus, was used to guide the writing and selection of test items aimed at assessing learning outcomes of biological concepts and principles. Two tests were planned; each with parallel forms based on content consistency. In addition, ‘anchor’ test items were placed on the two forms. The test forms were spiralled (i.e. arranged in a fixed alternate order) in the data collection from a representative sample of about 2000 students from a population of Singapore students who were candidates for the 1994 GCE ‘O’ level biology examination. This paper reports on an application of the Rasch model to analyse the test items on parallel forms of the two tests, and to assess the model-fit of the items.
A computerised adaptive testing approach: What are the benefits to students?
Student readiness for EduTOOLS use: An investigation of their IT-related skills and behaviours
EduTOOLS, a R&D project of the Centre for Learning Technologies which is jointly set up by the Ministry of Education, Kent Ridge Digital Labs and Infocomm Development Authority, involves an innovative use of computer technology. With EduTOOLS, students can make use of a suite of on-line tools for project work. Also available to students are a collection of recommended project resources and a list of experts. This project will be piloted in three Singapore schools - two secondary schools and one junior college. For an implementation of a high technology project such as EduTOOLS, one main concern is student readiness in terms of IT skills. This paper reports on the findings of an investigation of students' experience with computers, namely their use of computers in and outside of school, the types of computer applications used and of activities engaged with a computer, and their frequency of computer use. Self-reports of students' working behaviours on the computer as well as their affective characteristics were also captured. The implications of the findings for EduTOOLS implementation will be discussed.
Getting ready to use technology for school-based testing
In Singapore schools, testing of student learning is serious business but teachers have yet to get in full swing to exploit technology for school-based testing. What then is stopping them? Among other things, there is a imperative need to first understand and ensure the harmonious interplay of three related factors of hardware & infrastructure, software, and people-ware.
Test reporting of English language proficiencies of teacher trainees: Towards a profiling assessment system
A single overall grading is the hallmark of test reporting practices in education. This assessment system has provided an efficient and convenient means of reporting test results for a summative purpose such as certifying students on completion of a course of study or selecting students who can benefit from the next level of education. However it is less useful for reporting results for a formative purpose such as monitoring students' learning progress or diagnosing their learning difficulties. The National Institute of Education has developed a computerised assessment tool known as "NIE Computerised English Language Test" (NIECELT) for testing the English language proficiencies of prospective students in pre-service teacher training programmes. NIECELT, however, can be used for creating appropriate tests meant for any other grade level as well – primary, secondary or college. This paper will describe a profiling assessment system to be incorporated into the test reporting of language proficiencies. NIECELT, an interactive mode of computerised testing, presents a number of sub-tests for assessing a student’s level of competence in English sentence structure, collocation, text cohesion, text meaning, editing skills as well as semantic awareness. By harnessing the computer's unique capabilities in data management, the student's performance in specific language skill areas can be profiled.