Using computer vision and machine learning with a view to building children's vocabulary

Abstract

This paper describes an independent research project carried out by a pair of junior college students under the mentorship of a Research Scientist at the National Institute of Education. Our project aimed to design a working device to aid kindergarten children in learning about common household equipment such as tables, chairs, and so on. We hoped that this device would help to solve the problem of kindergarten children not being able to learn during the Covid-19 pandemic due to the closure of kindergartens. We trained an Object Detection model and ran it on a Raspberry Pi with screen, speakers and camera connected to it. We took photos of different household equipment and labelled them using software in order to train these images for our Object Detection model. We made use of the Google Open Images and COCO image datasets to sift out the related images of the objects we wanted to train in the model. Overall, the use of the software such as TensorFlow helped us with the training of the different types of objects for the model and the use of Text-to-Speech software allowed us to incorporate the use of sound to project the pronunciation of the objects. Eventually, there are many more aspects we could explore with our prototype. We could make it so that it can say out the object names in different languages to help teach Mother Tongue to kindergarten children.