Using Computer Vision to Improve Disability Access in Music Education

Mehek Gosalia, a high school student from Sammamish, Washington, developed a disability-accessible rhythm education tool named the Mehek Box, initially as a physical device using tactile blocks to represent rhythmic notes, which she later expanded to include Braille for accessibility. During the pandemic, she further innovated by creating app and web versions that utilize audio, vibration, and animation to provide a multisensory learning experience. Gosalia is collaborating with various programs to test and refine these tools, particularly focusing on bridging the physical and digital versions of the Mehek Box. She introduced an image recognition feature in the app, using a Tensorflow object detection model trained on a custom dataset of rhythm blocks, subsequently augmented with Roboflow's tools to expand the dataset to 10,000 images. Testing different datasets showed surprising results where the unaugmented dataset performed best, leading to further plans for enhancement, such as adding an alignment frame onscreen to assist users in capturing images accurately.