AI-generated live voice-driven storytelling
Tools
p5.js
ml5.js
Google Teachable Machine
Context
India's storytelling traditions are as diverse as the culture of the country. Storytelling is not limited to oral recitation; it takes form through drawings, paintings, dance, or even puppet shows. I wanted to bring innovation in tribal storytelling with the help of machine learning. It was the final project of the Creative Computing course.
Background
Pithora is a ritualistic painting done on the walls by Rathwa, Bhils, and Bhilala tribes. It is popular in the Gujarat and Madhya Pradesh states of India. These paintings are distinguished by the lucky and sacred mascots, the horses, the sun, the birds, the animals, and the trees.
It began by sourcing an original folk story from the Rathwa tribe of Gujarat, narrated in their native Gujarati language and rooted in a moral lesson. Visuals were then created by my teammate to match the narrative, with characters and scenes designed in the traditional Pithora art style to preserve cultural authenticity. To enable interaction, a machine learning model was trained using a large set of Gujarati language samples for speech recognition using Teachable Machine. The code was developed using p5.js and the ml5.js library, leveraging sound classification to detect and respond to narration. These components, visuals, ML model, and interaction logic, were combined to animate the story in real time.
During the presentation, as the story is narrated live in the Gujarati language, the code plays corresponding visuals, creating a responsive, immersive storytelling experience that bridges oral tradition with modern machine learning techniques.
