We build AIris!

Following my first post about Gamuda AI Academy, I now wanted to share my capstone project that we presented during the demo day. We delivered AIris - A camera-first language learning application, an app designed to help kids struggling with learning and pronouncing words in Bahasa Melayu. It was a team project consisting of six people.

AIris - A Camera-First Malay Learning Application

The idea is simple: help kids learn to converse in Bahasa Melayu in a fun and engaging way—through the camera lens. Think of it as Pokemon Go, but educational and meaningful.

Core Feature of AIris

The camera feature allows kids to point at an object, take a picture, and learn its name in Bahasa Melayu. For example, if a kid doesn’t know the word for “cat,” they can take a picture, and AIris will recognize the object and teach them how to pronounce it in Bahasa Melayu.

In the background, the image is sent to Gemini API, which recognizes the object and generates a response to teach the user.

VocabCard - Comprehensive Flashcard System

• After processing the image, the word is displayed in VocabCard.
• VocabCard includes:
  • The word itself.
  • A simple description of the object.
  • Pronunciation guide through Text-to-Speech (TTS).
  • Sample sentences demonstrating usage in English and Bahasa Melayu.

Kids can either take another picture or save the word as a flashcard. Saved words are stored in the wordbank, which they can review later.

Leitner System - The Flashcard Learning Method

To test kids’ understanding of words, AIris offers three types of quizzes:

1. Naming Test - Name the object in Bahasa Melayu based on the picture.
2. Listening Test - Listen to the pronunciation and type the correct answer.
3. Speaking Test - Pronounce the word, and the app checks if it’s correct.

The mixed test combines all three quiz types. Questions are drawn from the word bank based on their exploration of words.

The Leitner system is a flashcard-based method for memorization and retention. Flashcards are sorted into categories based on familiarity, and reviewed at increasing intervals. Difficult cards are reviewed more frequently, while easier ones are reviewed less often. This helps focus study time on areas needing improvement.

This system was a proof of concept for the capstone project. While we encountered errors implementing the algorithm, we managed to create a simple working version.

Leaderboard - Gamified Learning Experience

To keep kids engaged, we implemented a leaderboard. Kids earn points by taking pictures and answering quizzes. They can see their ranking, encouraging them to keep learning new words in Bahasa Melayu.

ChatBot - KSSR-Based Chatbot for Interactive Learning

The chatbot helps kids learn abstract words like “freedom” or “happiness” when they can’t snap images. Kids can ask the chatbot for meanings and pronunciations in Bahasa Melayu. The chatbot generates kid-friendly responses.

We also implemented Retrieval Augmented Generation (RAG) concepts tailored to the Kurrikulum Standard Sekolah Rendah (KSSR). This allows the chatbot to retrieve information from the KSSR curriculum and generate responses. Kids can ask questions like:

• What is “Peribahasa”?
• What is “Rukun Negara”?
• Teach me about “penjodoh bilangan”.

Scavenger Hunt - IRL Word-Finding Game

We also implemented a scavenger hunt game to enhance learning. Kids can go outside, find objects, and take pictures using AIris. For example, during a Raya gathering, they can take a picture of “Baju Kurung,” and AIris will recognize the object and teach them about it. This feature is localized to Malaysian culture, helping kids learn about their heritage.

The Tech Stack We Used

• Frontend:
  • Ionic Framework
• Backend:
  • FastAPI
  • Google TTS and STT
• Database:
  • PostgreSQL
  • CloudSQL
• AI Model:
  • Gemini Model
• Vector Database:
  • MongoDB
• Deployment:
  • Google Cloud Platform
  • Firebase
  • Cloud Run
• Version Control:
  • Git

Meet the Team

Here are five of my talented friends who made AIris possible. Feel free to connect with them and ask about their experiences with Gamuda AI Academy and full-stack development:

Conclusion

AIris is a fun and engaging application that helps kids learn Bahasa Melayu interactively. The use of cameras, flashcards, quizzes, leaderboards, and chatbots makes the learning experience enjoyable and effective. We are proud of what we achieved in such a short time and hope AIris positively impacts how kids learn languages in the future.

In the next post, I will share my personal experience during the demo day. Stay tuned!