Bicara Isyarat

Problem

Learning sign language (American Sign Language / ASL) often lacks accessible, interactive feedback loops for self-learners. Most existing detection tools rely on continuous network calls and cloud server APIs, introducing high latency, operating costs, and privacy concerns. Furthermore, sign language poses an interesting technical challenge: 24 letters (A–Y excl. J/Z) are static hand poses, whereas letters J and Z are dynamic gestures requiring trajectory tracking over time. Combining both static pose classification and temporal motion gesture recognition into a smooth, battery-efficient mobile application that functions completely offline requires a well-structured on-device AI architecture.

My role

This was a five-member capstone team project (PBL Semester 5). I was responsible for the core Flutter mobile architecture, on-device AI pipeline integration, offline data persistence, and the learning gamification engine:

• Hybrid detection pipeline architecture: Integrated the on-device TFLite CNN model for 24 static alphabet poses and coordinated the landmark-trajectory GRU pipeline for dynamic gestures (J and Z).
• Performance optimization & frame scheduling: Implemented TFLite isolate execution, throttled frame evaluation (~260 ms for CNN analysis, ~320 ms for hand presence detection), and structured generation-tagged cancellation to avoid thread starvation and memory leaks.
• Local storage & database schema migrations: Designed the local SQLite database (sqflite v2) supporting offline training histories, confidence ratings, input source provenance (live stream, photo, gallery file), and media file paths.
• Gamification & learning progression engine: Built state-driven Riverpod providers tracking active training streaks, total earned XP, 26-letter mastery metrics, and milestone achievements with custom SVG badges and unlock modals.
• Application modules & UI: Developed the multi-mode camera screen (live practice, photo snapshot, file upload), interactive sign dictionary, training history records, and progress dashboard.

System architecture

The application operates 100% offline on client devices with zero external backend dependencies:

• Flutter + Dart (Riverpod State Management): Coordinates camera hardware lifecycles, target practice vs free-form practice state machines, and reactive gamification metrics.
• Static Detection Pipeline (24 Alphabet Letters): Live camera frames (YUV420) pass to a CPU-based MediaPipe Hand Landmarker extracting 21 XYZ landmarks. Bounding-box regions are padded, orientation-adjusted, converted into 28×28 grayscale images, and passed to a lightweight CNN model (model_float16.tflite, only 318 KiB). Letters are committed when prediction confidence is $\ge 60%$ across at least 2 consecutive frames.
• Dynamic Temporal Pipeline (Letters J and Z): Dynamic gestures are sampled with a monotonic clock (Stopwatch) at a target rate of up to 15 FPS. Hand coordinate sequences are normalized relative to wrist landmarks and interpolated into a fixed 48-timestep sequence of 64 features ([1, 48, 64]), then classified by a compact Recurrent Neural Network (GRU TFLite, 224 KiB) returning J, Z, or OTHER.
• Local SQLite Persistence (sqflite): Stores complete records of all completed sessions, tracking timestamps, recognized letters, confidence percentages, input modalities, and generated media paths.

Key services & architectural contracts

Because the app is fully local, internal service modules provide clear interface boundaries:

Technical challenges

1. Eliminating collisions between static and dynamic recognizers. When a user begins drawing the motion path for letter J or Z, the starting hand position often mimics static letters (such as I or D). If the static pipeline stayed active, the CNN would falsely recognize and commit intermediate static letters before the motion concluded. To solve this, the architecture uses a generation token mechanism: triggering the dynamic gesture state instantly invalidates all enqueued static inferences and background callbacks. Furthermore, after completing a dynamic gesture, the pipeline requires the hand to leave the camera frame (hand release barrier) before re-enabling static letter recognition.

2. Sustained real-time inference without thermal throttling. Running MediaPipe landmarker and TFLite neural networks frame-by-frame on raw 30–60 FPS video streams quickly leads to frame stutter and high thermal loads on mobile hardware. We decoupled the acquisition and inference loops:

• Hand landmark presence checks run at a 320 ms cadence.
• Full CNN classification is throttled to 260 ms intervals and offloaded to an IsolateInterpreter, keeping the main UI thread at a consistent 60 FPS.
• Incoming frames arriving while a previous inference is active are dropped immediately (busy skip), preventing latency queue accumulation.

3. Mitigating false positives in dynamic motion. The GRU model incorporates an explicit negative class (OTHER) to reject accidental hand movements. Captured clips are strictly validated: minimum duration of 0.25 seconds, maximum 10-second capture window, at least 6 valid hand observations, and a hand detection ratio of $\ge 60%$. If hand visibility drops for more than 0.55 seconds mid-gesture, the sample is discarded automatically without corrupting user stats.

4. Offline-first gamification integrity. To guarantee streak and XP reliability in offline environments, the gamification state is derived deterministically from historical SQLite transaction entries. Active streak days are computed from calendar date continuity in the history records, eliminating edge cases caused by unexpected device shutdowns or background process terminations.

Future improvements

• Continuous Sentence Recognition: Implement automated temporal transition segmentation to detect multi-letter words naturally without requiring manual session split triggers.
• Model Fine-Tuning with Diverse Real Camera Data: Retrain the GRU model with expanded real-world negative motion datasets (OTHER) across varied ambient lighting conditions.
• Optional Cloud Synchronization: Provide optional user account sync via Supabase or Firebase to preserve learning progress and streaks across device changes.