https://docs.google.com/document/d/1ECsH3NbSoXkwm2zR4XM7vHkbddldqOXDLmq82ORsNQw/edit?tab=t.0
2025 ICT On-Device Practical Project
2025 . 10 . 30 ~ 2025 . 11 . 14
Pleasant Charles
[ Why I Picked This Topic ]
While actually running an unmanned store, I was experiencing a lot of friction. Looking to ease that friction, I searched for related products and services in Korea and abroad, but couldn't find any. On top of that, I reached out to well-known related manufacturers (CCTV, kiosk, etc.) and to related stakeholders (distributors) and asked, but although they all recognized the inconvenience, none of them had any active intent to solve it.
However, I'm actually running this kind of business and plan to keep doing so. Combined with the current size of the unmanned-retail market across various categories and its expansion potential going forward, I judged this to be sufficiently worthwhile and decided to push the project myself.
[ Project Overview ]
Build an MVP of a payment system using visual intelligence.
[ Goals ]
1. MVP implementation goal (revision, advancement)
Implement a dedicated admin for tuning the model and an entire pipeline for automating device sensing and application execution, so that the feasibility and usability of the MVP can be evaluated.
2. MVP implementation scenario (revision, advancement)
Administrator
1) After the model is tuned, it is registered in the database.
2) The end user's application checks the model and application versions. Updates are run optionally.
User
1) The user taps the start button on the device (the checkout-counter smartphone app).
2) The user places items on the checkout counter.
3) Once the counter recognizes the items,
- a success message (TTS / chime / lighting) is emitted
- the device (the checkout-counter smartphone app) shows the product name and price
- the conveyor belt runs and moves the items to the packaging zone
4) The user taps the end button on the device (the checkout-counter smartphone).
5) The device (the checkout-counter smartphone app) presents the total amount.
[ Schedule ]
By 11/02: implementation of unit functions
By 11/09: software (admin, fine-tuned model, app) feature implementation
By 11/13: hardware (camera, conveyor belt) and software integration, automation pipeline assembly
By 11/14: documentation work
[ Tech Stack ]
Hardware & Embedded Android
On-device ML optimization (TFLite + NNAPI) and real-time vision processing (30–60fps)
ESP32-based IoT system configuration
ESP32-CAM (OV2640, JPEG stream ~5fps)
L298N-based DC motor control (PWM + direction control)
WebSocket-based bidirectional communication (JSON + JPEG binary)
Design and firmware development of a simple state machine (IDLE → SCAN → PROCESSING)
Android / Mobile Development
Language & UI
Kotlin, Jetpack Compose (Material3)
SDK: Min 24 / Target 34
Mobile ML/AI
TensorFlow Lite (MobileNetV2, int8 quantization)
NNAPI Delegate (NPU acceleration)
GPU Delegate (Adreno) and ML Kit Object Detection
Lightweight model OTA update design (Firebase Remote Config + Storage)
Camera & real-time processing
Real-time frame processing pipeline built on CameraX
WebSocket client implementation (receiving the ESP32-CAM stream)
Distribution & operations
Firebase Authentication (anonymous auth)
Background updates of model/resources via WorkManager
CI/CD on GitHub Actions
ESP32 Firmware Development
Development environment
Arduino Core for ESP32 (3.x), esp_camera, ArduinoJson, WebSocketsServer
Major features implemented
Camera capture (JPEG), WebSocket frame streaming
Motor control (PWM, direction control)
JSON-based command protocol ("START_SESSION", "MOTOR_RUN", "STATE", etc.)
Designed real-time integration architecture between ESP32 and Android.
System Architecture
Structure
Android (ML inference, UI, control) ↔ ESP32-CAM (sensing/motor)
An integrated system combining real-time streaming + on-device AI + IoT control.
Functions
ESP32 handles camera and motor control.
Android handles AI vision processing and higher-level control logic.
Real-time bidirectional data pipeline built on WebSocket.
https://app.eraser.io/workspace/0UpgXOZiXCBAeFqFNky4
Everyone's Checkout Counter
Created with Eraser
app.eraser.io
[ Software Demo Video ]
https://drive.google.com/file/d/10CVS8S6XO9EUQi-n6R6jrM9jR2YQ8nij/view?usp=drive_link
[ Hardware Implementation Status ]
[ Progress Status ]
Model fine-tuning 100%
Building the admin for model fine-tuning 1-0%
Mechanical design 100%
Hardware unit testing 100%
Android app development 100%
