This blog post is from Brenda Zhuang, Software Development Manager, and Jayden Shin, Senior Application Engineer at MathWorks.

 

Ready to pick up a childhood game and play it with a robot—on stage at a tech conference?

At this year’s Korea MATLAB Expo, while most demos showcased industrial automation, deep learning pipelines, or high-performance simulations, Jayden brought something a little different: a small blue robot trained to play Cheonggi-Baekgi—a classic Korean game of strategy, speed, and team spirit.

Video: Robot playing Cheonggi-Baekgi with Jayden

The project was more than just a playful throwback. It demonstrated how accessible and expressive embedded AI has become. With tools like MATLAB, Simulink, NVIDIA® Jetson™, and Arduino®, Jayden turned a nostalgic idea into a live, interactive demo of real-time AI in action.

And let’s be honest—there’s something undeniably delightful about watching a robot try to master a game that once ruled the schoolyard.

 

What’s the Game?

Cheonggi-Baekgi is a children’s game that is similar to “Simon Says” but with flags, popular in Korea and Japan. Players hold a blue flag and a white flag in each hand. “Simon” calls out commands like “White flag up!” or “Blue flag down!”  The players follow the instructions by raising or lowering their blue and white flags accordingly. If they make a mistake or respond too slowly, they are out of the game. The game continues with increasingly rapid and tricky commands. It is exciting and fun!

 

Behind the Scenes: How the Robot Learns to Play

At its core, this demo is an example of an on-device AI agent—a compact system that can understand speech, make decisions, and control the motors in a robot to respond to the voice commands. Like a good companion, the agent is conversational, responsive, and in control of physical systems.

Here’s how the system works:

• Automatic Speech Recognition (ASR), natural language processing (NLP), and language model integration are handled using open-source frameworks.
• The decision-making logic and robot control are implemented using MATLAB and Simulink tools.
• By separating the decision model from the language model, the system reduces training complexity and avoids the unpredictability that can come from relying solely on large language models (LLMs) for control tasks.

This hybrid approach makes the system more stable, efficient, and easier to adapt for other applications—whether you're building a game-playing robot or a voice-controlled assistant.

Figure: System of open-source models, AI tools, and Model-Based Design for edge GPU and Arduino

 

Tools at work: Challenges and Iterations

The development of the control logic was quickly accomplished with Model-Based Design. Using GPU Coder and Jetson Support Package, the entire system was easily tested and deployed to the Nvidia Jetson platform. Using Simulink, the decision-making model and the control logic were simulated on desktop and with the Hardware-in-the-loop (HIL).

For ASR, we used OpenAI™ Whisper. While Whisper provides high recognition accuracy for relatively long sentences, it had difficulty accurately recognizing short and fast voice commands, such as those used in the game. In some cases, short utterances were misrecognized as similar but incorrect words. Since the model could not be fine-tuned, we implemented the exception handling logic to manage these misrecognitions.

 

Beyond the Game

What makes this demo more than just a fun experiment is its potential as a template for embedded intelligence, that is a modular, scalable architecture that can help you build complex AI-driven systems for robotics, autonomous vehicles, and innovative embedded applications with multimodal interface. The main building blocks in this template are:

• Perception (e.g., speech recognition, gestures, touch, and computer vision)
• Understanding (e.g., NLP and intent recognition)
• Decision-making (e.g., rule-based logic and neural networks)
• Control (e.g., motor commands and system actuation)

This layered approach mirrors how many intelligent systems are designed in industry, and it’s fully supported by the MathWorks ecosystem.

• MATLAB & Simulink: Ideal for modeling, simulation, and rapid prototyping of control systems and AI workflows with open-source integration.
• Deep Learning Toolbox: Enables training and deployment of neural networks for tasks like classification, regression, and decision-making.
• Embedded Coder & GPU Coder: Allow you to generate optimized code for deployment on embedded CPUs and GPUs, including platforms like NVIDIA Jetson.
• Hardware Support Packages: Enable direct testing on hardware targets such as Raspberry Pi, Arduino, and Jetson by providing dedicated interfaces.

Try It Yourself

Curious to build your own AI-powered robot? You can get started by downloading the example files and watching the demo introduction video. Once you’ve unzipped the files, open the ReadMe.mlx file in MATLAB to walk through the setup.

Here’s what you’ll need:

• A Jetson board (e.g., Orin)
• MATLAB with Deep Learning Toolbox
• A basic robot platform (or even a simulated one)
• A bit of curiosity and a willingness to experiment

What will you build?

In a world where AI is often framed in terms of productivity and performance, it’s easy to forget that it can also be personal. It can reflect who we are, where we come from, and what we care about. Whether it’s a childhood game, a local tradition, or a personal challenge, embedded intelligence gives us the tools to bring those stories to life in new and unexpected ways.

And the best part? With platforms like MATLAB, Jetson, and open-source models and MBD tools, the barrier to entry has never been lower.

What would your robot learn to do? Whatever it is, the tools are here—and the possibilities are wide open.

Leave your ideas in the comments section!