Student Project Showcase logoStudent Project Showcase
← Back to all projects
Computer VisionAug 2026

AI Rock Paper Scissors — Real-Time Gesture Recognition

by Chaemin Jun

AI Rock Paper Scissors — Real-Time Gesture Recognition
PythonTensorFlowKerasOpenCVNumPyCNNComputer VisionDeep Learning

Hold your hand up to the camera and the computer sees it, names it, and beats it. Behind that simple moment is a complete machine-learning pipeline that Chaemin built from nothing — the same sequence of steps a professional ML engineer follows.

**He built his own dataset.** Rather than downloading one, he gathered and labelled images of rock, paper and scissors gestures under varying lighting, then standardised, resized and normalised them into consistent image matrices with NumPy. He also applied data augmentation — rotating, shifting and flipping the images — so the model learned the *shape* of a gesture instead of memorising his training photos. Understanding why augmentation prevents overfitting is a concept many university students meet for the first time in their second year.

**He trained a neural network.** Using TensorFlow and Keras, he implemented a Convolutional Neural Network — the architecture designed to pick out spatial features in images, and the same family of model behind face unlock and medical imaging. He trained it with categorical cross-entropy loss and the Adam optimiser, then exported the trained weights to an `.h5` model file so it could run fast enough for live video.

**He deployed it to real hardware.** With OpenCV he opened a live webcam stream and ran every frame through the model as it arrived, drawing bounding boxes, the predicted gesture and its confidence score straight onto the video. Running a heavy deep-learning model on live video without lag is a real engineering constraint, not a tutorial exercise.

**He solved genuine problems.** Cluttered backgrounds and different skin tones both break naive gesture models. He worked through those by tuning his image thresholding — the kind of messy, real-world debugging that separates a working product from a demo that only behaves on the author's desk.

Along the way the project exercises the whole Python pathway: object-oriented design, vectorised operations over multi-dimensional arrays, and a clean architectural separation between data ingestion, model inference and rendering — three independent layers, which is precisely how production ML systems are structured.

For a student, this is the moment machine learning stops being an abstraction. The model isn't downloaded or called through somebody else's API; it was trained on his own data, and it runs on his own webcam.

More like this

Ready to start building?

Book a free trial class with Harvey.

Enroll