Frame2Puzzle is an interactive, real-time Computer Vision application and puzzle game powered by OpenCV, MediaPipe Hand Tracking, and a modern Glassmorphism UI. It transforms a live camera capture into an interactive 3x3 puzzle game that can be captured, played, dragged, and solved entirely using natural hand gestures—without needing a mouse or physical controller.
- 1.2s Gesture Hold Verification: Hold a 2-Finger Gesture (Peace / Victory Sign) stably for 1.2 seconds to initiate capture. Features a modern circular progress ring with percentage feedback.
- 3-Second Countdown: Features a glassmorphic central card with a glowing countdown timer and background camera blur.
- Zero-Disk RAM Storage: Photos are stored entirely in RAM as NumPy ndarrays for high performance and privacy.
- Shutter Flash & Photo Preview: Includes a realistic white shutter flash overlay and a sleek Photo Preview Card with drop shadows and a mint accent border.
- Gesture Retake: Show an OPEN PALM gesture (or press
R) at the preview screen to retake the photo.
- Seamless 3x3 Image Slicing: Dynamically slices the captured frame into 9 equal puzzle tiles.
- Seamless Tile Layout: Zero-gap, borderless puzzle tiles for a clean, unified visual puzzle experience.
- 5-Finger GRAB Gesture: Clustered 5-finger centroid gesture recognition for natural tile picking and dragging.
- Real-Time Drag & Snap: Float grabbed tiles across the screen with soft drop shadows and active cyan border highlights.
- Collision Swap Mechanics: Releasing a tile over an occupied grid slot automatically swaps positions with the occupant piece and snaps both tiles into grid alignment.
- Visual Feedback & Animations: Includes origin slot placeholders, hover slot highlights, swap flash animation, and an animated celebration pulse upon completing the puzzle.
- Frosted Glass Panels: Top HUD and bottom status bars feature real-time
GaussianBlurbackground blur and translucent color tinting. - Aesthetic Hand Skeleton Overlay: Minimalist 21-landmark hand skeleton drawn with warm silver translucent lines, subtle joint dots, and teal glowing fingertip rings.
- Glowing Teal Cursor: Interactive GRAB cursor rendered as a soft glowing cyan/teal ring instead of harsh solid shapes.
- Performance Metrics: Integrated MM:SS game timer and move counter.
The codebase is organized into a clean, modular package structure (src/):
Frame2Puzzle/
├── src/
│ ├── tracking/ # Hand tracking & gesture recognition subpackage
│ │ ├── __init__.py
│ │ ├── hand_tracker.py # MediaPipe 21-landmark detector & skeleton overlay renderer
│ │ └── gesture_recognizer.py # Geometric gesture classification & temporal smoothing filter
│ │
│ ├── game/ # Game mechanics & UI rendering subpackage
│ │ ├── __init__.py
│ │ ├── capture_manager.py # Pre-puzzle state machine (STREAMING -> COUNTDOWN -> CAPTURED)
│ │ └── puzzle_manager.py # 3x3 Puzzle engine, grid layout, drag interaction, timer & HUD
│ │
│ ├── __init__.py
│ └── main.py # Main application entry point & OpenCV webcam event loop
│
├── scripts/
│ └── get_model.sh # Automatic downloader for MediaPipe model asset
├── hand_landmarker.task # MediaPipe Hand Landmarker binary model asset
├── Makefile # Build, setup, run, and clean workflow targets
├── requirements.txt # Project dependencies
├── LICENSE # Open-source MIT License
└── README.md # Project documentation
- Core Language: Python 3.10+
- Computer Vision & UI Rendering: OpenCV (
opencv-python) - Hand Tracking ML: MediaPipe Tasks Vision (
mediapipe) - Image & Data Processing: NumPy & Math
- Build & Automation: GNU Make & Bash
- Operating System: Linux / macOS / Windows
- Python 3.10 or higher installed
- Webcam connected to your system
- GNU
makeutility
Simply run make in your terminal to automatically set up the virtual environment, download the MediaPipe model asset, and launch the application:
make# 1. Create Virtual Environment & Install Dependencies
make setup
# 2. Download MediaPipe Hand Landmarker Model Asset
make get-model
# 3. Launch the Application
make run- Action: Show a 2-Finger Gesture (Index + Middle finger extended up, Ring & Pinky folded).
- Feedback: Hold the gesture stably for 1.2 seconds. A circular progress ring will fill up to 100%.
- Action: Hold your pose and get ready!
- Feedback: A 3-second countdown timer (
3... 2... 1...) will run over a frosted glass panel with camera blur depth.
- Photo Preview: View the captured photo preview card on the right side.
- Start Puzzle: Show a GRAB gesture (or press
SPACE) to generate and shuffle the 3x3 puzzle. - Retake Photo: Show an OPEN PALM gesture (all 5 fingers extended spread out) or press
Rto retake.
- Grab Tile: Clench your hand into a GRAB gesture (bring all 5 fingertips close together) over any puzzle tile.
- Drag Tile: Move your hand while holding the GRAB gesture to float the tile across the board.
- Swap Tile: Move the floating tile over your target grid slot and release the GRAB gesture (open your fingers) to drop and swap positions with the occupant tile.
- Complete Puzzle: Arrange all 9 tiles into their original positions. The timer will stop and a celebration glow will border the completed board!
| Key | Action |
|---|---|
SPACE |
Start Puzzle game from CAPTURED preview screen |
R |
Retake photo or Restart puzzle game |
Q / ESC |
Quit the application |
The game lifecycle is governed by a strict 4-state state machine:
stateDiagram-v2
[*] --> STREAMING
STREAMING --> COUNTDOWN: 2-Finger Gesture Held for 1.2s
COUNTDOWN --> CAPTURED: 3s Timer Expired (RAM Capture)
CAPTURED --> STREAMING: OPEN PALM Gesture or 'R' Key
CAPTURED --> PUZZLE_GAME: GRAB Gesture or SPACE Key
PUZZLE_GAME --> STREAMING: 'R' Key (Restart)
| Command | Description |
|---|---|
make / make all |
Full pipeline: Downloads model asset, sets up .venv, and launches application |
make run |
Runs the application using .venv and src/main.py |
make setup |
Provisions Python .venv and installs packages from requirements.txt |
make get-model |
Downloads hand_landmarker.task binary asset if missing |
make clean |
Removes .venv and Python __pycache__ directories |
make help |
Displays Makefile targets and help menu |
The MediaPipe Hand Landmarker model file (hand_landmarker.task) is a binary asset (~7.8 MB) and is gitignored to keep the repository lightweight.
If hand_landmarker.task is not found locally when launching, the application and ./scripts/get_model.sh script will automatically fetch it from official Google MediaPipe storage candidates.
This project is open-source under the MIT License. Feel free to use, modify, and distribute it!