✋ Gesture-Controlled Infotainment System

A touch-free gesture-controlled infotainment system that uses real-time hand tracking and gesture recognition to provide an interactive interface for Music, Navigation, Settings, and Calls.

Developed as a team mini-project, the system explores gesture-based human-computer interaction using MediaPipe Hands, OpenCV, JavaScript, and web technologies.

✨ Features

• Real-time hand tracking using a webcam
• Hand landmark detection using MediaPipe Hands
• Index-finger-controlled virtual pointer
• Thumb and index finger pinch detection
• Gesture-based interaction with supported interface elements
• Music playback and media controls
• Volume adjustment
• Navigation interface with interactive maps
• Destination search
• Calls and contacts interface
• Settings and display controls
• Keyboard and mouse-based interaction for testing
• Real-time visualization of detected hand landmarks

🧠 How It Works

The project contains both a Python-based gesture prototype and a browser-based infotainment interface.

Python Gesture Interface

1. OpenCV captures live video from the webcam.
2. Each frame is processed using MediaPipe Hands.
3. MediaPipe detects and tracks hand landmarks in real time.
4. The index fingertip coordinates are used as a virtual pointer.
5. The distance between the thumb and index finger is used to detect a pinch gesture.
6. Detected hand landmarks and the virtual cursor are displayed on the infotainment interface.
7. Music, Navigation, Settings, and Calls screens can be switched using keyboard controls.

Browser Interface

The browser interface uses MediaPipe Hands through JavaScript to process webcam input and support gesture-driven interaction.

The interface includes:

• Gesture-controlled pointer interaction
• Pinch detection
• Music playback controls
• Playlist handling
• Volume controls
• Playback progress control
• Interactive navigation
• Destination search
• Calls and contacts interface
• Settings controls
• Keyboard and mouse interaction for testing

🛠️ Tech Stack

Computer Vision

• Python
• OpenCV
• MediaPipe

Frontend

• HTML5
• CSS3
• JavaScript

Maps and Navigation

• Leaflet.js
• OpenStreetMap
• Nominatim

📂 Project Structure

Gesture_App/
├── index.html
├── infotainment_gestures.py
├── script.js
├── style.css
├── song1.mp3
├── song 2.mp3
├── song 3.mp3
├── .gitignore
└── README.md

🚀 Getting Started

1. Clone the Repository

git clone https://github.com/parnika0110/Gesture_App.git
cd Gesture_App

2. Install Python Dependencies

Install the dependencies required for the Python gesture prototype:

pip install opencv-python mediapipe

3. Run the Python Gesture Interface

python infotainment_gestures.py

Make sure your computer has a working webcam and allow camera access when required.

4. Run the Web Interface

Open index.html using a local web server.

For example, using Python:

python -m http.server 8000

Then open the local server address displayed in the terminal.

Allow camera access in the browser when prompted so that MediaPipe can process the webcam feed.

Some optional browser functionality references PHP API endpoints for songs and contacts. These backend API files are not included in the current repository.

🎮 Controls

Python Interface

The Python prototype supports the following keyboard controls:

• M — Music
• N — Navigation
• S — Settings
• C — Calls
• Q — Quit application

Gesture Controls

• Move index finger — controls the virtual pointer
• Bring thumb and index finger together — detects a pinch gesture

The browser implementation additionally uses gesture detection for interaction with supported interface controls.

🎵 Music

The infotainment interface provides music-related functionality including:

• Play and pause
• Previous and next track
• Stop playback
• Volume adjustment
• Playback progress
• Playlist handling
• Local audio support

Sample audio files are included in the project for demonstrating the music interface.

🗺️ Navigation

The browser interface uses Leaflet.js with OpenStreetMap tiles to provide interactive map visualization.

Destination search uses Nominatim to search for locations entered by the user.

📞 Calls

The infotainment interface includes a Calls section designed to demonstrate an in-vehicle calling interface.

The Python prototype provides simulated incoming-call controls, while the browser interface contains functionality for working with contact information.

⚙️ Settings

The Settings interface demonstrates controls for infotainment preferences and display-related options.

The browser implementation also contains interactive controls that can be manipulated through standard input or supported gesture interactions.

🔒 Privacy

Camera input is used for real-time hand tracking and gesture recognition.

Users should allow webcam access only when running the gesture-controlled functionality and should review browser camera permissions when using the web interface.

🔮 Future Enhancements

• Improve gesture recognition accuracy
• Add additional custom gestures
• Support multi-hand gesture recognition
• Integrate a complete backend for contacts and media
• Add real calling functionality
• Improve navigation and route guidance
• Expand gesture-based media controls
• Improve responsiveness across different devices
• Add more configurable gesture actions

🤝 Collaboration

This project was developed collaboratively as a team mini-project. The original Git commit history is preserved to maintain accurate attribution of contributions.

👨‍💻 Contributors

• Parnika S M
• Jayashankar P

📄 Note

This project was developed for educational and learning purposes.