Predicto is a full-stack web application that turns a user's live facial expression into a mood-aware music playlist. The browser accesses the device camera, feeds the video stream into a MediaPipe Face Landmarker model running in VIDEO mode, and continuously extracts facial blendshape scores — numeric signals for smile intensity, jaw openness, brow movement, and frown depth. Those raw signals are evaluated against explicit threshold rules to classify the current expression into one of six application-level emotional states. The detected state is mapped to a supported mood category, which becomes the query parameter for a backend music request.
The backend is a Node.js / Express 5 REST API that authenticates users with JWT tokens delivered as HTTP-only cookies, stores user accounts and song metadata in MongoDB via Mongoose, writes revoked tokens to Redis on logout, and orchestrates uploaded audio through a media-processing pipeline before persisting assets in ImageKit. The pipeline reads embedded ID3 tags, extracts any embedded album artwork, uploads audio and poster separately, normalizes the track title, and saves the resulting song record with its mood category. Authenticated users who land on the detection or music pages are served mood-matched songs that stream directly from ImageKit CDN URLs into an in-browser audio player with full playback controls.
- Overview
- Core Capabilities
- Computer Vision Pipeline
- Emotion-to-Mood Classification
- Recommendation Flow
- System Architecture
- Frontend Architecture
- Backend Architecture
- Data Architecture
- Security & Session Management
- Media Architecture
- Tech Stack
- Engineering Highlights
- Product Experience
- Responsive Design
- Future Roadmap
- Author
Predicto sits at the intersection of browser-based computer vision and music discovery. Rather than asking a user to self-report a mood, Predicto observes it through the camera in real time. MediaPipe performs sub-second facial landmark and blendshape inference directly in the browser, the application layer classifies the resulting signals into an emotional state, and that state is translated into one of five supported mood categories. A credentialed API request fetches the corresponding playlist from the backend, and the user immediately enters an interactive music experience — all within a cohesive, visually polished single-page application.
The project demonstrates a complete engineering story: real-time vision inference in the browser → application-level emotion classification → authenticated REST API → mood-filtered data retrieval → cloud media delivery → interactive audio player. Every layer — frontend architecture, routing protection, backend controllers, data modeling, upload processing, and media storage — is implemented from scratch without abstraction shortcuts.
When a user navigates to the detection page (a protected route), the browser requests camera access via the MediaDevices API. The live video stream is assigned to an HTML video element and fed into MediaPipe Face Landmarker running in VIDEO mode with blendshape output enabled. An animationFrame-driven inference loop continuously calls detectForVideo against the current video timestamp, extracts the blendshape score map from the first detected face, and evaluates those scores through a classification function. The resulting emotion label updates in real time on screen. When the component unmounts, all active animation frames are cancelled and every camera track is explicitly stopped, releasing the hardware resource.
The classification layer reads five groups of blendshape signals from the MediaPipe output:
| Signal Group | Blendshapes Sampled |
|---|---|
| Smile intensity | mouthSmileLeft, mouthSmileRight (max of pair) |
| Jaw openness | jawOpen |
| Brow raised | browOuterUpLeft, browOuterUpRight (max of pair) |
| Brow pressed down | browDownLeft, browDownRight (max of pair) |
| Frown depth | mouthFrownLeft, mouthFrownRight (max of pair) |
These numeric scores (0–1) are evaluated against explicit threshold rules in a priority-ordered chain:
| Detected Emotion | Primary Conditions |
|---|---|
| 😁 Very Happy | Smile > 0.75 and Jaw > 0.30 |
| 😊 Happy | Smile > 0.45 |
| 😲 Surprise | Jaw > 0.50 and Brow raised > 0.38 |
| 😠 Angry | Brow down > 0.45 and Frown > 0.30 |
| 😢 Sad | Frown > 0.0045 |
| 😐 Neutral | Default fallback |
A separate mapping layer then converts each emotional state string into one of five supported mood category keys — very happy, happy, surprised, sad, neutral — that serve as the query parameter to the backend. "Angry" and "Surprise" both route to the surprised music bucket, reflecting the application's music-category taxonomy rather than a one-to-one psychological mapping.
Once an emotion is detected and its mood key resolved, the user triggers a navigation event that carries the mood key as a URL search parameter. The music page reads that parameter, calls the authenticated song API with the mood value, and receives an ordered array of song records from the backend. Each record contains a track title, an audio CDN URL, and a poster CDN URL. The full playlist renders immediately into the player interface, with the first track auto-selected and ready to play.
The music experience delivers a fully functional audio player built with the native HTML <audio> element, managed through React state and refs:
- Play / Pause toggle with live icon state
- Previous / Next track navigation with wrap-around
- Seek bar with real-time scrubbing and timestamp display (M:SS)
- Duration tracking via
onLoadedMetadataandonTimeUpdateevents - Auto-advance to the next track when the current track ends via
onEnded - Poster artwork rendered from the ImageKit CDN URL with a graceful fallback to a default poster image
- Active track highlighting and a live playing indicator in the playlist
- Glowing poster animation toggled by playing state via a CSS class
User registration and login are handled by the backend, which issues a short-lived JWT (1-hour expiry) stored as an HTTP-only cookie. On every page load the frontend's AuthContext silently calls the profile endpoint using the cookie; if the token is valid, the user state is hydrated and the session is considered active. If the profile request fails, the user state is set to null.
Protected routes — the emotion detector and the music page — are wrapped by a Protected component that consults the auth context. If the session has not loaded, a loading screen is displayed. If the session is absent, the component redirects to /login using React Router's <Navigate>. Authenticated users proceed directly to the protected experience.
Administrators can upload audio tracks through an authenticated upload endpoint. The pipeline processes each upload through the following stages:
- Multipart reception — Multer accepts the audio file as an in-memory buffer (20 MB limit, no disk writes)
- ID3 metadata extraction —
node-id3reads the buffer synchronously and extracts embedded tags including the track title and any embedded image buffer - Parallel cloud upload — the audio buffer and the poster image buffer (when present) are uploaded concurrently to ImageKit using
Promise.all, each routed to dedicated folders - Fallback poster — tracks without embedded artwork receive a default poster reference
- Title normalization — the extracted ID3 title undergoes multi-step cleaning: BOM stripping, underscore expansion, removal of bitrate and quality suffixes, bracket keyword removal, and whitespace normalization
- Metadata persistence — the cleaned title, audio CDN URL, poster CDN URL, and mood category are saved as a new Song document in MongoDB
The complete vision flow, from hardware to music query:
Camera Stream
→ MediaPipe Face Landmarker (VIDEO mode, 1 face, blendshapes enabled)
→ 52+ Blendshape Category Scores per Frame
→ 5-Group Signal Extraction (smile, jaw, brow-up, brow-down, frown)
→ Priority Threshold Chain → Emotional State Label
→ Mood Key Mapping
→ Backend Music Query
The vision layer runs entirely in the browser — no camera frames are transmitted to the server. MediaPipe's WebAssembly runtime is loaded from the jsDelivr CDN and the Face Landmarker model weights are fetched from Google's MediaPipe model repository on first initialization. The model runs in VIDEO mode, which enables frame-timestamp-based inference optimized for continuous live input rather than single-image processing.
The inference loop is driven by requestAnimationFrame, which ties execution to the display refresh cycle. At each frame, detectForVideo is called with the current performance timestamp. If a face is detected, the first face's blendshape category array is reduced into a score lookup object. The classification function then reads five composite signal values and applies the ordered threshold chain. The result propagates through React state to update the live emotion label and the derived mood display panel.
On component unmount, cancelAnimationFrame halts the inference loop and each camera track is explicitly stopped — preventing the browser's camera-in-use indicator from persisting after the user navigates away.
The complete path from facial expression to playing music:
1. Detection — MediaPipe processes the live video stream and produces blendshape scores per frame.
2. Classification — Application threshold logic converts blendshape scores into a named emotional state (Very Happy, Happy, Surprise, Angry, Sad, or Neutral).
3. Mood Mapping — A deterministic lookup function converts the emotional state string into one of five supported mood category keys used by the backend.
4. Navigation — The user confirms their mood and navigates to /songsbymood?mood=<key>, carrying the mood as a URL parameter.
5. API Request — The music page reads the mood parameter and calls the authenticated backend song endpoint with it as a query string.
6. Data Retrieval — The Express controller queries MongoDB for all Song documents matching the requested mood category and returns the array.
7. Playback — The returned song records populate the playlist. The first track auto-selects and the full player UI activates.
| Layer | Responsibility | Technologies |
|---|---|---|
| Presentation | Interactive web experience | React 19, Vite 7, Sass |
| Routing | Client-side navigation and protected routes | React Router 7 |
| Computer Vision | Facial expression inference from live camera | MediaPipe Tasks Vision |
| State / Context | Authentication session and song playback state | React Context API |
| API Communication | Frontend-to-backend requests with credentials | Axios |
| API Layer | Authentication and music REST endpoints | Node.js, Express 5 |
| Authentication | Stateless session tokens and access control | JWT, HTTP-only cookies, bcrypt |
| Data Layer | Persistent user accounts and song metadata | MongoDB, Mongoose |
| Session Revocation | Logout token blacklist storage | Redis via ioredis |
| Upload Processing | Multipart audio handling and metadata extraction | Multer, node-id3 |
| Media Storage | Audio and poster asset hosting and CDN delivery | ImageKit |
The frontend is organized into four feature-scoped domains, each encapsulating its own components, pages, hooks, services, and styles. This structure enforces clear boundaries between distinct product areas and avoids shared-logic entanglement.
Auth — Handles user registration and login pages, the Protected route wrapper, the useAuth custom hook that abstracts login/register/logout actions, the AuthContext provider that maintains global session state across the application, and a dedicated Axios API layer for authentication endpoints. All HTTP requests include credentials so cookies are transparently transmitted.
Expression — Owns the emotion detection page, the live video display component, and the emotion.js utility module that encapsulates the full MediaPipe initialization, detection loop, classification logic, and cleanup sequence. Keeping this logic outside the React component tree makes the vision pipeline self-contained and cleanly separable from rendering concerns.
Home — Contains the marketing landing page with hero section, step-by-step flow, feature showcase, and emotion grid. Also houses the songCatalog.js domain module — the single source of truth for the moodCatalog array and the getMoodKeyFromEmotion mapping function shared across the Expression and Song features.
Song — Manages the mood-matched music page, the useSong hook that wraps song fetching, the SongContext provider for cross-component song state, and the song API service. The music player is fully self-contained within the page component, managing audio state, playback controls, track selection, and progress tracking locally.
- Global Sass (
main.scss) establishes base reset, typography, scrollbar theming in electric violet, and smooth scroll behaviour - Design tokens (
_variables.scss) define the full color palette — deep navy#0D1B2A, electric violet#9D4EDD, crimson#BF092F— alongside spacing, radius, and transition values - Sass mixins (
_mixins.scss) provide reusable patterns:glass()for glassmorphism cards,gradient-text()for violet-to-crimson headline gradients,glow()for radial aura effects, and flex shorthand helpers - Route-level protection is implemented through a
Protectedcomponent that reads auth loading state before rendering guarded content, preventing flash-of-unauthorized-content on page load
The backend follows a layered architecture with strict separation between routing, controller logic, middleware, data modeling, service abstraction, and configuration.
Route Layer — Two Express routers handle all application traffic. The auth router exposes public registration and login endpoints alongside token-protected profile and logout handlers. The song router applies the verifyToken authentication middleware globally at the router level, ensuring every song endpoint — retrieval and upload — requires a valid session.
Controller Layer — Auth and song controllers contain all business logic. The auth controller handles user lookup, bcrypt comparison, JWT signing, cookie management, and Redis blacklist writes on logout. The song controller orchestrates the full upload pipeline — ID3 extraction, concurrent ImageKit uploads, title cleaning, and MongoDB persistence — as well as mood-filtered song retrieval.
Middleware Layer — The verifyToken middleware extracts the JWT from the request cookie, verifies it against the application secret, and attaches the decoded payload to the request object. The upload middleware configures Multer with in-memory storage and a 20 MB file size cap.
Model Layer — Mongoose schemas define the User model (username, email, hashed password with select: false) and the Song model (title, audio URL, poster URL, mood with an enum constraint). Both models enforce runtime validation.
Service Layer — A dedicated storage.service.js module abstracts all interaction with the ImageKit SDK. Controllers call this service without any direct SDK dependency, keeping the upload orchestration decoupled from the storage provider.
Configuration Layer — Separate config modules initialize the Mongoose connection and the ioredis Redis client, each with connection event logging and error handling.
The User document stores the account identity required for authentication: a unique username, a unique email address, and a bcrypt-hashed password. The password field is excluded from default query projections (select: false) and must be explicitly requested during login verification — preventing accidental password exposure in profile or listing responses.
The Song document stores everything needed to render and play a track: a cleaned, human-readable title; a direct audio URL pointing to the ImageKit CDN; a poster URL pointing to the extracted album artwork or the default fallback; and a mood category field constrained to the application's supported values (neutral, happy, sad, surprised, very happy, or an empty string for uncategorized). The mood field's enum constraint enforces data integrity at the database layer, preventing invalid categories from being inserted.
| Mechanism | Implementation |
|---|---|
| Password hashing | bcrypt with a cost factor of 10 |
| Authentication tokens | JSON Web Tokens signed with an application secret, 1-hour expiry |
| Token transport | HTTP-only cookies; secure flag enabled in production |
| Route protection (frontend) | Protected component blocks unauthenticated access and redirects to /login |
| Route protection (backend) | verifyToken middleware applied globally to all song routes and private auth routes |
| CORS policy | Credentials-enabled CORS with an explicit allowlist of frontend origins |
| Session revocation | On logout, the token is written to Redis with a matching 1-hour TTL as a revocation record |
| Upload size limiting | Multer enforces a 20 MB maximum per uploaded file |
Accuracy note on Redis: The
verifyTokenmiddleware validates the JWT signature and expiry but does not currently query the Redis blacklist on each incoming request. Redis is written to during logout, providing a persistent revocation record. Extending the middleware to consult the blacklist per request would enable true immediate token invalidation — a natural future enhancement.
Audio and artwork travel through a multi-stage pipeline before reaching the database:
Multipart Upload (≤ 20 MB)
→ Multer memory storage (buffer only, no disk writes)
→ node-id3 reads ID3 tags synchronously from the buffer
├── Track title extracted
└── Embedded image buffer extracted (when present)
→ Parallel Promise.all upload to ImageKit:
├── Audio buffer → ImageKit /Predicto/songs/
└── Poster buffer → ImageKit /Predicto/posters/
(fallback: default poster reference if no embedded art)
→ CDN URLs returned from ImageKit
→ Title normalization applied:
strips BOM / zero-width characters
expands underscores to spaces
removes bitrate / quality suffixes (MP3, 320K, kbps)
removes media-type bracket keywords (Official Video, Lyrical, HD)
collapses repeated delimiters, trims edges
→ Song document persisted in MongoDB:
{ title (cleaned), url, posterUrl, mood }
At playback time, audio streams directly from ImageKit's CDN to the browser — the backend is not in the media streaming path. This keeps backend load minimal and leverages ImageKit's global delivery infrastructure for audio delivery.
| Domain | Technology | Role |
|---|---|---|
| Frontend Framework | React 19 | Component-driven interactive UI |
| Build Tool | Vite 7 | Frontend bundling and development server |
| Routing | React Router 7 | Client-side navigation and protected-route logic |
| Styling | Sass | Modular design system with variables, mixins, and feature-scoped stylesheets |
| Computer Vision | MediaPipe Tasks Vision 0.10 | Face Landmarker model, VIDEO-mode blendshape inference |
| HTTP Client | Axios | API communication with credential-forwarding |
| Backend Runtime | Node.js | Server-side JavaScript runtime |
| API Framework | Express 5 | REST API with router-level middleware |
| Database | MongoDB | Document storage for users and songs |
| ODM | Mongoose 9 | Schema validation, data modeling, and query layer |
| Cache / Revocation | Redis via ioredis 5 | Token revocation blacklist storage |
| Authentication | JWT (jsonwebtoken 9) | Stateless identity assertions |
| Password Security | bcrypt 6 | Adaptive password hashing |
| Upload Handling | Multer 2 | Multipart form data processing with memory storage |
| Metadata Processing | node-id3 0.2 | ID3 tag and embedded artwork extraction from audio buffers |
| Media Storage | ImageKit (Node.js SDK 7) | Audio and poster CDN hosting and delivery |
| Typography | Google Fonts — Outfit, Inter | Display and body typefaces |
Browser-based computer vision without a backend roundtrip. MediaPipe runs as a WebAssembly module in the browser. Each inference frame processes the camera stream locally, producing blendshape scores in real time without sending any image data to the server.
Continuous animation-frame inference loop. The detection cycle is tied to requestAnimationFrame rather than a polling interval, aligning inference with the display refresh rate and avoiding unnecessary CPU work between frames.
Five-signal, threshold-based expression classification. Rather than treating MediaPipe's output as a black box label, the application reads raw blendshape scores and applies an explicit, readable priority chain. The logic is transparent, auditable, and adjustable without any model retraining.
Feature-oriented React architecture. Each product domain — Auth, Expression, Home, Song — owns its own components, pages, hooks, services, and styles. There are no monolithic files or sprawling shared directories. Each feature composes its own context, custom hook, and API module.
Layered backend with service abstraction. Controllers, routes, middleware, models, services, and configuration are separated into distinct layers. The storage.service.js module abstracts the ImageKit SDK entirely; swapping storage providers would require changes in exactly one file.
Parallel media uploads. When an audio file contains embedded artwork, the audio buffer and poster buffer are uploaded to ImageKit concurrently via Promise.all rather than sequentially — reducing total upload latency without adding complexity.
ID3 metadata pipeline with title normalization. Rather than requiring manual metadata entry, the upload endpoint reads embedded ID3 tags from the audio buffer, recovers the track title and any embedded album art, then applies a multi-step normalization pass to produce clean, human-readable display titles free of encoding artifacts and quality-suffix noise.
Redis-backed revocation storage. Logout writes the current token to Redis with a TTL matching the token's original expiry, providing a persistent revocation record that survives server restarts within the token lifetime.
Credentialed Axios instances. Both the auth and song API modules create Axios instances with withCredentials: true, ensuring cookies are forwarded on every request without manual header management at each call site.
Sass design system. The UI is styled through a structured Sass architecture: global base styles, a color token file defining the full deep-navy / electric-violet / crimson palette, a mixin file providing reusable glassmorphism, gradient-text, glow-aura, and flex utilities, and feature-scoped stylesheets that consume those tokens consistently.
User journey:
Discover → Authenticate → Enable Camera → Live Detection → Mood Resolved → Fetch Playlist → Play Music
The landing page presents Predicto through a multi-section layout: an animated hero with floating emotion tags, a four-step "how it works" walkthrough, a feature showcase with a blendshape confidence visualization mockup, an emotion recognition grid, and a call-to-action banner — all rendered in the application's signature dark visual language.
Visual identity at a glance:
| Element | Implementation |
|---|---|
| Base canvas | Deep navy #0D1B2A |
| Primary accent | Electric violet #9D4EDD (scrollbar, badges, glow orbs, gradient) |
| CTA accent | Crimson #BF092F (buttons, gradient counterpoint) |
| Cards | Glassmorphism — backdrop-filter: blur(), translucent navy borders |
| Background depth | Large animated radial orbs in violet and teal |
| Texture | CSS grid overlay for subtle HUD-inspired visual layer |
| Active player poster | Glows and pulses via playing-state CSS class |
| Camera feed | Mirrored (scaleX(-1)) for natural selfie orientation |
| Headings | Outfit (700–800 weight) |
| Body text | Inter (400–600 weight) |
| Gradient headlines | Crimson-to-violet on hero and feature headings |
The application adapts its layout across desktop, tablet, and mobile breakpoints through media queries embedded in each feature's Sass stylesheet.
The detection page switches from a two-column grid (camera stage left, mood panel right) to a single stacked column at 992 px and below, with adjusted padding to maintain usability on smaller screens.
The music player page condenses its layout into a single column on narrower viewports, with the poster, controls, and progress bar remaining fully accessible and touch-friendly.
The authentication pages use a split-panel layout on desktop — brand panel left, form right — and collapse to a single centred card on mobile, replacing the brand panel with a compact header inside the form card.
The navigation bar maintains its logo, link group, and CTA layout across standard breakpoints.
The following areas represent natural extensions of the current architecture, not existing features:
- Redis blacklist enforcement at request time — extending
verifyTokento query the Redis blacklist on each protected request, enabling true immediate token invalidation after logout - Refresh token rotation — short-lived access tokens paired with longer-lived refresh tokens to allow session extension without re-login
- Richer mood taxonomy — expanding beyond five mood buckets with more granular emotional states and corresponding song categories
- Confidence-weighted track selection — using blendshape score magnitude to influence playlist weighting rather than purely categorical lookup
- Multi-frame averaging — aggregating blendshape scores across consecutive frames to increase classification stability and reduce single-frame noise
- Per-user play history — storing playback events to surface personalization signals within a mood category
- Admin song management dashboard — a protected interface for uploading, tagging, and reviewing the music catalog
- Analytics layer — tracking mood detection frequency, track play duration, and mood-shift patterns
- Accessibility enhancements — keyboard-navigable player controls, screen reader announcements for emotion state changes, and reduced-motion alternatives for animated backgrounds
Raza — Full-Stack Engineer
Built Predicto as a portfolio project demonstrating the integration of browser-based computer vision, real-time inference, full-stack authentication, cloud media processing, and a polished interactive product experience.
🌐 Live deployment referenced at
predicto.skramizraza.tech
Predicto — where your face picks the playlist.
