Dhwani: Deepfake Audio Detection API 🎙️

Welcome to Dhwani, an advanced API and machine learning pipeline engineered to detect deepfake and synthetically generated audio. By leveraging state-of-the-art transformer architectures and graph attention networks, Dhwani provides a robust defense against AI voice cloning, audio spoofing, and synthetic media.

---

🧠 Model Architecture & Training Process

Our detection engine is built upon a hybrid neural network architecture, combining self-supervised learning with an advanced graph-based backend classification system.

1. Base Model: Wav2Vec2 XLS-R (300M)

• The front-end feature extractor utilizes Facebook's Wav2Vec2 XLS-R-300m, a massive multilingual self-supervised model.
• During our training phase, the CNN feature extractor layers were frozen to preserve general acoustic representations, while the transformer layers were fine-tuned for anomaly detection.

2. Backend: AASIST (Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention)

• The raw hidden states (1024-dimensional) from XLS-R are passed into a custom Graph Attention Network (GAT) backend.
• This AASIST backend captures complex temporal and spectral artifacts left behind by vocoders and text-to-speech synthesis engines, effectively distinguishing between human vocal tract naturalness and machine generation.

3. Training Optimization

• Hardware & Framework: Trained using PyTorch with Mixed Precision (AMP) to accelerate convergence and reduce VRAM usage.
• Augmentation: To ensure real-world robustness against telephony degradation and compression, our dataset.py pipeline dynamically injects random noise (SNR 10-20dB) and simulates 8kHz GSM/VoIP downsampling.
• Calibration: Post-training, the model undergoes Temperature Scaling calibration to produce reliable probability scores, reducing false positives in production environments.

---

📊 Comprehensive Multilingual Dataset Construction

Because high-quality datasets for deepfake audio in regional languages are scarce, we built a highly diverse, custom multilingual dataset from scratch across five languages (English, Hindi, Tamil, Telugu, Malayalam).

1. Data Sources

We sourced and aggregated audio from several massive open-source repositories to create a balanced dataset:

• Mozilla Common Voice (v24.0): Extracted hundreds of hours of high-quality, genuine human speech recordings for Indic languages directly from raw .tar.gz archives.
• HuggingFace (vdivyasharma/IndicSynth): Streamed thousands of synthetic, AI-generated Text-to-Speech (TTS) spoofing attempts across multiple Indic languages.
• HuggingFace (garystafford/deepfake-audio-detection): Extracted both genuine human speech and generative AI spoofs for the English language baseline.

2. The Audio Processing Pipeline

To ensure our model didn't fail on weird encodings or variable lengths, we engineered a strict standardization pipeline:

1. Decoding & Format Unification: All audio streams and archives were decoded from various formats (MP3, FLAC, WAV) and strictly converted to 16kHz, Mono, Float32 arrays.
2. Silence Truncation (VAD): Voice Activity Detection and energy thresholding logic were applied to trim excess silence and dead air from the ends of clips.
3. Amplitude Normalization: Every clip was normalized to a standard -3dBFS peak amplitude to ensure consistent volume across different microphones and TTS engines.
4. Windowing: Clips were dynamically padded or cropped to a fixed 3-second evaluation window (48,000 samples) to create uniform PyTorch tensors.

3. Data Preparation Scripts (scripts/)

We have included the exact scripts used to harvest, synthesize, and augment this dataset so the community can replicate our work:

• download_real.py: Automatically extracts and standardizes high-quality genuine human voices directly from the massive Mozilla Common Voice archives and HuggingFace streams.
• download_fakes.py: Connects to HuggingFace to stream and download synthetic Text-to-Speech (TTS) spoofing attempts.
• augment_data.py: A robustness engine that deliberately degrades the clean data to simulate real-world conditions. It injects random Gaussian noise and performs speed/pitch perturbations (0.9x and 1.1x resampling) to simulate telephone line distortion and adversarial attacks.

---

📁 Repository Structure

We have cleanly separated our production deployment code from our research and training code to maintain an elegant and scalable repository.

• src/ - Production API. Contains the FastAPI application and ONNX model inference logic (inference.py, main.py).
• training/ - Research & ML Pipeline. Contains the training_notebook.ipynb, dataset loaders, and PyTorch training scripts (train.py, calibrate.py) used to build the model.
• scripts/ - Data Generation. Contains the pipeline to download, extract, and augment the massive multilingual dataset from scratch.
• models/ - Directory designated for storing the trained model weights.
• Dockerfile & docker-compose.yml - Infrastructure configuration to run the API container seamlessly in isolated environments.

---

🚀 How to Run the API (Deployment)

You can spin up the full inference API locally using Docker.

1. Environment Setup & Cloud Logging (.env)

The API integrates with Supabase to maintain a persistent, cloud-based audit trail of all inference requests. Every time an audio file is processed, the system logs the prediction results, confidence scores, and extraction metadata to the database for monitoring and analytics. You must create a .env file in the root directory to connect to your Supabase instance before running the container.

Create a file named .env and add your keys:

SUPABASE_URL=your_supabase_url_here
SUPABASE_KEY=your_supabase_key_here

(Note: Do not commit your .env file to version control. It is already ignored in our .gitignore.)

2. Run with Docker

1. Ensure Docker and Docker Compose are installed on your system.
2. Download Model Weights: Download the pre-trained best_model.onnx file from our HuggingFace Repository and place it directly into the models/ folder.
3. Open a terminal in the root directory and build the container:

   docker-compose up --build

4. The API will initialize and be accessible via your localhost (check src/api/config.py for exact port details, typically port 8000).

---

🔬 Exploring the Training Pipeline

If you want to understand how the model was trained, reproduce the results, or fine-tune it on a new dataset:

1. Navigate to the training/ folder.
2. Open training_notebook.ipynb for the interactive training execution.
3. The underlying logic is highly modularized into dataset.py (data loading and augmentation), train.py (training loop), and calibrate.py (temperature scaling) for readability.

Note: The core model architecture class is shared and strictly maintained in src/model/architecture.py to ensure total consistency between PyTorch training and ONNX deployment.