🔮 PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning
A research-driven library with high-level APIs, tightly integrated with HuggingFace, and state-of-the-art tools for self-supervised learning.
- 📍 Overview
- 📝 Reference Paper
- 🧠 What is Self-Supervised Learning?
- 🔄 Overall Training Pipeline
- 🧩 System Architecture
- 🚀 Supported Methods
- 📦 Installation
- 🛠️ Usage Tutorial
- 🖥️ Run the Dashboard (Beta)
- 📊 Benchmarks
- 🔧 Extra Superpowers
- 🧬 HuggingFace Example
- 🤝 Collaborators and Advisors
- 📜 License
- 📚 Citation
Say hello to PrismSSL — a library born from late-night debugging sessions, too much coffee, and the realization that self-supervised learning didn’t need to feel like solving a Rubik’s cube in the dark. In our research, we bounced between half-finished repos, clashing APIs, and “it worked on my machine” moments. Out of that chaos, we decided to build something cleaner: one place where SSL across audio, vision, graph, and cross-modal data actually makes sense.
At its core, PrismSSL is a unified playground for SSL. Imagine a command center where you can test state-of-the-art methods, swap modalities with a single line change, and still keep your sanity intact. Everything is modular, transparent, and reproducible — because science should be fun, not frustrating.
We also wanted PrismSSL to be welcoming. Whether you’re a student curious about representation learning, a researcher hunting for benchmarks, or a practitioner putting SSL into production, this library has your back. With HuggingFace baked in, plus support for distributed training, hyperparameter tuning, and lightweight fine-tuning, you’ll spend less time wrestling with setup and more time exploring ideas.
PrismSSL also represents an improved version of an earlier research project, AK_SSL, developed by two previous students. That library included implementations of other SSL methods, and the good news is: everything from AK_SSL is now accessible directly within PrismSSL using the same syntax. If you’d like to read more about AK_SSL or revisit those original methods, you can check the link above — but for practical use, everything has been consolidated here into one unified framework.
In short: PrismSSL is where rigor meets playfulness. Built from academic struggles but polished for the community, it lowers the barriers to SSL while giving you the tools to push the boundaries further.
This repository accompanies the paper
PrismSSL: A Modular Framework for Self-Supervised Learning Across Modalities
Kianoosh Vadaei, et al., 2025.
Self-Supervised Learning (SSL) is basically the art of teaching machines to make up their own homework and then solve it. Instead of us spoon-feeding models with expensive, hand-labeled data, SSL lets them invent clever tasks using only the raw input. Mask part of an audio signal and predict it? Shuffle an image and put it back together? Align speech with text? All of these are ways for models to get smarter without needing humans to sit down and annotate millions of examples.
From an academic angle, SSL has become a game-changer. It powers breakthroughs in speech recognition for low-resource languages, revolutionizes medical imaging where labels are scarce, and even helps scientists model molecules and proteins. At the same time, it’s the secret sauce behind today’s most powerful foundation models — making it both theoretically fascinating and practically indispensable.
But SSL isn’t just serious science — it’s also a bit of fun. There’s something delightful about watching a model reconstruct missing audio or fill in the gaps of an image, almost like it’s playing puzzles at scale. That blend of rigor and playfulness is exactly why we built PrismSSL: to give you a sandbox where curiosity, research, and real-world applications all come together.
The following diagram shows the complete training and evaluation flow:
The framework is built in modular layers to maximize reusability and extensibility:
| 🧩 Domain | 🧠 Method | 💡 Core Idea | ⚙️ Key Mechanism | 🚀 Applications / Strengths |
|---|---|---|---|---|
| Audio | Wav2Vec2 | Contextual speech from raw audio | Mask raw waveform; predict latent targets | Low-resource ASR; robust context |
| HuBERT | Pseudo-labeled speech embeddings | Iterative k-means clustering + masked prediction | Transferable, robust speech reps | |
| SpeechSimCLR | Contrastive audio representations | Augmentations (noise, speed, warp) + contrastive loss | Speaker verification; noise-robust | |
| COLA | Temporal coherence in speech | Align nearby segments; separate distant ones | Dialogue modeling; segmentation | |
| EAT | Acoustic structure via reconstruction | Masked spectrogram patches; transformer reconstructor | Music understanding; large-scale pretrain | |
| CLAP | Align audio–text semantics | Contrastive joint embedding | Retrieval; semantic audio understanding | |
| AudioCLIP | Tri-modal alignment | Shared audio–image–text embedding space | Multimodal search; generative apps | |
| Wav2CLIP | Map audio into CLIP space | Audio encoder guided by frozen CLIP | Audio→image retrieval; creative tasks | |
| Vision | MAE | Visual reps via reconstruction | Mask image patches; decode to reconstruct | Transfer learning; segmentation |
| Barlow Twins | Invariance without negatives | Align embeddings; decorrelate dimensions | Classification; detection | |
| BYOL | Self-bootstrapped representations | Online/target encoders with prediction | Strong SSL without negatives | |
| SimCLR | Simple contrastive baseline | Augmented views; contrastive objective | Strong baseline; transfer | |
| DINO | Self-distillation with no labels | Teacher–student ViTs; centering + sharpening | Strong features; segmentation; detection | |
| MoCo v2 | Momentum contrast with improved augmentations | Momentum encoder; large queue memory bank | Contrastive learning; general vision reps | |
| MoCo v3 | ViT-based momentum contrast | Siamese ViTs; no queue; improved stability | Robust transformer pretraining | |
| SimSiam | Negative-free siamese learning | Stop-gradient branch; predictor head | Lightweight SSL; avoids collapse | |
| SwAV | Online clustering without negatives | Sinkhorn-Knopp prototypes; multi-crop training | Efficient SSL; scalable | |
| Cross-Modal | CLIP | Vision–language alignment | Contrastive training on image–caption pairs | Zero-shot tasks; multimodal retrieval |
| SLIP | Hybrid SSL + CLIP | Language–image contrast + visual SSL | Robust transfer; low-data performance | |
| BLIP | Vision–language via bootstrapped captioning | Caption generation + contrastive alignment | Image captioning; VQA; retrieval | |
| ALBEF | Align before fuse vision–language modeling | Contrastive alignment + multimodal fusion transformer | VQA; retrieval; grounding | |
| SimVLM | Simple multimodal pretraining | Prefix language modeling on image-text sequences | Captioning; VQA; generative multimodal tasks | |
| UNITER | Unified vision–language representation | Multi-task pretraining on large VL datasets | VQA; grounding; retrieval | |
| VSE / VSE++ | Visual–semantic embedding | Image and text encoders aligned via ranking loss | Image–text retrieval | |
| Graph | GraphCL | Graph invariance learning | Graph augmentations; contrastive alignment | Molecular/biological/social graphs |
pip install prism-sslRequirements:
- Python ≥ 3.8
- PyTorch ≥ 1.12
- CUDA-enabled GPU recommended for large-scale training
With PrismSSL, you can go from raw data to results in minutes. The design philosophy is plug-and-play, letting you switch methods or modalities seamlessly.
from PrismSSL.audio.Trainer import Trainer
trainer = Trainer(
method = 'wav2vec2',
backbone = None,
save_dir = './',
wandb_project = 'wav2vec2-pretext',
wandb_mode = "online",
use_data_parallel = True,
checkpoint_interval = 5,
verbose = True,
reload_checkpoint=False,
mixed_precision_training=False
)trainer.train(
train_dataset=train_dataset,
val_dataset=val_dataset,
batch_size=16,
epochs=100,
lr=1e-4,
weight_decay=1e-2,
optimizer="adamw",
use_hpo=True,
n_trials=20,
tuning_epochs=5,
use_embedding_logger=True,
logger_loader=logger_loader
)trainer.evaluate(
train_dataset=train_dataset,
test_dataset=test_dataset,
num_classes=39,
batch_size=64,
lr=1e-3,
epochs=10,
freeze_backbone=True
)The PrismSSL dashboard lets you inspect datasets, preview configs, and launch training/evaluation from a simple UI.
Default port:
5123(changeable). The dashboard runs locally athttp://127.0.0.1:5123.
If you are working from the repository (recommended for the dashboard):
# 1) Clone and enter the repo
git clone https://github.com/PrismaticLab/PrismSSL
cd PrismSSL
# 2) Create & activate a virtual environment (Python 3.10+)
python -m venv .venv
source .venv/bin/activate # on Windows, activate the venv via PowerShell Scripts/Activate.ps1
# 3) Install the package in editable mode (and Flask if needed)
pip install -e .
pip install flask # only if not already installed
# 4) Launch the dashboard (default port 5123)
chmod +x run_dashboard.sh # first time only, to give execution permission
./run_dashboard.sh
# Windows: run_dashboard.batOpen your browser at http://127.0.0.1:5123.
The default port is 5123. To use a different port, set the PORT environment variable before running the script.
macOS/Linux
PORT=5124 ./run_dashboard.shWindows (PowerShell)
$env:PORT=5124
./run_dashboard.batThe dashboard launch scripts currently live only in the repository. If you installed via PyPI, clone the repo to use the dashboard scripts.
PrismSSL is designed for reproducible benchmarking across domains.
Wav2Vec2 pretrained with PrismSSL.
| Task | Dataset | Model | Accuracy |
|---|---|---|---|
| Emotion Clf | Speaker Recognition (2 speakers) | Speech SimCLR | 72.5% |
| Emotion Clf | TESS | COLA | 88.39% |
| Speaker Clf | TESS | EAT | 93.21% |
Wav2CLIP learns powerful joint embeddings, enabling intuitive cross-modal retrieval.
Vision models pretrained on CIFAR-10 with PrismSSL yields competitive performance with limited fine-tuning.
| Method | Linear-prob Top1 | Fine-tune Top1 |
|---|---|---|
| MAE | 61.84% | 87.98% |
| SimCLR v2 | 73.07% | 81.52% |
| BarlowTwins | 70.92% | 79.50% |
| MoCo v2 | 70.08% | 78.71% |
| SwAv | 33.36% | 74.14% |
| MoCo v3 | 59.98% | 74.20% |
| SimSiam | 19.77% | 70.77% |
| BYOL | 71.06% | 71.04% |
| SimCLR v1 | 73.09% | 72.75% |
| DINO | 9.91% | 9.76% |
GraphCL learns molecular-level embeddings competitive with supervised baselines.
| Dataset | Accuracy | AUC |
|---|---|---|
| BBBP | 89.76% | 92.62% |
| Tox21 | task0: 96.61% | – |
| Tox21 | task1: 97.25% | – |
| Tox21 | task2: 87.28% | – |
| Tox21 | task3: 91.39% | – |
| Tox21 | task4: 86.73% | – |
| Tox21 | task5: 96.30% | – |
| Tox21 | task6: 96.11% | – |
| Tox21 | task7: 76.65% | – |
| Tox21 | task8: 94.61% | – |
| Tox21 | task9: 91.71% | – |
| Tox21 | task10: 83.11% | – |
| Tox21 | task11: 88.78% | – |
| Tox21 | 12-task avg: 90.54% | – |
PrismSSL isn’t just a collection of SSL methods — it’s armed with extra superpowers that make your research life smoother, faster, and a lot more fun. Think of these as the cheat codes we always wished existed when we were wrestling with messy experiments:
- 🖥️ Distributed Deep Learning (DDL) — Scale your experiments across multiple GPUs or nodes without needing to summon a cluster-wrangling wizard. Big models? Big data? Bring it on.
- 🎯 Hyperparameter Optimization (HPO) — Stop playing guessing games. Automated tuning with Optuna helps you find the sweet spots without losing weeks of your life.
- 🧠 LoRA Finetuning — Efficiently adapt giant models with lightweight parameter updates. It’s like upgrading your model’s brain without burning your GPU.
- 📊 WandB Integration — Track, visualize, and share every training run like a pro. Who doesn’t love pretty dashboards?
- 🧾 Logging System — Clean, colorful, and customizable logs that won’t make your terminal cry.
- 🤗 HuggingFace Compatibility — Plug and play with transformers and pretrained backbones. Because reinventing the wheel is overrated.
- 🎥 Dynamic Visualizations — Watch your embeddings evolve over time with animated plots. It’s science, but make it art.
In other words: PrismSSL doesn’t just help you run experiments — it helps you run better experiments, with less pain and more insight!
from transformers import BertForPreTraining, AutoTokenizer
model = BertForPreTraining.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
trainer = GenericSSLTrainer(
model=model,
loss_fn=bert_loss_fn,
dataloader=dataloader,
optimizer_ctor=optimizer,
epochs=10
#loRA
use_lora: bool = False,
r=8,
lora_alpha=32,
target_modules=["query", "key", "value"],
lora_dropout=0.1,
bias="none",
task_type="FEATURE_EXTRACTION",
)
trainer.fit()This project was made possible through our collaborative research and academic mentorship. The main contributors are:
Our combined efforts shaped the design, implementation, and structure of PrismSSL. The project was further enriched by the guidance of Dr. Peyman Adibi and Dr. Hossein Karshenas, whose academic mentorship ensured rigor and practical impact.
We’re keeping things chill with the MIT License. In plain English: do whatever you want with this code — use it, remix it, build something wild on top of it. Just don’t sue us if your GPU explodes or your cat walks across your keyboard mid-training and somehow invents AGI. Fair game? Cool. 🚀
If you use this work in your research, please cite:
@article{vadaei2025prismssl,
title={PrismSSL: A Modular Framework for Self-Supervised Learning Across Modalities},
author={Vadaei, Kianoosh and Others},
journal={arXiv preprint arXiv:XXXX.XXXXX},
year={2025}
}