Sidon

Liquid AI fork

This is a fork of sarulab-speech/Sidon maintained for Liquid AI's duplex_chat_english audio-etl pipeline. The optimizations branch targets distributed inference on nodes with 8 AMD MI325X GPUs while preserving the public DialogueSidon weights, 30-step DPM-Solver++ sampler, and native 120-second chunks with 10-second overlap.

The fork adds dynamic batching, padding masks, duration-aware batch packing, OOM batch splitting, and a reusable inference API. Measured on one MI325X at 120 s chunks and 30 steps, batching raises throughput from 86x to 113x real time; memory is bound by the O(T^2) attention of the SSL encoder at T=6000, so batch 16 needs 135 GB and batch 32 does not fit.

BF16/FP16 inference and torch.compile were removed: bf16 costs 19 dB SI-SNR and fp16 25 dB against the fp32 reference, and every torch.compile mode fails on ROCm (NoValidChoicesError, cudagraph assertion, or OOM). Inference runs in fp32, matching upstream DuplexChat. The upstream training and model documentation is retained below.

Liquid cluster setup

There are two separate environments, and they cannot be merged:

	Used for	transformers
audio-etl's .venv	running ingestion (sidon.inference)	5.x, from audio-etl's lockfile
a throwaway export venv	export_diffusion_dialogue.py	<5 — peft imports HybridCache, removed in v5

audio-etl depends on sidon as a normal git dependency, so make install is all that is needed for ingestion — do not uv pip install extra packages into that venv. Doing so re-resolves torch against audio-etl's configured wheel index and can leave torch and torchaudio from different builds, which breaks libtorchaudio.so at import time.

git clone --branch duplex_chat_english \
  git@github.com:Liquid4All/audio-etl.git audio-etl
cd audio-etl && make install          # installs sidon (inference deps only)

Re-exporting the model artifacts

The public DialogueSidon artifacts have a fixed batch dimension and a 3-argument diffusion head. Batched inference needs dynamic batch/sequence dims and a padding mask, so re-export once on one MI325X. This writes supports_attention_mask: true into metadata.json; without that flag DialogueSidonSeparator silently falls back to one chunk at a time and every batching gain is lost.

The artifacts are already built and shared at /home/shared_tw/pretrained/dialogue-sidon-optimized, which is what duplex_chat_english points at, so most users do not need this step.

To rebuild them, create the isolated export environment (this is the one that needs transformers<5) and run the export:

git clone --branch optimizations git@github.com:Liquid4All/Sidon.git Sidon
cd Sidon
uv venv --python 3.12 .venv-export
uv pip install --python .venv-export/bin/python \
  --index-strategy unsafe-best-match \
  --extra-index-url https://wheels.vllm.ai/rocm/0.25.1/rocm723 \
  -e ".[train]"

PYTHONPATH=src .venv-export/bin/python export_diffusion_dialogue.py \
  --source-model sarulab-speech/DialogueSidon \
  --output-dir /home/shared_tw/pretrained/dialogue-sidon-optimized \
  --device cuda:0

PyTorch uses the cuda device name on ROCm. The command downloads the public model; it does not require a private training checkpoint. It must produce:

/home/shared_tw/pretrained/dialogue-sidon-optimized/
├── ssl_encoder.pt2
├── diffusion_head.pt2
├── vae_decoder.pt2
└── metadata.json

Prepare the remaining audio-etl models if they are not already shared:

hf download nvidia/parakeet-tdt-0.6b-v3 \
  parakeet-tdt-0.6b-v3.nemo \
  --local-dir /home/shared_tw/pretrained/parakeet-tdt-0.6b-v3

test -f /home/teams/audio/audio_tokenizer/tok_finetune_nq12_1715528/100000/model.pt

No ffmpeg binary is required: audio-etl decodes source episodes and encodes the separated stereo output through PyAV, which bundles the ffmpeg libraries. The compute nodes have no ffmpeg on PATH, and the shared build at /home/shared_tw/opt/ffmpeg-7.1.1 is built without libmp3lame, so shelling out is not an option here.

Download the pinned English manifest and submit ingestion from the audio-etl checkout. No PYTHONPATH is needed — sidon is a declared dependency, so it is importable in every Ray worker:

cd audio-etl
source .venv/bin/activate
audio-etl duplex-chat-english download

sbatch --nodes=<NUMBER_OF_NODES> --export=ALL \
  slurm/run_ingestion.sh duplex-chat-english ingest

After ingestion completes, export the two directional duplex rows with the FSQ nq12 tokenizer:

sbatch --nodes=<NUMBER_OF_NODES> --export=ALL \
  slurm/run_ingestion.sh duplex-chat-english export-v2

Ingested Parquet is written under s3://multimodal/audio-data/env=prod/source=duplex-chat-english/. Exported training data is written under /home/shared_tw/datasets/audio/exported/source=duplex-chat-english/task=duplex/.

Large-scale text-to-speech (TTS) systems are bottlenecked by the scarcity of clean, multilingual recordings. Sidon tackles this by pairing a fast, open-source speech restoration model with reproducible tooling so researchers can turn noisy in-the-wild corpora into studio-quality datasets that scale across dozens of languages.

Sidon consists of two stages: a w2v-BERT 2.0 feature predictor finetuned to cleanse representations from degraded speech, and a vocoder trained to synthesise restored waveforms from those features. The stack achieves restoration quality comparable to Miipher—Google's internal speech restoration pipeline—while running up to 500× faster than real time on a single GPU. We also observe that training downstream TTS models on Sidon-cleansed automatic speech recognition corpora improves zero-shot synthesis quality. This repository releases the code, configs, and models needed to reproduce Sidon's dataset cleansing workflow for the community.

This repository ships two models:

• Sidon (arXiv:2509.17052) — the single-speaker speech restoration pipeline described above.
• DialogueSidon (arXiv:2604.09344) — a diffusion-based two-speaker dialogue separator that reuses the Sidon backbone. See the DialogueSidon section.

Requirements

• Python 3.10+
• Recent PyTorch / CUDA stack (tested with torch>=2.8, torchaudio>=2.8)
• uv for dependency management (or an equivalent toolchain you are comfortable with)

Install project dependencies:

uv sync

If you rely on a different environment manager, replicate the dependencies listed in pyproject.toml.

Repository layout

• src/sidon/model/sidon/lightning_module.py — Feature predictor, decoder, and discriminator Lightning modules.
• src/sidon/data — WebDataset helpers, preprocessing augmentations, and the PreprocessedDataModule used for training.
• src/sidon/preprocess.py — Parallel writer that turns augmented samples into on-disk shards.
• config/ — Hydra configuration tree with defaults for preprocessing, data, models, and trainer settings.
• scripts/ — Utility scripts plus PBS job templates for batch processing.

Preparing data

Training consumes WebDataset shards that contain tensors expected by the PreprocessedDataModule:

• input_wav.pth and noisy_input_wav.pth — paired clean / degraded waveforms stored as 1D float tensors.
• Optional SSL features (ssl_inputs.pickle, noisy_ssl_inputs.pickle) that provide contextual embeddings for the model.
• sr.index and other metadata entries produced by the preprocessing pipeline.

Update config/data/preprocessed.yaml with the locations of your prepared shards. You can point the train_urls and val_urls entries at directories of .tar / .tar.gz files, or text manifests containing S3 URIs. Set is_s3=true to stream from object storage via the AWS CLI.

Generating preprocessed shards

Use the Hydra-driven preprocessing entrypoint to convert raw WebDataset collections into the tensorised format described above.

1. Choose the base configuration in config/preprocess.yaml (e.g. webdataset_preprocess_24k or webdataset_preprocess_48k). These configs reference the augmentation pipeline, SSL encoders, and noise sources defined in config/data/webdataset_preprocess_*.yaml.

2. Set output parameters in config/preprocess/default.yaml (target directory, shard size, number of writer processes).

3. Launch preprocessing locally:

   uv run python -m sidon.preprocess \
     data=webdataset_preprocess_24k \
     preprocess.writer_name=my_preprocessed_run

   Hydra creates run-specific subdirectories under outputs/ and writes shards into ${preprocess.output_root}/{writer_name}/{split}/{job_id}.

4. On PBS-based clusters, adapt the templates in scripts/pbs/ (e.g. preprocess_24k.sh) to submit distributed jobs. The scripts activate a local virtual environment, set MPI-friendly environment variables, and forward Hydra overrides to the preprocessing entrypoint.

Utilities such as scripts/summarise_shard_durations.py can help audit the duration distribution of generated shards before training.

Training pipeline

Sidon training runs in three sequential stages. Every invocation of python -m sidon.train resolves a Hydra config and writes artefacts under outputs/<timestamped_run>/.

1. Feature predictor pretraining — LoRA-adapts the SSL encoder to denoise representations before they are fed to the vocoder.

   uv run python -m sidon.train \
     model=sidon_feature_predictor \
     data=preprocessed

   The resulting checkpoint (e.g. outputs/<run>/checkpoints/last.ckpt) becomes the model.cfg.ssl_model_name input for the finetuning stage.

2. Vocoder pretraining — Trains the decoder and discriminator while the SSL encoder remains frozen on clean features.

   uv run python -m sidon.train \
     model=sidon_vocoder_pretrain \
     data=preprocessed

   Capture the checkpoint path; it will be referenced as model.cfg.pretrain_path during finetuning.

3. Vocoder finetuning — Warm-starts from the pretraining weights and swaps in the denoised SSL features predicted by the feature predictor.

   uv run python -m sidon.train \
     model=sidon_vocoder_finetune \
     data=preprocessed_48k \
     model.cfg.ssl_model_name=/path/to/feature_predictor.ckpt \
     model.cfg.pretrain_path=/path/to/vocoder_pretrain.ckpt

Adjust optimiser, scheduler, or trainer parameters via the files in config/model/ and config/train/, and use train.ckpt_path to resume a run.

DialogueSidon — diffusion-based dialogue separation

Full-duplex dialogue audio, in which each speaker is recorded on a separate track, is an important resource for spoken dialogue research, but is difficult to collect at scale. Most in-the-wild two-speaker dialogue is available only as degraded monaural mixtures, making it unsuitable for systems requiring clean speaker-wise signals. We propose DialogueSidon, a model for joint restoration and separation of degraded monaural two-speaker dialogue audio. DialogueSidon combines a variational autoencoder (VAE) operates on the speech self-supervised learning (SSL) model feature, which compresses SSL model features into a compact latent space, with a diffusion-based latent predictor that recovers speaker-wise latent representations from the degraded mixture. Experiments on English, multilingual, and in-the-wild dialogue datasets show that DialogueSidon substantially improves intelligibility and separation quality over a baseline, while also achieving much faster inference.

This repository implements DialogueSidon on top of the Sidon feature backbone: a diffusion transformer head predicts per-speaker latents over a frozen SSL-VAE, conditioned on features from a LoRA-adapted w2v-BERT encoder.

Architecture

• SSL encoder — a LoRA-adapted facebook/w2v-bert-2.0 student encodes the noisy mixture into frame-level features.
• SSL-VAE — a pretrained SSLVAE (loaded from cfg.vae_checkpoint_path) provides the target latents; its weights are frozen during training.
• Conditioning heads — two linear projections (output_linear1, output_linear2) map SSL features to per-speaker VAE latents used as a conditioning signal.
• Diffusion transformer head — a DiT with AdaLN conditioning, RoPE attention, and sinusoidal timestep embeddings predicts the noise (or v target) for the concatenated two-speaker latents.
• DDPM training — noise is sampled with a DDPMScheduler (prediction_type=v_prediction by default, 1000 training timesteps). Speaker assignment is resolved with Permutation-Invariant Training on the conditioning heads.
• Latent normalisation — running mean/std buffers are initialised from the first training batch and re-used at inference to stabilise diffusion.

The matching inference script is infer.py (not infer_geneses.py, which is reserved for the flow-matching GENESES separator).

Model variants

Available under config/model/:

Config	Head hidden	Head layers	Heads	Notes
diffusion_dialogue_sidon	768	8	12	default
diffusion_dialogue_sidon_small	384	12	6	small
diffusion_dialogue_sidon_xsmall	384	6	6	xsmall
diffusion_dialogue_sidon_ac	768	8	12	activation checkpointing
diffusion_dialogue_sidon_wo_diffusion_head	—	—	—	baseline without diffusion head
diffusion_dialogue_sidon_wo_vae_latent	768	8	12	ablation without VAE latent conditioning
diffusion_dialogue_sidon_decoder_finetune	—	—	—	decoder finetuning stage

Training

DialogueSidon requires a pretrained SSL-VAE checkpoint. Train it first with model=ssl_vae, then pass the resulting checkpoint into the diffusion run via model.cfg.vae_checkpoint_path.

1. SSL-VAE pretraining — learns the latent space that the diffusion head will predict over.

   uv run python -m sidon.train \
     model=ssl_vae \
     data=dialogue_preprocessed

2. Diffusion training — point model.cfg.vae_checkpoint_path at the SSL-VAE checkpoint from step 1.

   uv run python -m sidon.train \
     model=diffusion_dialogue_sidon \
     data=dialogue_preprocessed \
     model.cfg.vae_checkpoint_path=/path/to/ssl_vae.ckpt

PBS templates for each variant are provided in scripts/pbs/diffusion_dialogue_sidon*.sh.

Inference

infer.py batches chunks across files and resolves speaker permutation in the waveform overlap before crossfading. The same pipeline is available as sidon.inference.DialogueSidonSeparator.

For the batched runtime, prepare dynamic artifacts once and load the resulting directory with --model:

python export_diffusion_dialogue.py \
  --source-model sarulab-speech/DialogueSidon \
  --output-dir /home/shared_tw/pretrained/dialogue-sidon-optimized \
  --device cuda:0

# Batch mode — directory of wav files
python infer.py \
  --model /home/shared_tw/pretrained/dialogue-sidon-optimized \
  --input-dir ./wavs \
  --output-dir ./out \
  --device cuda:0 \
  --num-steps 30 \
  --chunk-seconds 120 \
  --overlap-seconds 10 \
  --batch-size 32 \
  --bf16 \
  --compile

# Single audio or video file (replaces the audio track when given a video)
python infer.py \
  --checkpoint sidon/<run_id> \
  --input-video input.mp4 \
  --output-wav separated.wav \
  --output-video output.mp4

Use scripts/pbs/infer_dialogue.sh to submit the same job on an rt_QG (single GPU) PBS queue.

Validation and troubleshooting

• Perform a quick syntax sweep with python -m compileall src before submitting jobs.
• Ensure the PyTorch build matches the installed CUDA or ROCm runtime.
• If streaming from S3, check that the AWS CLI is installed and accessible in your job environment.
• The stack is ported from an internal codebase and only partially smoke-checked; if something breaks, please open an issue with details so we can follow up.