Fine-tuned Microsoft's Phi-3 Mini (3.8B) with QLoRA to translate natural language questions into SQL, on the Spider benchmark - trained and evaluated end-to-end on a consumer 8GB laptop GPU.
| Execution Accuracy (Spider dev, 1034 examples) | |
|---|---|
| Zero-shot baseline | 57.83% |
| Fine-tuned (QLoRA, 1 epoch) | 63.54% |
| Improvement | +5.71 points |
Execution accuracy runs both the model's predicted SQL and the gold SQL against the real SQLite database and compares result sets - not string matching. This means semantically correct SQL that differs from the gold query in casing, formatting, or column aliasing still counts as correct, which string-match metrics would incorrectly penalize.
Fine-tuned on:
- Phi-3 Mini (3.8B), 4-bit NF4 quantization
- LoRA rank 16, targeting
qkv_proj,o_proj,gate_up_proj,down_proj(0.65% of total parameters trainable) - Full Spider train split (7,000 examples, 160+ distinct databases), 1 epoch
- RTX 2070 Super (8GB VRAM, Turing architecture)
build_dataset.py- reads schemas directly from Spider's SQLite files (nottables.json) so training and execution-accuracy evaluation share a single source of truth. Builds schema-augmented (question, SQL) pairs.stage2_baseline.py- zero-shot execution accuracy baseline, same evaluation machinery used for the fine-tuned model.stage3_train.py- QLoRA fine-tuning via TRL'sSFTTrainer.stage4_eval.py- fine-tuned model execution accuracy, directly comparable to the baseline.inference.py- CLI for generating (and optionally executing) SQL against any SQLite database.
Schema read from the actual SQLite files, not tables.json. Spider's
official metadata file and its SQLite databases can drift. Reading schemas
directly from the same .sqlite files used for execution-accuracy scoring
guarantees training and evaluation never see a mismatched schema.
bf16, not fp16, despite training on a pre-Ampere GPU. The initial plan
was fp16 (Turing lacks bf16 tensor-core acceleration). In practice, PEFT's
LoRA-delta computation against a 4-bit quantized base layer consistently
produced bf16 gradients regardless of explicit dtype overrides at every
level (load-time dtype, model config, per-parameter casts) - confirmed by
directly intercepting the failing gradient-unscale call and inspecting
tensor dtypes. Forward/backward compute in bf16 worked correctly throughout;
only fp16's GradScaler (which bf16 doesn't need, given its fp32-equivalent
exponent range) was actually incompatible. Training in bf16 removed the
mismatch entirely, at the cost of running without tensor-core acceleration.
Phi-3's fused projection layers. Phi-3 merges attention and MLP
projections differently than Llama-family models - qkv_proj and
gate_up_proj rather than separate q_proj/k_proj/v_proj/gate_proj/
up_proj. Targeting the wrong module names doesn't error; LoRA silently
attaches to nothing, and training "succeeds" while learning nothing. Verified
target modules directly rather than assuming Llama-style naming, and added a
fail-fast check that stops immediately if trainable parameter count is 0.
Read-only, timeout-bounded database connections for evaluation. A
zero-shot or lightly-trained model will occasionally generate a DELETE,
UPDATE, or unbounded cross join instead of a SELECT. Evaluation opens
every SQLite connection in read-only mode with a query timeout, so a bad
generation can't corrupt the shared dev databases or hang the evaluation run.
Assistant-only loss masking. Training loss is masked to the SQL completion only, not the schema and question tokens the model is given as input - otherwise the model spends training capacity partially learning to reproduce its own prompt.
- 1 training epoch. Eval loss/accuracy plateaued by roughly 17% into the
epoch (see
results/training logs); additional epochs on the same 7,000 examples showed no further improvement in loss and risked overfitting to the ~160 training schemas rather than improving generalization to the (entirely unseen) dev-set databases. - Single dataset. Trained and evaluated only on Spider. Real-world text-to-SQL usage would need broader schema and dialect coverage.
- 1024-token truncation. Roughly the 90th percentile of prompt+SQL length in the training data; the longest ~10% of schemas are truncated, which disproportionately affects the largest, most complex databases.
- Consumer GPU constraints. Batch size 1 with gradient accumulation, gradient checkpointing, and 4-bit quantization throughout - all deliberate trade-offs for training a 3.8B model on 8GB VRAM, not defaults.
transformers (5.x) · peft · trl (SFTTrainer) · bitsandbytes (4-bit
NF4 quantization) · torch · datasets · SQLite