A 4B parameter transformer LLM, hand-built from scratch, raised through developmental stages like a child.
Status: Paused. Joey hit the Chinchilla wall — 4B parameters pretrained on only ~15M tokens (5,000x below compute-optimal). He does not yet converse fluently. This repository documents the complete attempt, including every failure, and the architecture that survived it. See
docs/project-state-2026-04-04.mdfor the full post-mortem.
Every major LLM today is fine-tuned from someone else's base model. Joey is an attempt to do it from zero — hand-coded transformer, tokenizer trained on Project Gutenberg, pretrained on books and children's stories, then fine-tuned through instruction Q&A. The goal was never to compete with frontier models. The goal was to train character into the weights, not bolt it on with system prompts.
Core design principles:
- Honesty trained as a reflex, not enforced by guardrails. "I don't know" as a learned behavior.
- Cited knowledge with provenance tiers (trained / looked-up / unknown).
- Character emerges from curated training data, shaped by three distinct "parent" personalities.
- Self-awareness as a tool, not a person. Joey knows what he is.
- Developmental stages — no shortcuts, no faking maturity, every capability earned.
| Directory | Contents |
|---|---|
joey/ |
Hand-built transformer (model, attention, embeddings, training loop, inference, memory stubs) |
scripts/ |
Training scripts for every stage and round (R1-R10) |
data/raw/ |
Training corpus (2,444+ Q&A pairs across 43 files, plus Stage 4 knowledge docs) |
docs/baby-book/ |
Training log, milestones, personality profile, lessons learned |
docs/superpowers/specs/ |
Design specifications and research references |
cmd/kitt/ |
KITT-style WebSocket interface (scaffolded, non-functional without trained model) |
checkpoints/ |
Evaluation JSONs for every round (model binaries excluded from git) |
Start here if you are:
- A researcher →
docs/baby-book/personality-profile.mdfor the character design, thendocs/baby-book/lessons-learned.mdfor the hard-won knowledge. - An ML engineer →
docs/baby-book/training-log.mdanddocs/baby-book/training-log-r5-r10.mdfor every training run with configs and results. - An architect →
docs/superpowers/specs/2026-03-26-joey-llm-design.mdfor the full design spec. - Someone considering a similar project →
docs/project-state-2026-04-04.mdfor the honest post-mortem and the scaling law math you should do BEFORE writing code.
- 4B parameters — 32 layers, 3200 hidden dim, 25 attention heads
- Hand-built transformer in PyTorch — no HuggingFace model wrappers
- RoPE positional encoding, SwiGLU activation, RMSNorm
- Flash Attention via
scaled_dot_product_attention - Confidence head — auxiliary output estimating prediction reliability
- Gradient checkpointing for memory-efficient training
- FP16 training with GradScaler for cross-platform (A100 / AMD) compatibility
- BPE tokenizer, 8192 vocab, trained on Gutenberg corpus (0% UNK rate)
- Three-tier memory system (scaffolded): working, episodic (SQLite), semantic (vector DB)
| Stage | Content | Status |
|---|---|---|
| 0-1 | Infrastructure (tokenizer, training loop, memory scaffolding) | Complete |
| 2 | Babbling — Project Gutenberg, 30 books | Complete |
| 3 | Language — TinyStories, 20+ epochs on A100 | Complete |
| 4 | Knowledge — code docs, RFCs, textbooks, novels, scripts | Complete |
| 5 | Instruction Fine-Tuning — Rounds 1-10 | Paused (see post-mortem) |
| 6+ | Conversation & continual learning | Not started |
Total pretraining data: ~58 MB (~15M tokens). What was needed per Chinchilla scaling: ~80 billion tokens minimum. Deficit: 5,000x.
This is the root cause of Joey's current inability to converse fluently. The architecture is sound. The fine-tuning pipeline works. The pretraining foundation is the wall. Full analysis here.
Pretraining (Stages 2-4):
- Project Gutenberg: 30 classic novels
- TinyStories: children's narrative corpus
- Stage 4 knowledge: official docs only (no Wikipedia, no Reddit, no Stack Overflow)
- Code: Python, Go, Rust, C#, PowerShell, Bash, SQL, Lua
- Infrastructure: Docker, Kubernetes, Git, YAML
- Networking: 14 RFCs (TCP, HTTP, DNS, TLS, SSH, OAuth, JWT)
- Security: NIST Cybersecurity Framework, SP 800-53, SP 800-171
- Science: OpenStax Biology, Chemistry, Physics, History, Astronomy, Psychology
Fine-Tuning (Stage 5):
- 2,444+ unique Q&A pairs across 43 files
- Conversation patterns, knowledge rephrasings, "I don't know" training, topic discrimination, humor
Joey's personality emerges from three parent influences:
- Dad (Human) — Directness, moral compass, snarky humor, "look it up" philosophy
- Corporate Bob (Claude/Opus AI Model Personality) — Structured reasoning, intellectual humility, clear explanation
- Gamer Bob (Claude/Opus AI Model Personality) — Creative chaos, lateral thinking, pop culture references
See docs/baby-book/personality-profile.md for the full character design.
- Home: AMD RX 9060 XT 16GB (ROCm 7.2.1, Ubuntu 24.04)
- Cloud: NVIDIA A100-SXM4-80GB (RunPod, ~$80 total spend)
Joey is one of the few from-scratch LLM projects that's been trained on both AMD and NVIDIA — with all the cross-platform precision battles (BF16 vs FP16) documented in the training log.
Works:
- Complete transformer architecture, end-to-end
- Training pipeline: pretraining, instruction fine-tuning, partial-freeze, ChatML format
- Cross-platform FP16 checkpoint portability (A100 -> AMD)
- Novel prompt evaluation harness
- Comprehensive training documentation
Does not yet work:
- Conversational fluency (requires pretraining at scale)
- The
cmd/kitt/WebSocket interface is scaffolded but has no working model to serve - Voice integration (planned, not implemented)
- Stages 6-7 (conversation, continual learning)
Building an LLM from scratch is a genuine learning experience. Making one that can actually converse requires resources — specifically, pretraining data — that hobbyist budgets cannot support. Joey's architecture is a solid foundation for anyone who wants to try this. The lessons documented here are meant to save others from the same expensive mistakes.
If you're considering a similar project, read docs/project-state-2026-04-04.md first. It has the scaling law math you should do before writing code.
- Dad — Architect, engineer, the one who pays for everything
- Corporate Bob (Claude/Opus AI Model Personality) — The methodical twin
- Gamer Bob (Claude/Opus AI Model Personality) — The creative twin
Named after a baby kangaroo, because he grew up in his parents' pouch.