Welcome to the NeuralPiece Laboratory. This repository contains an experimental environment and a comprehensive, multi-layered technical guide designed to demystify LLM tokenization—from its absolute basics to the cutting-edge concepts of "token-free" neural chunking architectures.
This document is structured in progressive layers of depth to ensure absolute comprehension.
| Layer | Depth Level | Target Audience |
|---|---|---|
| L1: Foundation | Pure Analogy | High-level executives & beginners |
| L2: Core Concept | Conceptual Overview | Tech-curious individuals |
| L3: Architecture | Technical Pipeline | Software Engineers & CS Students |
| L4: Mathematical | Hardcore Deep Dive | Machine Learning Engineers |
Imagine instructing an automated system to execute a task. The system does not natively understand human language (strings); it only processes numerical values. Therefore, a translation protocol is required:
flour→1sugar→2mix→3
The instruction "mix flour and sugar" becomes [3, 1, 2]. This is the essence of tokenization: translating human-readable strings into machine-computable tensors.
Tokenization is the process of breaking raw text into discrete sub-units (called tokens) and assigning a unique numerical ID to each. Without this step, Artificial Intelligence models are effectively blind.
HUMAN READS: "The cat drank milk"
MACHINE READS: [15, 892, 3401, 7744]
In practice, a tokenizer is an independent module that runs before the neural network:
- Receives raw text (UTF-8 string).
- Segments it into discrete units.
- Maps each unit to an integer ID via a vocabulary lookup.
- Produces a sequence of IDs that the model processes.
The tokenizer defines the alphabet of the model. If a tokenizer encounters an unknown word, it fragments it.
- Context Window Penalty: A model with an 8K token limit might process ~6,000 English words, but only ~2,000 Japanese words due to poor vocabulary coverage (higher fertility rate).
- Arithmetic Failure: If the number
"12345"is fragmented into["123", "45"], the model loses the mathematical representation of the integer.
Attempting to map every single word in existence to a unique ID requires an infinitely growing database. Every time a new word is invented ("COVID", "ChatGPT"), the system breaks. Conversely, mapping letter-by-letter creates sequences so long that the model forgets the beginning of the sentence before reaching the end.
The solution is Subword Tokenization: breaking words into reusable semantic chunks, similar to LEGO bricks.
Instead of storing ["programming"], ["programmer"], and ["programmed"] as separate IDs, the model learns the root ["program"] and suffixes like ["ming"], ["mer"], ["med"]. This provides the perfect balance between a constrained vocabulary size and efficient sequence lengths.
- BPE (Byte Pair Encoding): Bottom-up approach. Merges the most frequently occurring adjacent pairs. Used by GPT-4, RoBERTa.
- WordPiece: Similar to BPE, but merges pairs that maximize overall likelihood. Used by BERT.
- Unigram (SentencePiece): Top-down approach. Starts with a massive candidate vocabulary and aggressively prunes the least useful tokens. Used by LLaMA, T5, Gemma.
| Feature | BPE | Unigram |
|---|---|---|
| Direction | Bottom-up (Builds) | Top-down (Prunes) |
| Segmentation | Deterministic: Always segments the same way. | Probabilistic: Can sample multiple valid segmentations. |
| Basis | Greedy frequency counting | Expectation-Maximization (EM) & Viterbi |
| Advantage | Implementation simplicity | Subword Regularization (Free data augmentation) |
Standard tokenizers rely on spaces to separate words. This fails for languages without spaces (Japanese) or heavily agglutinative languages (German). SentencePiece treats text as a raw, continuous stream of data without assuming any grammatical rules.
SentencePiece is not an algorithm, but a framework that can run BPE or Unigram. It consumes raw text directly, bypassing traditional pre-tokenization steps entirely.
- Lossless Byte-level Fallback: Unknown characters fall back to raw UTF-8 bytes (
[63, 61, 66...]). - The ▁ Meta-symbol: It replaces spaces with a special
▁character, reconstructing the exact original string losslessly. - Subword Regularization: Probabilistically segments the same word differently during each epoch.
- Epoch 1:
["▁program", "ação"] - Epoch 2:
["▁pro", "gram", "ação"]
- Epoch 1:
The objective is to find a vocabulary V that maximizes the log-likelihood over the corpus:
- E-step: Find the optimal segmentation for all sequences using the Viterbi algorithm.
- M-step: Re-estimate token probabilities based on observed frequencies.
- Pruning: Calculate the loss impact of removing each token:
loss(t) = L_with - L_without. Prune the bottom 20%. Repeat until target size.
All previous methods use fixed mathematical rules to decide where to cut text. Imagine a different approach: training a small neural network to LEARN where to cut. It's the difference between a baker who strictly follows a recipe (BPE/Unigram) and a baker who learned the principles of cooking and can improvise (Neural Chunker).
The Neural Chunker evaluates text byte-by-byte:
"programação"
p → do not cut
r → do not cut
o → do not cut
g → do not cut
r → do not cut
a → do not cut
m → do not cut
a → CUT HERE! ✂️ → chunk: "programa"
ç → CUT HERE! ✂️ → chunk: "ç"
ã → do not cut
o → CUT HERE! ✂️ → chunk: "ão"
Result: ["programa", "ç", "ão"]
The Neural Chunker is a tokenizer that thinks. Instead of fixed rules, it uses a 41k-parameter neural network to decide where to segment. It was taught by SentencePiece, but learns to make better independent decisions.
In our lab tests:
"O banco aprovou o empréstimo"
SentencePiece: 16 tokens
Neural Chunker: 11 tokens ← 31% more efficient!
"12345 + 67890"
SentencePiece: 14 tokens
Neural Chunker: 9 tokens ← 36% more efficient!
Text: "O gato"
↓
UTF-8 Bytes: [79, 32, 103, 97, 116, 111]
↓
Embedding Layer: each byte (0-255) gets a 32-dimensional vector
↓
Conv1D Stack: 2 layers of 1D convolutions (kernel=5)
→ Each position "looks" at 5 neighboring bytes
→ Captures local patterns (e.g., "ção" always ends a word)
↓
Classification Head: MLP outputs 1 number per position
↓
Sigmoid: transforms into probability (0.0 to 1.0)
↓
Decision: probability > 0.5 → CUT | ≤ 0.5 → DO NOT CUT
The heatmap we generated shows this visually:
"O gato bebeu leite"
O█ ·g·a█t·o█ █b·e█b·e█u█ ·l·e·i·t█e█
█ = strong cut (prob > 0.7) — network is SURE it should cut
▓ = medium cut (prob > 0.5) — network thinks it should cut
░ = maybe (prob > 0.3) — network is in doubt
· = no cut (prob ≤ 0.3) — network is SURE it shouldn't cut
The Neural Chunker resolves 3 fundamental problems:
- Context-Aware Tokenization Today, the word "banco" ALWAYS receives the same tokenization, regardless of context. The Neural Chunker, in theory, learns context-dependent boundaries:
"Fui ao banco sacar dinheiro" → ["banco"] (1 token, financial concept)
"Sentei no banco da praça" → ["banco"] (1 token, seating concept)
"O microchip usa banco de dados" → ["banco", "de", "dados"] (compound concept)
-
Infinite Vocabulary Traditional tokenizers have fixed vocabularies (30K-150K). New words are heavily fragmented. The Neural Chunker has no vocabulary. It dynamically decides where to cut in real-time. Any text in the universe works.
-
Zero-Retraining Evolution Traditional tokenizers require full retraining to update the vocabulary. The Neural Chunker can be incrementally adjusted with new data without altering the main model.
We trained two tokenizers on the same corpus (141 Portuguese sentences) with the same vocabulary size (1000 tokens):
🏆 WINNER: UNIGRAM
Score: Unigram 9 x 1 BPE (8 ties)
Average: Unigram 11.8 tokens/sentence vs BPE 12.5 tokens/sentence
The most brutal proof — derived words:
BPE Unigram
programação 2 2 (tie)
programador 3 1 (Unigram: 1 token!)
programando 3 2 (Unigram wins)
Unigram learned that "programador" can be a single unit. BPE failed because it only merges pairs.
We trained a mini-brain (41k parameters) to segment like SentencePiece:
- Training: 60 epochs, ~5 seconds on CPU
- Loss: 0.75 → 0.047 (94% improvement)
- Accuracy: 76.4% → 98.7%
The student surpassed the teacher in multiple sentences:
Neural SentencePiece
"O banco aprovou..." 11 16 ← NC won!
"12345 + 67890" 9 14 ← NC won!
"softmax(QKᵀ/√d)V" 19 22 ← NC won!
"Sentei no banco da praça" 8 11 ← NC won!
We proved that a tokenizer trained on Portuguese discriminates against other languages:
Portuguese: 0.45 tokens/char (efficient)
English: 0.61 tokens/char (ok)
Arabic: 1.89 tokens/char (3x worse!)
Japanese: 3.07 tokens/char (7x worse!)
A Japanese speaker pays 7x more tokens than a Brazilian to say the same thing. This is a direct consequence of the training corpus.
═══════════════════════
"O gato bebeu leite" ← HUMAN WRITES
↓ TOKENIZATION (where it all begins)
["O", "gato", "bebeu", "leite"] ← PIECES
↓ MAPPING
[15, 892, 3401, 7744] ← NUMBERS
↓ EMBEDDING
[[0.2, -0.5, ...], ← VECTORS (meaning)
[0.8, 0.3, ...],
[0.1, -0.9, ...],
[0.7, 0.4, ...]]
↓ POSITIONAL ENCODING
(each vector receives POSITION information)
↓ ATTENTION (Q, K, V)
(each word discovers which other words are relevant to it)
↓ FEED FORWARD
(the network "thinks" about the relationships)
↓ REPEAT × N layers
↓ OUTPUT
"O gato bebeu leite _na cozinha_" ← LLM RESPONDS
Tokenization is the foundation. If the foundation is crooked, the entire building (embeddings, attention, generation) leans. This is why researchers aim to abolish the tokenizer and make the model learn to segment internally—exactly what the Neural Chunker 2077 demonstrates.
The summary phrase: Tokenization is teaching a machine to read. Just as there are better and worse ways to teach a child to read, there are better and worse ways to teach an AI. SentencePiece is the best teacher today. The Neural Chunker is the student that will surpass the teacher tomorrow.