I trained a base smolLM3 checkpoint with the standard smolLM3 tokenizer + architecture,
but with the configuration adjusted to target ~500M parameters.
This is a raw pretrained base model (not instruction-tuned / not RLHF).
I’m sharing it in case it’s useful to the community, and I wanted to ask whether you’d
be open to linking it from the official repo (or listing it in a “community checkpoints”
section). No worries at all if not.
Links
Training summary
- ~19.8B tokens, packed seq_len=4096
- Multi-source mixture (FineWeb-Edu, DCLM, StackExchange, Wiki) with token caps
- Step-based training with Accelerate (4× RTX 6000)
- bf16, SDPA attention
- Apache-2.0 license
If this is something you’d consider including, I’m happy to:
- open a small docs PR that adds a link + short description, or
- adjust naming/formatting to match whatever you prefer.
I should also note that some parts of the training and data pipeline are
structured a bit differently from the official setup. This was intentional,
the choices I made were mainly to keep things simple and easier for me to
reason about while learning and experimenting. I’m not claiming this is a
better or more canonical approach, just one that worked well for me.
I trained a base smolLM3 checkpoint with the standard smolLM3 tokenizer + architecture,
but with the configuration adjusted to target ~500M parameters.
This is a raw pretrained base model (not instruction-tuned / not RLHF).
I’m sharing it in case it’s useful to the community, and I wanted to ask whether you’d
be open to linking it from the official repo (or listing it in a “community checkpoints”
section). No worries at all if not.
Links
Training summary
If this is something you’d consider including, I’m happy to:
I should also note that some parts of the training and data pipeline are
structured a bit differently from the official setup. This was intentional,
the choices I made were mainly to keep things simple and easier for me to
reason about while learning and experimenting. I’m not claiming this is a
better or more canonical approach, just one that worked well for me.