The 125M base fine-tuned on retrieval-augmented examples — golden passage plus distractors — to answer from the right source and refuse when the answer is absent.
RAFT teaches the model to answer from a passage handed to it among distractors, and to say "not stated in the context" when the answer is not there — the habit that makes retrieval (RAG) trustworthy.
Trained on Modal L4. Judged correctness 0.049 [0.033–0.067]: at 125M, RAFT mostly taught refusal — the model abstains on 84% of answerable questions and the grounding gap is +0.003 (not significant), so the right document didn't measurably help. It does correctly abstain 66% of the time when the answer is genuinely absent.
| Class | LlamaForCausalLM (from scratch) |
| Layers | 12 |
| Hidden size | 768 |
| Attention | 12 heads · dim 64 · full MHA |
| Feed-forward | SwiGLU · inner 3,072 |
| Positional | RoPE · θ 10,000 |
| Norm | RMSNorm · ε 1e-5 |
| Context | 1,024 tokens |
| Vocabulary | 16,384 · byte-level BPE |
| Embeddings | tied input/output |
| Init from | slm-125m-e4 base |
| Method | full fine-tune |
| Trainable params | 125.8M — all |
| Training data | 10,000 RAFT examples · 3 epochs |
| Training tokens | 25.2M (8.4M/epoch × 3) |
| Optimizer steps | 1,875 |
the full lineage. Unlike the 500M and Gemma sites, this base was pretrained from scratch here, so its cost is part of the total rather than someone else's.
| Stage | Detail | Cost |
|---|---|---|
| 125M pretraining (4 legs) | v1 + extended + e2 + e4 · 27.5 h on A100-40GB | $70.14 |
| RAFT fine-tune | 1,875 steps · 19.4 min on L4 | $0.41 |
| Evaluation (shared) | 13 versions on 500 held-out questions, this model's share | $1.48 |
| Total | $72.03 |
Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.