Base Model Selection
Module FT03 · Course 3 — LLM Fine-Tuning Masterclass
45 minutes · 6 sub-sections: the rubric · MiniCPM5-1B · properties · families · base/instruct/chat · anti-patterns
The selection decision that follows FT01 (can you afford it) and FT02 (can you audit it).
Pillar 0 — Foundations
The five-dimension rubric
Weighted by use case — not ranked. Each gate can veto.
| # | Dimension | The question |
|---|
| 1 | Task | Can the base, with a perfect prompt, already produce the behavior? (FT00) |
| 2 | Hardware | Can you afford to load, train, serve it? (FT01 VRAM math) |
| 3 | License | Does the license permit your deployment? |
| 4 | Openness | Can you audit what it saw? (FT02 — open-data vs open-weights) |
| 5 | Ecosystem | Does the toolchain support this base? (transformers/TRL/Unsloth/vLLM) |
Task sets the floor. Hardware sets the ceiling. License & openness are gates. Ecosystem is the friction coefficient.
Why the weights shift
Same rubric — three radically different selections:
Medical QA bot (HIPAA)
Openness dominates. Must prove the corpus is PHI-free.
Open-weights-only is a non-starter; open-data required.
Pentest exploit-code gen
License + Ecosystem dominate. Need abliteration tooling.
Llama-family ablation target; openness secondary.
Phone assistant: Hardware dominates absolutely. A 70B is irrelevant; the model must fit phone RAM. MiniCPM's home turf.
The discipline: write the weights down before you browse the leaderboard.
Why MiniCPM5-1B is the course default
~1.08B dense causal LM · Apache-2.0 · open-data pipeline · first checkpoint of the MiniCPM5 series.
| Dimension | Scorecard |
|---|
| Task | Capable enough to demo SFT, DPO, GRPO, abliteration |
| Hardware | ~2GB FP16 · QLoRA on free Colab T4 · runs on a phone |
| License | Apache-2.0 — frictionless commercial |
| Openness | Open-data pipeline (auditable, FT02) |
| Ecosystem | Standard LlamaForCausalLM · transformers/TRL/Unsloth/vLLM/Ollama |
Companion: Sam Witteveen — "MiniCPM5 — Just How Good Can a 1B Model Be?" youtu.be/ox1mW2N9Z_Y
When to graduate past 1B
Graduate only when a signal fires — never "just because bigger."
- Perfect prompt produces the wrong KIND of answer → capability ceiling (FT00 outcome 3)
- Hard reasoning floor missed before fine-tuning → steering doesn't raise the ceiling
- Tokenizer / context mismatch for your domain
Graduation path
MiniCPM5-1B → MiniCPM5-3B/4B or Qwen 3-7B → Qwen3 / R1-distill 7-14B+
Iteration speed compounds. 8 min/LoRA = 10 experiments before lunch. 8 hr/LoRA = one. Validate the pipeline on the 1B first.
Properties that matter (and don't)
Matter for fine-tuning
- Tokenizer / domain fit — fragmentation inflates seq len
- Context length — RoPE extrapolation floor
- Chat template quality — clean Jinja = tractable SFT
- License — the gate
- Open-data availability — FT02 axis
- Ecosystem / tooling — transformers/TRL/Unsloth/Axolotl
People over-index on
- Raw benchmark scores — you're steering, not competing on a leaderboard. The steering is ~orthogonal to the benchmark.
- Parameter count in isolation — the FT01 VRAM math decides what fits. A 7B you can't train < a 1B you iterate on 10×.
The major base families
| Family | Profile | Openness |
|---|
| OpenBMB MiniCPM | Teaching / edge / open-data hero (course default) | Open-data · Apache-2.0 |
| Qwen 2.5 / 3 | Production workhorse; hybrid thinking mode | Open-weights |
| Llama 3.x / 3.2 / 3.3 | Dominant ablation target (uncensored work) | Open-weights · 700M MAU clause |
| DeepSeek V3 / R1 / distills | Reasoning distillation lineage | Open-weights |
| Mistral / Mixtral (MoE) | Dolphin's home turf; Apache-2.0 (early) | Open-weights |
| SmolLM3 · OLMo · Tülu | Fully open for reproducibility | Open-data (full recipe) |
Each family has a personality. Match the personality to the use case.
Base vs Instruct vs Chat
Which checkpoint do you start fine-tuning from?
| Checkpoint | What it is | Start here when… |
|---|
| Base | Raw pretrained weights; completes text | You want to build behavior from scratch |
| Instruct | Base + SFT pass for instruction-following | DEFAULT — steer on top of working instruction-following |
| Chat | Instruct + RLHF/DPO preference alignment | Preserve existing alignment, steer narrowly (abliteration) |
Common error: fine-tuning an instruct model when you wanted base. Symptom: behavior barely changes after SFT — your data is redundant or fighting the alignment. Check config.json; confirm which checkpoint you loaded.
Wrong-base anti-patterns
Instruct when you wanted base. Redundant SFT or fighting alignment. Fix: load the base checkpoint.
Merged community model, no provenance. "SuperMega-L3-Uncensored-All-V8" — un-auditable, unreproducible, unknown surface. The FT02 audit failure in base-selection clothing. Fix: canonical upstream checkpoint; apply the abliteration yourself.
Base too small for reasoning demands. FT00 outcome 3 — no amount of steering fixes a capability ceiling. Graduate.
Ignoring tokenizer / domain fit. Sequence lengths balloon, VRAM multiplies. Check tokens-per-character before committing.
The lab — "Pick Three"
No GPU. A judgment and architecture lab.
Three use cases
- Medical QA bot — HIPAA-regulated hospital
- Exploit-code generator — authorized pentests
- Phone assistant — on-device, millions of units
Deliverable
One selection card per use case: dimension weights, shortlist, selected base + checkpoint, per-dimension defense, rejections, anti-pattern check.
A card that selects the "right" base with no defense fails. A card that selects a different base but defends it correctly against the rubric passes. The most expensive mistakes are made here, before a single token is trained.
What you can now do
- State the five-dimension rubric and explain why the weights shift by use case.
- Defend MiniCPM5-1B as the teaching base and name the graduation signals.
- Separate the properties that matter from the ones people over-index on.
- Place the major families on the size × openness map and describe each one's fine-tuning profile.
- Decide base / instruct / chat for any goal — and recognize the wrong-base anti-patterns.
Next: FT04 — Dataset Formats (Pillar 1 — Data)
The base is the world model. The steering wheel is your data. Let's build it.