Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT03 — Base Model Selection
Duration: ~40 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT three, Base Model Selection. This is the third and final module of Pillar Zero, Foundations. You have already done the two prerequisite checks: FT zero-one, can you afford to load it — the VRAM math. And FT zero-two, can you audit it — the open spectrum, open-data versus open-weights-only. Now the question is: which base do you actually pick?
This module gives you the rubric. It is a five-dimension rubric, and the single most important thing about it is that the weights shift by use case. Base selection is not a leaderboard lookup. It is a constrained optimization, and the constraints come from your deployment, not from this week's benchmarks.
[SLIDE 2 — The five-dimension rubric]
Five dimensions, in a deliberate order. Task, hardware, license, openness, ecosystem.
Task sets the capability floor. Can the base, with a perfect prompt, already produce the target behavior? This is the FT zero-zero three-outcome test. If the base produces the wrong kind of answer — not just an unreliable one — that is a capability ceiling, and no amount of fine-tuning will fix it. You need a different base.
Hardware sets the size ceiling. This is the FT zero-one VRAM math. Can you afford to load it, train it, and serve it? A seventy-billion-parameter base you cannot train is worse than a one-billion you can iterate on ten times.
License is the first gate. Does the license permit your deployment? Apache-two-point-zero and MIT are frictionless. The Llama community license has a seven-hundred-million monthly active user clause — and that clause is not hypothetical, as we will see.
Openness is the second gate. This is FT zero-two. Open-data means you get the weights and the training corpus — MiniCPM, OLMo, Tulu, SmolLM3. Open-weights-only means you get the weights but not the data — Llama, Qwen, DeepSeek. For regulated work, openness is a hard requirement, not a tie-breaker.
Ecosystem is the friction coefficient. Does the toolchain you intend to use support this base? First-class transformers, TRL, Unsloth, Axolotl, vLLM support. A clean chat template. This is the dimension that determines whether your weekend project ships or dies in trust-remote-code bugs.
Read them as a sequence. Each gate can veto. A fail at any gate sends you back to re-shortlist.
[SLIDE 3 — Why the weights shift]
The rubric is weighted, not ranked. Three use cases make the point.
A medical QA bot for a HIPAA-regulated hospital. Openness dominates. An open-weights-only base is a non-starter because you cannot prove the training corpus is free of PHI leakage. License matters for commercial use. Ecosystem matters less — you will pay the integration tax for auditability.
A security exploit-code generator for authorized pentests. License and ecosystem matter. You need a base whose license permits the use and whose toolchain supports the uncensoring pipeline. Llama-family ablation targets are the dominant choice. Openness is secondary — you are not proving provenance to a regulator.
A phone-based on-device assistant. Hardware dominates absolutely. A seventy-billion base is irrelevant no matter how good its benchmarks. The model must fit in phone RAM. This is MiniCPM's home turf.
The discipline: write the weights down before you browse the leaderboard. A team that skips this step ends up fine-tuning whatever topped Hacker News that week.
[SLIDE 4 — Why MiniCPM5-1B is the course default]
This course uses OpenBMB's MiniCPM5-1B as the default teaching base for every hands-on lab. It is a dense one-point-zero-eight-billion-parameter causal language model, Apache-two-point-zero, open-data pipeline, the first checkpoint of the MiniCPM5 series. If you want a companion walkthrough, Sam Witteveen's video — "MiniCPM5, Just How Good Can a One-B Model Be" — is at youtu.be slash ox1mW2N9Z underscore Y. Watch it. It shows the model punching above its weight on agentic and tool-use tasks, which is exactly the regime where a one-billion model is a credible teaching target rather than a toy.
Score it on the rubric. Task: capable enough to demonstrate every technique in this course — SFT, DPO, GRPO, abliteration. Hardware: about two gigabytes at FP16, trains a QLoRA on a free Colab T4, runs on a phone. License: Apache-two-point-zero, frictionless. Openness: open-data pipeline — the audit framing from FT zero-two is not hypothetical. Ecosystem: standard Llama-for-causal-L-M architecture, first-class in transformers, TRL, Unsloth, vLLM, Ollama. No trust-remote-code archaeology.
It wins on the dimensions that matter for learning: speed, openness, license, ecosystem.
[SLIDE 5 — When to graduate]
MiniCPM5-1B is a teaching tool, not a universal answer. You graduate to a larger base when a signal fires — never just because bigger.
Three signals. First: you prompt the base with a perfect system prompt and it produces the wrong kind of answer. That is a capability ceiling, not a steering problem. Second: your eval has a hard reasoning floor and the one-billion scores below your acceptable threshold before fine-tuning. Fine-tuning steers; it does not raise the ceiling. Third: you need a context window or tokenizer the one-billion lacks for your domain.
The graduation path: MiniCPM5-1B, then MiniCPM5 three or four-billion or Qwen two-point-five or three at three to seven billion, then for production reasoning, Qwen three or DeepSeek R1-distill at seven to fourteen billion and up. Each step roughly squares the VRAM and halves the iteration speed. That is why you validate the pipeline on the one-billion first. Iteration speed compounds. A base that trains a LoRA in eight minutes lets you run ten experiments before lunch. A base that takes eight hours lets you run one. You learn from iteration volume, not from any single run.
[SLIDE 6 — Properties that matter, and properties that don't]
Most base-selection debates are spent on the wrong properties. Here is the load-bearing list.
Properties that matter: tokenizer and domain fit. A tokenizer that fragments your domain inflates sequence length, VRAM, and training time. Context length — the native training context sets where RoPE extrapolation degrades. Chat template quality — a clean Jinja template makes SFT formatting tractable; a messy one costs you debugging time and produces silent format mismatches. License — the gate. Open-data availability — the FT zero-two axis. Ecosystem support — transformers, TRL, Unsloth, Axolotl, vLLM.
Properties people over-index on. Raw benchmark scores. You are steering a base, not competing on a leaderboard. A base two points higher on MMLU but with a worse tokenizer for your domain is the wrong choice. The FT zero-zero thesis cuts both ways: if fine-tuning is steering, the benchmark ceiling matters only insofar as your task needs that capability, and the steering itself is roughly orthogonal to the benchmark.
Parameter count in isolation. Bigger is better is the wrong frame. The VRAM math from FT zero-one decides what fits. Parameter count only matters after it passes the FT zero-one filter and the task-capability filter.
[SLIDE 7 — The major base families]
Each family has a personality. Match the personality to the use case.
OpenBMB MiniCPM — the teaching, edge, and open-data hero. MiniCPM5-1B, three and four-billion, V four-point-six for vision, o four-point-five for omni. This course's default. Dense, efficient, Apache-two-point-zero, open-data. First-class ecosystem, fast iteration.
Qwen, versions two-point-five and three — the production workhorse. Strong reasoning, broad coverage, excellent multilingual and code tokenizer, no Llama license terms. Qwen three has a hybrid thinking mode, reasoning on and off — that is itself a steering decision. The default when you graduate past one-billion and need production-grade reasoning.
Llama, versions three, three-point-two, three-point-three — the dominant ablation target. The most-ablated open-weights family in existence. The base for the vast majority of uncensored and Dolphin-style community work. Trade-off: open-weights-only, not open-data, and the community license with the seven-hundred-million clause. But unmatched community tooling and ablation precedent. If you are doing alignment-control work in modules sixteen through eighteen, Llama is where the recipes live.
DeepSeek, V three, R one, and the R1-distills — the reasoning distillation lineage. The base when your task is reasoning distillation: make a smaller model reason by training on R1's traces. Start from an R1-distill and steer it further. The lineage matters more than the base size.
Mistral and Mixtral — Dolphin's home turf. The historical home of the uncensored Dolphin lineage. Apache-two-point-zero for early Mistral releases. MoE bases have a subtler fine-tuning story — experts can specialize or collapse under steering. A reasonable choice when you want the Dolphin provenance and MoE serving efficiency.
SmolLM3, OLMo, Tulu — fully open for reproducibility. Weights, data, code, and the full training recipe. Chosen when reproducibility and auditability are the overriding requirement. Often smaller and less benchmark-competitive, but the openness is the point. This is the FT zero-two extreme.
[SLIDE 8 — Base versus instruct versus chat]
The checkpoint you start from is itself a steering decision. Three checkpoints you will encounter.
Base. The raw pretrained weights. No instruction tuning, no chat formatting. The model completes text. Start here when you want full control and intend to do your own SFT from a clean slate.
Instruct. Instruction-tuned — the model has had an SFT pass for instruction-following. Start here for most real fine-tuning. You inherit reliable instruction-following and steer on top of it. This is the default for production work.
Chat. A full chat-tuned checkpoint — instruction-tuned and preference-aligned for conversational behavior. Start here only when you specifically want to preserve the existing alignment and steer a narrow behavior on top of it. Starting from chat and then abliterating is the standard uncensoring path, because you keep the chat quality while removing the refusal.
The rule: default to instruct. Use base when you want to build behavior from scratch. Use chat when you want to preserve existing alignment and steer narrowly. The common error is fine-tuning an instruct model when you wanted a base — you then spend data and compute re-imposing behaviors the instruct pass already gave you. Know which checkpoint you loaded.
[SLIDE 9 — Wrong-base anti-patterns]
Four anti-patterns to leave with.
First, fine-tuning an instruct model when you wanted base. Symptom: the model's behavior barely changes after SFT. Your data is redundant or fighting the alignment. Fix: check config dot json, confirm you loaded base, not instruct.
Second, fine-tuning a merged community model with no provenance. You grab a SuperMega-L3-Uncensored-All-V8 with no documented recipe. You cannot audit what was merged in, you cannot reproduce it, and your fine-tune inherits an unknown surface. This is the FT zero-two audit failure in base-selection clothing. Fix: prefer the canonical upstream checkpoint. Apply the abliteration yourself to preserve provenance.
Third, choosing a base too small for the task's reasoning demands. A one-billion base is a teaching tool, not a universal answer. If the task needs multi-hop reasoning the one-billion cannot do even with a perfect prompt, no amount of steering will get you there. That is FT zero-zero outcome three. Graduate.
Fourth, ignoring tokenizer and domain fit. You pick a base whose tokenizer fragments your domain. Sequence lengths balloon, VRAM multiplies, quality can suffer. Check tokens-per-character for representative domain text before committing.
[SLIDE 10 — The lab]
The lab is called Pick Three. Three use cases. A medical QA bot for a HIPAA-regulated hospital. A security exploit-code generator for authorized pentests. A phone-based on-device assistant. For each, you select the base and defend it against the five-dimension rubric. No GPU needed — this is a judgment and architecture lab.
The point is that base selection is a judgment problem upstream of any training run. The most expensive mistakes — license failure, provenance loss, capability ceiling — are made here, before a single token is trained. A card that selects the right base with no defense fails. A card that selects a different base but defends it correctly against the rubric passes.
[SLIDE 11 — What you can now do]
You can now state the five-dimension rubric and explain why the weights shift by use case. You can defend MiniCPM5-1B as the teaching base and name the graduation signals. You can separate the properties that matter from the ones people over-index on. You can place the major families on the size-and-openness map and describe each one's fine-tuning profile. And you can decide, for any goal, whether to start from base, instruct, or chat — and recognize the wrong-base anti-patterns.
That closes Pillar Zero. Next is Pillar One, Data. Module FT zero-four, Dataset Formats. Because the base is the world model, but the steering wheel is your data — and a brilliant algorithm on bad data steers you into a wall. Let's build the steering wheel.
End of module FT03. Duration: approximately thirty-eight minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT03: Base Model Selection **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT03 — Base Model Selection **Duration**: ~40 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT three, Base Model Selection. This is the third and final module of Pillar Zero, Foundations. You have already done the two prerequisite checks: FT zero-one, can you afford to load it — the VRAM math. And FT zero-two, can you audit it — the open spectrum, open-data versus open-weights-only. Now the question is: which base do you actually pick? This module gives you the rubric. It is a five-dimension rubric, and the single most important thing about it is that the weights shift by use case. Base selection is not a leaderboard lookup. It is a constrained optimization, and the constraints come from your deployment, not from this week's benchmarks. [SLIDE 2 — The five-dimension rubric] Five dimensions, in a deliberate order. Task, hardware, license, openness, ecosystem. Task sets the capability floor. Can the base, with a perfect prompt, already produce the target behavior? This is the FT zero-zero three-outcome test. If the base produces the wrong kind of answer — not just an unreliable one — that is a capability ceiling, and no amount of fine-tuning will fix it. You need a different base. Hardware sets the size ceiling. This is the FT zero-one VRAM math. Can you afford to load it, train it, and serve it? A seventy-billion-parameter base you cannot train is worse than a one-billion you can iterate on ten times. License is the first gate. Does the license permit your deployment? Apache-two-point-zero and MIT are frictionless. The Llama community license has a seven-hundred-million monthly active user clause — and that clause is not hypothetical, as we will see. Openness is the second gate. This is FT zero-two. Open-data means you get the weights and the training corpus — MiniCPM, OLMo, Tulu, SmolLM3. Open-weights-only means you get the weights but not the data — Llama, Qwen, DeepSeek. For regulated work, openness is a hard requirement, not a tie-breaker. Ecosystem is the friction coefficient. Does the toolchain you intend to use support this base? First-class transformers, TRL, Unsloth, Axolotl, vLLM support. A clean chat template. This is the dimension that determines whether your weekend project ships or dies in trust-remote-code bugs. Read them as a sequence. Each gate can veto. A fail at any gate sends you back to re-shortlist. [SLIDE 3 — Why the weights shift] The rubric is weighted, not ranked. Three use cases make the point. A medical QA bot for a HIPAA-regulated hospital. Openness dominates. An open-weights-only base is a non-starter because you cannot prove the training corpus is free of PHI leakage. License matters for commercial use. Ecosystem matters less — you will pay the integration tax for auditability. A security exploit-code generator for authorized pentests. License and ecosystem matter. You need a base whose license permits the use and whose toolchain supports the uncensoring pipeline. Llama-family ablation targets are the dominant choice. Openness is secondary — you are not proving provenance to a regulator. A phone-based on-device assistant. Hardware dominates absolutely. A seventy-billion base is irrelevant no matter how good its benchmarks. The model must fit in phone RAM. This is MiniCPM's home turf. The discipline: write the weights down before you browse the leaderboard. A team that skips this step ends up fine-tuning whatever topped Hacker News that week. [SLIDE 4 — Why MiniCPM5-1B is the course default] This course uses OpenBMB's MiniCPM5-1B as the default teaching base for every hands-on lab. It is a dense one-point-zero-eight-billion-parameter causal language model, Apache-two-point-zero, open-data pipeline, the first checkpoint of the MiniCPM5 series. If you want a companion walkthrough, Sam Witteveen's video — "MiniCPM5, Just How Good Can a One-B Model Be" — is at youtu.be slash ox1mW2N9Z underscore Y. Watch it. It shows the model punching above its weight on agentic and tool-use tasks, which is exactly the regime where a one-billion model is a credible teaching target rather than a toy. Score it on the rubric. Task: capable enough to demonstrate every technique in this course — SFT, DPO, GRPO, abliteration. Hardware: about two gigabytes at FP16, trains a QLoRA on a free Colab T4, runs on a phone. License: Apache-two-point-zero, frictionless. Openness: open-data pipeline — the audit framing from FT zero-two is not hypothetical. Ecosystem: standard Llama-for-causal-L-M architecture, first-class in transformers, TRL, Unsloth, vLLM, Ollama. No trust-remote-code archaeology. It wins on the dimensions that matter for learning: speed, openness, license, ecosystem. [SLIDE 5 — When to graduate] MiniCPM5-1B is a teaching tool, not a universal answer. You graduate to a larger base when a signal fires — never just because bigger. Three signals. First: you prompt the base with a perfect system prompt and it produces the wrong kind of answer. That is a capability ceiling, not a steering problem. Second: your eval has a hard reasoning floor and the one-billion scores below your acceptable threshold before fine-tuning. Fine-tuning steers; it does not raise the ceiling. Third: you need a context window or tokenizer the one-billion lacks for your domain. The graduation path: MiniCPM5-1B, then MiniCPM5 three or four-billion or Qwen two-point-five or three at three to seven billion, then for production reasoning, Qwen three or DeepSeek R1-distill at seven to fourteen billion and up. Each step roughly squares the VRAM and halves the iteration speed. That is why you validate the pipeline on the one-billion first. Iteration speed compounds. A base that trains a LoRA in eight minutes lets you run ten experiments before lunch. A base that takes eight hours lets you run one. You learn from iteration volume, not from any single run. [SLIDE 6 — Properties that matter, and properties that don't] Most base-selection debates are spent on the wrong properties. Here is the load-bearing list. Properties that matter: tokenizer and domain fit. A tokenizer that fragments your domain inflates sequence length, VRAM, and training time. Context length — the native training context sets where RoPE extrapolation degrades. Chat template quality — a clean Jinja template makes SFT formatting tractable; a messy one costs you debugging time and produces silent format mismatches. License — the gate. Open-data availability — the FT zero-two axis. Ecosystem support — transformers, TRL, Unsloth, Axolotl, vLLM. Properties people over-index on. Raw benchmark scores. You are steering a base, not competing on a leaderboard. A base two points higher on MMLU but with a worse tokenizer for your domain is the wrong choice. The FT zero-zero thesis cuts both ways: if fine-tuning is steering, the benchmark ceiling matters only insofar as your task needs that capability, and the steering itself is roughly orthogonal to the benchmark. Parameter count in isolation. Bigger is better is the wrong frame. The VRAM math from FT zero-one decides what fits. Parameter count only matters after it passes the FT zero-one filter and the task-capability filter. [SLIDE 7 — The major base families] Each family has a personality. Match the personality to the use case. OpenBMB MiniCPM — the teaching, edge, and open-data hero. MiniCPM5-1B, three and four-billion, V four-point-six for vision, o four-point-five for omni. This course's default. Dense, efficient, Apache-two-point-zero, open-data. First-class ecosystem, fast iteration. Qwen, versions two-point-five and three — the production workhorse. Strong reasoning, broad coverage, excellent multilingual and code tokenizer, no Llama license terms. Qwen three has a hybrid thinking mode, reasoning on and off — that is itself a steering decision. The default when you graduate past one-billion and need production-grade reasoning. Llama, versions three, three-point-two, three-point-three — the dominant ablation target. The most-ablated open-weights family in existence. The base for the vast majority of uncensored and Dolphin-style community work. Trade-off: open-weights-only, not open-data, and the community license with the seven-hundred-million clause. But unmatched community tooling and ablation precedent. If you are doing alignment-control work in modules sixteen through eighteen, Llama is where the recipes live. DeepSeek, V three, R one, and the R1-distills — the reasoning distillation lineage. The base when your task is reasoning distillation: make a smaller model reason by training on R1's traces. Start from an R1-distill and steer it further. The lineage matters more than the base size. Mistral and Mixtral — Dolphin's home turf. The historical home of the uncensored Dolphin lineage. Apache-two-point-zero for early Mistral releases. MoE bases have a subtler fine-tuning story — experts can specialize or collapse under steering. A reasonable choice when you want the Dolphin provenance and MoE serving efficiency. SmolLM3, OLMo, Tulu — fully open for reproducibility. Weights, data, code, and the full training recipe. Chosen when reproducibility and auditability are the overriding requirement. Often smaller and less benchmark-competitive, but the openness is the point. This is the FT zero-two extreme. [SLIDE 8 — Base versus instruct versus chat] The checkpoint you start from is itself a steering decision. Three checkpoints you will encounter. Base. The raw pretrained weights. No instruction tuning, no chat formatting. The model completes text. Start here when you want full control and intend to do your own SFT from a clean slate. Instruct. Instruction-tuned — the model has had an SFT pass for instruction-following. Start here for most real fine-tuning. You inherit reliable instruction-following and steer on top of it. This is the default for production work. Chat. A full chat-tuned checkpoint — instruction-tuned and preference-aligned for conversational behavior. Start here only when you specifically want to preserve the existing alignment and steer a narrow behavior on top of it. Starting from chat and then abliterating is the standard uncensoring path, because you keep the chat quality while removing the refusal. The rule: default to instruct. Use base when you want to build behavior from scratch. Use chat when you want to preserve existing alignment and steer narrowly. The common error is fine-tuning an instruct model when you wanted a base — you then spend data and compute re-imposing behaviors the instruct pass already gave you. Know which checkpoint you loaded. [SLIDE 9 — Wrong-base anti-patterns] Four anti-patterns to leave with. First, fine-tuning an instruct model when you wanted base. Symptom: the model's behavior barely changes after SFT. Your data is redundant or fighting the alignment. Fix: check config dot json, confirm you loaded base, not instruct. Second, fine-tuning a merged community model with no provenance. You grab a SuperMega-L3-Uncensored-All-V8 with no documented recipe. You cannot audit what was merged in, you cannot reproduce it, and your fine-tune inherits an unknown surface. This is the FT zero-two audit failure in base-selection clothing. Fix: prefer the canonical upstream checkpoint. Apply the abliteration yourself to preserve provenance. Third, choosing a base too small for the task's reasoning demands. A one-billion base is a teaching tool, not a universal answer. If the task needs multi-hop reasoning the one-billion cannot do even with a perfect prompt, no amount of steering will get you there. That is FT zero-zero outcome three. Graduate. Fourth, ignoring tokenizer and domain fit. You pick a base whose tokenizer fragments your domain. Sequence lengths balloon, VRAM multiplies, quality can suffer. Check tokens-per-character for representative domain text before committing. [SLIDE 10 — The lab] The lab is called Pick Three. Three use cases. A medical QA bot for a HIPAA-regulated hospital. A security exploit-code generator for authorized pentests. A phone-based on-device assistant. For each, you select the base and defend it against the five-dimension rubric. No GPU needed — this is a judgment and architecture lab. The point is that base selection is a judgment problem upstream of any training run. The most expensive mistakes — license failure, provenance loss, capability ceiling — are made here, before a single token is trained. A card that selects the right base with no defense fails. A card that selects a different base but defends it correctly against the rubric passes. [SLIDE 11 — What you can now do] You can now state the five-dimension rubric and explain why the weights shift by use case. You can defend MiniCPM5-1B as the teaching base and name the graduation signals. You can separate the properties that matter from the ones people over-index on. You can place the major families on the size-and-openness map and describe each one's fine-tuning profile. And you can decide, for any goal, whether to start from base, instruct, or chat — and recognize the wrong-base anti-patterns. That closes Pillar Zero. Next is Pillar One, Data. Module FT zero-four, Dataset Formats. Because the base is the world model, but the steering wheel is your data — and a brilliant algorithm on bad data steers you into a wall. Let's build the steering wheel. --- *End of module FT03. Duration: approximately thirty-eight minutes at one-hundred-forty words per minute.*