{ "module": "FT03 — Base Model Selection", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 35, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the five dimensions of the base selection rubric, in their deliberate order.", "options": [ "Task → License → Openness → Hardware → Ecosystem", "Task → Hardware → License → Openness → Ecosystem", "Hardware → Task → Ecosystem → License → Openness", "Openness → License → Task → Ecosystem → Hardware" ], "answer_index": 1, "rationale": "Task sets the capability floor; Hardware sets the size ceiling (FT01 VRAM math); License and Openness are gates that can veto; Ecosystem is the friction coefficient. The order is deliberate — a fail at any gate sends you back to re-shortlist." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Why is MiniCPM5-1B the course's default teaching base?", "options": [ "It has the highest MMLU score of any open model under 2B parameters.", "It is the largest model that runs on a free Colab T4 without quantization.", "It wins on the dimensions that matter for learning: ~1B params for fast iteration, Apache-2.0 license, open-data pipeline, and first-class ecosystem support (transformers/TRL/Unsloth/vLLM).", "It is the only open-data model with a documented chat template." ], "answer_index": 2, "rationale": "MiniCPM5-1B wins on the dimensions that matter for LEARNING: speed (1B = fast iteration), openness (open-data, auditable — FT02), license (Apache-2.0, frictionless), and ecosystem (standard LlamaForCausalLM, no trust_remote_code rot). Benchmark scores are explicitly NOT the reason — that is an over-indexed property." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Which of the following is the correct default starting checkpoint for most real fine-tuning, and why?", "options": [ "Base — to build instruction-following from a clean slate.", "Instruct — you inherit reliable instruction-following and steer on top of it.", "Chat — to preserve existing alignment in all cases.", "Merged community checkpoint — for the best of all upstream behaviors." ], "answer_index": 1, "rationale": "Default to INSTRUCT. You inherit reliable instruction-following and steer on top of it. Use BASE when you want to build behavior from scratch; use CHAT only when you want to preserve existing alignment and steer narrowly (e.g. abliteration). Merged community checkpoints are an anti-pattern (no provenance)." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are building a phone-based on-device assistant that must run entirely in ~2GB of phone RAM at Q4, shipping on millions of devices. Which rubric dimension dominates, and which base is the defensible choice?", "options": [ "Task dominates; choose the highest-benchmarking 1B model.", "Hardware dominates (near-absolute); choose MiniCPM5-1B — purpose-built for on-device, Apache-2.0 (no MAU clause), ~2GB at Q4, dense for phone SoC latency.", "Openness dominates; choose OLMo at the largest size that fits.", "Ecosystem dominates; choose Llama 3.2 small for its tooling." ], "answer_index": 1, "rationale": "Hardware is near-absolute on a phone (RAM + SoC throughput set the ceiling). MiniCPM5-1B is purpose-built for this regime. Critically, the Llama community license's 700M MAU clause is triggered by shipping on millions of devices — that is exactly the scenario it covers, so Llama 3.2 is vetoed by the license gate here. OLMo is defensible on openness but loses on capability/ecosystem for this size class." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You are fine-tuning a model to remove refusals for an authorized-pentest exploit-code generator. The model already knows offensive security but refuses. Which checkpoint do you start from, and why?", "options": [ "Base — build the willingness from scratch.", "Instruct — inherit instruction-following, then steer.", "Chat — preserve the existing chat quality and steer narrowly (remove only the refusal via abliteration).", "A merged 'uncensored' community checkpoint — it already has no refusals." ], "answer_index": 2, "rationale": "Start from CHAT for abliteration (FT17). You want to PRESERVE the existing chat quality and steer narrowly — remove only the refusal direction. Starting from BASE throws away the alignment quality; starting from INSTRUCT loses some chat behavior. A merged uncensored checkpoint is the provenance anti-pattern (no documented recipe, un-auditable, unpredictable degradation)." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "A team picks the base that topped the MMLU leaderboard this week, then discovers its tokenizer fragments their CJK-heavy domain into many sub-tokens, inflating sequence length and VRAM. Which anti-pattern did they commit, and what should they have done?", "options": [ "Wrong-base anti-pattern 'ignoring tokenizer/domain fit.' They should have checked tokens-per-character for representative domain text before committing.", "Wrong-base anti-pattern 'choosing a base too small.' They should have picked a larger base.", "They violated the FT01 VRAM filter; they should have quantized first.", "They started from the wrong checkpoint; they should have used base, not instruct." ], "answer_index": 0, "rationale": "This is the 'ignoring tokenizer/domain fit' anti-pattern. Tokenizer fit is a load-bearing property: poor fit inflates sequence length, multiplies VRAM (FT01), and can hurt quality. The fix is to check tokens-per-character for representative domain text BEFORE committing. The deeper error was leaderboard-driven selection — benchmarks measure general capability, not domain tokenizer fit." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You prompt MiniCPM5-1B with a perfect system prompt on a multi-hop reasoning task, and it produces the wrong KIND of answer (not just an unreliable one). Per the FT00 three-outcome test, what does this mean, and what is the correct response?", "options": [ "Outcome 1 (steering) — fine-tune with SFT to make it reliable.", "Outcome 2 (preference) — use DPO or abliteration.", "Outcome 3 (capability ceiling / knowledge gap) — graduate to a larger base or different family; no amount of steering will reliably fix this.", "Increase the LoRA rank and retrain." ], "answer_index": 2, "rationale": "Producing the wrong KIND of answer (not just unreliable format) is outcome 3: a capability ceiling. Fine-tuning steers; it does not raise the ceiling. The correct response is to graduate to a larger base (MiniCPM5-3B/4B or Qwen3 7B) or a different family with stronger reasoning. This is the 'choosing a base too small for the task's reasoning demands' anti-pattern." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "A hospital's compliance team requires the ability to prove what the training corpus contained (no PHI leakage). Which rubric dimension dominates, and is an open-weights-only base acceptable?", "options": [ "Ecosystem dominates; open-weights-only is acceptable if the tooling is good.", "Openness dominates (hard gate); open-weights-only is NOT acceptable — you need open-data (MiniCPM, OLMo, Tülu, SmolLM3) to audit the corpus.", "License dominates; any Apache-2.0 base is acceptable regardless of data openness.", "Task dominates; pick the base that knows medicine best, regardless of openness." ], "answer_index": 1, "rationale": "For HIPAA-regulated work, openness is a HARD gate, not a tie-breaker. Open-weights-only (Llama, Qwen) means you trust the publisher's word — unacceptable when you must PROVE the corpus is PHI-free. You need open-data (MiniCPM, OLMo, Tülu, SmolLM3). The tension: Qwen wins on capability but fails this gate; you fall back to the open-data family and accept the capability/iteration trade-off." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You intend to build instruction-following behavior from a clean slate, but you accidentally loaded the instruct checkpoint. What symptom will you see, and what is the fix?", "options": [ "The model will refuse all requests; switch to the chat checkpoint.", "The model's behavior barely changes after SFT — your data is either redundant (it already follows instructions) or fighting the existing alignment. Fix: load the base checkpoint.", "VRAM usage will double; quantize the model.", "The chat template will be missing; reinstall transformers." ], "answer_index": 1, "rationale": "This is the 'fine-tuning an instruct model when you wanted base' anti-pattern. Symptom: behavior barely changes after SFT — your data is redundant (instruct already follows instructions) or fighting the alignment. Fix: check config.json and the model card, confirm you loaded -base not -instruct, and reload the base checkpoint." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the FT00 thesis ('fine-tuning steers behavior; it does not teach knowledge') imply that raw benchmark scores are over-indexed in base selection?", "options": [ "Because benchmarks are always wrong and should be ignored entirely.", "Because if fine-tuning is steering (redirecting existing capability), the base's benchmark ceiling matters only insofar as your task needs that capability — and the steering itself is roughly orthogonal to the benchmark. A base 2 points higher on MMLU but with a worse tokenizer for your domain is the wrong choice.", "Because steering always degrades benchmark scores, so they are irrelevant.", "Because benchmarks only measure instruction-following, not capability." ], "answer_index": 1, "rationale": "The thesis cuts both ways: if fine-tuning is steering, then the benchmark ceiling matters only when your task needs that capability, and the steering operation is roughly orthogonal to the benchmark. Benchmarks measure general capability, not steerability toward YOUR behavior. A higher-MMLU base with a worse tokenizer/domain fit is the wrong choice. (Benchmarks still matter for setting the capability FLOOR — they are not ignored, just not over-indexed.)" }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team selects a community merge ('SuperMega-L3-Uncensored-All-V8') with no documented merge recipe as their fine-tuning base. Which anti-pattern is this, and what is the core risk?", "options": [ "Anti-pattern: 'ignoring tokenizer fit.' Risk: sequence length inflation.", "Anti-pattern: 'fine-tuning a merged community model with no provenance.' Risk: you cannot audit what was merged in, cannot reproduce it, and your fine-tune inherits an unknown capability/alignment surface — unpredictable degradation and an un-debuggable model. This is the FT02 audit failure in base-selection clothing.", "Anti-pattern: 'choosing a base too small.' Risk: capability ceiling.", "Anti-pattern: 'wrong checkpoint.' Risk: redundant SFT." ], "answer_index": 1, "rationale": "This is the provenance anti-pattern — the FT02 audit failure applied to base selection. Without a documented merge recipe, you cannot audit the inputs, cannot reproduce the base, and inherit an unknown surface. The fix: prefer the canonical upstream checkpoint; if you must use a merge, require a documented recipe. Apply the abliteration yourself to preserve provenance." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is the Llama community license's 700-million-monthly-active-user clause a real GATE (not a tie-breaker) for a phone assistant shipping on millions of devices, but NOT a gate for an on-prem hospital deployment?", "options": [ "It isn't — the clause applies equally in both cases.", "The clause is triggered by deployment SCALE (MAU). A phone assistant shipping on millions of consumer devices hits the clause directly; an on-prem hospital deployment with thousands of clinician users is far below the threshold. The clause is scale-conditional, so it gates consumer-device shipping but not internal enterprise deployment.", "The clause only applies to open-data models, not Llama.", "The clause applies only to fine-tuned derivatives, not the base." ], "answer_index": 1, "rationale": "The Llama community license's 700M MAU clause is scale-conditional. A consumer-device deployment shipping on millions of phones plausibly approaches or exceeds the threshold — it is a real gate, and MiniCPM5-1B (Apache-2.0, no MAU clause) is the defensible choice. An on-prem hospital deployment with thousands of clinician users is far below the threshold, so the same license is acceptable there. The SAME license gates one use case and not the other — which is exactly why the rubric is weighted by use case." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Two teams are choosing between Qwen3 (open-weights-only, strong capability) and an OLMo open-data base (weaker benchmarks, fully auditable) for a regulated medical task. Team A picks Qwen3; Team B picks OLMo. The FT03 rubric says one of them is making a defensible choice and the other is not, depending on ONE factor. What is that factor?", "options": [ "Which base scores higher on MMLU.", "Whether the regulator requires PROVING the training corpus (open-data) or merely accepts open-weights. If proof is required, Team A (Qwen3) fails the openness gate; if open-weights is acceptable, Team A is defensible and Team B is paying an unnecessary capability tax.", "Which base has better ecosystem support.", "Which base is larger." ], "answer_index": 1, "rationale": "The decisive factor is the openness REQUIREMENT. If the regulator requires proving the corpus (open-data, FT02), Qwen3 fails the gate and OLMo is the only defensible choice — even at a capability cost. If open-weights-only is acceptable, Qwen3 is defensible (better capability/ecosystem) and OLMo is paying an unnecessary tax. The rubric is weighted by use case; the same two bases produce different correct answers depending on a single gate." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the course recommend validating the fine-tuning pipeline on MiniCPM5-1B BEFORE graduating to a 7-14B production base, rather than starting directly on the production base?", "options": [ "Because the 1B produces higher-quality fine-tunes than the 7B.", "Because iteration speed compounds: a 1B trains a LoRA in minutes, letting you run 10+ experiments to learn data quality, format, and hyperparameter sensitivity before lunch. A 7B takes hours per run, yielding one experiment. You learn from iteration VOLUME, not any single run — so validate the pipeline where iteration is cheap, then scale only when the task demands it.", "Because 7B models cannot be fine-tuned with LoRA.", "Because the 1B has a better chat template than the 7B." ], "answer_index": 1, "rationale": "The on-ramp principle: iteration speed compounds. A 1B lets you run 10+ experiments in the time a 7B yields one. You learn data quality, format, and hyperparameter sensitivity from iteration VOLUME. Validate the pipeline where iteration is cheap (1B), then graduate to the production base ONLY when a graduation signal fires (capability ceiling, reasoning floor, tokenizer mismatch) — never 'just because bigger.'" }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team is choosing a base for a reasoning-distillation task: they want a smaller model to learn to reason by training on DeepSeek-R1's traces. Two team members argue — one says start from any strong 7B base, the other says start from an R1-distill checkpoint. Who is right, and what FT03 principle decides it?", "options": [ "The first team member — a strong 7B base is always better for reasoning.", "The second team member — for reasoning-distillation, the LINEAGE matters more than the base size. You start from an R1-distill (R1-Distill-Qwen or R1-Distill-Llama) because it is already in the reasoning-distillation lineage; you steer it further. The family's fine-tuning profile (DeepSeek = reasoning distillation lineage) is the deciding principle.", "Neither — they should start from a merged community model.", "The first team member — base size is the only property that matters for reasoning." ], "answer_index": 1, "rationale": "For reasoning-distillation tasks, the DeepSeek family's profile is 'the lineage matters more than the base size.' R1-distills are already distilled from R1 and are the correct starting point to steer further toward reasoning behavior. Starting from a generic strong 7B ignores the lineage and requires re-establishing the reasoning distribution. This illustrates why base selection maps families to use cases rather than picking the largest available base." } ] }