Rejection sampling fine-tuning is often the simplest strong choice when you can generate many answers, score them well, and turn the winners into clean supervised training data. It sits between plain supervised fine-tuning and heavier preference methods like DPO or PPO. Teams like it because it is understandable, debuggable, and surprisingly effective when the reward signal is decent.
What rejection sampling fine-tuning actually does
Rejection Sampling Fine-Tuning, or RSFT, is a simple loop:
- Start with a base or instruction-tuned language model.
- Give it prompts from your target task.
- Sample multiple completions for each prompt.
- Score those completions using humans, rules, tests, or a reward model.
- Keep the best completions and reject the rest.
- Fine-tune the model on the accepted responses using standard supervised learning.
That is the whole idea. No mysterious magic. No complex policy update. The model learns to imitate the better responses from its own sampled output, or from a stronger teacher model.
The method works because language models often can produce good answers, even when they do not produce them consistently. RSFT turns occasional quality into training data. If one response out of ten is excellent, you keep that one and teach the model to make it more often.
How it differs from DPO
Direct Preference Optimization, or DPO, does not train only on the winning answer. It trains on pairs: one response is preferred, the other is rejected. The objective pushes the model to assign higher probability to the chosen answer than to the rejected one.
That difference sounds small. It is not.
- RSFT says: “Here is the good answer. Copy this style and behavior.”
- DPO says: “Prefer this answer over that answer, given the same prompt.”
RSFT is closer to supervised fine-tuning. DPO is closer to preference learning, but without the complexity of full reinforcement learning. DPO often handles subtle human preferences better, such as tone, refusal style, verbosity, and safety boundaries.
The catch is that DPO needs high-quality preference pairs. Bad pairs poison the signal. If your “chosen” answer is only slightly better, or the labelers disagree, the model may learn noise with confidence. That gets annoying fast. You run one training job, wait hours, inspect samples, and find the model became more polite but less useful.
Where RSFT shines
RSFT is strong when outputs can be scored clearly. Code tasks are a great example. If a generated function passes unit tests, keep it. If it fails, reject it. Math is similar when final answers can be checked. Structured extraction, SQL generation, tool use, and policy-grounded customer support also fit well.
It works especially well when you can sample many candidates cheaply. Suppose you have 100,000 prompts. You generate 6 answers per prompt. That gives you 600,000 candidates. If your scorer keeps the top 20%, you now have 120,000 higher-quality examples for supervised fine-tuning. Simple. Scalable. Easy to audit.
RSFT is also easier to explain to non-research teams. Product managers, compliance reviewers, and data teams can inspect accepted examples. They can ask why a sample was kept. They can spot repeated flaws. That matters in production, where explainability is not just a research luxury.
Where RSFT falls short
RSFT can overfit to the reward model. If the scorer likes long answers, the fine-tuned model may become wordy. If the scorer misses subtle factual errors, those errors get preserved. If the generator lacks diversity, sampling ten answers may only produce ten versions of the same mediocre reply.
It drives me a little crazy that teams sometimes call RSFT “automatic data cleaning” and stop thinking. The rejected samples matter too. They can reveal why the model fails. Throwing them away without analysis loses useful information.
Another issue is distribution shift. RSFT trains on polished winners. At inference time, the model still sees messy user prompts, vague requests, and adversarial phrasing. If the collected prompts are too clean, the model may look great in offline tests and then stumble with real users.
How DPO compares in practice
DPO is usually better when your main problem is preference alignment rather than basic skill. If the model already answers correctly but its answers are too blunt, too long, too risky, or poorly calibrated, DPO can help.
DPO uses both the good and bad responses. This gives it a richer signal than RSFT. It can learn that a concise answer is better than a bloated one, even if both are factually correct. It can learn that a safe refusal is better than a partial policy violation.
But DPO is not free. It is sensitive to data quality, prompt mix, and hyperparameters. A preference dataset with 50,000 carefully labeled pairs can beat a much larger sloppy one. Also, DPO may reduce output diversity if pushed too hard. The model can become cautious, repetitive, or oddly tuned to the preference dataset’s quirks.
Other fine-tuning techniques worth knowing
- Supervised Fine-Tuning, or SFT: The starting point for most custom LLM work. You train the model on prompt-response examples. It is simple, stable, and useful for format, domain style, and task basics.
- RLHF with PPO: A classic alignment method. A reward model scores outputs, and reinforcement learning updates the model. It can work very well, but it is harder to run and easier to destabilize.
- RLAIF: Similar to RLHF, but feedback comes from AI systems instead of only humans. It is cheaper at scale, though it inherits the judge model’s blind spots.
- KTO: Kahneman-Tversky Optimization trains from desirable and undesirable examples without needing strict pairs. It can be useful when feedback is binary, such as “acceptable” or “bad.”
- ORPO: Odds Ratio Preference Optimization merges supervised learning and preference optimization into one training objective. It can be attractive when you want a simpler pipeline than separate SFT plus DPO.
- IPO: Identity Preference Optimization is another preference method designed to reduce some DPO instability risks, especially around overconfident preference margins.
Choosing the right method
Pick based on your data, not on hype.
| Use case | Best first choice |
| You have gold responses written by experts | SFT |
| You can generate many candidates and score them | RSFT |
| You have chosen versus rejected pairs | DPO |
| You need strong policy alignment at scale | RLHF or DPO |
| You have pass or fail feedback | RSFT or KTO |
A practical training recipe often looks like this: start with SFT, apply rejection sampling to improve task quality, then use DPO to tune preferences. This staged approach is common because each step fixes a different problem. SFT teaches the task. RSFT raises the average answer quality. DPO shapes tradeoffs.
A realistic example
Imagine a legal research assistant. The team starts with 40,000 expert-written examples for SFT. The model learns structure, citation style, and expected tone. Then they run RSFT on 200,000 real anonymized research prompts. For each prompt, they sample 5 answers and keep only responses that cite approved sources and pass factual checks.
After RSFT, citation accuracy rises from 78% to 89% in offline review. Good. But users still complain that answers are too cautious and sometimes bury the main point. The team then collects 30,000 preference pairs. Lawyers choose the more useful response. DPO improves user preference win rate from 52% to 64% in A/B testing.
That sequence makes sense. No single method had to do everything.
The bottom line
Rejection sampling fine-tuning is best seen as a high-quality data generation method, not a full replacement for preference optimization. It is practical, transparent, and strong when scoring is reliable. DPO is better for teaching relative judgment. PPO and related reinforcement methods can go further, but they demand more care.
If you are building a domain model, start simple. Use SFT for foundations. Use RSFT to turn many rough samples into fewer excellent ones. Add DPO when users care about subtle choices that a pass-fail scorer cannot capture. That path saves time, reduces weird training failures, and gives you more control over what the model actually learns.