Post Training Qwen2.5-0.5B with RLHF on Heterogeneous, Long-Form Preference Data

Read technical notes here.

Technical notes

Project question: a small model, but not a small alignment problem.
I wanted to test whether the complete RLHF process could remain useful and inspectable under a modest compute budget without simplifying the data into a short, single-domain task. HelpSteer3 made that question concrete: its 40,476 preference records span general, code, STEM, and multilingual prompts, with responses ranging from compact answers to code and multi-step explanations. An early length audit showed that a 1,024-token total training budget would truncate roughly 40% of SFT and reward-model examples. Expanding it to 4,096 tokens reduced truncation to about 1%. The goal was not to claim that a 0.5B policy solves general alignment, but to train the full pipeline and make both its gains and its failures available for inspection.

The goal is to use preference data from NVIDIA HelpSteer3 to align the Qwen2.5-0.5B-Instruct LLM toward responses that a human would prefer over boring or mechanistic bags of words. The 3-step process begins with supervised fine-tuning (SFT) using the preferred responses in the training dataset. Then I train a reward model on HelpSteer3 prompt-response pairs, where one response is preferred over the other. The reward model learns to assign higher scores to preferred responses and lower scores to rejected responses. Finally, I use that reward model to train the LLM policy toward preferred responses using PPO, with a frozen SFT reference policy, KL control, and value estimates for token-level optimization.

I learned that making PPO run stably is only one part of LLM alignment; the quality of the preference data, the reward model, the stopping behavior, and the evaluation protocol can matter just as much as the optimizer. This project gave me a working system in which those interactions are visible and measurable. After training, I used the Base, SFT, and PPO policies to generate responses to all validation prompts and scored them with the same learned reward model. For human review, I built the response explorer at the top of this page so I could move from a number in a table to the actual text that produced it. Even though I used the smallest model in the Qwen2.5 family because of my own compute budget, the project became a substantial learning curve: I read through the development of RLHF and policy optimization, implemented the complete training sequence, and learned most from the places where a high reward did not mean a good answer.

The RLHF pipeline is implemented in my RLHF repository, where the concise report, technical companion, resolved configurations, and experiment records document the implementation in more detail.

Training pipeline

HelpSteer3 provides 38,459 training and 2,017 validation preference records spanning general, STEM, code, and multilingual prompts. After filtering invalid or tied preference rows, the final TRL training run used 36,264 rows for SFT and reward-model training and 1,917 reward-model validation pairs. Its breadth is operationally important: code, multi-step explanations, multilingual conversations, and long prompt histories make sequence handling part of the method rather than a cosmetic configuration choice.

1. Supervised fine-tuning. The preferred responses train a response-only LoRA adapter over Qwen2.5-0.5B-Instruct. Expanding the total sequence budget from 1,024 to 4,096 tokens reduced SFT truncation from 38.47% to 0.83% and reward-model pair truncation from 40.82% to 1.00%. The final SFT run used one epoch, LoRA rank 16, an effective batch size of 32, and reached 72.02% validation mean token accuracy.
2. Reward modeling. A scalar reward head is trained with a Bradley–Terry pairwise ranking loss so chosen responses score above rejected ones. The final reward model starts from the merged SFT model, uses LoRA rank 32 and two total epochs, and reached 65.62% validation pairwise accuracy across 1,917 usable pairs. This score is a learned preference proxy, not an oracle; its weaker STEM and general-domain accuracy became important when interpreting PPO.
3. Guarded PPO alignment. The SFT checkpoint initializes both the trainable policy and the frozen reference, while the reward model initializes the value model. The completed PPO run generated 12,032 on-policy responses over 188 updates, with every batch balanced across general, code, STEM, and multilingual prompts and each response allowed up to 768 new tokens.

The final PPO stage was designed around preventing obvious ways of gaming an imperfect reward model. Reward scores were clipped to empirical 0.5th and 99.5th percentile bounds estimated from a stratified sample; a response that missed EOS received the calibrated lower-bound score; a smooth 4-gram repetition penalty activated only beyond the preferred-response 95th percentile; and KL regularization kept the policy anchored to the SFT reference. The run also saved complete policy, value, optimizer, scheduler, RNG, rollout-position, and guardrail state for exact resume.

Long responses made GPU execution a separate problem. A dense batch × response × vocabulary score tensor would consume roughly 27.8 GiB by itself at the final batch and response sizes. The final path generated token IDs without retaining all vocabulary scores, used a rollout-only KV cache, recomputed log-probabilities only for sampled response tokens in bounded chunks, trimmed all-masked prompt columns, shared the reference backbone, and separated policy/value backward graphs. This kept the completed run to about 37.6 GiB peak allocated memory without changing the PPO objective or effective batch size.

Discussion

I started out resolved to understand every moving part of the RLHF process, so the project first used custom SFT, reward-model, and token-level PPO code. Those implementations were useful precisely because they failed in visible ways: truncated responses, empty outputs, wrong checkpoint paths, weak EOS handling, unstable reward scales, and high-reward repetition. I later moved the main trainer loops to Hugging Face TRL, while keeping the repository-owned preprocessing, model initialization, reward calibration, exact-resume wrapper, evaluation suite, and response explorer. The custom code exposed assumptions; TRL gave me a cleaner substrate on which to finish the experiment.

The final guarded run is the iteration I am freezing. It completed its declared 12,000-episode budget as 12,032 rollouts, rather than stopping at the earlier 100-update checkpoint. During the second half of training, EOS became more frequent, cap hits and response length declined, repetition eased, and the value loss settled near the reward scale. That is evidence that the PPO process learned without numerical or empty-output collapse. It is not evidence that the learned reward model captured every human preference.

With my evaluation code, I can perform Base vs SFT, Base vs PPO, and SFT vs PPO comparisons from the same generated response table. The quantitative judge is the trained reward model; the qualitative judge is the actual response. The latter matters because a policy can optimize what the reward model likes while becoming longer, repetitive, factually wrong, or merely well-formatted. That need for skeptical auditing motivated the response explorer and the 4-gram, EOS, cap, and reward-margin metadata shown with every row. I don’t have infinite compute, alright! It is also the reason I chose the half-a-billion-parameter model and made the evaluation resumable one policy at a time.

Evaluation

The final evaluator generated Base, SFT, and guarded PPO responses once for each of the 2,017 HelpSteer3 validation prompts. Every policy used the same tokenizer, prompt budget of 3,072 tokens, response budget of 768 new tokens, and sampled decoding settings. The same two-epoch reward model scored all responses. These are reward-model comparisons, not a blinded human preference study.

Qualitative analysis of responses

The automated audit uses reward margins, response length, cap hits, EOS behavior, repeated word-level 4-grams, and selected sensitive-term checks to create review queues. It found 9 qualified PPO candidates, 120 strong PPO-loss candidates, 326 repetition-risk rows, and 155 high-reward repetition or stopping mismatches. These groups overlap; they are routes into the data, not quality labels.

The guarded policy is much easier to inspect than the earlier checkpoint that maximized the proxy more aggressively. Its median response is 355 tokens, its cap-hit rate is 16.31%, and 16.16% of responses cross the 25% repeated-4-gram threshold. Base is still cleaner on repetition at 8.18%, so the guardrails alleviated rather than eliminated the problem. Manual inspection finds useful local improvements—clearer structure, better coverage of some multi-part prompts, supportive wording, and compact answers—but also invalid code, confused factual reasoning, prompt restatement, fabricated details, and occasional loops that the reward model still scores highly.

That is the main lesson I would defend. PPO can be numerically stable and remain close to its reference while still optimizing blind spots in a scalar judge. The final policy does not universally improve Qwen2.5-0.5B-Instruct, but the complete training and explorer make the distinction between reward, behavior, and visible response quality concrete. I am freezing this phase here rather than running another unqualified block of PPO against the same imperfect reward model.

Future work

Independent preference review: Run a blinded human or strong-LLM comparison on a stratified subset and measure where it agrees with the learned reward model.
Hard-negative reward modeling: Add audited loops, prompt restatements, fabricated citations, malformed code, and incorrect STEM answers to a new reward-model training set.
Task-specific checks: Execute code where possible and use factual or symbolic checks for correctness-sensitive prompts.
Controlled token-budget studies: Compare 512, 768, and 1,024 generated-token budgets while holding checkpoint, prompt order, seed, batch size, and software environment fixed.
Preference-objective baselines: Evaluate the prepared DPO path from the same SFT checkpoint only when it can be judged with the same full suite.
Scale carefully: Repeat the pipeline on a stronger model after the evaluator and reward model become more trustworthy.

I would also be able to reuse this response explorer, the heuristic queues, and the evaluation infrastructure to study other alignment methods without reducing the result to one aggregate score.