PPO vs DPO vs GRPO: which RL method should you use?

The right method follows from the training signal you have: preference pairs point to DPO, a program that scores answers points to GRPO, and a learned reward model points to PPO. Nodus runs the first two as managed training runtimes, nodus/dpo and nodus/grpo-lora, each priced by a dry run and stopped at a dollar cap you set. The cost gap between them is mostly about how many models sit in GPU memory and how much text the trainer has to generate.

What does each method actually train on?

Each method learns from a different kind of signal, and that signal is the first thing to check.

PPO (Proximal Policy Optimization) is a general reinforcement learning algorithm. It alternates between sampling from the model and optimizing a clipped surrogate objective. For language models it became the RL step of RLHF. InstructGPT collected human rankings of model outputs, trained a reward model on them, then fine-tuned with PPO against that reward model.

DPO (Direct Preference Optimization) skips the reward model and the RL loop. It trains straight on rows of a prompt, a chosen answer and a rejected answer with a classification-style loss. The paper describes it as "eliminating the need for sampling from the LM during fine-tuning."

GRPO (Group Relative Policy Optimization) was introduced in the DeepSeekMath paper as "a variant of Proximal Policy Optimization (PPO)" that also reduces PPO's memory use. It samples a group of answers per prompt and scores each one against its group's average, so it needs no value model. The DeepSeek-R1 paper used it with rewards based on whether final answers were correct.

Model lists come from the TRL PPO trainer, TRL DPO trainer and TRL GRPO trainer docs. Runtimes are from the Nodus training runtimes reference.

How much GPU memory does each method need?

For full training of an 8B model, a typical PPO setup needs about twice the memory of DPO or GRPO, because it trains a second network. The TRL PPO trainer takes four models: the policy, a reference model, a reward model and a value model. The DeepSeekMath authors note that the value model "is typically another model of comparable size as the policy model."

We worked out the weights and trainable state for Qwen3-8B, which has 8.2B parameters per its model card. Trained models cost 16 bytes per parameter with mixed-precision Adam, following the ZeRO paper. Frozen models in 16-bit cost 2 bytes per parameter. We assume the reward and value models are the same size as the policy.

• Trained 8B model: 8.2B × 16 bytes = 131.2 GB.
• Frozen 8B model: 8.2B × 2 bytes = 16.4 GB.
• PPO: trained policy and value model, frozen reference and reward model. 131.2 × 2 + 16.4 × 2 = 295.2 GB.
• DPO: trained policy plus frozen reference. 131.2 + 16.4 = 147.6 GB. TRL's precompute_ref_log_probs option scores the dataset with the reference model before training, so the reference "does not need to be kept in memory." That brings it down to 131.2 GB.
• GRPO: trained policy only, with the reward computed in code. 131.2 GB with the KL term off, which is the default in both TRL and the Nodus runtime (beta: 0). Turning the KL term on adds the 16.4 GB reference, for 147.6 GB.

The value model is what doubles PPO's footprint. DPO and GRPO both fit the budget of a single trained model plus at most one frozen copy.

These figures leave out activations and, for PPO and GRPO, the generation cache. With LoRA, the trainable state for any of the three shrinks to well under a gigabyte on an 8B model, and the frozen base weights dominate. Our LoRA vs QLoRA guide has that math.

Which method spends the most time generating text?

GRPO generates the most text per training batch, PPO less, and DPO none. DPO reads finished answers from your dataset, while PPO and GRPO must sample new answers from the current model before every update.

Take a batch of 64 prompts with completions of up to 512 tokens, which is TRL's default max_completion_length for GRPO. As PPO is usually run, with one sampled answer per prompt, the worst case is 64 × 512 = 32,768 generated tokens. GRPO with a group of 4, the Nodus default, generates up to 64 × 4 × 512 = 131,072 tokens. With a group of 8, the TRL default, that is 262,144 tokens.

GRPO trades the value model's memory for extra sampling: a group of 8 is up to 8x PPO's generation for the same prompts.

The extra sampling is the price of the group baseline. It is usually worth paying for verifiable tasks, because generation runs fast on an inference engine and the group replaces an entire trained network. It also explains why the group size and the completion length are the two GRPO settings that move the bill most.

When should you use DPO?

Use DPO when you already have, or can cheaply label, pairs of a better and a worse answer to the same prompt. Typical sources are human ratings, thumbs-up and thumbs-down logs turned into pairs, or a stronger model judging two candidates.

DPO is the cheapest of the three to run and the simplest to debug, because each step is a supervised-style pass over fixed data. Its limit follows from the same fact: it only learns from answers already in the dataset and never tries new ones during training.

One trap is easy to miss. The Nodus training guide warns that "lower DPO loss does not establish better generated answers." Hold out a test set and compare generated answers from the starting model and the trained one before you trust a run.

When should you use GRPO?

Use GRPO when you can write a function that scores an answer, and the base model already gets some answers right. Math with a known result, code with unit tests, structured output with a schema and games with a rule-based win condition all qualify.

The second condition matters as much as the first. If every answer in a group gets the same score, all of them right or all of them wrong, the group advantage is zero and the step teaches nothing. Run a baseline evaluation first. Our GRPO explainer walks through the advantage math and the Nodus runtime settings in detail.

Where is PPO still the better choice?

PPO remains the better choice when your reward is a learned model scoring open-ended answers and you want the policy to explore against it. Classic RLHF for helpfulness or tone fits that description, since no short program can judge a good answer.

The learned value model also gives PPO a per-token baseline, while GRPO gives every token in an answer the same advantage. That can matter for long answers where only part of the text earns the reward. The cost is the extra trained model. There is also the tuning effort the DPO authors had in mind when they called RLHF "a complex and often unstable procedure."

Nodus has no managed PPO runtime today. You can train the reward model with the managed nodus/reward-model runtime from prompt, chosen and rejected rows, then run your own PPO training script as a Nodus job. If you have the same preference pairs, try DPO on them first, as it is far less work.

How do you run DPO or GRPO on Nodus?

On Nodus each method is a TrainingJob manifest that names a runtime. A DPO run needs a model and a dataset of prompt, chosen and rejected rows. This is the example from the dpo runtime reference:

apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: {name: my-dpo} spec: runtime: nodus/dpo model: {uri: "hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca"} data: {volume: my-dataset, format: JSONL} maxCostUSD: "5.00" 

A GRPO run swaps the dataset for an Environment, a task set with a grader. The grpo-lora runtime reference uses environment: {name: nodus/graph-coloring@1.0.0, trainTasks: 200, heldOutTasks: 64, seed: 42} with runtime: nodus/grpo-lora. Price either one before it runs, then start it:

$ nodus apply -f my-dpo.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time $ nodus apply -f my-dpo.yaml 

The dry-run estimate is the number of steps times the runtime's measured seconds per step. Both runtimes currently list 0.45 s on an H100 and 1 s on an L40S.

A 2,000-step DPO run on an L40S therefore estimates at 2,000 s, or 0.556 hours. At the L40S rate of $0.793/hr on the Nodus pricing page, that is $0.44. Prices checked November 3, 2026.

Treat that as a floor. Long completions and big GRPO groups add generation time the per-step figure may not reflect, and evaluation adds GPU time on top. maxCostUSD stops the run at your cap either way, and runs checkpoint to /nodus/state so they resume after an interruption.

FAQ

Can you combine these methods in one pipeline?

Yes, and it is common. A model can come from one of your Volumes at a pinned revision, so a Nodus DPO run can start from the output of an earlier SFT run.

Is ORPO or KTO a better fit than DPO?

They suit slightly different data. KTO learns from single answers labeled good or bad, so it fits thumbs-up and thumbs-down logs without pairing them. ORPO combines SFT and preference training in one pass with no reference model, and both have managed Nodus runtimes.

Does DPO need a GPU as large as PPO?

No. Full DPO on an 8B model needs about half the memory of a typical PPO setup by the arithmetic above. With QLoRA, the nodus/dpo preset for Qwen3-8B runs on a single 48 GB or 80 GB card.

Is GRPO only for reasoning models?

No. GRPO works for any task with an automatic score, such as format compliance, tool-call correctness or test pass rates. Reasoning made it famous because math and code answers are easy to check.

Originally published on the Nodus blog.