Reinforcement fine-tuning with LLM-as-a-judge | Synthetic Intelligence

Massive language fashions (LLMs) now drive probably the most superior conversational brokers, inventive instruments, and decision-support techniques. Nonetheless, their uncooked output usually incorporates inaccuracies, coverage misalignments, or unhelpful phrasing—points that undermine belief and restrict real-world utility. Reinforcement Advantageous‑Tuning (RFT) has emerged as the popular technique to align these fashions effectively, utilizing automated reward alerts to interchange expensive guide labeling.

On the coronary heart of contemporary RFT is reward features. They’re constructed for every area via verifiable reward features that may rating LLM generations via a bit of code (Reinforcement Studying with Verifiable Rewards or RLVR) or with LLM-as-a-judge, the place a separate language mannequin evaluates candidate responses to information alignment (Reinforcement Studying with AI Suggestions or RLAIF). Each these strategies present scores to the RL algorithm to nudge the mannequin to resolve the issue at hand. On this submit, we take a deeper take a look at how RLAIF or RL with LLM-as-a-judge works with Amazon Nova fashions successfully.

Why RFT with LLM‑as‑a-judge in comparison with generic RFT?

Reinforcement Advantageous-Tuning can use any reward sign, easy hand‑crafted guidelines (RLVR), or an LLM that evaluates mannequin outputs (LLM-as-a-judge or RLAIF). RLAIF makes alignment way more versatile and highly effective, particularly when reward alerts are obscure and laborious to craft manually. In contrast to generic RFT rewards that depend on blunt numeric scoring like substring matching, an LLM choose causes throughout a number of dimensions—correctness, tone, security, relevance—offering context-aware suggestions that captures subtleties and domain-specific nuances with out task-specific retraining. Moreover, LLM judges supply built-in explainability via rationales (for instance, “Response A cites peer-reviewed research”), offering diagnostics that speed up iteration, pinpoint failure modes instantly, and scale back hidden misalignments, one thing static reward features can’t do.

Implementing LLM-as-a-judge: Six vital steps

This part covers the important thing steps concerned in designing and deploying LLM-as-a-judge reward features.

Choose the choose structure

The primary vital resolution is choosing your choose structure. LLM-as-a-judge presents two major analysis modes: Rubric-based (point- based mostly) judging and Choice-based judging, every suited to totally different alignment eventualities.

Outline your analysis standards

After you’ve chosen your choose kind, articulate the precise dimensions that you just wish to enhance. Clear analysis standards are the muse of efficient RLAIF coaching.

For Choice-based judges:

Write clear prompts explaining what makes one response higher than one other. Be express about high quality preferences with concrete examples. Instance: “Desire responses that cite authoritative sources, use accessible language, and instantly handle the consumer’s query.”

For Rubric-based judges:

We advocate utilizing Boolean (cross/fail) scoring for rubric-based judges. Boolean scoring is extra dependable and reduces choose variability in comparison with fine-grained 1–10 scales. Outline clear cross/fail standards for every analysis dimension with particular, observable traits.

Choose and configure your choose mannequin

Select an LLM with adequate reasoning functionality to guage your goal area, configured via Amazon Bedrock and referred to as utilizing a reward AWS Lambda perform. For frequent domains like math, coding, and conversational capabilities, smaller fashions can work nicely with cautious immediate engineering.

Refine your choose mannequin immediate

Your choose immediate is the muse of alignment high quality. Design it to provide structured, parseable outputs with clear scoring dimensions:

• Structured output format – Specify JSON or parseable format for easy extraction
• Clear scoring guidelines – Outline precisely how every dimension must be calculated
• Edge case dealing with – Handle ambiguous eventualities (for instance, “If response is empty, assign rating 0”)
• Desired behaviors – Explicitly state behaviors to encourage or discourage

Align choose standards with manufacturing analysis metrics

Your reward perform ought to mirror the metrics that you’ll use to guage the ultimate mannequin in manufacturing. Align your reward perform with manufacturing success standards to allow fashions designed for the proper aims.

Alignment workflow:

1. Outline manufacturing success standards (for instance, accuracy, security) with acceptable thresholds
2. Map every criterion to particular choose scoring dimensions
3. Validate that choose scores correlate together with your analysis metrics
4. Check the choose on consultant samples and edge circumstances

Constructing a sturdy reward Lambda perform

Manufacturing RFT techniques course of 1000’s of reward evaluations per coaching step. Construct a resilient reward Lambda perform to assist present coaching stability, environment friendly compute utilization, and dependable mannequin conduct. This part covers methods to construct a reward Lambda perform that’s resilient, environment friendly, and manufacturing prepared.

Composite reward rating structuring

Don’t rely solely on LLM judges. Mix them with quick, deterministic reward parts that catch apparent failures earlier than costly choose evals:

Core parts

Infrastructure readiness

1. Implement exponential backoff: Handles Amazon Bedrock API charge limits and transient failures gracefully
2. Parallelization technique: Use ThreadPoolExecutor or async patterns to parallelize choose calls throughout rollouts to cut back latency
3. Keep away from Lambda chilly begin delays: Set an acceptable Lambda timeout (quarter-hour beneficial) and provisioned concurrency (~100 for typical setups)
4. Error dealing with: Add complete error dealing with that returns impartial/noisy rewards (0.5) fairly than failing your complete coaching step

Check your reward Lambda perform for resilience

Validate choose consistency and calibration:

• Consistency: Check choose on the identical samples a number of instances to measure rating variance (must be low for deterministic analysis)
• Cross-judge comparability: Examine scores throughout totally different choose fashions to determine analysis blind spots
• Human calibration: Periodically pattern rollouts for human overview to catch choose drift or systematic errors
• Regression testing: Create a “choose take a look at suite” with recognized good/unhealthy examples to regression take a look at choose conduct

RFT with LLM-as-a-judge – Coaching workflow

The next diagram illustrates the whole end-to-end coaching course of, from baseline analysis via choose validation to manufacturing deployment. Every step builds upon the earlier one, making a resilient pipeline that balances alignment high quality with computational effectivity whereas actively stopping reward hacking and supporting production-ready mannequin conduct.

Actual-world case research: Automating authorized contract overview

On this part, we discuss with a real-world use case with a number one authorized business associate. The duty is to generate feedback on dangers, assessments, and actions on authorized documentation with respect to the insurance policies and former contracts as reference paperwork.

Problem

Accomplice was concerned about fixing the issue of automating the method of reviewing, assessing, and flagging dangers in authorized contract paperwork. Particularly, they wished to guage potential new contracts towards inside tips and laws, previous contracts, and legal guidelines of the nation pertaining to the contract.

Resolution

We formulated this drawback as one the place we’re offering a goal doc (the “contract” that wants analysis), and a reference doc (the grounding doc and context) and count on the LLM to generate a JSON with a number of feedback, remark sorts, and beneficial actions to take based mostly on the evaluation. The unique dataset out there for this use case was comparatively small that included full contracts together with annotations and feedback from authorized specialists. We used LLM as a choose utilizing GPT OSS 120b mannequin because the choose and a customized system immediate throughout RFT.

RFT workflow

Within the following part we cowl particulars of the important thing features within the RFT workflow for this use case.

Reward Lambda perform for LLM-as-a-judge

The next code snippets current the important thing parts of the reward Lambda perform.

Be aware: title of Lambda perform ought to have “SageMaker”, for instance, "arn:aws:lambda:us-east-1:123456789012:perform:MyRewardFunctionSageMaker"

a) Begin with defining a high-level goal

# Contract Evaluation Analysis - Unweighted Scoring
You're an professional contract reviewer evaluating AI-generated feedback. Your PRIMARY goal is to evaluate how nicely every predicted remark identifies points within the TargetDocument contract clauses and whether or not these points are justified by the Reference tips.

b) Outline the analysis strategy

## Analysis Strategy
For every pattern, you obtain:
- **TargetDocument**: The contract textual content being reviewed (the doc underneath analysis)
- **Reference**: Reference tips/requirements used for the overview (the analysis standards)
- **Prediction**: A number of feedback from the AI mannequin
**Essential**: The SystemPrompt reveals what directions the mannequin acquired. Take into account whether or not the mannequin adopted these directions when evaluating the prediction high quality.
**CRITICAL**: Every remark should determine a selected problem, hole, or concern IN THE TARGETDOCUMENT CONTRACT TEXT ITSELF. The remark's text_excerpt subject ought to quote problematic contract language from the TargetDocument, NOT quote textual content from the Reference tips. The Reference justifies WHY the contract clause is problematic, however the problem should exist IN the contract.
Consider EACH predicted remark independently. Feedback ought to flag issues within the contract clauses, not merely cite Reference necessities.

c) Describe the scoring dimensions with clear specs on how a selected rating must be calculated

## Scoring Dimensions (Per Remark)
**EVALUATION ORDER**: Consider on this sequence: (1) TargetDocument_Grounding, (2) Reference_Consistency, (3) Actionability
### 1. TargetDocument_Grounding
**Evaluates**: (a) Whether or not text_excerpt quotes from TargetDocument contract textual content, and (b) Whether or not the remark is related to the quoted text_excerpt
**MANDATORY**: text_excerpt should quote from TargetDocument contract textual content. If text_excerpt quotes from Reference as an alternative, rating MUST be 1.
- **5**: text_excerpt accurately quotes TargetDocument contract textual content AND remark identifies a extremely related, legitimate, and notable problem in that quoted textual content
- **4**: text_excerpt accurately quotes TargetDocument contract textual content AND remark identifies a sound and related problem in that quoted textual content
- **3**: text_excerpt accurately quotes TargetDocument contract textual content AND remark is considerably related to that quoted textual content, however concern has reasonable validity
- **2**: text_excerpt accurately quotes TargetDocument contract textual content BUT remark has weak relevance to that quoted textual content, or concern is questionable
- **1**: text_excerpt does NOT quote TargetDocument contract textual content (quotes Reference as an alternative, or no precise quote), OR remark is irrelevant to the quoted textual content
### 2. Reference_Consistency
...
...

d) Clearly outline the ultimate output format to parse

## Scoring Calculation
**Comment_Score** = Easy common of the three dimensions:
- Comment_Score = (TargetDocument_Grounding + Reference_Consistency + Actionability) / 3
**Aggregate_Score** = Common of all Comment_Score values for the pattern
## Output Format
For every pattern, consider ALL predicted feedback and supply:
```json
{ "feedback": [ 
        { "comment_id": "...",
          "TargetDocument_Grounding": {"score": X, "justification": "...", "supporting_evidence": "Verify text_excerpt quotes actual TargetDocument contract text and comment is relevant to it"},
          "Reference_Consistency": {"score": X, "justification": "...", "supporting_reference": "Quote from Reference that justifies the concern OR explain meaningful reasoning"},                   
          "Actionability": {"score": X, "justification": "Assess if action is clear, grounded in TargetDocument and Reference, and relevant to comment"},
          "Comment_Score": X.XX 
        } ],
  "Aggregate_Score": {
          "rating": X.XX,
          "total_comments": N,
          "rationale": "..." 
   }
}
```

e) Create a high-level Lambda handler, offering adequate multithreading for sooner inference

def lambda_handler(occasion, context): 
        scores: Record[RewardOutput] = []
        samples = occasion
        max_workers = len(samples)
        print(f"Evaluating {len(samples)} objects with {max_workers} threads...")
        with ThreadPoolExecutor(max_workers=max_workers) as executor:
                futures = [executor.submit(judge_answer, sample) for sample in samples]
                scores = [future.result() for future in futures]
        print(f"Accomplished {len(scores)} evaluations")
        return [asdict(score) for score in scores]

Deployment of the Lambda perform

We used the next AWS Identification and Entry Administration (IAM) permissions and settings within the Lambda perform. The next configurations are required for reward Lambda features. RFT coaching can fail if any of them are lacking.

a) Permissions for Amazon SageMaker AI execution function

Your Amazon SageMaker AI execution function will need to have permission to invoke your Lambda perform. Add this coverage to your Amazon SageMaker AI execution function:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Useful resource": "arn:aws:lambda:area:account-id:perform:function-name"
        }
    ]
}

b) Permissions for Lambda perform execution function

Your Lambda perform’s execution function wants fundamental Lambda execution permissions and the permissions to Invoke the choose Amazon Bedrock mannequin.

Be aware: This resolution follows the AWS shared duty mannequin. AWS is chargeable for securing the infrastructure that runs AWS companies within the cloud. You’re chargeable for securing your Lambda perform code, configuring IAM permissions, implementing encryption and entry controls, managing knowledge safety and privateness, configuring monitoring and logging, and verifying compliance with relevant laws. Observe the precept of least privilege by scoping permissions to particular useful resource ARNs. For extra data, see Safety in AWS Lambda and Amazon SageMaker AI Safety within the AWS documentation.

c) Add provisioned concurrency

Publish a model of the Lambda and to allow the perform to scale with out fluctuations in latency, we added some provisioned concurrency. 100 was adequate on this case, nevertheless, there’s extra room for value enhancements right here.

d) Set Lambda timeout to fifteen minutes

Customizing the coaching configuration

We launched Nova Forge SDK that can be utilized for your complete mannequin customization lifecycle—from knowledge preparation to deployment and monitoring. Nova Forge SDK removes the necessity to seek for the suitable recipes or container URI for particular strategies.

You should use the Nova Forge SDK to customise coaching parameters in two methods: present a full recipe YAML utilizing recipe_path or cross particular fields utilizing overrides for selective modifications. For this use case, we use overrides to tune the rollout and coach settings as proven within the following part.

# Launch coaching with recipe overrides
end result = customizer.prepare(
        job_name="my-rft-run",
        rft_lambda_arn="",
        overrides={
                # Coaching config
                "max_length": 64000,
                "global_batch_size": 64,
                "reasoning_effort": None,
                # Information
                "shuffle": False,
                # Rollout
                "kind": "off_policy_async",
                "age_tolerance": 2,
                "proc_num": 6,
                "number_generation": 8,
                "max_new_tokens": 16000,
                "set_random_seed": True,
                "temperature": 1,
                "top_k": 0,
                "lambda_concurrency_limit": 100,
                # Coach
                "max_steps": 516,
                "save_steps": 32,
                "save_top_k": 17,
                "refit_freq": 4,
                "clip_ratio_high": 0.28,
                "ent_coeff": 0.0,
                "loss_scale": 1,
        },
)

Outcomes

RFT with Amazon Nova 2 Lite achieved a 4.33 mixture rating—the best efficiency throughout all evaluated fashions—whereas sustaining excellent JSON schema validation. This represents a big enchancment, demonstrating that RFT can produce production-ready, specialised fashions that outperform bigger general-purpose options.

We evaluated fashions utilizing a “better of okay” single-comment setting, the place every mannequin generated a number of feedback per pattern and we scored the highest-quality output. This strategy establishes an higher sure on efficiency and allows a good comparability between fashions that produce single versus a number of outputs.

Determine 1 — JSON Schema Validation Scores (0–1 scale, increased is healthier)

Determine 2 — Combination LLM choose scores (1–5 scale, increased is healthier)

Key takeaways:

1. RFT achieved the best efficiency amongst evaluated fashions on this research.

Amazon Nova 2 Lite with RFT achieved a 4.33 mixture rating, outperforming each Claude Sonnet 4.5 and Claude Haiku 4.5, whereas additionally attaining excellent JSON schema validation.

2. Removes pointless coaching artifacts

Throughout SFT iterations, we noticed problematic behaviors together with repetitive remark technology and unnatural Unicode character predictions. These points, probably attributable to overfitting or dataset imbalances, didn’t seem in RFT checkpoints. RFT’s reward-based enhancements naturally discourages such artifacts, producing extra strong and dependable outputs.

3. Robust generalization to new choose standards

After we evaluated RFT fashions utilizing a modified choose immediate (aligned however not an identical to the coaching reward perform), efficiency remained sturdy. This demonstrates that RFT learns generalizable high quality patterns fairly than overfitting particular analysis standards. This can be a vital benefit for real-world deployment the place necessities evolve.

4. Compute concerns

RFT required 4–8 rollouts per coaching pattern, rising compute prices in comparison with SFT. This overhead is amplified when utilizing non-zero reasoning effort settings. Nonetheless, for mission-critical functions the place alignment high quality instantly impacts enterprise outcomes—similar to authorized contract overview, monetary compliance, or healthcare documentation, the efficiency good points justify the extra compute prices.

Conclusion

Reinforcement Advantageous-Tuning (RFT) with LLM-as-a-judge represents a strong strategy to aligning LLMs for domain-specific functions. As demonstrated in our authorized contract overview case research, this system delivers important enhancements over each base fashions and conventional supervised fine-tuning (SFT) approaches, with RFT attaining the best mixture scores throughout all analysis dimensions. For groups constructing mission-critical AI techniques the place alignment high quality instantly impacts enterprise outcomes, RFT with LLM-as-a-judge presents a compelling path ahead. The methodology’s explainability, flexibility, and superior efficiency make it significantly priceless for advanced domains like authorized overview (or Monetary Companies or Healthcare) the place refined nuances matter.

Organizations contemplating this strategy ought to begin small—validate their choose design on curated benchmarks, confirm infrastructure resilience, and scale regularly whereas monitoring for reward hacking. With correct implementation, RFT can rework succesful base fashions into extremely specialised, production-ready techniques that constantly ship aligned, reliable outputs.

References:

1. Amazon Nova Developer Information for Amazon Nova 2
2. Nova Forge SDK- GitHub
3. Reinforcement Advantageous-Tuning (RFT) with Amazon Nova fashions

Disclaimer:

The authorized contract overview use case described on this submit is for technical demonstration functions solely. AI-generated contract evaluation shouldn’t be an alternative choice to skilled authorized recommendation. Seek the advice of certified authorized counsel for authorized issues.

Concerning the authors