trinity.common.workflows.rubric_judge_workflow module

A workflow with LLM-as-a-judge.

class trinity.common.workflows.rubric_judge_workflow.RubricJudgeWorkflow(*, task: Task, model: ModelWrapper, auxiliary_models: List[ModelWrapper] | None = None)

基类：SimpleWorkflow

A workflow using LLM-as-a-judge and rubrics to get the reward.

Adapted from https://arxiv.org/pdf/2507.17746

__init__(*, task: Task, model: ModelWrapper, auxiliary_models: List[ModelWrapper] | None = None)

reset(task: Task)

Modified from SimpleWorkflow.reset

run() → List[Experience]

Modified from SimpleWorkflow.run

get_judge_reward(response: str, judge: OpenAI) → Tuple[bool, float]

Get rewards with LLM-as-a-judge The prompts are adapted from RAR-IMPLICIT method in https://arxiv.org/pdf/2507.17746

class trinity.common.workflows.rubric_judge_workflow.RubricJudgeWorkflowWithAPI(*, task: Task, model: ModelWrapper, auxiliary_models: List[ModelWrapper] | None = None)

基类：RubricJudgeWorkflow

Rubric judge workflow using an external OpenAI-compatible API as the judge.

Example of workflow_args:

judge_model_name: "gpt-4o" judge_api_base_url_env: "OPENAI_BASE_URL" judge_api_key_env: "OPENAI_API_KEY"

__init__(*, task: Task, model: ModelWrapper, auxiliary_models: List[ModelWrapper] | None = None)