trinity.common.models.mm_utils module

Utilities for processing multi-modal data (images/videos) for specific vision-language models.

Main capabilities: - Parse prompts with media tags (<image> / <video>) - Detect multi-modal content in conversation messages - Normalize legacy transformers-style media parts to vLLM/OpenAI-style schema - Build training-time multimodal inputs with model processor - Keep client-side parsing behavior aligned with vLLM server parsing

Provides functions to: 1. Parse prompts with media tags (<image>/<video>) 2. Validate multi-modal content in conversations 3. Normalize and preprocess media inputs for inference/training 4. Construct vLLM-compatible message formats

Note

Processor availability is detected from local model assets (e.g., processor/preprocessor config files). The implementation does not rely on model-class-name matching.

Compatibility:

MultiModalRender normalizes legacy transformers-style message parts (e.g., type=image with url/path/base64) into vLLM/OpenAI-style part schema before calling vLLM parse_chat_messages. vLLM is loaded only by its renderer, so text-only backends can import the shared helpers without installing the optional vLLM dependency.

trinity.common.models.mm_utils.should_use_processor(model_path: str) → bool

trinity.common.models.mm_utils.processor_or_tokenizer_cls(model_path: str) → Any

trinity.common.models.mm_utils.build_mm_message(prompt: str, images: List[str | Any], videos: List[str | Any]) → Dict[str, Any]

Construct multi-modal message by injecting media references at tag positions in prompt.

Parses prompt for <image>/<video> tags, replaces them with corresponding media references, and handles surplus media items. Extra media (beyond tag count) is prepended to content.

Parameters:

• prompt – Text containing optional <image> and <video> tags as media placeholders. Example: “First <image> then <video> and finally <image>”

• images – List of image references (file paths, URLs, or PIL images) in order of appearance.

• videos – List of video references (file paths, URLs) in order of appearance.

Returns:

{

“role”: “user”, “content”: [

{“type”: “image”, “image”: …}, # Surplus media first {“type”: “video”, “video”: …}, {“type”: “text”, “text”: “First “}, {“type”: “image”, “image”: …}, # Tag-replaced media …

]

}

Return type:

Message dictionary formatted for VL models

Raises:

ValueError – If prompt contains more <image> tags than provided images, or more <video> tags than provided videos.

Behavior details:

• Tags are case-sensitive and must be exact: “<image>”, “<video>”

• Empty text segments between tags are omitted

• Surplus media (images/videos beyond tag count) appears at START of content list

• Text segments preserve original prompt ordering around tags

trinity.common.models.mm_utils.has_multi_modal_content(messages: List[Dict]) → bool

Check if any message contains non-text (image/video) content.

Inspects message content structure to detect multi-modal elements. Handles both: - String content (text-only, returns False) - List content (multi-modal candidates)

Parameters:

messages – List of conversation messages. Each message must contain a “content” field. Content may be: - str: Plain text message - List[Dict]: Multi-modal content items (each with “type” key)

Returns:

True if any message contains at least one non-text content item (type != “text”), False otherwise.

Example

>>> msg = [{"role": "user", "content": [{"type": "text", "text": "Hi"}, {"type": "image", "image": "..."}]}]
>>> has_multi_modal_content(msg)
True

trinity.common.models.mm_utils.combine_output_token_ids(output_token_ids: List[int], multi_modal_inputs: dict[str, Tensor] | None = None)

class trinity.common.models.mm_utils.MultiModalRender(model_path: str, *args, **kwargs)

Bases: ABC

Client-side processor that mirrors server’s multimodal handling.

__init__(model_path: str, *args, **kwargs)

abstractmethod build_mm_input_for_training(*, messages: List[Dict[str, Any]] = None, tools: List[Dict[str, Any]] | None = None, input_ids: List[List[int]] = None, multi_modal_data: Dict[str, List] = None) → Dict[str, Any] | None

abstractmethod process_messages(messages: list[dict[str, Any]], *args, **kwargs)

abstractmethod async process_messages_async(messages: list[dict[str, Any]], *args, **kwargs)

class trinity.common.models.mm_utils.vLLMMultiModalRender(model_path: str, *, media_io_kwargs: dict[str, dict[str, Any]] | None = None, allowed_local_media_path: str = '', allowed_media_domains: list[str] | None = None, trust_request_chat_template: bool = False, mm_processor_kwargs: dict[str, Any] | None = None)

Bases: MultiModalRender

Client-side processor that mirrors vLLM server’s multimodal handling.

This class enables RL training endpoints to extract multimodal data with identical processing to the inference server, ensuring consistency. Legacy transformers-style content parts are normalized to vLLM-compatible OpenAI part schema before parsing.

__init__(model_path: str, *, media_io_kwargs: dict[str, dict[str, Any]] | None = None, allowed_local_media_path: str = '', allowed_media_domains: list[str] | None = None, trust_request_chat_template: bool = False, mm_processor_kwargs: dict[str, Any] | None = None)

Initialize the client-side multimodal processor.

Parameters:

• model_path – Path to the model

• media_io_kwargs – Media I/O configuration (mirrors –media-io-kwargs)

• allowed_local_media_path – Path to allowed local media directory

• allowed_media_domains – List of allowed media domains

• trust_request_chat_template – Whether to trust request-provided chat template

• mm_processor_kwargs – Additional processor kwargs for multimodal processing

build_mm_input_for_training(*, messages: List[Dict[str, Any]] = None, tools: List[Dict[str, Any]] | None = None, input_ids: List[List[int]] = None, multi_modal_data: Dict[str, List] = None) → Dict[str, Any] | None

Build multimodal tensors aligned with training token sequence.

Parameters:

• messages – Optional chat messages used to derive token ids and/or multimodal payloads when not provided explicitly.

• tools – Optional tool schema list passed to mm_processor.apply_chat_template when tokenizing messages.

• input_ids – Optional token ids (shape (seq_len,) or (batch_size, seq_len)). When omitted, token ids are generated from messages.

• multi_modal_data – Optional multimodal payload returned by process_messages (e.g., image/video media items). When omitted, it is inferred from messages.

Returns:

A dictionary containing multimodal training tensors, including mm_token_type_ids and media-processor outputs (for example image and/or video tensors), or None when multimodal processing is not available or no multimodal content exists.

Raises:

ValueError – If both input_ids and messages are missing.

Notes

• Requires an initialized multimodal processor (self.mm_processor). If unavailable for the current model, returns None.

• 1D input_ids are reshaped to (1, seq_len) to satisfy processor expectations.

process_messages(messages: list[dict[str, Any]], content_format: ChatTemplateContentFormat = 'string') → tuple[list[ConversationMessage], MultiModalDataDict | None]

Process chat messages and extract multimodal data.

This replicates the server-side parse_chat_messages behavior.

Parameters:

• messages – List of chat messages with potential multimodal content

• content_format – Chat template content format (“string” or “openai”)

Returns:

Tuple of (conversation, mm_data) matching server output

async process_messages_async(messages: list[dict[str, Any]], content_format: ChatTemplateContentFormat = 'string') → tuple[list[ConversationMessage], MultiModalDataDict | None]

Async version of process_messages for concurrent media fetching.