OpenWorldLib Developer Guidance
输入“/”快速插入内容
OpenWorldLib Developer Guidance
OpenWorldLib Developer Guidance
用户8551用户85513月21日修改
Method template
The OpenWorldLib repository encompasses a wide range of tasks. To better distinguish and manage different functional components within the framework, we have established the following modular division:
Operator — Accepts complex interaction signals and perception inputs (e.g., text instructions, visual observations, keyboard/mouse actions). The Operator validates, parses, and preprocesses these raw inputs into a standardized format consumable by downstream modules.
Synthesis — Consumes the processed output from the Operator and generates predicted perception outputs such as video frames, audio clips, or image sequences. This module is the primary generation backend for visual world simulation.
Reasoning — Consumes the processed output from the Operator and produces structured, formatted textual outputs. This module primarily targets VLM-based methods and can also serve as an auxiliary component to guide or condition the Synthesis module.
Representation — Consumes the processed output from the Operator and produces 3D or simulation results. This module targets 3D generation methods (e.g., point clouds, Gaussian splatting) and simulator-based environments (e.g., AI2-THOR).
Memory — Provides long-term and short-term memory management for multi-turn interactive sessions. The Memory module records, selects, compresses, and converts historical context into model-ready formats (e.g., KV caches, latent tokens) across generation turns.
Operator
The Operator is the entry point for all user interactions and perception signals. It is responsible for receiving, validating, and preprocessing inputs before they are dispatched to Synthesis, Reasoning, or Representation modules. The base template and required methods are as follows:
代码块
class BaseOperator(object):
def __init__(self):
"""
Initialize the operator with empty interaction state and a template
that defines valid interaction types.
Operations include:
- textual_instruction: natural-language commands or prompts.
- visual_instruction: image/video-based guidance signals.
- action_instruction: mouse, keyboard, or gamepad inputs used to
control trajectory and viewpoint.
"""
self.current_interaction = []
self.interaction_template = []
def get_interaction(self, interaction_list):
"""
Receive and validate a batch of interaction commands. Each element
in `interaction_list` is checked against `self.interaction_template`
before being appended to the current interaction buffer.
Args:
interaction_list (list): A list of interaction commands
(e.g., ["forward", "camera_up"]).
Raises:
ValueError: If any interaction is not found in the template.
"""
for act in interaction_list:
self.check_interaction(act)
self.current_interaction.append(interaction_list)
def check_interaction(self, interaction):
"""
Validate a single interaction command against the allowed template.
Args:
interaction (str): The interaction command to validate.
Returns:
bool: True if the interaction is valid.
Raises:
ValueError: If the interaction is not in the template.
"""
if interaction not in self.interaction_template:
raise ValueError(f"{interaction} not in template")
return True
def process_interaction(self):
"""
Convert the buffered raw interactions into a model-ready format.
This method handles encoding, tokenization, or any task-specific
transformation of interaction signals.
Called by Pipeline.process() before generation.
"""
pass
def process_perception(self):
"""
Process incoming visual and audio perception signals. Unlike
process_interaction(), this method is designed for real-time,
continuously updated sensory streams (e.g., live camera feeds
or streaming audio).
Called by Pipeline.process() when real-time perception updates
are required.
"""
pass