Introduced in Cosmos 3 - Omnimodal World Models for Physical AI
Dual-Stream Joint Attention
Two towers use independent parameters
Tokens from diffusion subsequence interacts with AR subsequence through dual-stream joint attention operation
Autoregressive subsequence attention
Tokens in the AR subsequence attend only to tokens within the AR subsequence using casual self-attention
Diffusion subsequence attention
Tokens in the DM subsequence uses full bidirectional attention
Union of AR and DM tokens serving as the keys and values. This allows each diffusion token to freely attend to the text prompts from AR subsequence
This applies to all conditional and diffusion tokens in the sequence, maintaining temporal and spatial consistency: