Blog post: https://si.inc/posts/fdm1/
FDM-1 is a foundational model for computer use designed by Standard Intelligence
FDM-1 is trained on “a portion” of a 11-million hour screen recording dataset
Their video encoder can compress 2 hours of 30 FPS video into 1M tokens
It trains and infers directly on video instead of screenshots and can learn unsupervised from the entirety of the internet
Training Recipe
The Inverse Dynamics Model (IDM) is trained on 40,00 hours of labeled screen recordings
Then they use the IDM to label the 11-million-hour video corpus
IDM-labeled videos are utilized to train a Forward Dynamics Model (FDM) which autoregressively predicts the next action
FDM’s output token space consists of key presses and mouse movement deltas (modeling any action taken on a computer)
Video Encoder
Honestly did not go into much detail about their video encoder
Trained their video encoder on a masked compression objective
Training is unsupervised so tasks like inverse dynamics, action prediction, frame reconstruction, and random text transcription measure the abilities of the encoder
The VJEPA paper is similar but not exactly the same (look at it for inspiration)
Video Encoder vs ViT they observe ~100x faster convergence during training
Inverse Dynamics
To train a model they need more data than contractors can provide
They train an IDM to predict high-quality labels letting them achieve similar efficiency when training on arbitrary videos as when training on human-gathered ground-truth data
Labeling video is fundamentally non-causal (eg. you can’t label a Cmd + C until you see the resulting pasted sequence)
Forward Dynamics
FDM predicts next action given the prior frames and actions
FDM operates directly on video and action tokens
Inference is low-latency and allows modeling a multitude of tasks that current designs cannot capture
Conclusion
“This brings computer action from a data-constrained regime to a compute-constrained one”