Revisiting Feature Prediction for Learning Visual Representations from Video

Paper Link: https://arxiv.org/pdf/2404.08471

JEPA §

Main idea behind a JEPA is to learn to predict the representation of an input from the representation of another input

Basic architecture is made of an encoder,

Which computes the representations of the inputs

and a predictor, which predicts the representation of from the representation of

the variable indicates the transformation or corruption between and

Training Objective §

Visual Encoder is used to satisfy the constraint that representations computed from one part of the video should be predictable from representations computed from another part of the video,

The predictor network maps the representation of x to the representation of y and is trained simultaneously with the encoder

Network Parameterization §

To process a video with transformer network, the video clip is split into 3D grid of L spatio-temporal patches (“tokens”)

The sequence of tokens is processed by stack of transformer blocks, inputs x and y correspond to masked regions of a video

A mask is applied by dropping a subset of tokens, take the input of the