Variation Autoencoder

Autoencoder focuses on learning a latent representation that compresses data and allows for reconstruction of that compressed data

More commonly called a Variational Autoencoder

Encoder doesn’t output one fixed code but a distribution over possible latent codes

Decoder says how data could be generated from a latent variable

Main Idea §

A VAE is a probabilistic generative model

Instead of mapping an input $x$ to one deterministic latent vector $z$, it learns:

• An encoder distribution: $q_{\varphi }(z\mid x)$
• A decoder distribution: $p_{\theta }(x\mid z)$
• A prior over latent variables: $p(z)$, usually $p(z)=\mathcal{N}(0,I)$

So the model assumes data is generated by:

1. Sample a latent vector $z\sim p(z)$
2. Sample an observation $x\sim p_{\theta }(x\mid z)$

The encoder approximates the posterior distribution $p_{\theta }(z\mid x)$, which is usually intractable to compute exactly.

Latent Distribution §

For each input $x$, the encoder outputs parameters of a Gaussian distribution:

$$q_{\varphi }(z\mid x)=\mathcal{N}(z;\mu (x),\mathrm{diag}({\sigma }^2(x)))$$

where:

• $\mu (x)$ is the mean vector
• ${\sigma }^2(x)$ is the variance vector
• $\varphi$ are the encoder parameters

This means each example is represented by a region in latent space, not a single point.

Objective Function §

We want to maximize the likelihood of the data:

$$\log p_{\theta }(x)$$

But this is difficult because:

$$p_{\theta }(x)=\int p_{\theta }(x\mid z)p(z)dz$$

The integral is typically intractable, so VAEs optimize the Evidence Lower Bound (ELBO):

$$\mathcal{L}(x;\theta ,\varphi )={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]-D_{KL}(q_{\varphi }(z\mid x)\Vert p(z))$$

and:

$$\log p_{\theta }(x)\ge \mathcal{L}(x;\theta ,\varphi )$$

Meaning of the Two Terms §

1. Reconstruction Term §

$${\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]$$

This encourages the decoder to reconstruct the input accurately from the latent sample.

If reconstruction is poor, this term becomes smaller.

2. KL Divergence Term §

$$D_{KL}(q_{\varphi }(z\mid x)\Vert p(z))$$

This pushes the encoder’s latent distribution to stay close to the prior, usually $\mathcal{N}(0,I)$.

This regularizes the latent space so nearby points decode to similar outputs and random samples from the prior can generate realistic data.

Loss Used in Practice §

Since training usually minimizes a loss, people often write:

$$\text{Loss}=-\mathcal{L}(x;\theta ,\varphi )$$

or equivalently:

$$\text{Loss}=\text{Reconstruction Loss}+D_{KL}(q_{\varphi }(z\mid x)\Vert p(z))$$

For a diagonal Gaussian posterior and standard normal prior:

$$D_{KL}\left(\mathcal{N}(\mu ,{\sigma }^{2}I)\Vert \mathcal{N}(0,I)\right)=\frac{1}{2}\sum _{j=1}^d\left({\mu }_j^2+{\sigma }_j^2-\log {\sigma }_j^2-1\right)$$

This closed form makes training efficient.

Reparameterization Trick §

We need to sample $z$ from $q_{\varphi }(z\mid x)$, but direct sampling breaks standard backpropagation.

So we rewrite the sample as:

$$ϵ\sim \mathcal{N}(0,I)$$

$$z=\mu +\sigma \odot ϵ$$

where $\odot$ is element-wise multiplication.

Now the randomness is isolated in $ϵ$, while $\mu$ and $\sigma$ remain differentiable functions of the encoder output.

This is the reparameterization trick.

Why VAE Works Better Than a Plain Autoencoder for Generation §

A plain Autoencoder may learn a latent space with no smooth global structure.

A VAE forces the latent encodings to match a simple prior distribution, which gives:

• Smoother latent space
• Better interpolation between latent points
• Ability to sample new data by drawing $z\sim \mathcal{N}(0,I)$

Intuition §

You can think of a VAE as learning:

• Where an input should live in latent space through $\mu$
• How uncertain the model is through $\sigma$
• How to reconstruct the input from sampled latent variables

The KL term prevents every input from being mapped to completely separate disconnected regions.

Common Tradeoff §

There is a tradeoff between:

• Good reconstruction
• Well-structured latent space

If the KL term is too strong, reconstructions may become blurry.

If the KL term is too weak, the model behaves more like a regular autoencoder and loses some generative quality.

Compact Summary §

VAE learns a distribution in latent space instead of a single code.

It trains by maximizing the ELBO:

$${\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]-D_{KL}(q_{\varphi }(z\mid x)\Vert p(z))$$

So it both:

• Reconstructs data well
• Keeps the latent space regular enough for sampling and generation