Autoencoders and Denoising Autoencoders

Let

$$x \in \mathbb R^d$$

be one input vector. A deterministic autoencoder has an encoder

$$f_\theta:\mathbb R^d \to \mathbb R^m$$

and a decoder

$$g_\phi:\mathbb R^m \to \mathbb R^d.$$

The latent code is

$$h = f_\theta(x) \in \mathbb R^m,$$

and the reconstruction is

$$\widehat x = g_\phi(h) = g_\phi(f_\theta(x)).$$

Given training examples $x_1,\ldots,x_n$, the standard reconstruction objective is

$$\min_{\theta,\phi} \frac{1}{n}\sum_{i=1}^n \ell\!\left(x_i, g_\phi(f_\theta(x_i))\right).$$

For real-valued vectors, a common loss is squared error:

$$\ell(x,\widehat x)=\|x-\widehat x\|_2^2.$$

If the code dimension $m$ is smaller than the input dimension $d$, the autoencoder is undercomplete. It cannot store every coordinate independently, so it must decide what information the code preserves.

This is the bridge back to PCA. In the special case where both encoder and decoder are linear, the loss is squared reconstruction error, and the code is undercomplete, the learned reconstruction subspace is the same principal subspace as PCA. That does not make nonlinear autoencoders globally solved PCA replacements, and it does not guarantee better downstream features.

If $m \ge d$, or if the encoder and decoder are powerful enough, the model may copy inputs without learning useful structure. Overcomplete autoencoders therefore need some other pressure: noise, sparsity, weight decay, dropout, contractive penalties, architectural limits, or a task-specific objective.

A denoising autoencoder changes the input side of the objective. Let

$$\widetilde x \sim q_D(\widetilde x \mid x)$$

be a corrupted version of $x$. The model receives $\widetilde x$, but the loss still compares the output with the clean $x$:

$$\min_{\theta,\phi} \mathbb E_{x\sim p_{\mathrm{data}}} \mathbb E_{\widetilde x \sim q_D(\widetilde x \mid x)} \ell\!\left(x, g_\phi(f_\theta(\widetilde x))\right).$$

The target has not changed. The model is not asked to reproduce the noise. It is asked to use the visible evidence and the learned regularities of the data distribution to reconstruct the clean example.

Three separations keep the concept honest.

First, reconstruction quality is not likelihood. A deterministic autoencoder trained with squared error does not define a prior over latent codes or a calibrated probability model over inputs.

Second, a VAE is not just an autoencoder with noise. A VAE trains a latent-variable generative model with an ELBO, a prior over latents, an approximate posterior $q_\phi(z \mid x)$, and a likelihood term. Reconstruction is only one part of a probabilistic objective.

Third, denoising autoencoders are historically and conceptually related to denoising and score ideas, but they are not diffusion models. Modern diffusion needs a separate route through noise schedules, score or noise prediction, and iterative generative sampling.