Variational Autoencoder (VAE)

VAE
Type Generative model
Introduced 2013 (Kingma & Welling)
Components Encoder, Latent space, Decoder
Training Reconstruction + KL divergence
AM Uses Design generation, design space exploration

A Variational Autoencoder (VAE) is a generative model that learns to encode data into a compressed representation (latent space) and then decode it back. Unlike regular autoencoders, VAEs learn a probability distribution over the latent space, allowing them to generate new, realistic samples by sampling from this distribution.

In additive manufacturing and design optimization, VAEs are used to learn the "design space" of valid structures. Engineers can then explore this space to find novel designs that meet specific performance criteria, or interpolate between existing designs to create variations.

Contents
  1. Core Concept
  2. Architecture
  3. Training Objective
  4. The Latent Space
  5. Applications in Design
  6. VAE vs. GAN
  7. References

Core Concept

The key idea behind VAEs is to learn a continuous latent space where similar inputs map to nearby points. This makes the latent space "smooth"—small changes in the latent variables produce small, meaningful changes in the output.

Why this matters for design: If you train a VAE on thousands of optimized wheel designs, you can generate new wheel designs by picking any point in the latent space. You can also "walk" between two designs by interpolating their latent representations, producing a smooth transition of intermediate designs.

Architecture

    Input (x)                                      Output (x')
       │                                              ▲
       ▼                                              │
┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   ENCODER   │───▶│ LATENT SPACE│───▶│   DECODER   │
│  (Neural    │    │   z ~ N(μ,σ)│    │  (Neural    │
│   Network)  │    │             │    │   Network)  │
└─────────────┘    └─────────────┘    └─────────────┘
       │                  │
       ▼                  │
    μ (mean)              │ ← Reparameterization trick:
    σ (std dev)              z = μ + σ × ε, where ε ~ N(0,1)

Encoder

The encoder network takes an input (e.g., an image of a design) and outputs two vectors: the mean (μ) and standard deviation (σ) of a Gaussian distribution in latent space. This is different from regular autoencoders, which output a single point.

Latent Space

The latent space is typically much smaller than the input space (e.g., 64 dimensions instead of thousands of pixels). Each dimension captures some abstract feature of the data.

Decoder

The decoder takes a point from the latent space and reconstructs the original input. During generation, you sample random points from the latent space to create new outputs.

Training Objective

VAEs are trained to minimize a loss function with two terms:

VAE Loss = Reconstruction Loss + KL Divergence

Reconstruction Loss: How well can the decoder reconstruct the input?
Lrecon = ||x - x'||² (mean squared error)

KL Divergence: How close is the learned distribution to a standard normal?
LKL = KL(N(μ,σ) || N(0,1))

The KL term acts as a regularizer, preventing the encoder from just memorizing inputs and ensuring the latent space is smooth and continuous.

The Latent Space

The latent space is where VAEs become powerful for design exploration:

Example: Oh et al. (2019) trained a VAE on topology-optimized wheel designs. By sampling from the latent space, they generated novel wheel designs that were structurally valid. They could also interpolate between a lightweight design and a stiff design to find Pareto-optimal solutions. [DOI]

Applications in Design

Generative Design

VAEs can generate thousands of design candidates from a learned design space. Engineers filter these candidates based on performance criteria, exploring possibilities that might not emerge from traditional optimization.

Design Space Exploration

By visualizing the latent space (using techniques like t-SNE), designers can understand the landscape of possible designs and identify clusters of similar solutions.

Design Interpolation

Interpolating between two known good designs often produces valid intermediate designs, enabling smooth trade-offs between competing objectives.

Conditional Generation

Conditional VAEs (CVAEs) can generate designs that satisfy specific constraints (e.g., "generate a bracket with load capacity > 500N").

VAE vs. GAN

Aspect VAE GAN
Training Stable, single network optimization Adversarial, can be unstable
Output quality Sometimes blurry Sharp, realistic
Latent space Smooth, continuous, interpretable Less structured
Interpolation Excellent Good but less predictable
Likelihood Can estimate probability Cannot easily estimate
Best for Design exploration, interpolation High-fidelity image generation

In practice, researchers often combine VAEs and GANs (VAE-GAN hybrids) to get the benefits of both: a smooth latent space with high-quality outputs.

See Also

References