Project 5: Fun with Diffusion Models

By Tony Wan

Part 0: Setup

The random seed I'm using is 777. The quality of the photo is already decent for n=10, but for n=5, there are still many visible noise particles.

a man wearing a hat (n=5).png — a man wearing a hat (left n=5, right n=10)

a man wearing a hat (n=10).png — a man wearing a hat (left n=5, right n=10)

a rocket ship (n=5).png — a rocket ship (left n=5, right n=10)

a rocket ship (n=10).png — a rocket ship (left n=5, right n=10)

an oil painting of a snowy mountain village (n=5).png — an oil painting of a snowy mountain village (left n=5, right n=10)

an oil painting of a snowy mountain village (n=10).png — an oil painting of a snowy mountain village (left n=5, right n=10)

Part 1.1: Forward Process

Clean Image at t=0.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Noisy Image at t=250.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Part 1.2: Classical Denoising

Noisy Image at t=250.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Noisy Image at t=500.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Denoised Image at t=250.png — Gaussian Blur Denoising at t=0, 250, 500, 750 (left to right)

Denoised Image at t=500.png — Gaussian Blur Denoising at t=0, 250, 500, 750 (left to right)

Part 1.3: One-Step Denoising

Denoised Image at t=0.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Denoised Image at t=250.png — Noisy Campanile at t=0, 250, 500, 750 (left to right)

Denoised Image at t=0.png — Denoised Campanile at t=0, 250, 500, 750 (left to right)

Denoised Image at t=250.png — Denoised Campanile at t=0, 250, 500, 750 (left to right)

Part 1.4: Iterative Denoising

Noisy Campanile at t=90, 240, 390, 540, 690 (left to right)

Noisy Campanile at t=90, 240, 390, 540, 690 (left to right)

Original, Iteratively Denoised, One-Step Denoised, Gaussian Blurred Campanile (left to right)

Original, Iteratively Denoised, One-Step Denoised, Gaussian Blurred Campanile (left to right)

Part 1.5: Diffusion Model Sampling

5 Samples

5 Samples

Part 1.6: Classifier-Free Guidance (CFG)

5 Samples with CFG

5 Samples with CFG

Part 1.7: Image-to-image Translation

Campanile SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Campanile SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Golden Gate SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Golden Gate SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Monterey SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Monterey SDEdit with i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Part 1.7.1 Editing Hand-Drawn and Web Images

Horse at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Horse at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Smile at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Smile at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Grass at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Grass at i_start=1, 3, 5, 7, 10, 20, and original (left to right)

Part 1.7.2: Inpainting

Campanile, Mask, Hole to Fill, Inpainted (left to right)

Campanile, Mask, Hole to Fill, Inpainted (left to right)

Golden Gate, Mask, Hole to Fill, Inpainted (left to right)

Golden Gate, Mask, Hole to Fill, Inpainted (left to right)

Monterey, Mask, Hole to Fill, Inpainted (left to right)

Monterey, Mask, Hole to Fill, Inpainted (left to right)

Part 1.7.3: Text-Conditional Image-to-image Translation

Rocket Ship at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Rocket Ship at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Pencil at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Pencil at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Snowy Mountain Village at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Snowy Mountain Village at noise level 1, 3, 5, 7, 10, 20, and original (left to right)

Part 1.8: Visual Anagrams

an oil painting of an old man and an oil painting of people around a campfire

an oil painting of an old man and an oil painting of people around a campfire

an oil painting of a snowy mountain village and a photo of a dog

an oil painting of a snowy mountain village and a photo of a dog

a lithograph of waterfalls and a lithograph of a skull

a lithograph of waterfalls and a lithograph of a skull

Part 1.10: Hybrid Images

Hybrid image of a lithograph of a skull and a lithograph of waterfalls

Hybrid image of a lithograph of a skull and a lithograph of waterfalls

Hybrid image of a photo of a dog and a photo of a man

Hybrid image of a photo of a dog and a photo of a man

Hybrid image of a rocket ship and a pencil

Hybrid image of a rocket ship and a pencil

Part B.1.2: Using the UNet to Train a Denoiser

Varying levels of noise on MNIST digits

Part B.1.2.1: Training

Varying levels of noise on MNIST digits

Results on digits from the test set after 1 epoch of training

Results on digits from the test set after 5 epochs of training

Part B.1.2.2: Out-of-Distribution Testing

Results on digits from the test set with varying noise levels.