DreamBooth - Reproduction & Extensions

Spring 2026 · Sean Lee

Built with Andrew Noviello, Eashan Vagish and Ronald Leung. The code, report and poster are public.

Text-to-image models know what a dog looks like. They do not know what your dog looks like, and no prompt will tell them. Personalization is the problem of teaching a pretrained model one specific subject from a handful of photos, so that it can place that subject in scenes it has never been photographed in.

DreamBooth (Ruiz et al., CVPR 2023) solves this by fine-tuning. It binds a rare token to the subject, and it adds a loss that stops the model forgetting the rest of its class. That raises an obvious question: how much of the model actually has to change? All of it, a small low-rank slice of it, or none of it, if the subject can be steered in at sampling time instead?

We reimplemented DreamBooth on Stable Diffusion 1.5 and tried all three. Full fine-tuning reproduced the paper closely, raising subject identity from 0.29 to 0.60 on DINO similarity against the paper's 0.668. LoRA kept about 85% of that gain while training 0.28% of the UNet. The same objective carried over unchanged to a video model. But no amount of guidance without weight updates worked: every variant we tried produced blurred images or copies of a training photo.

Prior Preservation

DreamBooth's training loss has two denoising terms. The first teaches the subject, using an instance prompt such as a [V] dog, where [V] is the rare token:

\mathcal{L}_{\mathrm{inst}} = \mathbb{E}\left[\left\lVert \epsilon - \epsilon_\theta(z_t, t, c_{\mathrm{inst}})\right\rVert_2^2\right]

Trained on that term alone, the model drifts: it starts to treat every dog as the subject, and loses the variety of the class it began with. The second term, the class-specific prior preservation loss, counters this. It trains on images the frozen base model generated for the plain class prompt, a dog, so the model keeps its general idea of the class:

\mathcal{L} = \mathcal{L}_{\mathrm{inst}} + \lambda\,\mathbb{E}\left[\left\lVert \epsilon' - \epsilon_\theta(z'_t, t, c_{\mathrm{class}})\right\rVert_2^2\right]

The weight λ sets the trade-off between learning the subject and remembering everything else, and it turned out to matter more than we expected.

Score Functions and Guidance

Both terms ask the UNet to predict the noise that was added to an image. It helps to see what that prediction is for, because it explains how sampling works, how a prompt steers it, and later why our attempt to skip fine-tuning failed.

Think of every possible image as a point in a very high-dimensional space. Real images occupy a thin region of it, a manifold where the data density p(x) is high, with almost nothing in between. Adding noise blurs that density. At a noise level σ the model sees p_σ, which is broad and smooth when σ is large and sharp around real images when σ is small. The quantity that matters is the score, the gradient of the log density, which points toward where images become more likely. Predicting the noise is, up to a scale, predicting the score (Song and Ermon, 2019; Song et al., 2021):

\epsilon_\theta(x_t, t) \approx -\sigma_t \, \nabla_{x} \log p_{t}(x_t)

So each denoising step is a step of noisy gradient ascent on the log density. In its simplest form, Langevin dynamics, it reads:

x \leftarrow x + \eta \, \nabla_x \log p_\sigma(x) + \sqrt{2\eta}\, z, \qquad z \sim \mathcal{N}(0, I)

Samplers run this from a large σ down to a small one. The early steps climb a smooth, nearly single-peaked landscape and settle the layout; the late steps climb a sharp one and add detail. The noise term is what makes this sampling rather than optimization. Without it, every sample would run uphill to the nearest peak and stop. With it, samples spread out over the density in proportion to it.

A prompt enters the same way. By Bayes' rule, the score of a conditional distribution is the unconditional score plus the gradient of a classifier, so steering toward a prompt means adding a second uphill direction:

\nabla_x \log p(x \mid c) = \nabla_x \log p(x) + \nabla_x \log p(c \mid x)

Classifier guidance takes that second term from a separate classifier trained on noisy images. Classifier-free guidance, or CFG (Ho and Salimans, 2022), which Stable Diffusion applies at every step, gets the same effect from the model itself. It predicts the noise with and without the prompt and extrapolates from one toward the other:

\tilde\epsilon = \epsilon_\theta(x_t, \varnothing) + w\,\big(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \varnothing)\big)

In score terms this climbs log p(x) + w log p(c | x). Each step is still a step uphill, but on a landscape reweighted toward images the prompt explains well. At w = 1 that is the ordinary conditional distribution. Above 1 it trades variety for adherence to the prompt, which is why Stable Diffusion's default of 7.5 gives vivid samples that resemble one another.

Here is the same process on a toy two-dimensional "image space" with three classes of images, drawn as a landscape whose height is the density. Particles start as noise and follow the score down 40 noise levels. The surface is the density being climbed at the current level: broad and almost featureless at first, then sharpening into one peak per mode as σ falls. The dashed rings on the floor mark where the real images are. Drag to rotate it:

Guidance weight w = 3.0

Noise level 1 of 40σ = 1.60

With no guidance, the samples land on all six modes in proportion to the density. CFG toward "dog" reweights the landscape so that the cats and birds disappear, while the samples still spread across all three dog modes. The two anchor settings are the experiment that failed, and we come back to them at the end.

Setup

We kept the VAE and the CLIP text encoder frozen and trained only the UNet, about 860M parameters. The subjects were four from the official DreamBooth dataset, a dog, a cat, a backpack and a bear plushie, with five photos each and 200 class-prior images per class sampled from the base model. Each model trained for 1,000 steps at batch size 1 with λ = 1.0, on a single RTX 4090.

We evaluated with the paper's three metrics, on 8 fixed prompts and 2 seeds per model. DINO uses self-supervised features that separate individuals within a class, so it is the main measure of subject identity. CLIP-I is a coarser image-to-image similarity. CLIP-T measures how well the image matches the prompt, and it is the counterweight: an overfitted model can score well on DINO by reproducing the subject while ignoring the requested scene.

Reproduction

Averaged over the four subjects, full fine-tuning landed close to the paper:

Method	DINO ↑	CLIP-I ↑	CLIP-T ↑
Paper, DreamBooth on Stable Diffusion (30 subjects)	0.668	0.803	0.305
Base SD 1.5, no fine-tuning	0.288	0.661	0.339
Full fine-tune (ours)	0.605	0.775	0.326
LoRA r = 16 (ours)	0.517	0.744	0.334

Our CLIP-I is close to the paper's and our CLIP-T is higher. DINO is 0.06 lower, which is within the spread we saw between subjects; the paper also averages over 30 subjects where we used four. The base model's high CLIP-T is a reminder of why all three metrics are needed. It follows prompts well precisely because it draws a generic member of the class.

That difference is easiest to see side by side. Here is the same prompt, "a photo of [V] <class> in Times Square at night," for every subject and method:

Dog, Reference — Seed 0 throughout. The base model has no way to draw the backpack at all.

Dog, Base SD 1.5 — Seed 0 throughout. The base model has no way to draw the backpack at all.

Both fine-tuned models keep small identifying details, such as the backpack's pins and the bear's blue hat. But identity recovery was far from uniform across subjects:

Base SD 1.5Full fine-tuneLoRAPaper, Stable Diffusion (0.668)

The cat reached 0.73 DINO, above the paper's average, while the backpack reached only 0.41. The backpack is also the one subject where LoRA beat full fine-tuning. A single averaged score hides most of this, and looking at the samples for each subject told us more than the mean did.

Plotting the two kinds of fidelity against each other shows the trade-off that DreamBooth is negotiating:

Mean over four subjectsSingle subject

Fine-tuning moves every subject to the right, toward the subject, and slightly down, away from the prompt. The paper's result sits furthest along that line. LoRA lands between the base model and full fine-tuning on both axes, which is what you would expect from a smaller update that does less to the model's existing behaviour.

LoRA

Full fine-tuning updates 860M parameters to learn one noun, which is a lot of weight to move and a 4.3 GB checkpoint to store for every subject. LoRA freezes each pretrained weight matrix and learns a low-rank update beside it:

h = Wx + \tfrac{\alpha}{r}BAx, \qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d, k)

We added adapters to the query, key and value projections of all 32 UNet attention layers, 96 adapter pairs at rank 16. Lower ranks lost subject detail. LoRA also needed a learning rate of 1e-4, about 20 times the full fine-tuning rate; at 5e-6 it barely moved.

The result trains 2.39M parameters instead of 860M, and the savings look very different depending on which cost you measure:

Full fine-tuneLoRA r = 16

Trainable parameters

360× fewer% of UNet

Training time

24% fasterminutes

Peak VRAM

36% lessGB

Checkpoint size

440× smallerMB

Storage falls by more than two orders of magnitude, from a 4.3 GB checkpoint per subject to 9.6 MB. Compute barely moves. Training time fell from 12 to 9 minutes and peak memory from 11.9 GB to 7.6 GB, because the frozen UNet still runs a full forward and backward pass at every step. The cost of a step barely depends on how many of its parameters are trainable.

Prior Weight

We had guessed that the low-rank constraint would act as a regularizer on its own, so that LoRA would need less prior preservation than full fine-tuning. A sweep over λ on the dog subject supported this:

LoRA, dog subjectFull fine-tuneBase SD 1.5

DINO · subject identity

CLIP-T · prompt fidelity

At λ = 1.0, the paper's setting, LoRA underfit the subject. At λ = 0.1 prompt fidelity started to drop. The best value sat near 0.5, where DINO rose from 0.477 to 0.575. This also means the LoRA numbers in the table above, trained at λ = 1.0, understate what LoRA can do.

Video Generation

We then applied the same objective to ModelScopeT2V, a latent video diffusion model built much like Stable Diffusion, with a stack of frames in place of a single image:

Adapted from Wang et al. (2023), with frames from our own clips.

An encoder compresses each frame into a latent, the diffusion process noises the whole stack, and a spatio-temporal UNet, conditioned on the prompt's text embedding, denoises it over many steps before the decoder turns it back into video. Because the clip is just a larger block of latents, the DreamBooth loss needs no changes at all. We only had the compute to fine-tune it with LoRA, on five short clips of one bird:

With the token, the model puts the training bird into a new scene. Without it, the same prompt produces a generic bird, so the identity lives in the token rather than leaking into every bird the model draws. We only judged these results by eye; we did not run the metrics on video.

Guidance Without Fine-Tuning

Everything so far changes weights. The last experiment asked whether that is necessary. Following classifier guidance, we left the base model untouched and added a term at each denoising step that pulls the current latent toward a subject latent noised to the same step:

z_t \leftarrow z_t - w_t \nabla_{z_t} \left\lVert z_t - \tilde{z}^{\,\mathrm{subj}}_t \right\rVert_2^2

We tried four ways to choose the target: the mean of all the subject's latents, one fixed reference per sample, a fresh reference at every step, and the reference nearest the current latent. For each we swept weights and schedules. Every run ended in one of two failures:

A fixed target made the sample collapse onto a training photo and ignore the prompt. The pooled, per-step and nearest-neighbour targets blurred the image.

The score picture from earlier explains both failures. Adding the gradient of −w‖z − z̃‖² to every step is adding the score of a Gaussian centred on the target. It multiplies the density being climbed by a bump around a single point. With one reference photo as the target, the bump sits on a training image, and a strong enough pull drags every sample onto it. That is the collapse. With the mean of the references as the target, the bump sits at the average latent. The average of several photos is not itself a photo, so the bump lies off the manifold, in a region that decodes to a blur.

Both are visible in the demo above. As w rises, the fixed anchor pulls every sample, cats included, into one dog mode, and the mean anchor pulls samples into the empty space between the dog modes. A latent target mixes identity with pose, crop, background and lighting, so no single point can stand for the subject. Fine-tuning works differently. It changes p(x | c) itself, moving probability onto the whole range of the subject's appearances, which is what lets the subject change pose and setting from prompt to prompt. The repository also includes a DINO feature-guidance variant that targets identity more directly, which we have not evaluated here.

Takeaways

So, do you need to fine-tune? On this evidence, yes, but much less of the model than DreamBooth's full fine-tuning touches. A rank-16 adapter on the attention projections keeps most of the identity gain, as long as it gets its own prior weight. Steering the sampler is not a substitute, because what personalization needs is a change in what the token means, and a target in latent space cannot express that.

The evaluation was small: four subjects, eight prompts, two seeds per prompt and one training seed per configuration, with the λ sweep run on a single subject. Differences as small as LoRA against full fine-tuning on the backpack should be read as directional. The questions we would pursue next are whether several subjects can share one model through separate rare tokens, and how much of this carries over to newer rectified-flow models.