Same objects, same colours, different scene
a red suitcase next to a black backpack
a black suitcase next to a red backpack
CLIP-style models often score these two captions almost the same, because their training never forces them to bind each attribute to the right object.
The common fix adds hard negatives on the text side only: swapped, negated, or perturbed captions. The text encoder learns the fine distinctions, but the image encoder never sees a matching near-miss. The shared embedding space becomes lopsided, even when retrieval scores look healthy.
Hubs
A few points become the nearest neighbour of almost everything and crowd out correct matches.
One-way neighbours
An image's nearest caption often has a different image as its own nearest match.
Brittle to small edits
Swapping an attribute or relation can leave the edited caption scoring above the true one.
Hard negatives for both modalities, made in embedding space
LACE edits embeddings instead of pixels. It needs no diffusion model, no image editor, and no retrieval mining, so image-side counterfactuals become as cheap as text-side ones.
Find the concepts in play
Sparse coding writes each embedding as a few active atoms, such as sheep, grass, white, on. Only atoms that are both strongly used and semantically aligned are allowed to be edited.
Change one thing
Attenuate, swap, or transplant a single concept with another of the same type: sheep → zebra, red → blue. The rest of the scene stays as it was.
Train both sides
Edited captions become negatives for image→text retrieval, and edited images become negatives for text→image retrieval. At test time all of this is gone: it is a plain dual encoder.
Three ways to make a near-miss
The bars show the active atoms of an image embedding for a photo of sheep in a field. Pick an edit to see how LACE changes the sparse code before rebuilding the embedding. Values are illustrative.
Edited embeddings retrieve the new concept
We transplant a concept inside a real COCO image embedding and look up its nearest real images.
- Before the editNeighbours show the source concept: sheep in one row, donuts in the other.
- After a transplantNeighbours switch to the target. Sheep → zebra brings in two zebras; donut → broccoli turns all three into broccoli. Grass, plates and tables stay.
- A random edit of the same sizeNever retrieves the target. The direction of the edit carries the meaning, not its size.
What the counterfactuals add
Every comparison below changes only whether LACE's counterfactuals are used.
The paper has full benchmark results, comparisons with prior compositional methods, and ablations.
What LACE doesn't need
No generated images. Counterfactuals are made in embedding space.
No inpainting or pixel edits to make image-side negatives.
No search over a corpus for near-miss pairs.
BibTeX
@inproceedings{udai2026lace,
title = {{LACE}: Latent Alignment via Counterfactual Embeddings},
author = {Udai, Harsh and Mopuri, Konda Reddy},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}