GPS-CLIP: Beyond Hard Negatives (NeurIPS 2026)

The idea

CLIP knows what is in an image but often not how it fits together. Most fixes train it to reject edited "hard negative" captions. GPS-CLIP does the opposite: it pulls the image toward its own grounded structure, its regions, their descriptions, and the relations between them.

Most fixes · hard negatives

Push the image away from edited captions

Learns to spot word swaps
Often costs zero-shot and retrieval accuracy

GPS-CLIP · grounded positives

Pull the image toward its own structure

Learns regions and who-does-what-to-whom
Keeps or improves zero-shot and retrieval

Scene graphs are used only while fine-tuning. The model you deploy is a plain CLIP dual encoder with the same architecture, the same inputs, and no extra inference cost.

Three losses, one embedding space

An image and its caption are decomposed into regions and a scene graph. Three complementary objectives then act on the same image and text encoders.

GPS-CLIP overview: image and text encoders trained with global, region, and triplet losses from a scene graph — **Structured contrastive training at a glance.** Region and sub-region crops, their descriptions, and ⟨subject, relation, object⟩ triplets are aligned in the shared embedding space alongside the usual image–caption pairs.

ℒ = + λ_r + λ_t

Click a term to focus on it.

GlobalThe standard image–caption contrastive loss. Kept as is, so zero-shot ability is preserved.

RegionImage sub-regions are matched to their text spans. Overlapping regions count as weighted positives instead of being pushed apart.

RelationEach image is aligned with its ⟨s, r, o⟩ triplets. Order matters: "dog chasing frisbee" is not "frisbee chasing dog".

Overlap is not a mistake

Regions cut from the same image often overlap. A standard contrastive loss would treat every other region as a negative, punishing the model for noticing that two boxes show the same thing. GPS-CLIP decides by overlap instead.

Similarity grid between image sub-regions and sub-region descriptions, marked positive, masked, or negative — **IoU-weighted region–phrase loss.** Pairs of regions from the same image are weighted by how much they overlap, instead of all being treated as negatives.

Drag to move region B

IoU: 0.00
Standard loss: negative
GPS-CLIP

Where the gain comes from

Average points gained at each step of the region-loss design, from the ablations.

Results

Fine-tuned from CLIP ViT-B/32, GPS-CLIP improves every benchmark we report, including zero-shot ImageNet and retrieval, which hard-negative methods often trade away.

CLIP ViT-B/32GPS-CLIP

Accuracy or Recall@1 (%), from Table 1. Comparisons with NegCLIP, TripletCLIP, DAC, CLoVe, CE-CLIP, FSC-CLIP and CLIC are in the paper.

Why it's practical

All the extra machinery lives on the training side. What you deploy is an ordinary CLIP.

At training time

Scene graphs: From an offline multimodal model
Extra losses: Region + relation
Noisy graphs: Degrades gracefully
Loss family: InfoNCE or SigLIP
Scale: ViT-B/32 to ViT-L/14

At inference time

Architecture: Same as CLIP
Parameters: +0
Latency: Unchanged
FLOPs: Unchanged
Inputs: Image + text only

Use GPS-CLIP

GPS-CLIP loads like any CLIP checkpoint. Swap it into existing code without other changes.

Cite

@inproceedings{udai2026gpsclip,
  title     = {Beyond Hard Negatives: Grounded Positive Supervision for Compositional {CLIP}},
  author    = {Udai, Harsh and Mopuri, Konda Reddy and Balasubramanian, Vineeth N.},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Beyond Hard Negatives: Grounded Positive Supervision for Compositional CLIP

Same words, different scene

What the models prefer