Short or polyunsaturated lipid variants of the NKT cell antigen α-galactosylceramide (αGC) exhibit decreased potency and a Th2 bias in vivo despite conserved TCR contact residues and stable binding to CD1d at neutral and acidic pH. Using reagents to directly visualize lipids in their free or CD1d-bound form, we determined that, contrary to predictions, these lipids reached the lysosome better than αGC. However, in contrast with αGC, they loaded CD1d at the cell surface and underwent immediate pH-dependent dissociation upon recycling to the lysosome. In cell-free assays, ultrafast dissociation of preformed complexes could be induced at acidic pH only when free competitor lipids were added, suggesting active lipid displacement. These findings provide a common cell biological explanation for the decreased stimulatory properties of short and polyunsaturated αGC variants. They also suggest that direct lipid displacement is a potent mechanism underlying highly dynamic lipid exchange reactions in the lysosomal compartment that shape the repertoire of lipids associated with CD1d.