Learning Transferable Visual Models From Natural Language Supervision

Given a batch of $N$ (image, text) pairs, CLIP learns a multi-model embedding space by jointly training an image encoder and text encoder to maximize the cosine similarity of the image and text embeddings of the $N$ correct pairs while minimizing the cosine similarity of the $N^2 - N$ incorrect pairs. (It is optimized using a symmetric cross entropy loss over these similarity scores)

400 million