MoCa - Modality-aware continual pretraining for multimodal embeddings

Adapting VLM for embeddings -> Use contrastive learning on pretrained VLM backboens on image text pairs. Problems!

1. labeled pairs are hard ot scale up
2. They don't like using causal attention for embeddings
3. lack of diversity in training data Proposal

• modality-aware continual pretraining - interleaved image and text, joint denoising (cross entropy for text and masked autoencoding for images) - two losses?
  • bidirectional attention
• heterogenous contrastive fine tuning
  • instead of using image-caption pairs, use long-form multimodal pairs and interleaved pairs
  • use task aware batching

.. it seems we have internet scale embeddings already so I am unsure how this is new, but they show SOTA post-training 3B QWEN. Seems they show compute scaling (can go much higher). I wonder how the bidirectional loss works on images and text