Home

GenAI

GenAI

Vision Language Captioning/Understanding

Standard Large Language Models (LLMs) are masters of text, but they are blind. Vision-Language Models represent a major evolution, giving models the ability to “see” and reason about the world through images, combining sophisticated visual perception with advanced language understanding.

Example Data

VLMs are trained on vast datasets of images paired with relevant text. For instruction-following, the data looks like a multimodal conversation.

  • Input (Multimodal):
    • Image: A photo of a capybara sitting calmly next to a pelican by a lake.
    • Text: "Describe the interaction between the two animals in this picture."
  • Target (Textual Answer):

    "This image shows a capybara and a pelican sitting peacefully next to each other. They seem unusually calm and comfortable in each other's presence, highlighting the capybara's famously sociable nature with other species."

Use Case Scenario

The goal is to understand, interpret, and generate content that involves a combination of images and text, enabling powerful real-world applications.

  • Advanced Visual Assistance: A user at a hardware store in San Jose takes a picture of a specific screw.
    • Prompt: "What type of screw is this and what screwdriver bit do I need?"
    • LLM Output: "This appears to be a Phillips head machine screw with a flat top. You will need a Phillips head screwdriver, likely a #2 size bit."
  • Describing the World for the Visually Impaired: A user can point their phone camera at any scene, and the VLM can describe in detail what is happening around them.
  • Creative Content Generation: A user uploads a sketch and prompts, "Write a short story opening based on this drawing."

How It Works:

VLMs are complex systems that bridge the world of pixels and the world of words. Their construction involves a multi-stage training process and a clever fusion of different model architectures.

1. The Core Architecture

A modern VLM is a combination of two powerful components:

  1. A Vision Encoder: This is the model’s “eye.” It is almost always a Vision Transformer (ViT). The ViT slices an image into a grid of patches, converts each patch into an embedding, and processes them through a Transformer to understand the relationships between different parts of the image. Crucially, it outputs a sequence of patch embeddings, not a single embedding for the whole image. This preserves spatial details.
  2. A Large Language Model (LLM): This is the model’s “brain.” It is typically a powerful, pre-trained Decoder-Only LLM (like Gemini, GPT, or Llama), which excels at reasoning and generating sequential text.
2. The “Bridge”: How Vision and Language Connect

For the LLM to understand the ViT’s output, they must “speak the same language.” This is achieved by a Projection Layer.

  • Role: This is a small, trainable neural network (an MLP) that sits between the Vision Encoder and the LLM.
  • Function: Its sole purpose is to act as a “universal translator.” It takes the sequence of image patch embeddings from the ViT and maps (projects) them into the LLM’s word embedding space.
  • Result: To the LLM, the entire image now looks like a series of special “word” tokens that are seamlessly integrated with the user’s text prompt.
3. The Two-Stage Training Process

Creating a VLM is not a single training run. It’s a sequential process of teaching different skills.

Stage 1: Alignment Pre-training (Learning to See and Read)

  • Goal: To teach the Vision Encoder and a Text Encoder to understand the same concepts. The model learns that a picture of a dog and the word “dog” should be close in a shared semantic space.
  • Architecture: A two-tower Siamese Network is often used.
    • Vision Tower: Image -> ViT -> Pooling Layer -> Single Image Embedding
    • Text Tower: Text Caption -> Text Encoder (e.g., BERT) -> Pooling Layer -> Single Sentence Embedding
    • Note on Pooling: In this stage, a pooling layer (often mean pooling) is used because the goal is to get one summary vector for the image and one for the text to compare them.
  • Loss Function: Contrastive Loss (like InfoNCE). The model is trained on millions of (image, text) pairs. The loss function’s job is to pull the embeddings of correct pairs together and push all incorrect pairs apart. For a given image i and its positive text t in a batch of N pairs, the loss is:
\[L_{i} = -\log \frac{\exp(\text{sim}(i, t) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(i, t_j) / \tau)}\] 
This trains the weights of the ViT and the Text Encoder to become powerful, aligned feature extractors.

Stage 2: Generative Fine-tuning (Learning to Talk and Reason)

  • Goal: To transform the feature extractors into a helpful, instruction-following assistant.
  • Architectural Switch:
    1. The pooling layers and the contrastive loss function are discarded. Their job is done.
    2. The pre-trained Vision Encoder (ViT) from Stage 1 is kept.
    3. A powerful, pre-trained Decoder-Only LLM is brought in to act as the brain.
    4. The crucial Projection Layer is inserted between the ViT and the LLM to bridge them.
  • Freezing vs. Fine-tuning: A key decision is made here. The ViT’s weights can either be frozen (computationally cheaper, more stable) or fine-tuned (more expensive, higher potential performance) along with the rest of the model. Many state-of-the-art models fine-tune the whole system end-to-end.
  • Loss Function: Cross-Entropy Loss. The model is trained on conversational data like (image, question) -> (answer). The loss is calculated only on the generated answer tokens, forcing the model to learn how to produce helpful textual responses based on the combined visual and text input.
4. The Inference Phase

Once fully trained, the VLM’s operation is straightforward:

  1. Input: The user provides an image and a text prompt.
  2. Processing: The image is passed through the ViT. The text is tokenized. Both are converted to embeddings. The image embeddings are passed through the projection layer.
  3. Combination: The LLM receives a single, combined sequence of text embeddings and aligned image embeddings.
  4. Generation: The LLM autoregressively generates the text answer, attending to both the user’s question and the visual information provided by the image “tokens” to form a coherent, context-aware response. ***