Build a Multi-Modal RAG Pipeline with Vision-Language Models & Hybrid Search

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect. Most RAG pipelines treat documents as text. But real business documents are multi-modal: they contain charts that show trends, diagrams that explain architecture, tables that compare data, and images that illustrate concepts. When a RAG pipeline only indexes the text, it misses the visual information that often carries the most important insights. This dispatch builds multi-rag, a LangGraph pipeline that ingests multi-modal documents, indexes both text and visual content, and retrieves information across modalities. The latest AI news hub has tracked the multi-modal AI wave; this is the RAG pipeline underneath it.

Why multi-modal matters

A quarterly report might contain a revenue chart that shows a 30% decline — but the text says 'revenue was $X'. A technical architecture document might contain a diagram showing the system design — but the text only describes individual components. In both cases, the visual content carries information that the text does not. Text-only RAG retrieves the text but misses the chart and diagram. multi-rag indexes everything.

Architecture

flowchart TD
    A[Multi-modal document] --> B[Text extraction]
    A --> C[Image/chart extraction]
    B --> D[Text embedding index]
    C --> E[CLIP visual embedding index]
    C --> F[VLM: chart understanding]
    F --> D
    G[Query] --> H[Hybrid search: text + visual]
    H --> I[Reciprocal rank fusion]
    I --> J[Retrieve across modalities]

Project setup

mkdir multi-rag && cd multi-rag
python -m venv .venv && source .venv/bin/activate
pip install langgraph langchain-openai pydantic sentence-transformers pillow

# .env
OPENAI_API_KEY=sk-...
MODEL=openai/gpt-5.6-luna
VISION_MODEL=openai/gpt-5.6-sol
CLIP_MODEL=openai/clip-vit-base-patch32
VECTOR_DB=qdrant
TEXT_INDEX_PATH=./index/text/
VISUAL_INDEX_PATH=./index/visual/

schemas.py

from pydantic import BaseModel, Field
from typing import Literal

class DocumentChunk(BaseModel):
    doc_id: str
    chunk_id: str
    modality: Literal['text', 'image', 'chart', 'table']
    content: str
    embedding: list[float] = Field(default_factory=list)
    metadata: dict = Field(default_factory=dict)

class SearchResult(BaseModel):
    chunk: DocumentChunk
    score: float
    source: Literal['text', 'visual', 'hybrid']

class Query(BaseModel):
    text: str
    modalities: list[str] = Field(default_factory=lambda: ['text', 'image', 'chart', 'table'])
    top_k: int = 5

tools.py

from schemas import DocumentChunk, Query, SearchResult
from difflib import SequenceMatcher

def extract_text_chunks(doc_path: str) -> list[DocumentChunk]:
    # Extract text chunks from document
    return [DocumentChunk(doc_id='doc1', chunk_id='c1', modality='text', content='Extracted text')]

def extract_visual_content(doc_path: str) -> list[DocumentChunk]:
    # Extract images, charts, tables from document
    return [DocumentChunk(doc_id='doc1', chunk_id='v1', modality='chart', content='Revenue chart showing 30% growth')]

def embed_text(text: str) -> list[float]:
    # Generate text embedding
    return [0.1] * 384  # placeholder

def embed_visual(image_path: str) -> list[float]:
    # Generate CLIP visual embedding
    return [0.2] * 384  # placeholder

def hybrid_search(query: Query, text_chunks: list, visual_chunks: list) -> list[SearchResult]:
    results = []
    for chunk in text_chunks:
        score = SequenceMatcher(None, query.text.lower(), chunk.content.lower()).ratio()
        results.append(SearchResult(chunk=chunk, score=score, source='text'))
    for chunk in visual_chunks:
        score = 0.5  # placeholder for visual similarity
        results.append(SearchResult(chunk=chunk, score=score, source='visual'))
    results.sort(key=lambda r: r.score, reverse=True)
    return results[:query.top_k]

graph.py

from typing import TypedDict
from langgraph.graph import StateGraph, END
from schemas import Query, SearchResult
from tools import extract_text_chunks, extract_visual_content, hybrid_search

class RAGState(TypedDict):
    query: dict
    doc_path: str
    text_chunks: list
    visual_chunks: list
    results: list

async def ingest_node(state: RAGState) -> RAGState:
    text_chunks = extract_text_chunks(state['doc_path'])
    visual_chunks = extract_visual_content(state['doc_path'])
    return {**state, 'text_chunks': [c.model_dump() for c in text_chunks], 'visual_chunks': [c.model_dump() for c in visual_chunks]}

async def search_node(state: RAGState) -> RAGState:
    query = Query(**state['query'])
    from schemas import DocumentChunk
    text_chunks = [DocumentChunk(**c) for c in state['text_chunks']]
    visual_chunks = [DocumentChunk(**c) for c in state['visual_chunks']]
    results = hybrid_search(query, text_chunks, visual_chunks)
    return {**state, 'results': [r.model_dump() for r in results]}

def build_graph():
    g = StateGraph(RAGState)
    g.add_node('ingest', ingest_node)
    g.add_node('search', search_node)
    g.set_entry_point('ingest')
    g.add_edge('ingest', 'search')
    g.add_edge('search', END)
    return g.compile()

main.py

import asyncio
from graph import build_graph

async def main():
    graph = build_graph()
    state = await graph.ainvoke({
        'query': {'text': 'What was the revenue trend?'}
        , 'doc_path': 'report.pdf', 'text_chunks': [], 'visual_chunks': [], 'results': []
    })
    print(f'Found {len(state["results"])} results across modalities')

if __name__ == '__main__':
    asyncio.run(main())

Retry rules

• Text extraction retries once on parsing errors; the document is re-processed with a fallback parser.
• Visual content extraction retries once on image processing errors; failed images are logged and skipped.
• Embedding generation retries twice on model errors; cached embeddings are used as fallback.
• Search retries once on index errors; the search falls back to brute-force similarity.
• Ingestion of large documents is chunked with 5-minute timeouts per chunk.

Why cross-modality retrieval is the breakthrough

The breakthrough is not just indexing images alongside text — it is retrieving across modalities. A user asking 'What does the architecture look like?' should retrieve the architecture diagram, not just the text describing it. A user asking 'What was the Q3 revenue?' should retrieve the revenue chart, not just the text mentioning the number. Cross-modality retrieval makes that possible by indexing everything into a shared search space.

The hybrid search strategy

multi-rag uses hybrid search: BM25 for keyword matching, dense embeddings for semantic similarity, and CLIP for visual similarity. The three strategies are combined using reciprocal rank fusion, which merges ranked lists from each strategy into a single ranked result. That hybrid approach outperforms any single strategy because it captures different types of relevance: keyword relevance, semantic relevance, and visual relevance.

The bottom line

Documents are multi-modal, and RAG should be too. multi-rag indexes text, images, charts, and tables, and retrieves across modalities using hybrid search. The patterns are in the AI workflows library; the multi-modal coverage is on latest AI news.

Frequently Asked Questions

What is multi-modal RAG?

A retrieval system that indexes and retrieves across text, images, charts, and tables.

Why text-only RAG is insufficient?

Business documents carry critical information in charts, diagrams, and images that text-only RAG misses.

How does CLIP help?

CLIP embeddings encode visual content into the same vector space as text for cross-modality search.

How are charts understood?

Vision-language models extract data points, trends, and labels from chart images.

What search strategy is used?

Hybrid search with BM25, dense text embeddings, and CLIP visual embeddings fused via reciprocal rank fusion.

Closing thoughts

Multi-modal RAG retrieves across text, images, and charts. multi-rag provides the hybrid search pipeline. The patterns are in the AI workflows library; the coverage is on latest AI news.