RAG: retrieval-augmented generation

Ask a model about your refund policy and it will give you a confident answer about refund policies in general. It was trained on public text. Your policy PDF wasn’t part of it, and you can’t retrain the model every time the policy changes.

Retrieval-augmented generation (RAG) fixes this without touching the model. Before it answers, your app searches your own documents and adds the most relevant passages to the prompt. The model then answers from those passages. Google’s RAG Engine docs describe the steps like this:

• Ingest and transform: load your documents and split them into chunks.
• Embed and index: turn each chunk into an embedding (a vector of numbers that captures its meaning) and store them in an index.
• Retrieve: when a question arrives, find the chunks closest in meaning to it.
• Generate: add those chunks to the prompt so the answer is grounded in them.

The running example is a shop’s help chat (the policy text is made up). Press Play to see what happens with no retrieval, then switch the Try it selector to the full RAG pipeline, and to a question your documents can’t answer.

import numpy as np
from google import genai
from google.genai import types

client = genai.Client()

# 1. Index (once): split your documents into chunks and embed each chunk.
chunks = [
    "Refunds for failed UPI payments go back to the source account within 5 working days.",
    "Cash on delivery orders are refunded as store credit or by bank transfer.",
    "You can cancel an order free of charge until it has been shipped.",
]
doc_vectors = np.array([
    e.values
    for e in client.models.embed_content(
        model="gemini-embedding-001",
        contents=chunks,
        config=types.EmbedContentConfig(task_type="RETRIEVAL_DOCUMENT"),
    ).embeddings
])


def answer(question: str, top_k: int = 2) -> str:
    # 2. Retrieve: embed the question, rank chunks by cosine similarity.
    q = np.array(client.models.embed_content(
        model="gemini-embedding-001",
        contents=question,
        config=types.EmbedContentConfig(task_type="RETRIEVAL_QUERY"),
    ).embeddings[0].values)
    scores = doc_vectors @ q / (np.linalg.norm(doc_vectors, axis=1) * np.linalg.norm(q))
    best = [chunks[i] for i in np.argsort(scores)[::-1][:top_k]]

    # 3. Generate: answer from the retrieved text, and only from it.
    context = "\n".join(f"- {c}" for c in best)
    interaction = client.interactions.create(
        model="gemini-3.8-flash",
        system_instruction=(
            "Answer only from the context. If the answer is not in the context, "
            "say that the information is not available."
        ),
        input=f"<context>\n{context}\n</context>\n\nBased on the context above: {question}",
    )
    return interaction.output_text


print(answer("My UPI payment failed. When do I get my money back?"))

What to notice

1. Two separate phases. Indexing runs once (and again when documents change). Retrieval and generation run on every question. Only the question is embedded at query time.
2. Search is by meaning, not keywords. The customer said “money back” and the chunk says “refunds”. Their embeddings are still close. Use the same embedding model for chunks and questions, with the matching task types (RETRIEVAL_DOCUMENT and RETRIEVAL_QUERY).
3. Retrieval always returns something. Top-k search hands back the k best chunks even when none is relevant. The instruction “answer only from the context, otherwise say it’s not available” is what stops the model from filling the gap with a guess.
4. The prompt is just a prompt. Retrieved chunks are context, so the usual rules apply: context first, the question last. See prompt engineering patterns.
5. Chunking is a design choice. Chunks that are too big bury the answer in noise. Chunks that are too small lose the sentence that explains it. Expect to tune this.

You don’t have to build it all yourself

The code beside the animation keeps every step visible, with a plain list as the index. In production you would normally use a managed service: