Large Language Models: Transformers, RAG, and Fine-Tuning

Large Language Models (LLMs) are the foundation of modern AI assistants. Understanding how they work and how to build with them is essential.

How Transformers Work

The Transformer architecture (2017, “Attention Is All You Need”) is the foundation of all modern LLMs.

Input Text → Tokenizer → Embeddings → Transformer Blocks → Output
                                            │
                                     Self-Attention
                                            +
                                     Feed-Forward

Key Concepts

Self-Attention (Simplified)

For each word, the model asks: “How much should I pay attention to every other word?”

"The cat sat on the mat"

For "sat": attention weights might be:
  The=0.1  cat=0.5  sat=0.1  on=0.2  the=0.0  mat=0.1

Types of LLM Models

Retrieval-Augmented Generation (RAG)

RAG solves the problem of LLMs not knowing your private data. Instead of training the model, you give it relevant documents at query time.

User Query
    │
    ↓
┌─────────────┐     ┌──────────────┐
│ Embed Query │────→│ Vector DB     │
└─────────────┘     │ Search        │
                    └──────┬───────┘
                           │ Top-K documents
                           ↓
                    ┌──────────────┐
                    │ LLM Prompt:   │
                    │ Context + Query│
                    └──────┬───────┘
                           │
                           ↓
                       Answer

RAG Pipeline

# 1. Index documents
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma

# Split documents into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)

# Store in vector database
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings)

# 2. Query
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
relevant_docs = retriever.get_relevant_documents("What is our return policy?")

# 3. Generate answer
prompt = f"Based on these documents: {relevant_docs}\n\nAnswer: {query}"
response = llm.invoke(prompt)

Fine-Tuning

Adapt a pre-trained model to your specific domain or task.

When to Fine-Tune vs RAG

LoRA (Low-Rank Adaptation)

Fine-tune efficiently by training only small adapter layers:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

model = get_peft_model(base_model, config)
# Train only ~1% of parameters

Vector Databases

Store and search text by meaning (semantic similarity), not keywords.

"How to deploy an app" → [0.2, 0.8, 0.1, ...] → Similar to:
  - "Application deployment guide" (0.92)
  - "Shipping containers to production" (0.85)
  - "Dog training tips" (0.12)

Popular Vector DBs

LLM Engineering Patterns

Prompt Engineering

# System prompt sets behavior
messages = [
    {"role": "system", "content": "You are a helpful medical assistant. "
     "Only answer based on provided context. Say 'I don't know' if unsure."},
    {"role": "user", "content": "What are the symptoms of..."}
]

Guardrails

Validate LLM outputs:

• Check for harmful content
• Validate JSON structure
• Ensure answers stay on topic
• Prevent prompt injection

Evaluation

Summary