LLM & VLM courseBlog
Track 6 · sessions 143–162

LLM applications

Classification, QA, summarization, metrics, retrieval, RAG, agents, evaluation and safety.

LLM applications and systems

Every classic NLP task with its metric, then retrieval, RAG, agents, evaluation, hallucination, interpretability and security.

143

Prompting and in-context learning

  • Zero-shot, few-shot, instructions, system prompts
  • Why in-context learning works: the main theories
  • Prompt sensitivity: order and format of examples
  • Writing prompts as clear specifications
Ready
144

Text classification: sentiment and beyond

  • Encoder + CLS head, or decoder + label words, or embeddings + logistic regression
  • Zero-shot by comparing label likelihoods (“positive” vs “negative”); calibration
  • Class imbalance, thresholds, metrics: accuracy, F1, ROC-AUC
  • Long documents: split, classify, combine
Ready
145

Natural language inference and question answering

  • NLI: entailment, contradiction, neutral (SNLI, MultiNLI); NLI models as zero-shot classifiers
  • Extractive QA: predict start and end positions in a passage (SQuAD)
  • Open-domain QA: retrieve, then read
  • Metrics: exact match (EM) and token F1
Ready
146

Named entities, information extraction and structured output

  • Token classification with BIO tags
  • Extraction with JSON schemas and function calling
  • Grammar-constrained decoding guarantees valid structure
  • When a small fine-tuned model beats a big prompted one
Ready
147

Summarization

  • Extractive vs abstractive
  • Encoder–decoder (BART, T5) vs decoder-only summarizers
  • Long documents: map-reduce, refine, hierarchical summaries
  • Faithfulness and hallucination in summaries
Ready
148

Translation, code generation and dialogue

  • Machine translation with encoder–decoders and with LLMs
  • Code generation: execution-based testing
  • Dialogue: multi-turn context, chat templates, persona and memory
  • What makes each task hard to evaluate
Ready
149

Metrics for text generation: perplexity, BLEU, ROUGE, METEOR, BERTScore, pass@k

  • BLEU: clipped n-gram precision + brevity penalty
  • ROUGE-N and ROUGE-L (longest common subsequence) for summaries
  • METEOR (synonyms, stems) and BERTScore (embedding similarity)
  • pass@k for code, with the unbiased estimator
Ready
151

Vector search at scale

  • Exact search vs approximate nearest neighbor (ANN) search
  • IVF (clusters), PQ (compression), HNSW (graphs)
  • Memory and speed math; the recall vs latency trade-off
  • FAISS on CPU and GPU; filters and updates
Ready
152

RAG, part 1: the basic pipeline

  • Why RAG: fresh, private and citable knowledge
  • Chunking, embedding, indexing; hybrid search (BM25 + dense)
  • Reranking and context assembly; citations
  • RAG vs long context vs fine-tuning
Ready
153

RAG, part 2: advanced methods and evaluation

  • Query rewriting, HyDE, multi-query, multi-hop
  • GraphRAG and Self-RAG
  • Evaluate retrieval (recall@k) and generation (faithfulness, relevance) separately
  • Failure modes: wrong chunk, right chunk ignored, conflicting sources
Ready
154

Agents and tool use

  • ReAct: think → act → observe loops
  • Function calling, tool schemas, MCP
  • Planning, memory and multi-agent orchestration
  • Failure modes: loops, tool errors, prompt injection through tools; evaluation
Ready
155

Evaluation of LLM systems

  • Static benchmarks (MMLU, GSM8K) vs arenas vs your own test set
  • LLM-as-judge and its biases: position, length, self-preference
  • Statistics: confidence intervals and enough samples
  • A regression suite for your product
Ready
156

Hallucination, calibration and uncertainty

  • Why models hallucinate: the training objective, missing knowledge, decoding
  • Detection: self-consistency, semantic entropy, retrieval checks
  • Calibration and abstaining (“I don’t know”)
  • Product designs that reduce hallucination
Ready
157

Interpretability: looking inside the model

  • Logit lens: decode the residual stream at every layer
  • Attention patterns and induction heads
  • Probing classifiers
  • Sparse autoencoders and features
Ready
158

Security and safety in LLM applications

  • Prompt injection, direct and indirect (through documents and tools)
  • Jailbreaks and adversarial suffixes
  • Data leakage and memorization
  • Guardrails, least-privilege tools, a human in the loop
Ready
159

one model, five classic tasks

  • Sentiment, NER, NLI, extractive QA and summarization on public datasets
  • Compare an encoder (BERT-style) with a small decoder LLM (prompted and LoRA-tuned)
  • Report the right metric for each task (F1, EM, ROUGE)
  • Note cost and latency per task
Ready
160

build and evaluate a RAG system

  • Index a document set with hybrid search and a reranker
  • Write a test set of 50 or more questions with known answers and sources
  • Measure retrieval recall@k and answer faithfulness
  • Fix the biggest failure mode and measure again
Ready
161

an agent with tools, and its evaluation

  • Build a ReAct agent with search, a calculator and code execution
  • Write 30 tasks with checkable answers
  • Measure success rate, steps, cost and failure types
  • Add one guardrail against prompt injection and test it
Ready
162

the whole LLM system on one whiteboard

  • Draw the full path: tokenizer → model → training → serving → application
  • Answer exam-style questions with numbers: memory, FLOPs, KV cache, latency
  • Find the bottleneck in 5 short case studies, and name the survey branch that fixes it
  • List what is still unclear; we fix it before Part E
Ready