Track 6 · sessions 143–162
LLM applications
Classification, QA, summarization, metrics, retrieval, RAG, agents, evaluation and safety.
LLM applications and systems
Every classic NLP task with its metric, then retrieval, RAG, agents, evaluation, hallucination, interpretability and security.
143Ready
Prompting and in-context learning
- Zero-shot, few-shot, instructions, system prompts
- Why in-context learning works: the main theories
- Prompt sensitivity: order and format of examples
- Writing prompts as clear specifications
144Ready
Text classification: sentiment and beyond
- Encoder + CLS head, or decoder + label words, or embeddings + logistic regression
- Zero-shot by comparing label likelihoods (“positive” vs “negative”); calibration
- Class imbalance, thresholds, metrics: accuracy, F1, ROC-AUC
- Long documents: split, classify, combine
145Ready
Natural language inference and question answering
- NLI: entailment, contradiction, neutral (SNLI, MultiNLI); NLI models as zero-shot classifiers
- Extractive QA: predict start and end positions in a passage (SQuAD)
- Open-domain QA: retrieve, then read
- Metrics: exact match (EM) and token F1
146Ready
Named entities, information extraction and structured output
- Token classification with BIO tags
- Extraction with JSON schemas and function calling
- Grammar-constrained decoding guarantees valid structure
- When a small fine-tuned model beats a big prompted one
147Ready
Summarization
- Extractive vs abstractive
- Encoder–decoder (BART, T5) vs decoder-only summarizers
- Long documents: map-reduce, refine, hierarchical summaries
- Faithfulness and hallucination in summaries
148Ready
Translation, code generation and dialogue
- Machine translation with encoder–decoders and with LLMs
- Code generation: execution-based testing
- Dialogue: multi-turn context, chat templates, persona and memory
- What makes each task hard to evaluate
149Ready
Metrics for text generation: perplexity, BLEU, ROUGE, METEOR, BERTScore, pass@k
- BLEU: clipped n-gram precision + brevity penalty
- ROUGE-N and ROUGE-L (longest common subsequence) for summaries
- METEOR (synonyms, stems) and BERTScore (embedding similarity)
- pass@k for code, with the unbiased estimator
150Ready
Text embeddings and semantic search
- Bi-encoder vs cross-encoder (reranker)
- Pooling, normalization, Matryoshka embeddings
- Public benchmarks (MTEB) vs your own data
- Metrics: recall@k, MRR, nDCG
151Ready
Vector search at scale
- Exact search vs approximate nearest neighbor (ANN) search
- IVF (clusters), PQ (compression), HNSW (graphs)
- Memory and speed math; the recall vs latency trade-off
- FAISS on CPU and GPU; filters and updates
152Ready
RAG, part 1: the basic pipeline
- Why RAG: fresh, private and citable knowledge
- Chunking, embedding, indexing; hybrid search (BM25 + dense)
- Reranking and context assembly; citations
- RAG vs long context vs fine-tuning
153Ready
RAG, part 2: advanced methods and evaluation
- Query rewriting, HyDE, multi-query, multi-hop
- GraphRAG and Self-RAG
- Evaluate retrieval (recall@k) and generation (faithfulness, relevance) separately
- Failure modes: wrong chunk, right chunk ignored, conflicting sources
154Ready
Agents and tool use
- ReAct: think → act → observe loops
- Function calling, tool schemas, MCP
- Planning, memory and multi-agent orchestration
- Failure modes: loops, tool errors, prompt injection through tools; evaluation
155Ready
Evaluation of LLM systems
- Static benchmarks (MMLU, GSM8K) vs arenas vs your own test set
- LLM-as-judge and its biases: position, length, self-preference
- Statistics: confidence intervals and enough samples
- A regression suite for your product
156Ready
Hallucination, calibration and uncertainty
- Why models hallucinate: the training objective, missing knowledge, decoding
- Detection: self-consistency, semantic entropy, retrieval checks
- Calibration and abstaining (“I don’t know”)
- Product designs that reduce hallucination
157Ready
Interpretability: looking inside the model
- Logit lens: decode the residual stream at every layer
- Attention patterns and induction heads
- Probing classifiers
- Sparse autoencoders and features
158Ready
Security and safety in LLM applications
- Prompt injection, direct and indirect (through documents and tools)
- Jailbreaks and adversarial suffixes
- Data leakage and memorization
- Guardrails, least-privilege tools, a human in the loop
159Ready
one model, five classic tasks
- Sentiment, NER, NLI, extractive QA and summarization on public datasets
- Compare an encoder (BERT-style) with a small decoder LLM (prompted and LoRA-tuned)
- Report the right metric for each task (F1, EM, ROUGE)
- Note cost and latency per task
160Ready
build and evaluate a RAG system
- Index a document set with hybrid search and a reranker
- Write a test set of 50 or more questions with known answers and sources
- Measure retrieval recall@k and answer faithfulness
- Fix the biggest failure mode and measure again
161Ready
an agent with tools, and its evaluation
- Build a ReAct agent with search, a calculator and code execution
- Write 30 tasks with checkable answers
- Measure success rate, steps, cost and failure types
- Add one guardrail against prompt injection and test it
162Ready
the whole LLM system on one whiteboard
- Draw the full path: tokenizer → model → training → serving → application
- Answer exam-style questions with numbers: memory, FLOPs, KV cache, latency
- Find the bottleneck in 5 short case studies, and name the survey branch that fixes it
- List what is still unclear; we fix it before Part E