LLM & VLM courseBlog
83Track 5: LLM inference and serving · session 6 of 65

The map: data-level, model-level and system-level optimization

There are hundreds of papers about making LLM inference faster. This session gives you one map to place all of them: data-level methods change the input or the shape of the output, model-level methods change the architecture or the weights, and system-level methods change the engine or the server. With one simple cost model you will see which part of the time each lever removes, which levers can hurt quality, and how the next 11 weeks of the course follow this map.

Time: 2 to 3 hours (reading 1.5 hours, exercises 1 hour)Level: IntermediateBefore this: Sessions 78 to 82 (the generation loop, the KV cache, inference metrics, the three root causes and the roofline)

What you will learn

  • The survey’s taxonomy: data-level, model-level and system-level optimization, and what each level is allowed to change
  • Why system-level methods are usually lossless, model-level methods are usually lossy and need training, and data-level methods change only the input or the output layout
  • A one-page cost model (TTFT, TPOT, memory) that shows which term each lever shrinks, with real Llama 3 8B and H100 numbers
  • Why a 4× shorter prompt can make a chat answer only 6% faster, and why INT4 weights do not speed up prefill (Amdahl’s law for inference)
  • How the three root causes (model size, quadratic attention, token-by-token decoding) map onto the three levels
  • How to classify a new method in one minute, and how to choose levers for a workload
  • How the next 55 sessions (11 weeks) walk through the map, one branch at a time

Why this matters

In the last five sessions we learned how generation runs and why it is slow. Now we want to make it fast. The problem is that there are hundreds of methods: prompt compression, MoE, GQA, GPTQ, AWQ, pruning, distillation, FlashAttention, speculative decoding, PagedAttention, continuous batching, and many more. Without a map, they look like a random list of names.

The survey by Zhou et al. (arXiv 2404.14294), which this track follows, sorts all of them into three levels [Certain: Section 3 of the survey]:

  • Data level: change the input (make the prompt shorter) or the shape of the output (let parts of the answer be written in parallel). The model stays exactly the same.
  • Model level: change the architecture (a cheaper design) or compress the weights (fewer bits, fewer weights, a smaller student). This is often lossy and may need training.
  • System level: change the engine that runs the model or the server that schedules many requests. The model and the input stay the same, so this is usually lossless.

This map is useful for three reasons. First, it tells you what you give up: a system-level change should not change the answers, a model-level change can, and you must measure it. Second, it tells you what you need: a new kernel needs engineering time, a quantized model needs a calibration set, an MoE needs a pre-training run. Third, the levers at different levels act on different parts of the time, so they can be combined. A real production stack usually uses all three at once: for example a quantized model (model level) served by vLLM with continuous batching (system level) behind a RAG pipeline that sends only the best passages (data level).

We continue from session 82, which named the three root causes of slow inference: the model size, the quadratic attention over the prompt, and the token-by-token decoding. Every method on the map attacks at least one of them. At the end of the session we walk through the next 55 sessions (11 weeks of five sessions each), which follow the map one branch at a time.

Three things you can change

Write the whole inference process as one function. An engine \(\mathcal{E}\) takes the weights \(\theta\) and the input tokens \(x\), and produces the output tokens \(\hat{y}\):

\[ \hat{y} \;=\; \mathcal{E}\big(\theta,\; x\big), \qquad \hat{y}_t \sim p_\theta\!\left(\cdot \mid x, \hat{y}_{\lt t}\right) \tag{83.1} \]
Read the symbols
\(x\)
the input: system prompt, retrieved documents, user question, image or video tokens
\(\theta\)
the weights of the model, together with its architecture (the shapes and the kind of layers)
\(\mathcal{E}\)
the engine and the server: the kernels, the order of the arithmetic, the KV-cache layout, the batching and scheduling, the hardware
\(\hat{y}_t\)
the output token at step \(t\), drawn from the model’s next-token distribution given the input and all earlier output tokens \(\hat{y}_{\lt t}\)

Each level of the map changes exactly one of the three arguments:

  • Data level replaces \(x\) by a shorter \(x' = g(x)\), or asks for an output with a different structure. \(\theta\) and \(\mathcal{E}\) stay.
  • Model level replaces \(\theta\) by \(\theta'\): a cheaper architecture, fewer bits per weight, fewer weights.
  • System level replaces \(\mathcal{E}\) by \(\mathcal{E}'\): faster kernels, a smarter cache, better batching, a different way to produce the same tokens.

Now we can say precisely what “lossy” means. Compare the next-token distributions of the original setup and the optimized setup, averaged over real inputs:

\[ \Delta \;=\; \E_{x}\Big[ \KL\big(\, p_\theta(\cdot \mid x) \,\big\|\, p_{\theta'}(\cdot \mid x') \,\big) \Big] \tag{83.2} \]
Read the symbols
\(\Delta\)
the quality gap: zero means the optimized system gives exactly the same distribution over next tokens
\(\KL(p \,\|\, q)\)
the KL divergence from session 3: how much information we lose if we use \(q\) where \(p\) is the truth
\(\theta', x'\)
the new weights (model level) and the new input (data level); for a system-level change, \(\theta' = \theta\) and \(x' = x\)

A system-level change keeps \(\theta' = \theta\) and \(x' = x\), so \(\Delta = 0\) in exact arithmetic. In floating point the order of additions changes the last bits, so \(\Delta\) is tiny but not always exactly zero. The survey says system-level methods are “typically lossless”, with a footnote that FlashAttention may cause small numeric deviations [Certain: survey Section 3]. Model-level and data-level changes make \(\Delta \gt 0\), and the whole question is whether the gap is small enough for your task. For output organization (Skeleton-of-Thought), the answer itself has a new structure, so we compare answer quality with a benchmark or a judge instead of a KL.

Figure 1 is the full map from the survey, with the parts of the course that teach each branch. Click any box.

data level (survey Section 4)model level (Section 5)system level (Section 6)
Figure 1. The map of efficient inference, following the taxonomy of Zhou et al. (2024). Gray labels are the survey’s groups; each box is one branch. The readout says what the branch changes, what it needs, whether it is lossy, which root cause it attacks, and where the course teaches it.
How to read Figure 1

Read the columns from left to right: they go from “change the input” to “change the weights” to “change the machine”. Click Quantization: it changes \(\theta\) (fewer bits per weight), needs only a small calibration set for post-training methods, is lossy, and attacks the model-size root cause; the course spends weeks 4 and 5 on it. Now click Speculative decoding: it does not change the model or the input, it is lossless, and it attacks token-by-token decoding. Notice how almost every system-level box says “lossless” and almost every model-level box says “needs training or calibration”.

The survey also says that data-level methods “typically do not change the original model”, so they avoid costly training, although “a small amount of training for auxiliary models might be required”. Model-level methods “often require costly pre-training or a smaller amount of fine-tuning” and are “typically lossy”. System-level methods do not involve costly model training [Certain: quoted from survey Section 3]. A second survey, by Wan et al. (arXiv 2312.03863), cuts the same field differently: model-centric methods, data-centric methods, and LLM frameworks [Certain]. It also covers efficient pre-training and fine-tuning, not only inference. The two maps agree on the main idea: separate what you change in the data, in the model, and in the software that runs it.

One cost model for every lever

To compare levers we need one simple model of time and memory. We reuse the roofline model from session 78 and session 82. A step takes at least the larger of its compute time and its memory time. A request takes one prefill and then \(n - 1\) decode steps:

\[ T_{\text{req}} \;\approx\; \text{TTFT} + (n - 1)\cdot \text{TPOT} \tag{83.3} \]
\[ \text{TTFT} \approx \max\!\left( \frac{2NP + 2 L d_{\text{a}} P^2}{\pi},\; \frac{b_w N + P s_{\text{kv}}}{\beta} \right) \tag{83.4} \]
\[ \text{TPOT} \approx \max\!\left( \frac{2NB + 4 L d_{\text{a}} T B}{\pi},\; \frac{b_w N + B\, T\, s_{\text{kv}}}{\beta} \right) \tag{83.5} \]
Read the symbols
\(P, n\)
prompt length and number of output tokens (the data level changes these)
\(N, L, d_{\text{a}}\)
number of parameters, number of layers, total width of the query heads (the model level changes these)
\(b_w\)
bytes per weight: 2 for BF16, 1 for INT8, 0.5 for INT4 (model level)
\(s_{\text{kv}}\)
KV-cache bytes per token, \(2 L n_{kv} d_h b_{kv}\), from session 80 (model level: GQA, MLA, KV quantization)
\(B\)
batch size: how many sequences share one decode step (the system level controls this)
\(T\)
tokens already in the cache of each sequence; it grows from \(P\) to \(P + n - 1\) during the answer
\(\pi, \beta\)
peak compute (FLOP/s) and memory bandwidth (bytes/s) of the GPU (system level: hardware, and how close the kernels get to the peaks)

Each symbol in these three formulas belongs to one level. That is the whole map in one line: the data level moves \(P\) and \(n\), the model level moves \(N\), \(b_w\) and \(s_{\text{kv}}\), and the system level moves \(B\), \(\pi\), \(\beta\) and the number of steps the engine needs per token.

Worked example: one RAG request on Llama 3 8B and one H100

Llama 3 8B: \(N = 8.03 \times 10^9\), \(L = 32\), \(d_{\text{a}} = 4096\), 8 KV heads of size 128, so \(s_{\text{kv}} = 131{,}072\) bytes per token in BF16 (from its config.json, as in session 80). H100 SXM: \(\pi = 989\) TFLOP/s dense BF16 and \(\beta = 3.35\) TB/s. The request: a prompt of \(P = 4000\) tokens (question plus retrieved passages) and an answer of \(n = 300\) tokens, batch size 1.

Prefill. FLOPs \(= 2 \times 8.03 \times 10^9 \times 4000 + 2 \times 32 \times 4096 \times 4000^2 = 6.424 \times 10^{13} + 4.19 \times 10^{12} = 6.84 \times 10^{13}\). At 989 TFLOP/s this is 69.2 ms of TTFT. The bytes (16.06 GB of weights) need only 4.8 ms, so prefill is compute-bound.

Decode. Each step reads the 16.06 GB of weights plus on average about 0.54 GB of cache (about \(4{,}150\ \text{tokens} \times 131{,}072\) bytes). That is \(16.6 \text{ GB} / 3.35 \text{ TB/s} \approx\) 4.96 ms per token.

Total. \(69.2 \text{ ms} + 299 \times 4.96 \text{ ms} \approx\) 1.55 s. Decode is 95.5% of the time, even though the prompt has more than 13 times as many tokens as the answer. Memory: 16.06 GB of weights plus 0.56 GB of cache for this one request.

All numbers are best-case roofline bounds, computed with a short Python script (shown later). Real engines are slower, but the shares of time are what matter for choosing a lever.

Data level: change the input or the shape of the output

The data level does not touch the model. It asks: does the model really need to read all these tokens, and does it really need to write the answer strictly one token after another?

Input compression (survey Section 4.1)

Long prompts are now normal: system prompts with rules and tools, conversation history, retrieved documents, few-shot examples, and in VLMs hundreds of image tokens. In formula (83.4), the prompt length \(P\) appears in the linear term and squared in the attention term, and every prompt token also stays in the KV cache for the whole answer. The survey lists four families:

  • Prompt pruning: delete tokens, words or sentences that carry little information. Selective Context and LLMLingua score tokens with a small language model and drop the predictable ones; LongLLMLingua uses the question to decide what to keep (session 88).
  • Prompt summary: rewrite the context into a shorter text, for example RECOMP compresses retrieved documents into a summary (session 89).
  • Soft-prompt compression: replace a long text with a few learned vectors that the model reads instead (Gisting, AutoCompressors, ICAE). These compressors are trained.
  • Retrieval-augmented generation (RAG): instead of putting a whole knowledge base in the prompt, retrieve only the passages that matter (session 90).

Output organization (survey Section 4.2)

Here the goal is the third root cause: one token per step. Skeleton-of-Thought (SoT) first asks the model for a short outline of the answer, then expands all points of the outline at the same time, as a batch of parallel requests. The number of sequential decode steps drops from the length of the full answer to roughly the length of the outline plus the length of the longest point. APAR teaches the model to fork parallel branches itself, and SGLang gives a programming language to write such structured generation programs (session 91). The price: the answer has a different structure, and points written in parallel cannot see each other.

What it costs. No training of the LLM. Sometimes a small helper model (a scorer, a summarizer, a retriever). The result is lossy: a deleted token might be the one fact the answer needed. In the Python demo later in this session, deleting the oldest half of the prompt leaves the answers unchanged, while deleting every second token destroys them: which tokens you delete matters more than how many.

Model level: change the architecture or compress the weights

The model level changes \(\theta\). It has two branches.

Efficient structure design (survey Section 5.1)

Design a model that is cheaper to run from the start:

  • Efficient FFN: a mixture of experts (MoE) keeps many FFN “experts” but sends each token to only a few of them, so the FLOPs per token depend on the active parameters, not on all parameters. Memory still holds all of them (session 93, then the three Efficient FFN sessions).
  • Efficient attention: multi-query and grouped-query attention (MQA, GQA) and multi-head latent attention (MLA) shrink \(s_{\text{kv}}\); low-rank attention (Linformer) and kernel attention (Performer) remove the \(P^2\) term (session 97).
  • Transformer alternatives: state space models (S4, Mamba), RWKV and RetNet keep a fixed-size state instead of a cache that grows with every token.

Model compression (survey Section 5.2)

Take a trained model and make it smaller:

  • Quantization: store weights (and maybe activations and the KV cache) in fewer bits: INT8, INT4, FP8. Post-training methods such as GPTQ and AWQ need only a small calibration set; quantization-aware training needs training (session 103).
  • Sparsification: remove weights (unstructured, 2:4, or whole heads and channels), or skip attention pairs (sparse attention). SparseGPT and Wanda prune in one shot; structured pruning usually needs recovery training (session 113).
  • Structure optimization: low-rank factorization of weight matrices and neural architecture search.
  • Knowledge distillation: train a small student to copy a large teacher (session 119).
  • Dynamic inference: early exit and layer skipping, so easy tokens use fewer layers (session 121).

What it costs. Training or at least calibration: from minutes of calibration for GPTQ, to a fraction of a pre-training run (the GQA paper converts a multi-head checkpoint by “uptraining” with 5% of the original pre-training compute [Certain: GQA paper]), to a full pre-training run for a new MoE or Mamba model. The result is usually lossy, so it must be evaluated. A new architecture is a different model; we compare it with the old one at the same training budget, not token by token.

Look at formula (83.4) and (83.5) again to see a subtle point. Weight-only quantization shrinks \(b_w\), which appears only in the memory terms. Decode is memory-bound, so it gets much faster. Prefill is compute-bound and its arithmetic still runs in BF16, so TTFT does not change at all. To speed up prefill with quantization you must also quantize the activations (W8A8, FP8) so the tensor cores run at a higher FLOP/s (session 104).

System level: change the engine or the server

The system level keeps the model and the input, and changes how the computation is done.

The inference engine (survey Section 6.1)

  • Graph and operator optimization: FlashAttention computes exact attention without writing the \(P \times P\) score matrix to memory; FlashDecoding splits the decode attention over the long cache across many GPU cores; fused kernels and CUDA graphs remove launch overhead (session 123, session 124).
  • Speculative decoding: a small draft model guesses \(\gamma\) tokens, and the big model checks all of them in one pass. Accepted guesses mean more than one token per read of the big weights (session 128).
  • Offloading: keep weights or cache in CPU memory or on disk and stream them to the GPU when needed (FlexGen, llama.cpp). It makes a big model fit on a small GPU, but it is slower (session 130).

The serving system (survey Section 6.2) and hardware (6.3)

  • Memory management: PagedAttention stores the cache in small blocks, so almost no memory is reserved and wasted; prefix caching reuses the cache of a shared prompt (session 134).
  • Batching: continuous batching lets requests join and leave the decode batch at every step; chunked prefill mixes prompt chunks with decode steps (session 84, next session).
  • Scheduling: which request runs next, preemption, fairness and latency targets.
  • Distributed systems: tensor and pipeline parallel inference, and prefill–decode disaggregation.
  • Hardware: GPUs, TPUs, edge chips; the engines (vLLM, SGLang, TensorRT-LLM, llama.cpp) that tie everything together.

Speculative decoding is the best example of “lossless but not trivial”. With the rejection-sampling rule of Leviathan et al. and Chen et al., the tokens it produces have exactly the distribution of the big model [Certain: proved in both papers]. With draft length \(\gamma\) and a probability \(\alpha\) that each draft token is accepted (assumed independent), one round produces on average

\[ \E[\text{tokens per round}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{TPOT}_{\text{spec}} \approx \frac{\gamma\, c\, t_{\text{step}} + t_{\text{verify}}}{\E[\text{tokens per round}]} \tag{83.6} \]
Read the symbols
\(\gamma\)
how many tokens the draft model proposes per round
\(\alpha\)
the acceptance rate: the chance that the big model accepts one draft token; it depends on the task and on how similar the two models are
\(c\)
the cost of one draft step relative to one big-model step
\(t_{\text{step}}, t_{\text{verify}}\)
the time of one normal decode step and of one verification step with \(\gamma + 1\) tokens per sequence; at small batch both are memory-bound, so they are almost equal

With \(\gamma = 4\) and \(\alpha = 0.8\), a round gives \((1 - 0.8^5)/0.2 = 3.36\) tokens. If the draft is Llama 3.2 1B (1.23B parameters, so \(c \approx 1.23/8.03 \approx 0.15\) by bytes), each round costs about \(1 + 4 \times 0.15 = 1.6\) big steps, and the speed-up is \(3.36 / 1.6 \approx 2.1\) times. With \(\alpha = 0.6\) it is only \(2.31/1.6 \approx 1.44\) times. The values of \(\alpha\) here are assumptions for illustration, not measurements.

What it costs. No training of the target model. Instead: engineering (kernels, schedulers), extra memory (a draft model, cache blocks), and sometimes a small trained part (Medusa heads, an EAGLE draft head). The survey places these under speculative decoding anyway, because the big model’s output distribution does not change.

Combining levers: which part of the time does each one remove?

A lever speeds up only the part of the time it acts on. If a part takes a fraction \(p\) of the total time and a lever makes that part \(s\) times faster, the whole request becomes faster by

\[ S \;=\; \frac{1}{(1 - p) + p / s} \tag{83.7} \]
Read the symbols
\(p\)
the share of the total time spent in the part the lever acts on (for example, the decode share)
\(s\)
how much faster that part becomes
\(S\)
the speed-up of the whole request; it can never exceed \(1/(1 - p)\)

This is Amdahl’s law, from computer architecture. It is the main reason you must measure before you optimize. In the worked example above, prefill is only 4.5% of the time. Even an infinitely fast prefill would make the request at most \(1 / 0.955 \approx 1.05\) times faster. So compressing that prompt 4 times makes the prefill 4.19 times faster but the request only about 1.06 times faster (a little more than Amdahl’s 1.035, because the shorter prompt also shrinks the cache that every decode step reads). INT4 weights make each decode step 3.64 times faster (not 4, because the cache is still BF16), and the request \(1/(0.045 + 0.955/3.64) \approx 3.26\) times faster.

Try it yourself in Figure 2. Every control belongs to one level.

Workload
Data: compress prompt
Model: weightsKV cache
System: speculative
prefill (TTFT)decode (\(n - 1\) steps)KV cache memoryweight memory
Figure 2. The cost model of formulas (83.3) to (83.6) for Llama 3 8B on one H100 SXM (80 GB). RAG: \(P = 4000\), \(n = 300\). Video QA: \(32 \times 576 = 18{,}432\) prompt tokens (32 frames of 576), \(n = 100\). Chat: \(P = 500\), \(n = 500\). Speculative decoding uses \(\gamma = 4\) and a draft that costs 15% of a big step; INT8/INT4 are weight-only. These are best-case bounds: real systems are slower, and quality effects are not modeled.
How to read Figure 2

The top bar is the baseline request (BF16, batch 1, no compression); the bar below is your setting, on the same time scale. Start with the RAG workload and press 4× compression: the blue part shrinks from 69.2 to 16.5 ms, but the total drops only from 1.55 s to 1.46 s. Now choose INT4: the orange part shrinks almost 4 times (total 0.388 s), while the blue part does not move. Add speculative decoding with \(\alpha = 0.8\): 0.193 s, about 8 times faster than the baseline. Then switch to Video QA: now prefill is 42% of the time, and 4× compression alone gives 1.63 times. Move the batch slider: total tokens per second grow fast while each user gets a little slower; at \(B = 128\) with BF16 cache the RAG requests no longer fit in 80 GB, and an 8-bit cache makes them fit again. The readout lists which changes are lossy.

Figure 3 adds one lever from each level, one at a time, to the RAG request. Watch how the size of each gain depends on what is left.

Figure 3. Stacking levers on the RAG request (Llama 3 8B, one H100, batch 1, roofline bounds). Step 1 is the baseline. Step 2 adds 4× prompt compression (data level), step 3 INT4 weights (model level), step 4 speculative decoding with \(\gamma = 4\), \(\alpha = 0.8\) (system level).
How to read Figure 3

Each bar is one request, split into prefill (blue) and decode (orange), on the same scale. Step 2 removes most of the blue but saves only 6% of the total. Step 3 attacks the large orange part and gives 3.77 times. Step 4 halves the orange again (2.0 times). The final request takes 0.193 s instead of 1.55 s: about 8.0 times faster. The speed-ups multiply here because each lever acts on a different term of the cost model. Notice that after step 4 the blue part is 8.5% of the time again: prefill would be the next target.

Levers do not always stack this nicely. Some examples you will meet later:

  • Speculative decoding and large batches. Speculative decoding uses the idle compute of a memory-bound step. At a large batch the step becomes closer to compute-bound, and the extra verification FLOPs are no longer free, so the gain shrinks or disappears [Likely: reported in several serving studies; the roofline model shows the trend].
  • Query-dependent prompt compression and prefix caching. If the compressor keeps different tokens for every question, two requests no longer share the same prefix, and the cache of that prefix cannot be reused [Likely].
  • Quantization and a draft model. If the big model becomes 4 times smaller in bytes, a draft model that was “15% of the cost” is now about 60% of it, unless it is quantized too.

The three root causes on the map

Session 82 named three root causes. The table shows which levers attack which cause. Most methods attack one cause at one level; a good system attacks all three.

Root causeData levelModel levelSystem level
Model size (weights to store and read)— (the model is unchanged)Quantization, pruning, distillation, low-rank, MoE (fewer active weights)Batching (share each weight read), offloading (fit in less GPU memory), tensor parallelism
Attention (quadratic in \(P\), KV cache grows)Prompt pruning, summaries, soft prompts, RAGGQA, MLA, low-rank and kernel attention, sparse attention, SSMs, KV-cache quantizationFlashAttention, FlashDecoding, PagedAttention, prefix caching
Decoding (one token per step)Skeleton-of-Thought, APAR (parallel parts of the answer)Early exit and layer skipping (cheaper steps)Speculative decoding (several tokens per step), continuous batching (more users per step)

Two things in this table are worth remembering for interviews. First, the KV cache sits under “attention” but is often the real limit on batch size, so it is attacked from all three levels: shorter prompts, fewer KV heads or fewer bits, and paged memory. Second, the borders are not perfectly sharp. KV-cache quantization is applied at serving time, but the survey puts it under model-level quantization (Section 5.2.1) because it changes the numbers the model computes with, and it is lossy. Medusa trains extra heads, but the survey files it under speculative decoding (system level). When in doubt, ask the question of formula (83.1): which argument did the method change?

The algorithms and the code

Algorithm 1: place a new method on the map
  1. input: a method description (a paper, a library feature, a config flag)
  2. if it changes the prompt, the retrieved context, or the requested structure of the answer: data level // \(x\) changes; \(\theta\) and the engine do not
  3. else if it changes the weights, their number format, or the architecture: model level // \(\theta\) changes: expect \(\Delta \gt 0\), plan an evaluation
  4. else: system level // only the engine changes: expect \(\Delta \approx 0\), test for exactness
  5. name the root cause it attacks: model size, attention, or decoding
  6. name the term of (83.3)–(83.5) it shrinks: \(P\), \(n\), \(N\), \(b_w\), \(s_{\text{kv}}\), steps per token, or it raises \(B\), \(\pi\), \(\beta\)
  7. name what it needs: nothing, a calibration set, fine-tuning, pre-training, a helper model, or new kernels
  8. return (level, root cause, term, cost, lossy or lossless)
Algorithm 2: choose levers for a workload
  1. input: a model, a GPU, a traffic sample (prompt and answer lengths), latency targets, a quality budget
  2. measure or estimate TTFT, TPOT, memory and the prefill share \(p\) with (83.3)–(83.5) // profile first
  3. if requests do not fit in memory: quantize weights or the cache, use GQA models, use paged memory, or offload
  4. if the prefill share is large (long prompts, RAG, video): compress the input, use prefix caching, chunked prefill, FP8 activations
  5. if the decode share is large and the batch is small: weight quantization, speculative decoding
  6. if throughput (cost) matters more than one user’s latency: raise \(B\) with continuous batching until the latency target is hit
  7. apply system-level levers first (lossless), then data and model levels, and re-measure with Amdahl’s law (83.7) after each change
  8. for every lossy lever, measure the quality gap on your own task // formula (83.2) or a task benchmark
  9. return the chosen stack and its measured speed and quality

The PyTorch program below makes the “lossless versus lossy” idea concrete. It trains a tiny decoder for a few seconds to continue arithmetic sequences (for example 3, 7, 11, 15, …, modulo 32). Then it applies one change from each level and measures the quality gap \(\Delta\) of formula (83.2), the share of identical argmax predictions, and the accuracy. It runs on a CPU in about 10 seconds.

import torch, torch.nn.functional as F
torch.manual_seed(0)
V, d, H, L, T = 32, 64, 4, 2, 16          # vocab, width, heads, layers, sequence length
dh = d // H

def batch(n, g):                          # arithmetic sequences mod V: x_t = (a + s*t) % V
    a = torch.randint(0, V, (n, 1), generator=g); s = torch.randint(1, 6, (n, 1), generator=g)
    return (a + s * torch.arange(T + 1)) % V

class Block(torch.nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = torch.nn.LayerNorm(d), torch.nn.LayerNorm(d)
        s.qkv, s.o = torch.nn.Linear(d, 3 * d), torch.nn.Linear(d, d)
        s.ff = torch.nn.Sequential(torch.nn.Linear(d, 4 * d), torch.nn.GELU(), torch.nn.Linear(4 * d, d))
    def forward(s, x, cache=None):
        B, t, _ = x.shape
        q, k, v = [z.view(B, t, H, dh).transpose(1, 2) for z in s.qkv(s.n1(x)).split(d, -1)]
        if cache is not None and "k" in cache:                 # decode: reuse stored keys/values
            k = torch.cat([cache["k"], k], 2); v = torch.cat([cache["v"], v], 2)
        if cache is not None: cache["k"], cache["v"] = k, v
        a = F.scaled_dot_product_attention(q, k, v, is_causal=(t > 1))
        x = x + s.o(a.transpose(1, 2).reshape(B, t, d))
        return x + s.ff(s.n2(x))

class TinyLM(torch.nn.Module):
    def __init__(s):
        super().__init__()
        s.emb, s.pos = torch.nn.Embedding(V, d), torch.nn.Embedding(T, d)
        s.blocks = torch.nn.ModuleList([Block() for _ in range(L)]); s.head = torch.nn.Linear(d, V)
    def forward(s, ids, caches=None, start=0):
        x = s.emb(ids) + s.pos(torch.arange(start, start + ids.shape[1]))
        for i, b in enumerate(s.blocks):
            x = b(x, None if caches is None else caches[i])
        return s.head(x)

model = TinyLM(); opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
g = torch.Generator().manual_seed(1)
for step in range(400):                                    # learn "continue the sequence"
    seq = batch(128, g)
    loss = F.cross_entropy(model(seq[:, :T]).reshape(-1, V), seq[:, 1:].reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
model.eval()

test = batch(1000, torch.Generator().manual_seed(2)); x, y = test[:, :T], test[:, T]
def report(name, logits, ref):
    kl = F.kl_div(F.log_softmax(logits, -1), F.log_softmax(ref, -1), log_target=True, reduction="batchmean")
    agree = (logits.argmax(-1) == ref.argmax(-1)).float().mean()
    acc = (logits.argmax(-1) == y).float().mean()
    print(f"{name:28s} KL {kl.item():.1e}  same argmax {100*agree:.1f}%  accuracy {100*acc:.1f}%")

with torch.no_grad():
    ref = model(x)[:, -1]                                    # baseline: full forward pass
    report("baseline", ref, ref)
    # SYSTEM level: same model, same input, different execution (prefill 15 + 1 decode step)
    caches = [dict() for _ in range(L)]
    model(x[:, :-1], caches, 0)
    dec = model(x[:, -1:], caches, T - 1)[:, -1]
    print("system: KV-cache decode      max |logit diff| %.1e" % (dec - ref).abs().max().item())
    report("system: KV-cache decode", dec, ref)
    # MODEL level: round every Linear weight to 8, 4 or 2 bits (per output row, symmetric)
    for bits in (8, 4, 2):                                  # 2 bits symmetric = only -1, 0, +1 times a scale
        q = TinyLM(); q.load_state_dict(model.state_dict()); q.eval()
        for m in q.modules():
            if isinstance(m, torch.nn.Linear):
                w = m.weight; qmax = 2 ** (bits - 1) - 1
                scale = w.abs().amax(1, keepdim=True) / qmax
                m.weight.copy_((w / scale).round().clamp(-qmax, qmax) * scale)
        report(f"model: INT{bits} weights", q(x)[:, -1], ref)
    # DATA level: drop half of the prompt tokens, two ways
    keep_recent = x[:, T // 2:]                              # drop the oldest 8 tokens
    report("data: drop oldest half", model(keep_recent)[:, -1], ref)
    every_other = x[:, 1::2]                                 # drop every second token
    report("data: drop every 2nd token", model(every_other)[:, -1], ref)
print("final train loss %.3f" % loss.item())

With PyTorch 2.8 on a CPU it prints:

baseline                     KL 0.0e+00  same argmax 100.0%  accuracy 100.0%
system: KV-cache decode      max |logit diff| 1.3e-05
system: KV-cache decode      KL 9.3e-10  same argmax 100.0%  accuracy 100.0%
model: INT8 weights          KL 5.8e-07  same argmax 100.0%  accuracy 100.0%
model: INT4 weights          KL 2.1e-04  same argmax 100.0%  accuracy 100.0%
model: INT2 weights          KL 1.7e+00  same argmax 68.7%  accuracy 68.7%
data: drop oldest half       KL 7.9e-04  same argmax 100.0%  accuracy 100.0%
data: drop every 2nd token   KL 8.0e+00  same argmax 19.2%  accuracy 19.2%
final train loss 0.106

Read the three groups. The system-level change (KV-cache decoding instead of a full pass) differs only by float32 rounding: \(\Delta \approx 10^{-9}\). The model-level change grows with the compression: invisible at 8 bits, small at 4 bits, and destructive at 2 bits with this simple rounding (better methods such as GPTQ, AWQ and QAT exist exactly for this). The data-level change depends on which tokens are removed: this task needs only the last few tokens, so dropping the oldest half is almost free, while dropping every second token doubles the apparent step size and breaks the answer. Real prompt compressors (LLMLingua and its successors) try to learn which tokens are safe to drop. The exact numbers may change with another PyTorch version; the pattern does not.

The cost model of Figures 2 and 3 is a few lines of plain Python. It reproduces every number in the worked examples:

# Roofline cost model of one request: formulas (83.3) to (83.6). Lower bounds, not measurements.
N, L, NKV, DH, DA = 8.03e9, 32, 8, 128, 4096        # Llama 3 8B
PEAK, BW = 989e12, 3.35e12                         # H100 SXM: dense BF16 FLOP/s, bytes/s

def step(m, T, B, wb, kb):                       # m new tokens per sequence, T cached tokens, batch B
    skv = 2 * L * NKV * DH * kb                    # KV bytes per token
    flops = 2 * N * m * B + 4 * L * DA * m * (T + m / 2) * B
    bytes_ = wb * N + B * (T + m) * skv
    return max(flops / PEAK, bytes_ / BW)

def request(P=4000, n=300, compress=1, wbits=16, kvbits=16, B=1, alpha=None, gamma=4, c=0.15):
    wb, kb, P = wbits / 8, kvbits / 8, round(P / compress)
    ttft = step(P, 0, 1, wb, kb)                   # prefill: compute-bound for long P
    decode = 0.0
    for i in range(1, n):
        t = step(1, P + i - 1, B, wb, kb)
        if alpha:                                  # speculative decoding, formula (83.6)
            tokens = (1 - alpha ** (gamma + 1)) / (1 - alpha)
            t = (gamma * c * t + step(gamma + 1, P + i - 1, B, wb, kb)) / tokens
        decode += t
    return ttft, decode / (n - 1), ttft + decode

for name, kw in [("baseline", {}), ("+ 4x prompt", dict(compress=4)),
                 ("+ INT4", dict(compress=4, wbits=4)), ("+ spec 0.8", dict(compress=4, wbits=4, alpha=0.8))]:
    ttft, tpot, total = request(**kw)
    print(f"{name:12s} TTFT {ttft*1e3:5.1f} ms  TPOT {tpot*1e3:5.2f} ms  total {total:.3f} s")
# baseline     TTFT  69.2 ms  TPOT  4.96 ms  total 1.551 s
# + 4x prompt  TTFT  16.5 ms  TPOT  4.84 ms  total 1.463 s
# + INT4       TTFT  16.5 ms  TPOT  1.24 ms  total 0.388 s
# + spec 0.8   TTFT  16.5 ms  TPOT  0.59 ms  total 0.193 s

How the next 11 weeks follow this map

The rest of this foundations block finishes the tools you need for all three levels: session 84 (next session), two sessions on GPU kernels (session 85 and session 86), and a lab where you measure your own GPU’s roofline. Then the course walks the map from left to right, in 55 sessions: 11 weeks of five sessions each. Every second week (weeks 1, 3, 5, 7, 9 and 11) ends with a lab.

data levelmodel levelsystem level
Figure 4. The roadmap: sessions 88 to 142, five per week, colored by level. Week 1 is the data level (survey Section 4), weeks 2 to 7 the model level (Section 5), weeks 8 to 11 the system level (Sections 6 and 7).

The order is deliberate. The data level comes first because it needs no new machinery: it is prompts, small helper models and retrieval. The model level takes the most time because it contains the most methods and the most math (quantization alone gets two weeks). The system level comes last because its methods (FlashAttention, speculative decoding, PagedAttention, continuous batching) are easiest to understand once you know what the model computes and where the bytes are. The final session of the track, session 141, puts all three levels together for real workloads.

Where this is used

ApplicationMain bottleneckTypical levers (level)
Chat assistant with long answersDecode (TPOT), cost per tokenINT4/INT8 or FP8 weights (model), continuous batching, speculative decoding (system)
RAG over company documentsPrefill and KV memory from long retrieved contextBetter retrieval and prompt compression (data), prefix caching of shared documents, chunked prefill (system)
Agents with tool callsRepeated prefill of a growing history, many sequential callsHistory summaries (data), prefix caching (system), small distilled models for simple steps (model)
On-device assistant (phone, laptop)Memory size and bandwidth4-bit weights, small distilled models (model), llama.cpp-style engines and offloading (system)
Video QA, video captioning and key-moment detectionPrefill: \(32 \times 576 = 18{,}432\) tokens (32 frames of 576); about 42% of a short-answer request in Figure 2Frame sampling and visual-token pruning (data), efficient attention (model), caching the video’s KV once for many questions (system)
Video retrieval and Video-RAGEmbedding many clips, then long prompts with retrieved clipsRetrieve fewer, better clips (data), quantized embedding models (model), large offline batches (system)
Offline batch jobs: synthetic data, RL rollouts, evaluationThroughput, not latencyVery large batches, paged memory (system); quantized weights (model)

Common mistakes

  • Optimizing the wrong phase. Compressing prompts when 95% of the time is decode, or quantizing weights (weight-only) when the problem is TTFT. Measure the prefill share first and use Amdahl’s law (83.7).
  • Calling a model-level change “free”. Quantization, pruning and distillation change \(\theta\). A small average perplexity change can hide large drops on some tasks (long context, math, code, rare languages). Always evaluate on your own task.
  • Assuming system-level changes give bit-identical outputs. They are lossless in exact math, but a new kernel or a different batch size can change the order of additions and flip a near-tie in greedy decoding. Test with a tolerance, not with exact string equality.
  • Multiplying speed-ups from papers. “2× from speculative decoding” and “3× from INT4” do not give 6× if both act on the same term or if the batch size differs from the papers’ setup. Recompute with your own workload.
  • Forgetting memory. A lever that speeds up one request may limit the batch: a draft model and its cache take GPU memory; longer outputs from Skeleton-of-Thought parallel branches use more cache.
  • Treating the map as strict law. KV-cache quantization, Medusa heads or MoE upcycling sit near the borders. Ask what the method changes (input, weights, or engine) and what it costs, not only which box it belongs to.
  • Comparing best-case bounds with measured numbers. The roofline numbers in this session are lower bounds. Real engines reach a fraction of peak bandwidth; compare like with like.

Interview questions

Name the three levels of inference optimization and give two examples of each.basic
Data level (change the input or the output shape, the model stays): prompt pruning such as LLMLingua, RAG, Skeleton-of-Thought. Model level (change the architecture or compress the weights): GQA or MoE, quantization (GPTQ, AWQ), pruning (SparseGPT), distillation. System level (change the engine or the server): FlashAttention, speculative decoding, PagedAttention, continuous batching, offloading.
Which levels are lossless, and why is speculative decoding lossless even though it uses a second model?core
System-level methods are typically lossless because they keep the same weights and input and only change how the computation is done (up to floating-point rounding). Model-level and data-level methods are typically lossy. Speculative decoding uses a draft model only to propose tokens; the big model checks them with a rejection-sampling rule, and when a token is rejected it samples a correction from an adjusted distribution. Leviathan et al. and Chen et al. prove that the resulting tokens have exactly the big model’s distribution. The draft affects only the speed (through the acceptance rate), not the output distribution.
Your RAG service has a high time to first token. Which levers do you try, at which levels?core
TTFT is mostly prefill, which is compute-bound and grows with the prompt length (linear plus a quadratic attention part). Data level: retrieve fewer and better passages, compress the prompt (LLMLingua, summaries). System level: prefix caching for shared system prompts and documents, chunked prefill so the queue does not wait behind long prompts, FlashAttention, more GPUs with tensor parallelism. Model level: FP8 or W8A8 quantization to raise the compute rate (weight-only quantization does not help prefill), or a smaller model. Also check queueing time, which is part of TTFT in a real server.
Why does INT4 weight-only quantization speed up decode but not prefill?core
Weight-only quantization reduces the bytes of the weights, so it shrinks the memory term of the roofline. Decode at small batch is memory-bound (about 1 FLOP per byte), so its time falls almost in proportion: for Llama 3 8B on an H100 the step goes from 4.96 to 1.36 ms in our model (not 4 times, because the cache is still BF16). Prefill is compute-bound, and the weights are dequantized to BF16 before the matrix multiply, so the number of BF16 FLOPs and the time stay the same. To speed up prefill you need to quantize activations too (W8A8, FP8) so the tensor cores run at a higher rate.
A colleague compressed prompts 4× and the end-to-end latency improved only 6%. Explain.core
Amdahl’s law: a lever speeds up only the part of the time it acts on. For a 4,000-token prompt and a 300-token answer on Llama 3 8B, prefill is about 4.5% of the request time and decode 95.5%. Compression makes prefill about 4.2 times faster, so the request gets at most \(1/(0.955 + 0.045/4.2) \approx 1.035\) times faster, plus a small extra gain because each decode step reads a shorter cache; together about 1.06 times. The lever was right for TTFT, but the wrong lever for total latency. For long answers, attack decode.
Map the three root causes of slow inference to the three levels.basic
Model size: model level (quantization, pruning, distillation, MoE) and system level (batching to share weight reads, offloading, parallelism). Quadratic attention and the growing KV cache: data level (shorter prompts), model level (GQA, MLA, sparse or linear attention, SSMs, KV quantization), system level (FlashAttention, PagedAttention, prefix caching). Token-by-token decoding: data level (Skeleton-of-Thought), model level (early exit), system level (speculative decoding, continuous batching).
Is KV-cache quantization a model-level or a system-level method?advanced
The survey places it under model-level quantization (Section 5.2.1). The reason is what it changes: it changes the numbers the attention layers compute with, so the output distribution changes and the method is lossy, like weight quantization. It is implemented in the serving engine, which is why people sometimes call it a serving feature. The useful habit is to ask formula (83.1)’s question, which argument changed, and then plan an evaluation because it is lossy.
How do you evaluate a lossy optimization before you ship it?core
Compare the optimized and the original system on the same inputs. Cheap checks: KL divergence or top-1 agreement of next-token distributions on real prompts, and perplexity on held-out text. Then task checks: the benchmarks and internal test sets that match your product (long context, code, math, multilingual, tool use), and human or LLM-judge comparisons for open-ended answers. Look at the worst slices, not only the average. For data-level methods such as prompt compression, also check faithfulness to the dropped context (did the answer lose a fact?). Measure speed under realistic batch and length distributions at the same time.
When do two optimizations not stack?advanced
When they act on the same term or change each other’s assumptions. Speculative decoding uses idle compute in memory-bound steps; at large batch sizes the step is closer to compute-bound and the gain shrinks. After INT4 quantization of the big model, a BF16 draft model becomes relatively more expensive. Query-dependent prompt compression breaks prefix caching, because requests no longer share an identical prefix. Two memory savings (paged KV and KV quantization) both raise the batch limit, but beyond some batch size the GPU is compute-bound and more batch does not help.
How does the taxonomy of Wan et al. differ from Zhou et al.?advanced
Wan et al. (Efficient Large Language Models: A Survey) use model-centric methods (compression, efficient pre-training, efficient fine-tuning, efficient inference, efficient architecture), data-centric methods (data selection, prompt engineering), and LLM frameworks. It covers the whole life cycle, including training. Zhou et al. focus on inference only and split it into data, model and system levels, with a detailed system level (engine and serving). They agree on the core separation: the data, the model, and the software that runs it.
Design question: make an 8B chat service 5 times cheaper per token without hurting quality much. Walk the map.advanced
First measure: tokens per second per GPU, batch sizes, prompt and answer lengths, TTFT and TPOT. Cost per token is GPU time divided by tokens, so throughput is the target. System level first (lossless): continuous batching and paged KV memory to raise the batch from a few to dozens, which alone can give several times more tokens per second while decode is memory-bound; prefix caching for the system prompt. Model level: FP8 or INT8 weights, then INT4 with AWQ or GPTQ if quality allows, and an 8-bit KV cache to fit more sequences; evaluate each step. Data level: trim system prompts and history, summarize old turns. Check latency targets at the new batch size and the worst-case quality slices, then report the measured gain.

Exercises

  1. Place each method on the map (level and root cause): GPTQ, PagedAttention, LLMLingua, Medusa, FlashAttention, Skeleton-of-Thought, Mamba, continuous batching, Minitron-style pruning plus distillation.
  2. In the RAG request of the worked example (Llama 3 8B, H100, \(P = 4000\), \(n = 300\)), switch to INT8 weights. Use Figure 2 or the Python model to find TPOT and the total time. Why is the speed-up below 2 times?
  3. Decode is 95.5% of the baseline RAG request. Some lever makes decode exactly 2 times faster and does nothing else. What is the speed-up of the whole request?
  4. Speculative decoding with \(\gamma = 4\), \(\alpha = 0.6\) and a draft that costs 15% of a big step: compute the expected tokens per round and the speed-up at batch 1.
  5. A video QA request has 18,432 prompt tokens and \(n = 100\). Use Figure 2 to find the prefill share and the speed-up from 4× prompt compression alone. Compare with the RAG request and explain.
  6. How many RAG requests (4,300 tokens of cache each, BF16 cache) fit next to the BF16 weights of Llama 3 8B in 80 GB, ignoring activations? How many with an 8-bit cache? Which levels did you use?
Answerscheck
1. GPTQ: model level, model size. PagedAttention: system level, attention/KV memory (larger batches). LLMLingua: data level, attention (shorter prompts). Medusa: system level in the survey (speculative decoding with trained heads), decoding. FlashAttention: system level, attention (memory traffic). Skeleton-of-Thought: data level (output organization), decoding. Mamba: model level (transformer alternative), attention. Continuous batching: system level, decoding and model size (shares weight reads). Pruning plus distillation: model level, model size. 2. TPOT 2.56 ms and total 0.834 s (from 4.96 ms and 1.55 s). Weights shrink 2 times, but the 0.54 GB of cache per step stays the same, and the 69.2 ms prefill does not change. 3. \(S = 1/(0.045 + 0.955/2) \approx 1.91\). 4. \((1 - 0.6^5)/0.4 = 2.31\) tokens per round; cost per round \(1 + 4 \times 0.15 = 1.6\) big steps; speed-up \(2.31 / 1.6 \approx 1.44\). 5. TTFT about 389 ms of 0.936 s, so about 42%. 4× compression gives about 1.63 times (0.573 s), much more than the 1.06 times for RAG, because prefill is a much larger share. 6. Each request needs \(4300 \times 131{,}072 \approx 0.564\) GB. \((80 - 16.06)/0.564 \approx 113\) requests; with an 8-bit cache about 226. The KV bits are a model-level change (lossy, needs evaluation); fitting more requests into one batch is the system level.

Key takeaways

  • Inference is \(\hat{y} = \mathcal{E}(\theta, x)\) (83.1). The data level changes \(x\) (or the output structure), the model level changes \(\theta\), the system level changes \(\mathcal{E}\).
  • System-level methods are typically lossless (\(\Delta \approx 0\) in (83.2)) and need no training; model-level methods are typically lossy and need training or calibration; data-level methods keep the model but change what it reads.
  • Every symbol of the cost model (83.3)–(83.5) belongs to one level: \(P, n\) to data; \(N, b_w, s_{\text{kv}}\) to model; \(B, \pi, \beta\) and steps per token to system.
  • A lever speeds up only its part of the time (Amdahl, 83.7). On a long-answer RAG request, 4× prompt compression gives 1.06×, INT4 weights about 3.3×, and stacking one lever per level gives about 8× in the roofline model.
  • Speculative decoding is lossless and gives \((1 - \alpha^{\gamma+1})/(1 - \alpha)\) tokens per round (83.6); its gain depends on the acceptance rate and the draft cost.
  • The next 55 sessions follow the map: week 1 data level, weeks 2 to 7 model level, weeks 8 to 11 system level.

Key formulas

\[ T_{\text{req}} \approx \text{TTFT} + (n-1)\,\text{TPOT}, \quad \text{TPOT} \approx \frac{b_w N + B T s_{\text{kv}}}{\beta} \;\; (\text{small } B) \]
\[ \Delta = \E_x\, \KL\big(p_\theta(\cdot \mid x) \,\|\, p_{\theta'}(\cdot \mid x')\big), \quad S = \frac{1}{(1-p) + p/s}, \quad \E[\text{tokens}] = \frac{1-\alpha^{\gamma+1}}{1-\alpha} \]

References for this session

  1. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  2. Wan et al. (2023). Efficient Large Language Models: A Survey
  3. Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
  4. Wan et al. (2023). Efficient Large Language Models: A Survey
  5. Miao et al. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  6. Yuan et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights
  7. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures
  8. Llama Team, Meta (2024). The Llama 3 Herd of Models
  9. Leviathan et al. (2022). Fast Inference from Transformers via Speculative Decoding
  10. Chen et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling
  11. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  12. Jiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  13. Ning et al. (2023). Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation
  14. Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
  15. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  16. Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
  17. Frantar & Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
  18. Hinton et al. (2015). Distilling the Knowledge in a Neural Network
  19. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention
  20. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models
  21. Sheng et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  22. Gu & Dao (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  23. Liu et al. (2023). Improved Baselines with Visual Instruction Tuning