DeepSeek V4.1-Flash.
Every layer, connected.

The first generation step needs decoder local caches. Replaying the final 128 prompt tokens bounds their reconstruction cost; it does not mean the decoder is never evaluated during prefill.

Each generated token then traverses both encoder and decoder. The “8B / 16B active” distinction describes the optimized phase-dependent path, rather than a fixed per-token active count under every execution strategy.

Proxy: baseline = 40N; CED = 20N + 20min(N,128). This counts block–token evaluations, not FLOPs or latency. It excludes projection cost, cache-hit behavior, vision, and kernel differences.

Encoder replay after a cache miss

If global prefix KV exists but encoder SWA state is missing, replay the last 128 cached prefix tokens along with the uncached suffix. Replayed positions regenerate local SWA only, without recomputing or overwriting their global KV. New suffix positions generate both kinds of state.

Short-lived versus persistent state

The deployment keeps encoder SWA in a short-TTL distributed memory pool, while global KV stays in the long-lived persistent cache. Decoder SWA is reconstructed at prefill and used for decoding. Removing SWA from persistent storage plus shrinking global KV explains the reported ~8× persistent-cache reduction under the stated workload.

The query and output paths

Query and output projections are low-rank/grouped. The reference applies inverse RoPE to the output’s rotary coordinates before the grouped projection. Each head also has a learned attention sink.

The global and index paths

Indexer scores combine rectified query–key products across 32 weighted heads. Its selection is causal. Main attention uses the selected global records together with the current layer’s local KV.

B = batch size; T = processed token positions. For cache accounting, a KV entry is one shared latent, not separately stored per-head K and V.

L20 selects from the whole visible context and constructs the shared candidate pool.

Visualization scale: 128 toy positions, blocks of 4, 8 candidate blocks, and Top-8. Synthetic deterministic scores illustrate the mechanism; they are not the model’s activations. Actual budgets are shown alongside.

Each block’s score is the maximum score among its positions. Selecting the best blocks creates a pool larger than the final attention set. Later indexers share the pool and keys, but use their own queries and can select different records.

The hierarchy is training-aware and was introduced during post-training. It bounds later indexers’ search work for a fixed pool size. The first indexer still scans the full visible range. §2.3.2 · pp. 11–12

Here l indexes attention/FFN sublayers in the mixing equation, not necessarily one complete two-sublayer Transformer block. X is 4 × 5,120 per token.

Why one index shift matters

Input mixing uses coefficients already produced by the previous sublayer. It no longer waits for the current coefficient predictor’s reduction over the hidden dimension, allowing residual update, input mixing, and coefficient prediction to be fused.

The deployment kernel Mega-mHC also incorporates input pre-normalization and FP8 conversion. The report gives activation traffic of (2n + 2)d for Single-Pass versus (4n + 4)d in the original implementation: at n = 4, that is 10d versus 20d values moved.

This is a reduction in the described residual-mixing activation traffic, not a 2× speedup for the entire model. Training retained a multi-kernel implementation. The released configuration uses 20 Sinkhorn-Knopp iterations for the manifold constraint.

§2.4.1 · pp. 12–13 Block.hc_mixes / Block.forward

All 384 boxes are shown. These are synthetic logits and biases. Switching modality holds logits fixed to isolate the effect of correction bias; real image and text hidden states also differ. Expert numbers are not human-assigned specialisms.

Selection and weighting are separate.

Selected routing weights sum to 1.5.

Correction bias changes which experts are chosen; the original unbiased scores determine their combination weights. The training procedure updates image and text biases separately to balance loads.

Each expert is a 5,120 → 2,304 → 5,120 SwiGLU FFN. The gate/up projections are distinct: the gate is clamped above at 10, and the up branch is clamped to [−10, 10], before SiLU(gate) × up. Gate / Expert / MoE

The phrases represent conceptual token windows, not actual tokenizer output. Real hashes use compressed token IDs, distinct prime-sized tables, and checkpoint-specific hash parameters.

What gets looked up—and what does not

Each module uses orders {2, 3, 4}. Each order has eight 256-dimensional hash heads, totaling 2,048 dimensions per order. Per-head tables have approximately 16M entries, with distinct prime sizes. The released packed tables have 384,006,168 and 384,016,682 rows of width 256.

Retrieved features are projected to a key for each residual stream and a shared value. A context-aware sigmoid gate controls how much of that value is added. The reference uses a signed square-root transform of the normalized query–key score before the sigmoid.

Image positions do not receive Engram contributions and do not participate in the n-gram history. The released tokenizer-compressed vocabulary size is 99,092. Both lookup tables and key/value projections use FP8 in the report.

Engram layout and hashing Engram.forward

Draft generation is semi-autoregressive: the heavy forward pass covers all five positions, while Markov-conditioned token sampling proceeds through them. Main-model verification remains responsible for deciding which draft tokens can be accepted.

The checkpoint stores these blocks under mtp.*, but DSpark is not the jointly pre-trained MTP recipe used in V3. §2.4.3 · pp. 13–14 DSparkBlock / forward_spec

Synthetic conditional acceptance probabilities: [0.94, 0.88, 0.81, 0.73, 0.61]. Bars show prefix survival: P(all first j accepted) = ∏ᵢ≤ⱼ pᵢ. Expected accepted prefix length through L is the sum of these survival probabilities.

The slider is an explanatory control, not the real scheduling policy. The production scheduler also uses profiled engine throughput and current system load to choose verification lengths; confidence alone does not determine the optimum. Bonus/correction tokens are excluded from this toy expectation.