Mamba & State Space Models, Playable

EP 01

Pack the suitcase — a memory that never grows

Here's a state space model reading a sentence, one word at a time. Its entire memory is the 8 slots below — that's it, forever, whether the text is 10 words or 10 million. Each incoming word updates the slots (a weighted blend of "keep what's there" and "write something new"), then vanishes. Step through and watch old contents fade as new ones move in. Nothing is ever re-read.

✓ good case: the sentence's gist survives in the slots — recent + salient content readable at the end✗ bad case: ask about word #3 after 40 more words — its trace has decayed below the noise floor; the suitcase kept the trip, not the packing list

—

The slot dynamics are a real linear recurrence h ← A·h + B·x running live (A decays, B writes), with the decay rate shown per slot — a faithful miniature of the S4/Mamba state update, minus the dimensions.

Recurrence is back. RNNs always worked this way — SSMs are their rehabilitated descendants, with the math fixed so training parallelizes and the memory doesn't explode or vanish uncontrollably. The suitcase is the whole bet: constant memory, constant per-token cost, at any context length.

EP 02

Selectivity — the doorman that made Mamba work

Early SSMs treated every word identically — same decay, same write strength. Mamba's breakthrough: the update amounts depend on the input. Important word? Open the gates, write hard, hold long. Filler word? Barely touch the state. Watch the same sentence pass through both: the fixed SSM lets "the" and "very" overwrite the crucial name; selective Mamba slams the gate on filler and the name survives 40 words of noise.

✗ bad case: fixed gates — "Eleanor" gets diluted by 30 filler words; recall of the name fails✓ good case: selective gates — filler bounces off, "Eleanor" still crisp in the state at the end

—

Both runs use the same recurrence; only the gate policy differs (fixed constants vs input-dependent). The "name signal strength" trace is measured from the actual state vector each step.

Selectivity is attention's job done at the door instead of in the library. A transformer keeps everything and searches later; Mamba filters at intake and never searches. That's the deep trade of the whole architecture — and the reason the paper's title word is "selective."

EP 03

The memory bill — constant vs a cache that eats the GPU

Now the economics. A transformer generating with a long context must hold KV entries for every past token — memory grows linearly, and per-token compute grows with it. Mamba holds the suitcase — a flat line. Drag the context length from a tweet to a codebase and watch the two curves and the serving arithmetic (how many concurrent users fit on one 80GB GPU) diverge.

✓ good case: 1M-token context — Mamba's state: same 20 MB as always; hundreds of concurrent streams per GPU✗ bad case: transformer at 1M tokens — ~100+ GB of KV cache per stream; one user doesn't even fit on the card

context length 8k —

KV math: 2 × layers(60) × kv-heads(8) × head-dim(128) × 2 bytes per token ≈ 0.25 MB/token (7B-class, GQA). Mamba state: layers × d_state × d_model × 2 bytes, fixed. Real numbers vary by model; the shapes don't.

This flat line is why SSMs refuse to die. Every time context lengths jump another 10×, the KV bill jumps with it and the suitcase looks better — long-audio, DNA, logs, always-on agents. Serving cost, not benchmark glory, is the beachhead.

EP 04

The recall test — what a suitcase cannot do

The honest weakness, measurable in one game: needle-in-a-haystack. Hide a random passcode early in a long text, ask for it verbatim at the end. The transformer flips back to the exact page — perfect recall at any distance (it kept everything). The suitcase must have decided, hundreds of words ago, that a random string was worth keeping — with no idea it would be asked. Run the test at growing distances and watch the two curves separate.

✓ good case: transformer — 99% verbatim recall at every distance; the library never forgets✗ bad case: SSM at 8k-word distance — recall collapses toward chance; the passcode was unpacked to make room somewhere around word 2,000

—

Simulation encodes the structural fact (finite state must lossily compress arbitrary content); the curve shape matches published associative-recall benchmarks where attention dominates pure SSMs.

Verbatim recall is attention's home turf, and no suitcase fully closes that gap. Copying, exact quoting, long-range lookups — tasks where the answer is "go re-read it" — punish any architecture that threw the original away. Know this failure mode and the hybrid in EP 05 explains itself.

EP 05

The hybrid — a few librarians among the travelers

The 2024-26 answer isn't either/or. Production models (Jamba, Zamba, Nemotron-H and friends) stack mostly Mamba layers with a few attention layers sprinkled in — travelers for speed, a couple of librarians for exact recall. Design your own stack below: choose the attention ratio, then read all three meters — recall, memory bill, throughput. Find your own sweet spot; the published ones cluster around 1:7.

✗ bad case ×2: 0% attention — cheap but flunks recall · 100% attention — you rebuilt the expensive transformer✓ good case: ~12% attention layers — recall within 3 points of full attention at ~1/5th the memory bill

attention layers 0% —

Meters computed from the component models: recall follows a saturating curve in attention ratio (a few librarians catch most lookups); memory and throughput are weighted blends of the two layer types' costs at 256k context.

Architecture wars end in ratios, not winners. The suitcase carries the bulk, the librarians handle the lookups, and the interesting engineering question became "what's the cheapest mix that passes the recall bar for THIS workload." When you see a 2026 model card bragging about tokens/sec at 1M context, check the layer list — there's almost certainly a suitcase inside.