The roofline tells the story
There is a standard analytical tool in computer architecture called the roofline model. It plots achievable performance against arithmetic intensity: the ratio of floating-point operations to bytes of memory transferred. Every computation falls somewhere on this plot. If the computation is compute-bound, it sits near the peak FLOPS ceiling. If it is memory-bandwidth-bound, it sits below that ceiling, constrained by how fast you can move data in and out of memory.
Transformer inference at token generation time is memory-bandwidth-bound for virtually every model size larger than a few billion parameters when you are using today's GPU hardware. The arithmetic intensity of a matrix-vector multiply during autoregressive generation is approximately equal to the batch size. At batch size 1 (single-request inference), each weight parameter is loaded from memory to perform roughly 1 multiply-accumulate operation. At batch size 64, you get 64 MACs per weight byte loaded, which is much more favorable.
The implication: if you are running a 70-billion-parameter model and serving one request at a time, you are spending most of your GPU's time and power shuffling weights from HBM to compute units, not doing actual math. Even at batch sizes of 8 to 16, many deployments remain bandwidth-bound. The GPU has the compute capacity to run much faster, but the memory bus cannot keep up with feeding it weight matrices.
What the memory bandwidth numbers actually look like
An H100 SXM5 has 3.35 TB/s of HBM3 bandwidth. A 70-billion-parameter model in BF16 requires loading roughly 140 GB of weights per forward pass. At 3.35 TB/s, that takes about 42 milliseconds just for the weight loads, ignoring everything else. The GPU's 989 TFLOPS of BF16 peak compute could theoretically run that forward pass in roughly 0.14 milliseconds if weights were instantly available. You are paying 300x the compute-optimal time because of memory bandwidth.
More recent HBM generations and higher-bandwidth interconnects (NVLink, multi-GPU tensor parallelism) help, but they push the cost up substantially. Multi-GPU deployments for a single model increase capital cost and power consumption by the same factor as the number of GPUs. The memory wall does not disappear; it gets more expensive to move.
This is not a criticism of GPU vendors who have done extraordinary engineering work on HBM bandwidth. It is a structural architectural consequence of a von Neumann model where weights live in memory separate from compute. No amount of HBM3e or HBM4 fixes the fundamental topology: weights must travel from storage to compute on every use.
The photonic weight bank bypasses this topology
A photonic MZI mesh encodes the weight matrix as phase values set into optical phase shifters. Those phase values do not live in DRAM. They live in the physical state of the optical circuit. When an activation vector arrives at the chip's optical input, the computation happens as the light propagates through the mesh. The weights do not move during inference; the data (activations) moves through the weights.
This is compute-in-place at the point of weight storage. The von Neumann bottleneck of "fetch weight, multiply, discard" does not apply. The weight was never in DRAM in the first place during inference. It is stored as a physical configuration of the optical hardware.
The analogy from RF engineering is an analog filter. A passive LC filter does not fetch its frequency-response coefficients from memory before applying them to an input signal. The coefficients are the physical circuit. The processing happens as the signal passes through the physical structure. Photonic weight-stationary computation is the equivalent for matrix-vector products: the filter coefficients are the waveguide phase configuration, and the signal (activation vector) propagates through them.
Where the memory bandwidth savings are realized
For a large transformer model running on a GPU-plus-photonic-coprocessor setup, the projection layers (Q, K, V, output projections in attention, plus the two linear layers per feedforward block) account for the majority of parameter count and MAC operations. These are the layers we hold in optical weight banks. The GPU handles the KV cache, which still lives in HBM and still requires bandwidth, and all the non-linear operations, which are not weight-bandwidth-dominated.
The KV cache grows with context length and batch size. For very long context lengths (100K+ tokens), the KV cache bandwidth can itself become a significant bottleneck. The photonic co-processor does not help with KV cache bandwidth; that remains a GPU and HBM problem. We are not saying photonics eliminates the memory wall entirely. We are saying it eliminates the weight-matrix component of the memory wall for the layers we handle, which in a large autoregressive decoder running standard attention is a substantial fraction of the total memory bandwidth used per token.
The arithmetic intensity of photonic inference
What is the effective arithmetic intensity of a photonic matrix-vector multiply? In some sense, it is infinite for the weight-bandwidth dimension: zero bytes of weight data move per MAC operation, because the weights are stored in place. The bandwidth cost is in programming the weights initially (once per model load) and reading out the results from the photodetectors.
The activation data still moves: input activations travel over PCIe to the co-processor, and output activations travel back. That PCIe bandwidth is real and is the binding bandwidth constraint for the photonic path. But PCIe bandwidth is cheap relative to HBM bandwidth in terms of both dollars and watts. PCIe Gen 4 x16 at 32 GB/s consumes a fraction of the power of HBM3 at 3.35 TB/s, and the activation vectors are orders of magnitude smaller in data volume than the weight matrices they interact with.
The counterpoint: programming cost and thermal drift
Honest accounting requires acknowledging the cost side. Programming the weight banks at model load time consumes energy and time. For a model loaded once and then serving thousands of tokens, that programming cost amortizes well. For scenarios where you are switching between many models rapidly or running many tiny batches, it does not, and the memory wall comparison becomes less favorable for the photonic approach.
Thermal drift in silicon photonics also means that phase values set at programming time drift over tens of seconds to minutes as chip temperature changes. On-chip thermal sensors and feedback control loops can correct for this, but they add circuit complexity and consume chip real estate. The phase correction overhead shows up in the total system power budget and should not be excluded from memory-wall comparisons.
The memory wall argument for photonic inference is strongest in exactly the deployment profile that describes high-throughput data-center inference: a fixed model, high batch sizes, sustained throughput over long periods, and a buyer who pays the power bill. In that profile, the weight-stationary photonic architecture eliminates the most expensive memory bandwidth in the system. That is the claim we are making, and it is bounded to that specific operating regime.