Four Macs Run Muse Spark at 36.7 tok/s, and a Fixed 14 ms per Token Sets the Ceiling

Meta released Muse Spark Open this week, a model with 816 billion parameters, and Apple published a report on running it across several Mac Studios.¹² The headline is that four Macs generate over 36 tokens per second. The same report says something more useful in passing: generation speed “is memory-bandwidth bound.”¹

That sentence is a testable claim. If decoding is limited by how fast weights can be read from memory, then speed should follow from two numbers, the bytes read per token and the bandwidth, and the cluster results should be predictable from them. I checked the rule on my own hardware, then took Apple’s tables and asked how close each run gets to the ceiling. The single-machine runs are close to it. The multi-machine runs are not, and the shortfall has a very regular shape: a fixed cost of roughly 9 to 16 milliseconds per token that adding Macs does not shrink.

The rule in plain terms

A language model produces text one token (a word or word fragment) at a time. Each new token, called a decode step, needs the model’s weights to flow through the processor, and those weights live in memory. A chip can only pull data from memory at some rate, its memory bandwidth. If a model needs 54 GB of weights per token and the chip delivers 1,200 GB/s, one token takes at least 45 ms, so the speed cannot exceed about 22 tokens per second. That ceiling is the whole rule.

Two things shrink the bytes per token. A mixture of experts (MoE) model has many specialist sub-networks and uses only a few per token, so Muse Spark reads about 42 billion of its 816 billion parameters each step. Quantization stores each weight in fewer bits, which Apple did here with a 4-bit format called NVFP4.¹

When a model is too big for one machine, there are two ways to split it. Pipeline parallelism puts the first half of the layers on one Mac and the second half on another, so they take turns. Tensor parallelism splits every layer across the machines, so all of them read memory at once, but they must exchange results many times per token.¹

Does the rule hold on my machine?

My test machine has a Core Ultra 7 258V with 8 LPDDR5X-8533 memory chips, each 16 bits wide, which gives a theoretical peak of 136.5 GB/s.³ I wrote a small C benchmark that does what a decode step does: one pass over a 2 GiB buffer of weights, multiplying each by an input vector.⁴ A pass over the whole buffer plays the role of one token for a model that is exactly that big. I ran it with the weights stored as 32-bit floats, 16-bit bfloat16, 8-bit integers and 4-bit integers.

The best sustained read bandwidth I could get was 94.0 GB/s with six threads, which is 69% of the spec sheet. The three final runs were noisier, with peaks of 67.8, 90.5 and 81.6 GB/s, so treat any single number here as plus or minus 15%.

Eight times fewer bits per weight bought between 6.8 and 8.6 times more weights per second across the three runs. The rule holds here: on a big enough buffer, time per token is bytes divided by bandwidth, with a few percent lost to unpacking the smaller formats.

Apple’s single machines sit close to the ceiling

Take the weights of each model, multiply by its decode speed, and you get the bandwidth the run actually used. I used 54 GB for Qwen 3.8 (27 billion parameters at 2 bytes) and 128 GB for Mistral Medium 3.5 (128 billion at 1 byte), and compared against the chips’ stated bandwidth of 1.2 TB/s for the M5 Ultra and 819 GB/s for the M3 Ultra.¹

A single Mac running a dense model uses 77 to 89% of what the chip claims to deliver. That is a higher fraction than the 69% my own machine reaches against its spec, though the two platforms are not comparable, and it fits the report’s line about bandwidth. The bars on the right are the surprise. With the same weights split across four machines, the fraction drops to 57 to 62%. Cluster bandwidth is not being used as well as chip bandwidth.

A two-number model that predicts the unseen point

Say each token costs a part that shrinks with the number of machines N, the memory reads, plus a part that does not. Then the time per token is t = A/N + B. The B term covers whatever repeats on every machine regardless of N: launching GPU work, sampling the next token, and the many small exchanges between machines for every layer.

flowchart LR
  A["Each machine reads<br/>1/N of the weights<br/>time A/N"] --> B["All machines exchange<br/>partial results<br/>every layer"]
  B -->|"next layer"| A
  B --> C["Sample token<br/>launch overhead"]
  C -->|"next token"| A

I fitted A and B using only the two-machine and four-machine tensor runs, then used the model to predict the one-machine speed, which was not in the fit. For Muse Spark there is no one-machine run, so the comparison is against the two-machine pipeline run, which Apple says is “very similar” to a single machine in decode speed.¹

The predictions land within 5% of the measurement in all six cases. Two numbers that I fitted from two points correctly reproduce a third point I held out, so the structure is real even if the explanation of what B contains is not proven. The fixed part is 9 to 16 ms per token, and it grows with model size and context.

The last column is the interesting one. With A shrinking toward zero as machines are added, speed approaches 1/B, around 70 tokens per second for Muse Spark whatever the cluster size. The fit implies Muse Spark would reach about 48 tokens per second on 8 machines and 57 on 16. Those two points are extrapolations, not measurements, and a Thunderbolt mesh of that many Macs is not something Apple demonstrates.

Is the MoE reading pattern the problem?

A reasonable suspicion is that Muse Spark, which reads a different handful of experts for every token, wastes bandwidth on scattered reads. I tested that on my machine by reading eight randomly chosen experts of a given size per token against reading the same number of bytes in one contiguous block.

Scattering made no consistent difference at any expert size from 64 KB to 16 MB: the ratio of scattered to contiguous time stayed between 0.92 and 1.12, within the run-to-run noise. What did matter was how much data each step moved. A step that reads 0.5 MB took about 18 µs, which is only 30 GB/s, while a step that reads 128 MB ran at 86 GB/s. Bandwidth alone would put the small step near 6 µs. Small steps leave bandwidth unused because fixed per-step costs dominate, not because of scattering.

That is the same shape as the Apple result, at a much smaller scale. I do not know Muse Spark’s expert sizes, because the model’s config.json is behind Hugging Face’s access gate, so I cannot say which row of my table it resembles.

What “4-bit” weighs

The NVFP4 checkpoint is not 4 bits per parameter. From the dtype counts in its Hugging Face metadata, it stores 773 billion 4-bit weights, 48.3 billion 8-bit scale factors (exactly one per 16 weights, which makes the 4-bit part 4.5 bits per weight), and 48.8 billion parameters left in 16-bit BF16.⁵ The file totals 533 GB for 822 billion parameters in the base model, which is 5.19 bits per parameter, and 18% of the bytes are the BF16 parts. The conversion settings list the attention layers and the projections around each expert block among the modules that were not converted.⁵

That matters for the ceiling. Reading 42 billion parameters at a pure 4 bits is 21 GB. If Muse Spark’s memory term A of 51 ms ran at the 82 to 89% efficiency that the dense models show, it would be moving 50 to 55 GB per token. Either a large share of the active weights are in BF16, or the Muse kernels run at only 34 to 44% of the chip’s bandwidth. I cannot tell which from public information.

Where this can mislead

• B is a bucket. It collects synchronization, kernel launches, sampling and any unparallelized layers. The fit says the bucket exists and is stable, not which of those costs fills it.
• Two parameters, two points. The held-out check is encouraging but small. Six cases within 5% is not a proof that the model is right at 8 or 16 machines.
• Weights-only bytes. My efficiencies ignore the KV cache, the memory that remembers the context. That is part of why the 65,536-token numbers look worse.
• Stated bandwidth, not measured. I used Apple’s 1.2 TB/s and 819 GB/s as given. Nobody here measured them on those machines.
• My machine is not theirs. A hybrid CPU in a WSL2 virtual machine is noisy. I report best-of runs for peaks and ranges for the rest.
• I did not run Muse Spark or touch any Apple hardware. Everything about the clusters comes from Apple’s published tables.

Takeaways

• Decode speed on one Mac is about bandwidth divided by weight bytes, and Apple’s dense runs reach 77 to 89% of the chip’s stated number.
• Adding machines shrinks the memory term but not a fixed 9 to 16 ms per token, so each doubling helps less: 1.45 to 1.7 times from two to four machines.
• For Muse Spark the fitted ceiling is near 70 tokens per second regardless of cluster size.
• “4-bit” checkpoints average over 5 bits per parameter, and the attention and expert projections stay in 16-bit, so bytes per token run higher than the headline suggests.
• Before buying a second Mac for speed, ask what B is for your setup, not what the bandwidth is.

References