Machine-generated. The original Metal kernels were written by GLM-5.3-Flash, a local open-weights model served on this machine. The follow-up optimisation, and this post, were written by an AI assistant running inside the pi coding-agent harness under human direction. No human wrote the prose; the measurements are real.

Some AI models are enormous — hundreds of billions of parameters, far too big for one computer. Hy4-preview, from Tencent, is one of these: 770 billion parameters in total. The trick that makes it usable at all is that it only calls on a small slice of itself for each word it writes — eight “experts” out of a huge library, rather than the whole staff. That’s what Mixture of Experts means, and it’s why a machine with 512 GB of memory could ever hope to hold the thing.

It almost did.

The model that refused to start

Someone in the community did everyone a favour and converted Hy4-preview into the format that the popular open-source program llama.cpp reads, then published the 229 GB of weights. My Mac — an M3 Ultra with 512 GB of unified memory — is one of the few machines in the world with room for it. On paper, this was mine.

On paper.

The model uses a clever compression trick: the expert weights aren’t small numbers, they’re just −1, 0 and +1, which squeezes them to about 1.3 bits each. That’s what lets a 770B model fit into 229 GB. The conversion had full support for NVIDIA GPUs and for the CPU. What it didn’t have was support for Apple’s GPU. And without that, the model refused to start. Not “ran slowly”. Would not start — on the only Mac with enough memory for it.

The missing pieces are called kernels. A kernel is a tiny, specialised program that tells the GPU how to do one specific piece of heavy maths. A format like this needs roughly a dozen of them, and none existed for Apple. That’s the whole story of the wall.

The part where a small model wrote the code

I didn’t write those kernels. A much smaller model did — running on the very machine the big one was meant for, at about a dozen words a second, on my own hardware. No third party involved. It read the existing code, wrote about 160 lines of GPU shading language plus the plumbing, built it, and iterated until the big model ran.

Two things from that session stuck with me.

The first is a bug we almost shipped. The GPU likes to read memory in chunks of four bytes, but this format packs 256 weights into exactly 42 bytes. Forty-two doesn’t divide by four, so every other block of data sat at a misaligned address, and the same instruction quietly read garbage on half of them. We only caught it because we refused to trust anything we couldn’t check: encode a matrix, run it on the GPU, compare against a careful reference. Fast and wrong is worse than slow and wrong, because fast and wrong travels much further before anyone looks.

The second is a lesson in humility. There was a clever way to decode those weights with pure arithmetic, skipping the lookup table entirely. Smaller, smarter, obviously better. It measured slower. The boring version won by a wide margin. A benchmark is allowed to overrule a beautiful idea — that was the part I didn’t expect.

Then we went back for more

The first pass got the model running at roughly 9–10 words per second — about 2× faster than the first working version. That’s the number people actually feel.

This week we went back to the code to see if we could squeeze out more. The honest answer: the path that runs on every single word is already close to the hardware’s ceiling. We found a few extra percent, but the real headroom there is small — the per-word speed is mostly set by other parts of the model that were already well-optimised. The one genuinely big opportunity left is the path that reads a long prompt up front, which is still running below where it should be. We mapped exactly what it needs and left it as the clear next step, rather than pretend we’d beaten a wall we’d measured.

Why this is worth writing down

Writing a GPU kernel for a format that had never run on this family of GPUs used to mean you were the kind of person who’d spent years inside graphics programming. What it actually took this time: being willing to measure things, refusing to believe a result you couldn’t verify, and knowing that data is allowed to win over cleverness. All the syntax, all the conventions, all the kernel signatures — retrieved and written by a model running on the hardware it was writing for.

Two local models were in that house that day. One wrote the code that let the other one run. Neither phoned anyone.

That’s not a demo. It’s a Tuesday. But it’s a Tuesday I wouldn’t have predicted five years ago, and I’d like to remember what it looked like from inside.

Credits

  • Tencent — Hy4-preview, Apache 2.0, including the ternary quantisation method.
  • AngelSlim — the GGUF conversion, the weights, and the llama.cpp patches. Without their work there is nothing to port.
  • The llama.cpp / ggml project — the codebase these kernels went into.
  • GLM-5.3-Flash (open weights, served locally) — wrote the original Metal kernels.
  • Me — one prompt, the measurements, and the stubbornness about not shipping unvalidated.

Kernels, benchmark harness and correctness test: github.com/JordiPosthumus/hy4-preview-metal.