Machine-generated. This is the AI news section of Jordi’s AI’s Blog. Stories were surfaced by a sourcing pipeline (HuggingFace trending + newest, drilled down to the authoritative model cards, plus curated X signals from the local X cache) and written by an AI assistant under human curation. Speeds are measured where a benchmark is cited, and flagged as estimates where they aren’t.

The lead: decoding ahead of the model

A cheerful small robot draft model sprinting ahead of a big serious robot, laying down a trail of glowing stepping-stone tokens into the future; the big robot hops along quickly using the stones, while a tiny skeptical critic robot stamps each stone with a green checkmark before it can be used; a small sign reads SPECULATIVE DECODING.
One small model, many tokens per step — the speculative-decoding pitch.

What it is. Today is a thin day for headline models — nothing new beats what you already run — but the sharpest thread is speculative decoding, and it’s been quietly getting better for local AI all week. The idea: a small draft model predicts several tokens ahead of the big one; the big model then verifies all of them in one parallel pass, keeping only the ones that were right. You get multiple tokens per forward pass instead of one, so decode feels faster without changing the quality of the main model. The freshest example is DwarfStar’s DSpark — its speculative-decoding path — which its maintainer notes “now works much better overall” with DeepSeek-V4-Flash after a couple of weeks of wide improvements. This is the local-stack angle: it’s not a new model, it’s a free-ish speedup on hardware you already own.

Models

MiniCPM5-2B — from OpenBMB, Apache-2.0, a 2B text-generation model (LlamaForCausalLM, safetensors). Brand-new edge/on-device release, first coverage here. Evidence: 2,879 downloads / 601 likes since Sep 6 — strong engagement for a 2B, and an official GGUF build at 7,760 downloads / 107 likes. The MiniCPM line is the edge family (compact, Apache-2.0, on-device), so this is a small, permissive, runnable-everywhere release. Verdict: Real deployment option — compact 2B, Apache-2.0, tiny footprint for edge/on-device work; the likes-to-downloads ratio says real interest, not a press release.

Quantizations

No brand-new quant build cleared the 200-download gate today — the freshest Sep 8 repos are all at 0 downloads. Two developments worth a line: WarmBloodAban/Minimax-h3_Singularity (58,060 downloads / 167 likes, a Singularity-container build of the covered MiniMax H3 — a derivative of an already-covered model, not a new one) and an in-progress DwarfStar experiment: compressing MoE experts with Q2_K instead of mxfp4 to free roughly 5 GB and allow larger contexts on 64 GB machines. No numbers yet, but it’s the direction to watch for tight-memory local boxes.

Fine-tunes

No new fine-tune cleared the 200-download gate today. The interesting item is empirical, not download-gated: a measured test of Qwopus3.8-27B-Flash (a Qwen3.8-27B fine-tune trained to reason more efficiently) on an RTX 5090. The result is roughly 2× efficiency: GPQA-diamond 70.7% at 4,229 tokens per correct answer (greedy, 198 questions, ~2.7 h) against its base. That’s a measured delta on the incumbent — real signal, not a vendor claim. I’d slot it as inference/context rather than a headline: it’s a follow-up on the covered Qwen3.8-27B base, and the efficiency win is the story.

Technical term of the day: speculative decoding (Draft / DSpec / MTP)

What it is. Normal decoding is strictly sequential: one token per forward pass of the big model, each waiting on the last. Speculative decoding breaks that. A small draft model (or a single-token head, or a multi-token-prediction / MTP head) generates a short sequence of candidate tokens in one cheap pass; the main model then scores that whole candidate sequence in one parallel pass, and you keep the longest prefix that matches its own predictions. Correct tokens are kept, wrong ones discarded, and the next draft restarts from the last correct token. The result: many tokens per step instead of one, often 2–4× decode throughput for the same model.

Why it matters for local AI. For anyone running a big model on fixed hardware, decode speed is usually the bottleneck, and it’s memory-bandwidth-bound: each token is a full pass over the weights. Speculative decoding converts part of that into batched parallelism — one pass verifies many candidates — which is exactly what a GPU/Apple-Silicon batch does well. It’s a quality-preserving speedup: the main model still decides, so you don’t trade accuracy for speed. That’s why it keeps showing up in local engines (DwarfStar’s DSpark, MTP builds in llama.cpp/Ollama, GGUF quant cards) — it’s the closest thing to free performance.

The trade-offs. It’s not free. You need a draft model (extra memory and an extra load path), and the speedup depends on how often the draft guesses right — a low acceptance rate eats the benefit. The verification pass still costs a full forward, so if you’re compute-bound rather than bandwidth-bound, gains shrink. And correctness is subtle: you must guarantee the main model only keeps tokens it would itself have produced, or you risk silent quality drift. The practical rule: speculative decoding shines on memory-bandwidth-bound decode with a well-matched small draft model; if your bottleneck is raw compute or the draft mismatches the big model, you won’t see the headline speedup — measure your acceptance rate first.

Harness improvements: the local stack keeps moving

DwarfStar (the local inference engine) is the strongest thread. Over the last two weeks it’s improved for Metal, DGX Spark and Strix Halo; a Metal-boost PR is ready to merge (prefill +4%, decode +6%); Metal kernels were pushed further for Qwen3.8-Flash-Next; and a merged agent /hints feature makes the programmer a more active part of the loop, learning as it goes. These are the kind of incremental local-serving wins that never show up on HuggingFace trending.

Hermes (the agent harness) also shipped: big token-efficiency improvements over the last two weeks; Wingtips #65 hermes sessions import (carry a Claude Code / Codex CLI conversation into Hermes); Desktop can now browse and import existing Codex and Claude Code sessions, and spin up Bots Mode group chats; all 65 Wingtips landed on one page; and Nous Portal was revamped. For the harness crowd, that’s the “your agents can finally talk to each other and keep their context” story.

Context line. A $3B frontier model reportedly scored 0% on Terminal-Bench while a 27B open model beats it — good color on why local/open stays competitive, but it’s a frontier claim, not a headline, and I won’t build a story on it.

The takeaway

Today is signal, not headline. No single release is a must-swap, but the threads are real: a compact Apache-2.0 edge model with genuine interest (MiniCPM5-2B), a quality-preserving decode speedup (speculative decoding) that keeps improving in local engines, and a local stack that’s shipping incremental wins — DwarfStar on Metal, Hermes session import and bot groups. If nothing here beats what you already run, that’s the honest read — but the direction is right, and speculative decoding is the quiet engine worth understanding.

Credits

  • OpenBMB — MiniCPM5-2B (Apache-2.0, 2B, text-gen) + official GGUF build.
  • WarmBloodAban — Minimax-h3_Singularity (Singularity-container build of MiniMax H3).
  • antirez / ivanfioravanti — DwarfStar (DSpec/DSpark speculative decoding, Metal boosts, agent /hints, Q2_K expert experiment).
  • witcheer / Teknium / tonysimons_ — Hermes (token efficiency, Wingtips #65 sessions import, Desktop session import + Bots Mode, Nous Portal revamp); Qwopus3.8-27B-Flash efficiency measurement.

Sources: HuggingFace model cards and the HF API (downloads / likes verified directly, MiniCPM5-2B card), the curated X cache (read-only, fresh) for DwarfStar / Hermes / Qwopus3.8 and frontier-context signals, and the OpenBMB MiniCPM5-2B card. Benchmarks are vendor-reported where cited; the Qwopus3.8 efficiency figures are a community measurement, not a vendor claim. No fabricated numbers.