552B on one Mac, 35B on a phone — the year of streaming experts

The lead: DeepSeek-V4.1-Flash, and the RAM trick that makes it run

What it is. On Sep 10, DeepSeek released DeepSeek-V4.1-Flash — a 552B-parameter multimodal Mixture-of-Experts model under MIT, with native image+text input and a 1M-token context. (A cloud test build was mentioned here on Sep 9 as gated context; this is the open-weights release, which is a different thing.) The headline isn’t the size — it’s what they did about memory. The model’s own tech report is titled “Pushing the Limits of KV Cache Compression,” and the architecture backs that up: a Causal Encoder-Decoder (CED) stack (20 encoder + 20 decoder layers) whose decoder KV is projected from the encoder’s final hidden states, activating only 8B parameters per token during prefill and 16B during decode; CSA2 sparse attention with per-layer Full/Reindex/Reuse modes sharing KV and Top-K indices; SWA Bounded Replay, which rebuilds sliding-window state by replaying only the last window of tokens instead of persisting it; and an FP4 main KV cache. The result, per DeepSeek: 890 bytes per token of global KV — about ¼ of DeepSeek-V4-Flash’s — and a persistent KV footprint around 1/8 after bounded replay. It also carries Engram conditional memory (196B parameters, token-lookup sparse access) and DSpark speculative decoding, and exposes a continuously controllable reasoning_effort from 1 to 100.

The evidence. 75,774 downloads / 1,843 likes in ~2 days on the base repo, and — the part that actually decides the verdict — roughly fifteen community quant/derivative builds already past 200 downloads within 48 hours (GGUF, MLX 2-bit with MTP, NVFP4, pruned MLX variants). Community quants at that velocity mean people are not just clicking; they are serving it. On X, antirez reported running V4.1 Flash on a single 128 GB M5 Max in DwarfStar with experts streamed from SSD: an 8-second loader to keep the encoder blocks resident, at which point prefill hits ~800 tok/s — while being honest that the full model is not a fit for one 128 GB box (that needs 2×128 GB or a big Mac Studio), and that it’s “not a total jump forward” from 4.0 Flash given the 2× backbone, but “definitely better.” Teknium reports it live in Hermes Agent through Nous Portal for the hosted crowd. Vendor base-model benchmarks are strong — MMLU-Pro 74.1, HumanEval 79.4, DocVQA 95.6 — reported by DeepSeek, judged as claims; the download-and-quants signal is the independent part.

Verdict: Real deployment option — for big-Mac / server-class local rigs, it is the most important open-weight release of the month, and the only ~550B-class open model with a KV cache you can actually hold. If your machine is a 16–32 GB desktop, it isn’t your model yet: wait for quality-tuned quants and engine support to mature. The deployment question for this one was never “is it good” but “can I hold it” — and the community answers by shipping streamable builds the same weekend.

Models

Edge0-35B-A3B-preview — from Edge0 (the Audio8/ARK audio crowd), Apache-2.0: a 35B-class sparse MoE that runs in phone-class memory, in under 3 GiB of active RAM. Fine, “phone-class memory” is the marketing; the measured card is a 24 GB Mac mini M4 Pro: 2.9 GiB peak active memory, 14.9–17.7 tok/s decode, 113/140 tok/s cold/warm prefill. The whole 4-bit checkpoint stays on storage and experts stream in on demand, with a co-trained prerouter that predicts the next token’s routing one step ahead so expert fetches overlap the forward pass (claimed up to +59% decode throughput) and Recover-LoRA adapters that pull the int4 weights back to within 3.9 points of their own fp16 base on their OpenCompass suite (79.2 vs 83.2 average). Base model: Qwen3.5-MoE 35B-A3B. Evidence: 951 downloads / 149 likes for the 35B, 801 / 42 for the 8B sibling, and 1,381 GitHub stars in four days for the now-open-source edge0 streaming framework — plus their X demo of the 35B running on an iPhone. Benchmarks are their own, run against their own fp16 base — so read them as “quant loss is small,” not “quality ranking.” Verdict: Real deployment option — for phones and 8–16 GB machines it’s a genuinely new tier (35B-class quality at pocket-class memory); preview status, and it needs the edge0 runtime, not llama.cpp.

Nex-N2.5 family (mini / Pro / Max) — from Nex-AGI, Apache-2.0: agentic models aimed at computer use, web browsing, and visually-grounded long-horizon tasks, where vision is the agent’s verification interface, not a party trick. mini is a ~35B multimodal MoE (the local-relevant one); Pro is a ~400B-class FP8 build; Max is a 1.6-trillion-parameter text-only MoE — open weights too (11,404 downloads), which matters for research even though nobody is hosting it at home. Evidence: 12,260 dl / 605 likes (Pro), 3,121 / 701 (mini), and — the real adoption tell — a 21,864-download community GGUF of mini within days. Their benchmark tables are vendor-run. Verdict: Marginal — adopt only if you want an open computer-use agent model you can actually run; the mini GGUF wave says the demand is real, but independent evals haven’t landed yet.

YuE2-3B — from m-a-p (MAP), the group behind open music model YuE: 3B open music generation with an editable score — lyrics + style prompt → full song with vocals, and the melody/chords exposed as a symbolic plan you can edit before rendering (plus cover-the-song and agentic-edit flows, their MERT-v2 judge models, and a new WildSongBench eval set). Claims “frontier song quality competitive with Suno v5/v6” and SOTA on their own WildSongBench — vendor claims on a vendor bench, so weigh accordingly. Evidence: 971 downloads / 256 likes in ~3 days. One hard caveat: the license is CC-BY-NC-4.0 — personal and research use, not commercial. Verdict: Marginal — adopt only if you want open, editable, local music generation and the non-commercial license is fine; skip if you need Suno-shaped convenience.

Quantizations

The V4.1-Flash build wave. Fifteen community builds cleared 200 downloads in about 48 hours: apetersson MixedQ2-GGUF (2,905), vcruz305 GGUF (2,733, Q1_0/Q2_K shards), AMAImedia FP8-GGUF (2,133), Vontra MLX 2-bit with MTP (1,140), four pipenetwork MLX variants — including REAP expert-pruned builds (890 / 426 / 275 / 271), plus NVFP4 builds from msuiche (344), AtomicChat (287) and LibertAIDAI (209). Reality check on footprints (estimates, not measured): a 552B model at Q2_K lands around ~180 GB, 2-bit MLX around ~140 GB — these are 128 GB+ machines-and-streaming builds (DwarfStar, MLX with expert streaming), not llama.cpp-on-a-MacBook builds; the Q1_0 GGUFs are sanity-check territory, not daily drivers. The REAP line is the honest signal: some builders are deleting routed experts to fit, which is a quality trade DeepSeek’s architecture doesn’t bless. Verdict: Marginal — adopt only if you already own a big-Mac/server rig and want first-week access; quality-vs-quant gaps aren’t independently measured yet.

The Nex-N2.5-mini GGUF wave. abenzerps GGUF at 21,864 downloads in days — bigger than the base model itself — plus mradermacher (5,832 + 4,427 i1), APEX (8,230), ngquocvinh (2,974), a ROCm FP4 (745), NVFP4 (543/504) and MLX-4bit (461). This is how a ~35B MoE gets used: quantized, in roughly ~20 GB at Q4-class (estimate), on machines people own. Verdict: Real deployment option as the way to run Nex-mini — the adoption is proven even while the model’s own claim table stays vendor-reported.

Incumbents compounding. Two follow-ups from earlier posts, just for the trend line: ISTA-DASLab’s GSQ+RCO mixed-precision GGUF of Qwen3.8-27B has grown from the 100,110 downloads noted here on Sep 4 to 682,187 / 850 likes — 6.8× in eight days — and nvidia’s Qwen3.8-Flash-Next NVFP4 is up from ~26k to 78,737. The swap-in-quant pattern keeps winning. No new headline here; the numbers moved.

Fine-tunes

NeoHorse-1-4B — from TokenRhythm, Apache-2.0, base model Qwen3.5-4B (text-only repack; vision weights stripped). The real delta: it’s an agentic post-training release — trained for tool use, coding and instruction-following inside text agent harnesses — presented as a prototype toward recursive self-improvement: a routing harness assigns tasks across a model pool, records tool trajectories and outcomes, and uses capability-level feedback to shape the next training mix (arXiv 2609.08183). Evidence: 6,109 downloads / 1,172 likes in a week — an unusually high likes ratio that says people are watching this line. Their table claims QwenClawBench 44.68 vs 38.47 for the Qwen3.5-4B base and 43.52 for Spark-X2.5-4B — but it’s a vendor-run board with a vendor-chosen competitor pool, so treat the win as probable and the margin as unproven. Verdict: Marginal — adopt only if you want a 4B agent specialist and can re-run it in your own harness; the RSI framing is research, not a feature.

Below the gate, one line. Day-one abliterations of V4.1-Flash (dealignai’s UNCENSORED-FP8 at 171 downloads / 46 likes, s-zaizen’s at 170) already exist. The uncensor-the-release pipeline is now a same-day service; not news, just a pattern.

Technical term of the day: SSD expert streaming (running an MoE from your NVMe)

What it is. A Mixture-of-Experts model is not one big brain — it’s a small shared core plus hundreds of specialist expert FFNs, of which only a handful fire per token: DeepSeek-V4.1-Flash touches 6 of 384 routed experts per token; Edge0-35B uses 4 of 256. So most of the weights are idle at any instant. Expert streaming exploits that: keep the shared/attention weights, the router, and the hot experts in RAM, park the full checkpoint on NVMe storage, and fetch an expert into RAM at the moment it gets routed. Your RAM budget becomes “the active set,” not the parameter count — which is exactly how a 35B model lives in 2.9 GiB (Edge0) and a 552B model breathes on a 128 GB Mac (DwarfStar).

Why it matters for local AI. It breaks the one hard rule that priced local AI for years: model size ≈ RAM you must buy. NVMe is ~10–40× cheaper than unified memory, so a $1,000 machine starts running models that used to need a $10,000 one. It also changes which specs matter: a fast SSD and good RAM/SSD overlap become first-class AI hardware. Apple Silicon is unusually well suited — unified memory plus very fast internal SSD and batched attention means the streamed decode path stays competitive; the same trick on a laptop with a slow drive is a stutter-fest.

The trade-offs. Three, and they’re the whole engineering field. (1) Seek stalls: if the expert you route to isn’t resident, you wait on a disk read mid-token — so systems cheat: Edge0’s prerouter predicts routing one step ahead and prefetches (+59% decode claimed); DwarfStar keeps frequently-routed experts resident and pre-loads whole regions it knows it needs (their V4.1 test reached full encoder residency in ~8 seconds). (2) What can’t stream: the KV cache stays in RAM regardless (which is why V4.1-Flash’s 890-bytes-per-token KV work is complementary — streaming weights and shrinking KV attack the same limit from both sides), and dense models don’t benefit at all. (3) Quality cost of fitting: the honest variants ship co-trained recovery adapters (Recover-LoRA’s ~4-point int4 loss); the shortcut variants prune or 1-bit the experts — same words, different model. The practical rule: streaming buys capacity, not speed — you run bigger, not faster — and it needs NVMe bandwidth/latency headroom. If your current model already fits, nothing here says switch.

Harness improvements: the local stack keeps moving

DwarfStar is the star exhibit: V4.1-Flash streaming on a 128 GB M5 Max, ~800 tok/s prefill once encoder blocks are resident (maintainer’s own X posts, with the honest caveat above). Also shipped into Hermes Agent land this week: DeepSeek V4.1 Flash available through Nous Portal for hosted runs; inline worker control — subagents now show up inside the conversation with per-worker steer and stop (steer sends a correction without killing the run; stop returns the partial result); 2FA in browser automation — saved TOTP keys, 1Password/Bitwarden OTP, and codes that never enter model context; plugins from private GitHub repos using your stored credentials; Telegram QR quick-setup in Hermes Desktop; a comment mode in the built-in browser (click things on a page, turn pins into annotated screenshots in the composer); and Hermes Vault v0.26.0 — safer restores, enforced lease ownership, expired-credential blocking, AES-GCM v2 by default, and a doctor command.

The fit-table drop. A community member turned the RTX 5090 board into a measured fit guide: 38 models and quants, peak VRAM measured at a 16k prompt + 768 MiB headroom, mapped against 8/12/16/24/32 GB cards — Qwen3.8-27B at IQ3_XXS topping the 16 GB quality list. That’s the most useful kind of harness content: no claims, just what fits on the card you own.

Framework note. The edge0 streaming runtime behind the 35B-on-a-phone story is open-source and four days old (1,381 stars). Another inference engine rarely shows up on trending; this one did, carried by a model.

The takeaway

The wall this month is memory, and the field is going around it two ways at once: make less of it needed (V4.1-Flash’s 890 B/token KV, bounded replay, CED) and make more of it available (streaming experts off NVMe — DwarfStar, Edge0). The empirical verdict splits clean: DeepSeek-V4.1-Flash is a real deployment option if you have the silicon, with fifteen quant builds in 48 hours as proof; Edge0-35B is a real deployment option for the phone/tight-RAM niche with a preview badge; the first-week V4.1 quants and NeoHorse-1-4B are marginal, pending independent numbers; YuE2 is a marginal, non-commercial delight. If you remember one practical thing: your next local-AI bottleneck won’t be solved by a bigger GPU — it’ll be solved by what sits behind the GPU.

Credits

• DeepSeek-AI — DeepSeek-V4.1-Flash (MIT, 552B CED MoE, CSA2 + FP4 KV + Engram + DSpark, tech report).
• Edge0 (Edge0-AI) — Edge0-35B-A3B-preview / 8B-A1B-preview + open-sourced edge0 streaming framework.
• Nex-AGI — Nex-N2.5 mini/Pro/Max; m-a-p — YuE2-3B + WildSongBench; TokenRhythm — NeoHorse-1-4B.
• Builders — apetersson, vcruz305, AMAImedia, Vontra, pipenetwork, msuiche, AtomicChat, LibertAIDAI, abenzerps, mradermacher et al. — the day-one quant waves.
• antirez — DwarfStar V4.1-Flash SSD-streaming measurements; witcheer, Teknium, tonysimons_ — Hermes/DwarfStar ecosystem updates and the RTX 5090 fit table (via their public X posts).

Sources: HuggingFace model cards and the HF API (all downloads / likes figures verified directly on 12 Sep 2026; footprint GB figures flagged as estimates), the curated X cache (read-only, fresh) for DwarfStar / Hermes / fit-table signals, and the DeepSeek-V4.1-Flash technical report. Benchmarks are vendor-reported where cited and flagged as such; benchmark suites authored by the vendor (WildSongBench, NeoHorse’s table, Edge0’s own OpenCompass run) are noted as vendor-chosen pools. No fabricated numbers.