SHELFCRITTER AI Digest

MODEL

Needle

modeltopic-noteopen-source

Overview

Needle is a 26M-parameter open-source function-calling model trained on synthetic data generated with Gemini (a teacher-data pipeline, not weight distillation). It targets extreme edge inference for tool-use tasks, claiming 6,000 tok/s prefill and 1,200 tok/s decode on consumer hardware.

Timeline

  • 2026-05-13-AI-Digest — Needle was open-sourced by its team and surfaced on r/LocalLLaMA (262 points / 33 comments). The team’s argument is that tool calling is fundamentally retrieval-and-assembly and does not require frontier-scale models; training signal comes from Gemini-generated synthetic data. At 26M parameters, Needle is an extreme outlier — production edge inference runs mostly on 1–7B (Llama 3.2, Phi-4, Gemma 3) — so it is best read as a thesis test rather than a representative shift in the edge market.
  • 2026-08-11-AI-Digest — Cactus Compute ships Needle2 as a 45M-param sibling to the original 26M Needle — 2-bit-compressed, single 14MB binary, running a full session in 28MB RAM at 500 tok/s decode on a Raspberry Pi 5, tuned for tool calls and structured extraction (Show HN 249 pts / 97 cmts on cactuscompute.com/needle). Concrete on-device agent that could actually run inside firmware. Corpus framing: Needle’s ultra-compact tool-use thesis (26M in May) now has a 45M sibling with production-shaped ergonomics — the “agent-tuned size is the design axis, not raw parameter count” framing lands alongside Meta‘s Muse Glimmer (30B, prosumer-hardware) the same week the frontier labs are gating their cyber SKUs behind human vetting. Two directions of travel on the same “who deploys the agent” question.

Key Developments

  1. Ultra-Compact Tool-Use Thesis (May 2026): 26M-parameter model claiming function-calling capability at 6,000 tok/s prefill on consumer hardware, trained on Gemini synthetic data. Positions itself as proof that tool calling does not require frontier-scale parameters — a hypothesis worth monitoring as production edge benchmarks emerge.

  2. Needle2 45M / 14MB Binary on Raspberry Pi 5 (August 2026): Follow-on release from Cactus Compute — 45M params, 2-bit-compressed, 14MB single binary, 28MB RAM session, 500 tok/s decode on a Pi 5, tuned for tool calls and structured extraction. Production-shaped ergonomics (single-binary deployment, sub-30MB RAM ceiling, decode throughput on a $80 SBC) turn the “ultra-compact tool-use” thesis into a concrete firmware-scale deployment envelope, not just a benchmarking demo. Lands the same week Meta ships Muse Glimmer as the 30B prosumer-hardware entrant — two distinct size classes making the same “agent-tuned size is the design axis” argument against frontier-gated cyber SKUs.