Needle

Overview

Needle is a 26M-parameter open-source function-calling model trained on synthetic data generated with Gemini (a teacher-data pipeline, not weight distillation). It targets extreme edge inference for tool-use tasks, claiming 6,000 tok/s prefill and 1,200 tok/s decode on consumer hardware.

Timeline

• 2026-05-13-AI-Digest — Needle was open-sourced by its team and surfaced on r/LocalLLaMA (262 points / 33 comments). The team’s argument is that tool calling is fundamentally retrieval-and-assembly and does not require frontier-scale models; training signal comes from Gemini-generated synthetic data. At 26M parameters, Needle is an extreme outlier — production edge inference runs mostly on 1–7B (Llama 3.2, Phi-4, Gemma 3) — so it is best read as a thesis test rather than a representative shift in the edge market.

Key Developments

1. Ultra-Compact Tool-Use Thesis (May 2026): 26M-parameter model claiming function-calling capability at 6,000 tok/s prefill on consumer hardware, trained on Gemini synthetic data. Positions itself as proof that tool calling does not require frontier-scale parameters — a hypothesis worth monitoring as production edge benchmarks emerge.