SHELFCRITTER AI Digest

COMPANY

Zyphra

companytopic-note

Overview

Zyphra is an AI research lab working on long-context attention primitives for language models. In November 2026 the team published ZHA — a subquadratic attention mechanism built as a tree of recursively compressed key summaries — framed as a retrofit onto existing pretrained models via a few-billion-token distillation budget rather than a from-scratch architecture, with reported gains beyond 1M-token context.

Timeline

  • 2026-11-02-AI-Digest — Lufkin, Dawes, Speicher, Krishnamurthy, Figliolia, Millidge publish Subquadratic Attention with a Hierarchical Tree-Structured Indexer (arXiv:2610.42112) introducing ZHA — a tree of recursively compressed key summaries that is retrofittable to pretrained models with a ~a-few-billion-token distillation budget, reporting gains beyond 1M-token context. Why it matters: a drop-in long-context primitive that doesn’t require pretraining from scratch is exactly the shape of practical capacity extension developers can actually ship.

  • 2026-11-03-AI-Digest — ZHA re-anchors today’s Papers section, re-contextualised against a visible 2025–2026 wave of pretrained-model grafting via KL-distillation (Hauzenberger et al., Foreign Sparse Attention, SeerAttention, SpargeAttention2) where sparse/linear attention becomes a drop-in upgrade rather than a from-scratch retrain. Narrow read: ZHA is not a novel method on its own; the structural datum is that both the sparse-attention retrofit and the faithful-CoT axes now have practitioner-ready recipes rather than diagnostic-only tooling. Robust up to 6-level trees per the paper; reporting “dramatically outperforms competing methods beyond 1M context.” Log against MOC - AI Infrastructure.

Key Developments

  1. ZHA — Subquadratic Attention as a Retrofit, Not a From-Scratch Architecture (November 2, 2026): Zyphra’s ZHA proposes a hierarchical tree-structured indexer of recursively compressed key summaries, retrofittable to pretrained models with a few-billion-token distillation budget. If the >1M-context behaviour holds in independent reproduction, long-context capacity extension moves from pretrain-scale project to finetune-scale project.

See also: MOC - AI Infrastructure, MOC - Open Source Models.