Epoch AI starts Benchmark Reviews audits
Epoch AI put numbers on benchmark quality instead of model quality, launching reviews across 15 benchmarks with 4 verified, 9 flawed, and 2 too underspecified to judge.
Technical AI signals · Daily at 8 PM ET
Friday, September 18, 2026
Today’s strongest operator signals were Epoch AI auditing benchmark quality, NVIDIA shipping a realistic LLM serving benchmark, a broad libheif exploit chain hitting AI-adjacent stacks, and new runtime surfaces for long-running agents.
Coverage update: Some YouTube sources were unavailable for this edition.
Epoch AI put numbers on benchmark quality instead of model quality, launching reviews across 15 benchmarks with 4 verified, 9 flawed, and 2 too underspecified to judge.
AIPerf targets deployed inference rather than lab demos, measuring TTFT, ITL, latency, and throughput under repeatable load patterns that should make serving regressions easier to catch.
One image-library flaw reportedly propagated into OpenAI, Slack, Meta, GitHub Enterprise, Rails, Next.js, and ImageMagick, a sharp reminder that multimodal pipelines inherit ordinary dependency risk at large blast radius.
AWS is turning long-running agents into managed infrastructure with a runtime aimed at multi-step work that can coordinate systems, write code, and run for hours unattended.
CUA-Bench pushes beyond static GUI tasks by requiring models to emit live keyboard and mouse actions in six games from video input, testing timing and control instead of just planning.
The model supports 1M-token context and auto-edits vlogs, translates short videos, and turns movies into recaps.
The architecture separates what the agent knows from how it reasons and adds a self-improvement loop that turns expert feedback into verified updates without retraining.
The method adds diffusion to K2-Horizon-7B and reports up to a 2.2× speedup without quality loss.
Anthropic and Accenture say they will each invest at least $1 billion over five years to build evaluator capacity.
The CLI can run commands and tests, fix errors, use subagents, MCP, plugins, and resume previous sessions.
The system is built on MiniMax AI H3 and claims lower inference cost without external training data.
Claude Code 2.1.277 now checks AGENTS.md when a folder does not contain CLAUDE.md.
The post describes using NVIDIA hardware video decoders to scale video captioning and description across multiple GPUs.
Together AI says DMI gave engineering teams direct control over scaling, models, and testing.
The author reports a linear probe that caught 86% of colluding transcripts at a 1.4% false-positive rate in a monitor-policy collusion setup.
Adam Harvey and the crates security team say attackers are using video calls and fake audio-codec installs to compromise devices and accounts.
The author argues that pretrained LLMs are better at judging math arguments than fiction writing because of the pretraining data.
Commenters say the Cloudflare article’s writing is unusually good, and one comment questions whether AI-written text still reflects the company’s internal complexity.
Commenters say the paper is old but still fascinating, and they note that cache-based model communication has not appeared in production models.
Commenters say the attack uses a $250k lab setup and may still be reproducible for far less in a home lab.
Commenters report odd behavior from the models, including misreading commands like "I need a wee" and turning on the bathroom vacuum.
Commenters say the article alleges ZCode uploaded Git history and that the company later apologized to affected users.
Commenters argue Delta is sharing edit history rather than replacing pull requests, and one says it lacks CI or merge gating.
Commenters discuss FEX and debate claims about x86 and ARM memory-ordering overhead under emulation.
Commenters note that Starship flight test 13 was aborted after several Raptor 3 engines failed to start.
Commenters say the system combines a search-and-inference database with a simulator that needs to know what "correct" looks like.
The paper introduces a multimodal MoE model with 552B backbone parameters and contexts up to one million tokens.
The method scales auto-research loops across many environments and keeps four mechanisms that survive selection.
The study evaluates 176 matched settings across four models, five context strategies, and targeted ablations on SWE-Bench Verified and Terminal-Bench 2.1.
The paper ties length inflation to mismatched stopping probabilities on different EOS tokens across Qwen3, Llama, and Gemma.
The method revises GUI skills from execution feedback at deployment time without additional training.
The paper proposes an index that adapts to its retrieval environment instead of relying on fixed human-driven optimization.
ActObs supervises observation tokens during SFT so the policy learns action consequences without extra data, parameters, or forward passes.
The study shows that the same candidate budget can be executed batched or sequentially, changing both energy use and reasoning performance.
VA-Bench tests observe-reason-act-revise behavior with 14 task families and a fixed controller that executes model-specified metric targets.
Video DeltaNet combines local Softmax attention with bidirectional linear memory that updates once per frame.
The paper proposes a decoupled, skill-conditioned teacher and a skill-free student trained jointly with RL after finding that teacher supervision is stage-dependent.
WeVisDoc uses a two-stage framework with heterogeneous data, structure-preserving degradation synthesis, and a held-out probe for residual errors in fixed visual-structural clusters.
The video focuses on the one IMO 2025 problem that AI missed and walks through the construction and proof.
Bernt Børnich says 1X plans to manufacture and ship 50,000 NEO units in 2027.
Ignacio Martinez argues that an agent is a model plus a harness, and that harness engineering turns a nondeterministic core into something controllable.
Stefano Ermon describes applying diffusion beyond images and video to discrete text and code generation.
Ali Ghodsi says the bigger bottleneck for enterprise adoption is context, not model capability.
The episode points to persistence, goal building, smart defaults, and progressive disclosure as the design patterns behind Meta's Muse.
No additional details are provided in the item text.
The episode walks through a Codex conversation from first idea to stack and architecture decisions, with verification planned so the agent can keep building autonomously.
TypeSafe says Jev answers in about 100 milliseconds and costs $42 per billion input tokens with free output.
The episode explains client delegation and how it connects a voice conversation to a reasoning model and a word bank.
The preview model uses a sparse MoE design that runs with under 3 GiB of active memory and about 15 tok/s.
The ternary GGUF checkpoint is about 9.3× smaller than FP16 and claims 98.2% of FP16 intelligence retained.
The model uses agentic post-training with a routing harness aimed at recursive self-improvement.
The dense 2B model is tuned for on-device use and reports strong results in coding, mathematics, long-context understanding, and tool use.
Unsloth says Dynamic V3.0 GGUFs improve Qwen3.8 accuracy by 10% at the same size versus other providers.
The model is tagged for image-to-video, text-to-video, video-to-video, and audio-to-video generation.
The dataset contains about 500,000 agent trajectories covering tool use, search agents, code agents, and general agents.
The dataset contains 509 end-to-end Windows CAD task recordings across seven CAD software families.
The browser workspace runs Pi’s agent loop, local WebGPU inference, and shell commands without an inference API or shell server.
The model repository says the weights are compatible with Transformers, vLLM, SGLang, and TokenSpeed, and points users to Qwen Cloud for managed inference.
The dataset contains more than 85,000 samples for MiniCPM5-2B post-training, covering mathematical reasoning, scientific reasoning, long-context understanding, and code generation.
The skill runs six phases, including reconnaissance, coverage-led hunting, candidate validation, and independent record verification.
Claude Code is a terminal agentic coding tool that can execute routine tasks, explain code, and handle git workflows.
The CLI reads Git diffs, sends changed files to a configurable LLM, and returns line-level review comments.
BrowserSkill lets agents borrow an already logged-in browser tab and return it when the task is done.
The repo packages 9 slash commands that map to development stages like spec, plan, build, and test.
Octop runs a multi-agent assistant locally and exposes a web dashboard, CLI, and IM integrations.
OpenSpec uses plain Markdown specs and an artifact-guided workflow where the AI writes the plan before code is written.
Coder defines workspaces in Terraform and connects them through a secure Wireguard tunnel with automatic shutdown when idle.
The memory layer claims 95% Recall@15 with 99.4% context reduction and about 50 ms user profiles.
RustFS is an Apache 2.0 distributed object storage system with S3-compatible APIs for data lakes, AI, and big data.
The repository bundles skills, connectors, slash commands, and sub-agents to tailor Claude for specific job functions.
Hister indexes pages and files for search from the web interface, terminal, or an AI assistant connected through MCP.
The same feed, delivered daily at 8 PM ET. No spam, unsubscribe anytime.