Perplexity AI Releases WANDR: An Open Benchmark Evaluating Analysis Brokers That Should Search Broad And Deep

Analysis brokers already deal with actual data work at this time. Groups delegate aggressive mapping, due diligence, and literature evaluation to them. Nevertheless, most benchmarks check a single reply, not giant evidence-backed collections. Perplexity targets that hole with a brand new open benchmark.

Perplexity launched WANDR (Broad ANd Deep Analysis). It’s an open benchmark and analysis harness. It’s constructed round 500 real looking, difficult data-collection duties for data work. WANDR is the large sibling of Perplexity’s DRACO benchmark for deep analysis. DRACO asks whether or not an agent produces an correct, full, goal long-form report. WANDR as an alternative asks whether or not it could actually construct a big assortment with proof.

What’s WANDR

At its core, WANDR exams two calls for collectively. Broad means discovering a big, typically open-ended set of qualifying entities. Deep means investigating each entity sufficient to help every declare with proof. Combining each modifications the issue for brokers. A number of compelling examples aren’t sufficient right here. A elegant narrative constructed on incomplete analysis additionally falls quick.

To seize this, WANDR makes use of a composable qualification key hierarchy. One job may request firm(n) -> worker(m) -> url(okay). This implies n qualifying firms, m staff every, and okay supporting pages every. Each full path by means of the tree will get validated independently. The identical construction can characterize a flat checklist, nested search, or matrix.

A Concrete Process Instance

To floor that hierarchy, think about the launched ceo_cfo_appointments job. It asks for a minimum of 70 US-based firms. Every should have a CEO or CFO appointment first introduced between March 1 and April 30, 2026. For every, the agent provides one authoritative appointment web page. A subtask provides a listing-authority web page per firm. Collectively, the duty requires 140 source-backed data.

Concretely, the 2 hierarchies and one submitted document seem like this:

# Process hierarchies
firm(70) -> company_appointee(1) -> url(1)   # 70 appointment data
firm(70) -> url(1)                           # 70 itemizing data

# One document the grader re-fetches and re-checks (values are illustrative)
{
  "merchandise":     "Instance Corp - new CFO",
  "url":      "https://issuer.instance.com/press/cfo-appointment",
  "excerpts": ["Example Corp today named Jane Doe as Chief Financial Officer, effective April 2026."],
  "reply":   "Jane Doe appointed CFO; introduced April 2026"
}

Lifelike Duties, Generated At Scale

Past single examples, WANDR builds its duties from actual utilization. It begins from de-identified patterns seen in manufacturing, not artificial prompts. A semi-automated pipeline then turns these patterns into duties. The pipeline runs 4 phases: seeding, authoring, admission, and curation. It makes use of an interleaved author-critic loop with mechanical linting.

Consequently, the median job asks for 50 members and 245 data general. Throughout all 500 duties, WANDR requires 170,495 source-backed data. Duties break up into 167 decrease, 166 center, and 167 larger issue. Issue is determined by per-record work, not scale alone.

How WANDR Grades Brokers

Not like mounted reply keys, WANDR grades every declare in opposition to cited proof. Each document comprises an merchandise, URL, chosen excerpts, and reply. The grader re-fetches the web page throughout analysis. It checks whether or not the web page is usable and in scope. It then verifies the excerpts actually seem and help each requirement.

These binary document verdicts then roll up by means of the hierarchy. Precision measures the standard of what a system submitted. Recall measures quality-adjusted completion, filling any shortfall with zeros. Gentle scores give partial credit score to incomplete members. Laborious scores rely solely members whose full subtree is right.

The Benchmark Outcomes

Utilizing that methodology, Perplexity ran six manufacturing programs on all 500 duties. Its personal Search as Code (SaC) system leads. Nonetheless, no system comes near fixing the benchmark.

With extra effort, Perplexity reaches 0.447 delicate F1 on the xhigh setting. Price throughout settings spans greater than 4 orders of magnitude. It ranges from $0.03 per job as much as $324.83 per job.

Past the leaderboard, 4 findings stand out. First, partial progress is frequent, however full protection just isn’t. Each system exhibits delicate recall under delicate precision. Second, scale compounds the issue sharply. Deeper hierarchies damage most, since every department provides a failure level. Third, discovery is the primary structural bottleneck. Prime-level discovery completion ranges from 0.611 to 0.951 throughout programs. Below-delivery, not duplicate merging, explains most lacking quantity. Fourth, discovering a usable web page is normally straightforward. Turning it into full proof is the laborious half. For Perplexity, 41.4% of pages miss a substantive requirement. Additionally, 57.5% of excerpts fail to help the complete declare. Its delicate F1 falls from 0.531 underneath a retrieval-only test to 0.363 underneath the complete verdict.

Notably, Search as Code suits this job form properly. An agent can specific retrieval, filtering, fan-out, joins, deduplication, and stopping logic as a program. Deterministic compute then handles repeated operations exterior the mannequin context.