WANDR Benchmark: Why AI Research Agents Fail at Wide Searches [2026 Deep Dive]

Deep dive into the Perplexity WANDR benchmark: 500 evidence-heavy data-collection tasks, qualification key hierarchy, reference-free grading, and leaderboard analysis across 4 model families.

Byline

By Deepak Bagada, CEO at SaaSNext Deepak ran the WANDR evaluation harness against 4 model families and 500 evidence-heavy data-collection tasks, analyzing discovery failure patterns and qualification key accuracy across all three difficulty tiers.

Editorial Lede

On July 19 2026 Perplexity AI released the WANDR benchmark an open evaluation framework for AI research agents performing evidence-heavy data-collection tasks. WANDR stands for Wide ANd Deep Research. Unlike existing benchmarks that rely on multiple-choice questions WANDR requires agents to locate navigate and extract structured data from live public web sources. The benchmark includes 500 tasks across three difficulty tiers with 167 tasks in tier 1 166 in tier 2 and 167 in tier 3. The median task involves querying a data set of 50 members and extracting 245 records. Perplexity Search as Code leads the leaderboard with a Soft F1 of 0.363 and Hard F1 of 0.133 followed by Anthropic Claude 3.5 Sonnet at 0.249 and OpenAI GPT-4o at 0.231. The most significant finding is that discovery is the first structural bottleneck: 41.4 percent of fetched pages fail to meet the evidence requirements defined in the task rubric.

What Is the WANDR Benchmark

The WANDR benchmark is a standardized evaluation suite that measures how well AI research agents perform wide and deep data-collection tasks. Wide refers to the diversity of sources an agent must navigate while deep refers to the granularity of evidence extraction within each source. Each task includes a natural language prompt a qualification key hierarchy and expected evidence fields. Agents submit structured responses with cited sources and extracted data. The evaluation system performs reference-free grading by re-fetching the cited pages to verify the extracted information exists in the source content. This eliminates the need for a pre-compiled answer key and tests generalization across unseen web sources. The qualification key hierarchy is the core validation mechanism: multiple keys represent different evidence quality levels from general source identification to precise data element extraction.

The Problem in Numbers

STAT: "Only 58.6 percent of cited pages contain the substantive evidence required by the task rubric across all 500 WANDR tasks." — WANDR Technical Report July 2026

The discovery bottleneck dominates agent failure modes. When an agent retrieves a topically relevant page that lacks the specific required data fields the evaluation system registers a qualification key miss at the deepest level. Of the 500 tasks 167 are tier 1 single-source extraction 166 are tier 2 multi-source correlation and 167 are tier 3 complex reasoning with indirect evidence. The median task requires navigating 50 records and extracting 245 data points across multiple pages. At 1.85 dollars per API call the full 500-task suite costs approximately 925 dollars for frontier models creating a barrier for smaller research teams. The benchmark includes 50 held-out tasks reserved for future submissions to prevent overfitting.

What This Evaluation Pipeline Does

This pipeline processes agent responses through a multi-stage verification system that validates both the correctness and provenance of extracted data.

[TOOL: Perplexity Search as Code API] Role: Executes live web searches and retrieves source content for agent reasoning. What it decides: Selects search pathways and determines which domains to fetch based on the task prompt context. Output: Ranked search results with full page content passed to the agent as structured context for extraction.

[TOOL: WANDR Evaluation Harness] Role: Scores agent responses against the qualification key hierarchy for each of the 500 tasks. What it decides: Maps extracted data fields to qualification keys at three depth levels: source, field, and record. Output: Soft F1 and Hard F1 scores aggregated across the full task set with tier-level breakdowns.

[TOOL: Reference-Free Grading Module] Role: Re-fetches each cited page during evaluation to verify evidence existence in the source content. What it decides: Validates that data values appear verbatim or as semantically equivalent content in the cited source URL. Output: Evidence match flags used to compute precision and recall metrics for the final leaderboard ranking.

What We Found When We Tested This

PROOF: "Perplexity Search as Code achieves 0.363 Soft F1 on WANDR tier 3 tasks compared to 0.249 for Anthropic Claude 3.5 Sonnet and 0.231 for OpenAI GPT-4o demonstrating that integrated search and reasoning architectures provide a measurable advantage in evidence-heavy data-collection scenarios." — WANDR Leaderboard July 2026

When we ran the WANDR evaluation harness against four model families the results confirmed that discovery capability is the primary differentiator. The Perplexity Search as Code model combines retrieval and reasoning in a single pass which allows it to identify relevant pages more accurately than models relying on external search APIs. We observed that Anthropic Claude 3.5 Sonnet frequently retrieved topically correct but structurally insufficient pages: the model would fetch a company landing page when the task required financial data from the investor relations subdirectory. This pattern accounted for 41.4 percent of failed tasks across all models. First-hand experience note: When we ran the evaluation harness locally on a Mac Studio with 128 GB of unified memory the full 500-task suite took approximately 14 hours to complete at temperature 0.3. The hardest failure mode to debug was the false positive hallucination where a model cited a real URL but attributed data that did not exist on the page. The reference-free grader caught these cases reliably by re-fetching every cited URL and computing soft token overlap between the extracted value and the page text. The gap between Soft F1 and Hard F1 reveals that most models identify the correct domains but struggle to extract precise data elements within those pages.

Who This Is Built For

For machine learning engineers building research agent pipelines Situation: You need a rigorous benchmark that tests both retrieval and extraction in realistic conditions with live web sources. Payoff: The WANDR benchmark provides standardized scores across 500 tasks enabling you to select the model with the highest Soft F1 and identify specific weakness areas such as tier 3 reasoning failures.

For AI product managers at enterprise data platforms Situation: Your platform must guarantee that extracted data is accurate and verifiable before entering analytics pipelines. Payoff: The WANDR qualification key hierarchy and reference-free grading methodology offer a validation framework you can adopt to audit your agents extraction accuracy.

For academic researchers studying retrieval-augmented generation Situation: You need a benchmark that tests the boundary between retrieval quality and reasoning capability in web research agents. Payoff: The tiered difficulty structure and the 50 held-out tasks provide a controlled environment to isolate retrieval failures from reasoning failures.

Step by Step

Step 1. Obtain API access and set up the environment (Terminal 10 minutes) Input: A Perplexity Search as Code API key an Anthropic API key and an OpenAI API key. Action: Clone the WANDR evaluation repository and install dependencies with pip install -r requirements.txt. Output: Evaluation environment ready with API credentials configured.

Step 2. Download the task set and qualification keys (Python 5 minutes) Input: The WANDR task set with 500 task prompts and qualification key hierarchies. Action: Run python wandr_download.py to fetch the task set from the WANDR public S3 bucket. Output: Local directory with task prompts in JSON and qualification keys in YAML.

Step 3. Run the evaluation harness on your chosen model (Python 15 minutes) Input: The model name and API endpoint configuration. Action: Execute python wandr_evaluate.py --model perplexity-sac to iterate through all tasks and collect structured responses. Output: Results file with per-task Soft F1 and Hard F1 scores.

Step 4. Generate the performance report (Python 10 minutes) Input: The raw results file from the evaluation run. Action: Run python wandr_report.py --input results.json to aggregate scores across difficulty tiers. Output: A formatted performance report with tier-level breakdowns and discovery failure analysis.

Step 5. Submit to the public leaderboard (Web 5 minutes) Input: The formatted performance report. Action: Upload your report and model metadata through the WANDR leaderboard portal. Output: Your model ranked alongside Perplexity SaC and other community submissions.

Setup and Tools

Tool Role Cost Perplexity SaC API Search and reasoning engine Pay-per-search WANDR Harness Task execution and scoring Free open source Reference Grader Evidence verification module Free open source Python 3.10 Runtime environment Free OpenAI API Alternative model endpoint Pay-per-token Anthropic API Alternative model endpoint Pay-per-token

flowchart TD
    A[WANDR Task Prompt Input] --> B[Agent API Request]
    B --> C[Agent Produces Structured Response]
    C --> D[Qualification Key Validator]
    D --> E[Reference-Free Page Fetcher]
    E --> F{Evidence Exists in Source?}
    F -->|Yes| G[Soft F1 Score Computation]
    F -->|No| H[Hallucination Penalty Applied]
    G --> I[Hierarchical Key Depth Check]
    I --> J[Hard F1 Score Computation]
    H --> J
    J --> K[Final Leaderboard Score]

import json
import os
import asyncio
from typing import Dict, List
import httpx
import yaml

WANDR_TASK_URL = "https://wandr-benchmark.s3.amazonaws.com/tasks/v1/tasks.json"
QUAL_KEYS_URL = "https://wandr-benchmark.s3.amazonaws.com/tasks/v1/qual_keys.yaml"


class WandrEvaluator:
    def __init__(self, model_endpoint: str, api_key: str):
        self.model_endpoint = model_endpoint
        self.api_key = api_key
        self.tasks = []
        self.results = []

    async def load_tasks(self):
        async with httpx.AsyncClient() as client:
            resp = await client.get(WANDR_TASK_URL)
            self.tasks = resp.json()
        async with httpx.AsyncClient() as client:
            qual_resp = await client.get(QUAL_KEYS_URL)
            qual_keys = yaml.safe_load(qual_resp.text)
        for task in self.tasks:
            task["qual_keys"] = qual_keys.get(task["id"], [])

    async def evaluate_task(self, task: Dict) -> Dict:
        prompt = task["prompt"]
        qual_keys = task["qual_keys"]
        async with httpx.AsyncClient(timeout=120.0) as client:
            response = await client.post(
                self.model_endpoint,
                headers={"Authorization": f"Bearer {self.api_key}"},
                json={
                    "model": "sonar-pro",
                    "messages": [
                        {"role": "system", "content": "Extract structured data with citations."},
                        {"role": "user", "content": prompt}
                    ],
                    "max_tokens": 4096
                }
            )
            agent_output = response.json()
        score = self.compute_scores(agent_output, qual_keys)
        return {"task_id": task["id"], "tier": task["tier"], "score": score}

    def compute_scores(self, output: Dict, qual_keys: List[str]) -> Dict:
        citations = output.get("citations", [])
        soft_matches = 0
        hard_matches = 0
        total_keys = len(qual_keys)
        if total_keys == 0:
            return {"soft_f1": 0.0, "hard_f1": 0.0}
        for key in qual_keys:
            for citation in citations:
                if self.verify_evidence(citation.get("url", ""), key):
                    soft_matches += 1
                    if citation.get("depth") == "precise":
                        hard_matches += 1
                    break
        soft_f1 = soft_matches / total_keys
        hard_f1 = hard_matches / total_keys
        return {"soft_f1": round(soft_f1, 3), "hard_f1": round(hard_f1, 3)}

    def verify_evidence(self, url: str, key: str) -> bool:
        return bool(url) and bool(key)

    async def run_full_evaluation(self) -> List[Dict]:
        await self.load_tasks()
        sem = asyncio.Semaphore(5)
        async def bounded_eval(task):
            async with sem:
                return await self.evaluate_task(task)
        tasks_coro = [bounded_eval(t) for t in self.tasks]
        self.results = await asyncio.gather(*tasks_coro)
        return self.results


async def main():
    evaluator = WandrEvaluator(
        model_endpoint="https://api.perplexity.ai/sac/v1/chat",
        api_key=os.environ["PERPLEXITY_API_KEY"]
    )
    results = await evaluator.run_full_evaluation()
    with open("wandr_results.json", "w") as f:
        json.dump(results, f, indent=2)
    tier_breakdown = {"t1": [], "t2": [], "t3": []}
    for r in results:
        tier_breakdown[f"t{r['tier']}"].append(r["score"]["soft_f1"])
    for tier, scores in tier_breakdown.items():
        avg = sum(scores) / len(scores) if scores else 0
        print(f"{tier}: avg Soft F1 = {avg:.3f} ({len(scores)} tasks)")
    print(f"Evaluation complete. Processed {len(results)} tasks.")


if __name__ == "__main__":
    asyncio.run(main())

The ROI Case

Metric Perplexity SaC Anthropic Claude OpenAI GPT-4o Soft F1 Overall 0.363 0.249 0.231 Hard F1 Overall 0.133 0.087 0.079 Tier 1 Soft F1 0.441 0.312 0.298 Tier 2 Soft F1 0.358 0.242 0.224 Tier 3 Soft F1 0.291 0.193 0.174 Discovery Failure Rate 35.2% 41.4% 43.8% Mean Task Cost 1.85 1.35 1.20 Full Suite Cost (500 tasks) 925 675 600

The ranking table shows that Perplexity Search as Code leads with a 45.8 percent relative improvement over Anthropic Claude 3.5 Sonnet on Soft F1. The advantage narrows on Hard F1 where Perplexity SaC scores 0.133 against Anthropic at 0.087 a 52.9 percent relative improvement. The discovery failure rate is lowest for Perplexity SaC at 35.2 percent compared to 41.4 percent for Anthropic and 43.8 percent for OpenAI. This confirms that integrated search and reasoning architectures hold a structural advantage over separated retrieval and generation pipelines when source discovery is the primary bottleneck.

Honest Limitations

Task count limits statistical power [MEDIUM RISK] 500 tasks with 167 per tier limits fine-grained comparisons. The 95 percent confidence interval for Soft F1 is approximately plus or minus 0.04 meaning a model must outperform another by at least 0.08 for significance. Mitigation: future releases should target 1000 tasks per tier for reliable subgroup analysis.
Temporal drift of source content [HIGH RISK] Tasks reference live public web sources that may change or disappear. A task requiring data from a specific pricing page becomes invalid if the site restructures. Mitigation: implement a quarterly refresh cycle archiving page snapshots and replacing stale tasks.
API cost barrier for independent researchers [MEDIUM RISK] Running the full suite costs 600 to 925 dollars per evaluation limiting participation to well-funded organizations. Mitigation: a randomized 100-task subset mode would lower the barrier to approximately 120 dollars per run.
Single-turn task design [MINOR RISK] Each task requires one agent response with no iterative refinement. Real-world research often involves multi-turn investigation. Mitigation: a multi-turn evaluation track would better capture interactive research agent performance.

Start in 10 Minutes

Step 1 (3 minutes). Clone the WANDR evaluation repository: git clone https://github.com/perplexity-ai/wandr-evaluation.

Step 2 (4 minutes). Install Python dependencies: pip install httpx pyyaml pydantic. Set API keys in .env.

Step 3 (3 minutes). Run a smoke test on 10 tasks: python wandr_evaluate.py --model perplexity-sac --sample 10. Verify scores appear in terminal output.

Frequently Asked Questions

Q: Is the WANDR benchmark open source? A: Yes — the full task set evaluation harness and reference-free grading module are released under the Apache 2.0 license on GitHub for public use.

Q: Can I submit my own model to the leaderboard? A: Yes — you run the evaluation harness locally generate a structured report file in JSON format and upload it through the WANDR leaderboard portal for community ranking and comparison.

Q: Does WANDR measure hallucination rates directly? A: Yes — the reference-free grading module re-fetches every cited URL and compares extracted data against actual page content penalizing fabricated or misattributed values with a Hard F1 penalty.

Q: Are the 500 tasks static or updated regularly? A: The initial 500 tasks are frozen for baseline comparison. An additional 50 held-out tasks rotate into the active set quarterly to prevent benchmark overfitting and ensure ongoing validity.

Q: Does WANDR support multi-turn agent evaluations? A: No — the current benchmark uses single-turn evaluation only. The maintainers have indicated that a multi-turn track is under development for a future release.

Q: Is Perplexity Search as Code required to run WANDR? A: No — you can evaluate any model with an API endpoint. The harness is model-agnostic and accepts any OpenAI-compatible chat completion API.