Build an Agent-Traffic Analytics Workflow with Server-Log Fingerprinting & AEO Reporting

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

Introduction

On August 13, 2026, OtterlyAI announced Agent Analytics — a feature that reads a website's server log data to report which AI agents are visiting, what they are crawling, and how the site is being used by answer engines and agentic browsers. It is a small launch with a big implication: agent traffic is the new analytics blind spot, and server logs are the only reliable window into it. Traditional web analytics runs on JavaScript tags, and most AI agents do not execute JavaScript. The GPTBots, ClaudeBots, and PerplexityBots of the world show up in your server logs and nowhere else — invisible to every dashboard your marketing team looks at.

This dispatch builds the engine behind agent visibility: a LangGraph analytics workflow, agent-audit, that ingests raw server logs, fingerprints AI agent traffic from user-agent patterns, IP reputation, and behavior signals, aggregates visits by agent family, and generates AEO (answer engine optimization) reports for the teams that need them. The same pipeline that tells your content team which pages answer engines actually read also tells your infrastructure team who is hammering your origin. If you are doing any agentic search work, this is the observability layer you are missing — the analytics equivalent of the tool catalog in the MCP directory.

Why agent traffic is invisible — and why it matters

The visibility gap is structural, not accidental. Web analytics platforms were built for humans: they inject a JavaScript tag, and only browsers that execute JavaScript report back. AI agents and crawlers are built for efficiency: they fetch raw HTML, rarely execute JavaScript, and move fast. Result: every agent visit is a request your server handled and your analytics never counted. The gap is growing as agentic browsers and answer engines multiply — the AI search ecosystem now includes dozens of agent families crawling the open web at scale.

Why it matters is twofold. First, content strategy: if answer engines are surfacing your content, you want to know which pages, how often, and from which agents — that is the raw material of AEO, the discipline of optimizing content for AI answers rather than just ranked links. Second, infrastructure: agents consume crawl budget, bandwidth, and origin compute, and a misconfigured agent loop can hammer your site harder than any human traffic spike. You cannot manage either problem — content surfacing or infrastructure load — until you can see the traffic. That is exactly what OtterlyAI's launch made visible as a product category, and what this workflow makes operational in your own stack.

Architecture overview

graph TD
  subgraph Ingest[Log Ingestion]
    L1[Raw Server Logs] --> L2[Parser & Normalizer]
    L2 --> L3[(Normalized Events)]
  end
  L3 --> F1[Agent Fingerprinter]
  F1 --> F2{Signals}
  F2 -->|user-agent| UA[Pattern Match]
  F2 -->|IP reputation| IP[ASN Lookup]
  F2 -->|behavior| BH[Behavioral Score]
  UA --> A1[Aggregator]
  IP --> A1
  BH --> A1
  A1 --> A2[(Agent Visits Table)]
  A2 --> R1[AEO Report Generator]
  R1 --> R2[Content Team Report]
  R1 --> R3[Infra Team Report]

The pipeline has five stages. Stage one — raw server logs are parsed into normalized events: timestamp, IP, user agent, path, status code, bytes. Stage two — the fingerprinter scores each event with three signals: user-agent pattern match, IP/ASN reputation, and a behavioral score. Stage three — events above the agent threshold are tagged with an agent family. Stage four — the aggregator rolls visits into an agent-visits table keyed by family, path, and date. Stage five — the report generator produces two views: an AEO report for content teams and a load report for infrastructure teams. The design goal: turn an invisible traffic class into an observable, actionable dataset.

Part 1 — The ingestion and fingerprinting schema

.env

LOG_SOURCE=s3://logs/dailyaiworld/
AGENT_THRESHOLD=0.7
AGGREGATION_WINDOW_HOURS=24
AEO_REPORT_PATH=reports/aeo/
INFRA_REPORT_PATH=reports/infra/
IP_ASN_DB_PATH=./data/asn.csv

schemas.py

from pydantic import BaseModel, Field
from typing import List, Literal
from datetime import datetime

class LogEvent(BaseModel):
    ts: datetime
    ip: str
    user_agent: str
    path: str
    status: int
    bytes_sent: int
    referer: str = ""

class Fingerprint(BaseModel):
    event: LogEvent
    agent_score: float = 0.0      # 0 (human) to 1 (certain agent)
    family: Literal["gptbot", "claudebot", "google-extended", "perplexitybot",
                    "agentic-browser", "unknown-bot", "human"] = "unknown-bot"
    signals: List[str] = Field(default_factory=list)  # which signals fired

class AgentVisit(BaseModel):
    family: str
    path: str
    date: str
    request_count: int
    distinct_ips: int
    avg_bytes: int
    last_seen: datetime

The LogEvent is the normalized unit — one request, one row. The Fingerprint carries the score, the assigned family, and the list of signals that fired, so every classification is explainable. AgentVisit is the aggregated unit that reports and dashboards consume: family, path, date, request count, distinct IPs, average payload, and last seen. The schema is deliberately small — agent analytics is a high-volume, low-cardinality problem, and the aggregation should happen in the pipeline, not in the reporting layer.

Part 2 — The fingerprinter

tools.py

import re, csv

AGENT_PATTERNS = {
    "gptbot": re.compile(r"GPTBot|ChatGPT-User|OAI-SearchBot", re.I),
    "claudebot": re.compile(r"ClaudeBot|Claude-Web|anthropic", re.I),
    "google-extended": re.compile(r"Google-Extended|GoogleOther", re.I),
    "perplexitybot": re.compile(r"PerplexityBot", re.I),
    "agentic-browser": re.compile(r"AgentBrowser|Kitesurf|BrowseAgent|Agent-", re.I),
}
BOT_ASNS = set()  # loaded from IP_ASN_DB_PATH

def load_asns(path: str):
    with open(path) as f:
        for row in csv.DictReader(f):
            BOT_ASNS.add(row["asn"])

def fingerprint(event: LogEvent) -> Fingerprint:
    score = 0.0
    signals = []
    family = "unknown-bot"

    # Signal 1: user-agent pattern (0.5 pts)
    for fam, pat in AGENT_PATTERNS.items():
        if pat.search(event.user_agent):
            family = fam
            score += 0.5
            signals.append(f"user-agent:{fam}")
            break

    # Signal 2: IP/ASN reputation (0.3 pts)
    asn = lookup_asn(event.ip)
    if asn in BOT_ASNS:
        score += 0.3
        signals.append(f"asn:{asn}")

    # Signal 3: behavioral cues (up to 0.2 pts)
    if not event.referer and event.status == 200 and event.bytes_sent < 500_000:
        score += 0.1
        signals.append("no-referer")
    if "/robots.txt" in event.path or "/.well-known" in event.path:
        score += 0.1
        signals.append("robots-fetch")

    return Fingerprint(event=event, agent_score=min(1.0, score),
                       family=family if score >= 0.3 else "human", signals=signals)

The fingerprinter combines three independent signals so no single one decides the verdict. A user-agent match is the strongest signal and names the family; ASN reputation catches agents that disguise their user agent; behavioral cues (no referer, robots.txt fetches, small payloads) catch the rest. The robots.txt heuristic is sneaky and effective — any well-behaved agent fetches robots.txt first, and humans almost never do. Families are keyed to the visible agent ecosystem, and the pattern table is a config you extend as new agents launch — the same way the MCP directory catalog grows as new tool servers ship.

Part 3 — The LangGraph agent-audit workflow

graph.py

from langgraph.graph import StateGraph, END
from typing import TypedDict, List

class AuditState(TypedDict):
    events: List[LogEvent]
    fingerprints: List[Fingerprint]
    visits: List[AgentVisit]
    aeo_report: str
    infra_report: str

def ingest(s: AuditState) -> AuditState:
    s["events"] = parse_logs(fetch_logs())   # S3 -> normalized events
    return s

def fingerprint_all(s: AuditState) -> AuditState:
    s["fingerprints"] = [fingerprint(e) for e in s["events"]
                         if e.status < 500]  # skip server errors
    return s

def aggregate(s: AuditState) -> AuditState:
    s["visits"] = rollup(s["fingerprints"], window_hours=24)
    return s

def report(s: AuditState) -> AuditState:
    s["aeo_report"] = build_aeo_report(s["visits"])
    s["infra_report"] = build_infra_report(s["visits"])
    return s

g = StateGraph(AuditState)
g.add_node("ingest", ingest)
g.add_node("fingerprint_all", fingerprint_all)
g.add_node("aggregate", aggregate)
g.add_node("report", report)
g.set_entry_point("ingest")
g.add_edge("ingest", "fingerprint_all")
g.add_edge("fingerprint_all", "aggregate")
g.add_edge("aggregate", "report")
g.add_edge("report", END)
app = g.compile()

main.py

if __name__ == "__main__":
    result = app.invoke({})
    print("Agent visits (24h):", len(result["visits"]))
    for v in result["visits"][:8]:
        print(f"  {v.family:16s} {v.path:40s} {v.request_count:6d} reqs")

Run it against a day of logs and the output is the visibility layer OtterlyAI is selling as a product: a table of which agent families hit which pages, how often, and from how many IPs. The workflow is a batch pipeline by default — run it hourly or daily — but the same nodes compose into a streaming version for high-traffic origins that need near-real-time load visibility.

Retry rules: log ingestion retries up to 3 times with exponential backoff on S3 fetch failures (transient network errors), with idempotent re-parsing — re-processing a log chunk must produce the same events. Fingerprinting is deterministic and never retried. Aggregation runs on a fixed window and is recomputed on reprocess, not patched incrementally, so reports are always reproducible from the raw logs. Report generation failures fail the run loudly: a missing AEO report is worse than a delayed one, because teams will act on stale data if the pipeline goes quiet. The same retry philosophy — retry transiently, never silently, fail loudly on outputs — runs through every AI workflow we document.

Part 4 — The AEO report and the two audiences

The report generator produces two views from the same visit data:

For content teams — the AEO report. Which agent families visit, which pages they crawl most, which content surfaces (extractable answers, FAQ blocks, structured data), and which high-value pages agents never visit. The optimization loop is direct: pages answer engines actually read get more investment; pages that matter but never get crawled get structured-data and internal-link fixes. This is the operational version of the agentic SEO discipline — AEO is not a theory, it is a feedback loop, and agent analytics is the sensor.

For infrastructure teams — the load report. Which families consume the most crawl budget and bandwidth, which paths get hammered, and which agents need rate limits or robots.txt adjustments. A runaway agent loop is a real incident class in 2026 — the same way a misbehaving bot loop can burn through an API quota, an agent loop can burn through your origin. The infra report turns that from a mystery into a dashboard.

Both reports share one foundation: the fingerprinting pipeline. If you cannot see agents, you cannot optimize for them or defend against them — the visibility problem is upstream of every AEO and infrastructure decision. The tool surface here — logs in, reports out — is exactly the pattern of a well-scoped MCP server tool: narrow, observable, and auditable end to end.

The production checklist

1. Log everything, keep it raw. You cannot fingerprint what you deleted. Raw server logs are the source of truth for agent visibility — retain them on the same schedule as your security logs.
2. Fingerprint with three signals, never one. User agent alone is spoofable; add ASN reputation and behavioral cues. Explainable scores beat confident guesses.
3. Aggregate before you report. High-volume logs collapse into family-by-path visit rows; dashboards should never scan raw events.
4. Separate the audiences. Content teams get the AEO report; infrastructure teams get the load report. One dataset, two views, zero confusion.
5. Act on the loop. A report nobody acts on is a dashboard. Wire the AEO findings into your content calendar and the load findings into your rate-limit rules.
6. Extend the pattern table. New agents launch every month; add their user-agent patterns to the config and re-run. The catalog discipline is the same as the MCP directory.

Frequently Asked Questions

Q: What did OtterlyAI announce on August 13, 2026?

A: OtterlyAI announced Agent Analytics on August 13, 2026, a feature that reads a website's server log data to report which AI agents and answer engines are visiting the site, what they crawl, and how the content is being used.

Q: Why can't traditional analytics see AI agent traffic?

A: Traditional web analytics rely on JavaScript tags, and most AI agents and crawlers do not execute JavaScript. Server logs capture every request, agent or human, which makes them the reliable source for agent visibility.

Q: How do you fingerprint an AI agent from a server log?

A: Combine three signals: user-agent patterns (GPTBot, ClaudeBot, Google-Extended, PerplexityBot), IP reputation (verified bot ASNs), and behavior (no JS execution, uniform request pacing, deep crawl depth, low interaction with media).

Q: What is an AEO report and what goes in it?

A: An AEO (answer engine optimization) report covers which agents visit, how often, which pages they crawl, what content gets surfaced, and optimization opportunities — the visibility layer for AI search and answer engines.

Q: What should a team do with agent analytics?

A: Two audiences: content teams use it to structure content for answer engines (FAQ blocks, schema, extractable facts), and infrastructure teams use it to manage crawl budgets, rate limits, and content freshness.

Closing thoughts

OtterlyAI's Agent Analytics launch names the category: agent traffic is the new analytics blind spot, and server logs are the window into it. The workflow in this dispatch makes that visibility operational — ingest logs, fingerprint agents with explainable signals, aggregate by family, and generate the AEO and infrastructure reports that turn visibility into action. Content teams get a feedback loop for answer-engine optimization; infrastructure teams get crawl-budget and rate-limit intelligence; and both start from the same raw logs. If you are serious about AI search visibility in 2026, agent analytics is not optional — it is the foundation. Build the pipeline, extend the pattern table, and start optimizing for the audience you could not see before. Track more AEO and analytics builds in the AI workflows library and on latest AI news.