The web was built for people who search, click, and read. Agents do none of those things quite the same way. They make many small queries, gather evidence across sources, and return an answer or take an action without necessarily sending a person back to the page. That shift changes who pays for information, who gets found, and what a useful website looks like.
Our view is that better models make retrieval more important, not less. A model can reason over what it knows, but its stored knowledge is an imperfect, aging compression of the world. The details that make an answer useful - this week's regulation, a supplier's live inventory, an obscure technical specification - live outside it. As agents become cheaper and more capable, they will ask for those details more often. The contest moves from the chat interface to the infrastructure that finds, licenses, evaluates, and serves reliable information.
More intelligence, more looking things up
A stronger agent can break a broad task into dozens of specific questions. The better it gets at deciding what it needs to know, the more it has reason to search. This is especially true in law, insurance, science, and industrial workflows, where a plausible answer is not enough: it must be grounded in the right document, updated to the right date, and good enough for an expert to rely on.
This changes the economics of search. Human searches can support advertising; an agent making hundreds of background queries cannot simply inherit that price structure. Parallel argues that search APIs need to become dramatically cheaper as agent usage grows. We think lower prices can expand demand, but that is a hypothesis about usage, not a guarantee of a particular market size. The agent-search figures discussed by Parallel's founder are estimates, and published forecasts define the category differently enough that a precise total would imply more certainty than exists.
Horizontal search is already a crowded, capital-intensive contest. Exa announced a $250 million Series C in May 2026, and large platforms have their own crawls, indexes, and distribution. We think the more interesting question is what must be built around search to make agent answers trustworthy and economically sustainable.
The web's old bargain is breaking
For years, publishers let search engines crawl in return for visitors. An agent can use a page to answer a question without sending that visitor back. That is useful to the person asking, but it weakens the bargain that paid for the page in the first place. Brookings describes a licensing market in which major publishers negotiate one-on-one deals, intermediaries meter and sell access, and smaller publishers largely remain outside the market. It also warns that the companies distributing AI answers may come to control the payment rails for the content they consume.
The rules are already changing. Cloudflare's pay-per-crawl proposal gives site owners a way to allow, block, or charge AI crawlers. The New York Times and Amazon signed an AI licensing agreement. Perplexity launched Comet Plus as one attempt to pay publishers whose work supports answers. These are different mechanisms, not one settled standard. Access to current, high-quality information is becoming an input with negotiated terms.
Exclusive access may improve one index and starve another. It may also draw scrutiny if only the largest publishers can make deals. The missing infrastructure is not merely a payment button. It is credible measurement: which content was used, how it contributed to an answer, and what compensation follows. Publishers need a way to verify those claims, and AI services need a way to buy rights at scale.
Six layers of the agent web
We map the stack from the data source to the agent that acts on it.
Crawl and indexing produce a fresh, searchable copy of the web. This is expensive, scale-sensitive infrastructure, where freshness and coverage matter as much as raw model performance.
Horizontal search APIs package that index for agents. The competition is on relevance, speed, citations, and cost per query. As prices fall, volume may grow, but margins may also compress.
Vertical retrieval connects specialized corpora to specific work: an underwriting file, a clinical study, a regulatory update. Here a general index is only a starting point. The product needs domain-specific quality tests and a place in the workflow where the answer is used.
Data rights and payout rails determine who can use a source, on what terms, and how value returns to its owner. The hard problem is attribution and settlement across many contributors, not only access control.
Agent experience, or AX, is how a site or service works for machine visitors. Netlify describes AX as the counterpart to user experience: can an agent discover what is offered, understand the terms, use the right interface, and complete its task? Analytics, structured endpoints, and tests for agent behavior will matter more than a page that merely looks good to a crawler.
Always-on monitoring lets an agent notice meaningful changes without repeatedly searching everything. It is valuable in regulation, supply chains, and competitive tracking; whether it becomes a separate company or a feature of retrieval is still open.
The layers interact. A cheaper search result is not useful if it misses the decisive source. A licensed source creates little value if the answer cannot be evaluated. A site optimized for agents still needs to know which visits produced real outcomes.
Where new companies may emerge
In vertical retrieval, we are interested in a tight loop between a hard-to-access corpus, expert evaluation, and actual use. The relevant test is not whether a demo finds documents. It is whether domain experts can measure answer quality on their own tasks, whether production use improves the system, and whether the value survives cheaper general-purpose search.
In content licensing, the opportunity is a neutral measurement and settlement layer. Large bilateral deals do not solve the problem for a regional newspaper or a specialist database. Aggregating rights from the long tail could make indexes more complete while giving smaller sources a way to participate, but only if attribution is credible and the economics work for both sides. Collective licensing or statutory rates could reshape this market, so a business cannot assume today's private deals will set permanent terms.
AX is earlier still. Most analytics tell a site what a human clicked. Agent traffic asks different questions: which agents arrived, whether they could parse the offering, what they cited, where they failed, and whether a machine-initiated interaction turned into a useful outcome. We expect teams to treat those questions as a product discipline rather than another SEO checklist.
We are less convinced by undifferentiated consumer answer engines or another general web index on its own. Distribution and crawl costs favor large incumbents. We also would not build on a simple return to referral traffic. The new bargain has to work even when an answer is consumed away from the original site.
What could make this wrong
Google has the index, the model, and distribution. If it bundles high-quality agent search cheaply enough, independent search providers may struggle before higher query volume pays off. Agent adoption itself may grow more slowly than expected. Search prices could fall faster than usage grows. The licensing market might be set by law rather than by private infrastructure, reducing what an intermediary can earn. And exclusive data deals may raise barriers while inviting antitrust or compulsory-licensing responses.
There are concrete signals to watch. Can a vertical retrieval product show expert-measured quality that a horizontal API cannot match? Can content owners and AI services agree on a verifiable pay-per-use or contribution-based agreement? Can search prices fall while providers retain healthy margins? Conversely, broad bundling by Google, binding rate-setting in a major market, or sustained stagnation in agent queries would weaken this thesis.
The point is not that every website becomes an API or that every publisher signs an exclusive deal. It is that an agent cannot do useful work without reliable information, and reliable information cannot remain free to collect, opaque to measure, and unsupported by the people who create it. The next web will have to make all three sides work together: retrieval quality, data rights, and agent experience.
Sources
- Brookings, Same gatekeepers, new tollbooths in the AI content licensing market
- Cloudflare, Introducing pay per crawl
- Reuters, The New York Times and Amazon sign an AI licensing deal
- Perplexity, Introducing Comet Plus
- Netlify, Agent Experience
- Exa, Series C announcement
- Digiday, AI Overviews and publisher referral traffic