Meta's new crawler indexes the web for Meta AI answers

Meta has a new way of reading the web, and brands are the raw material. Meta’s developer documentation now lists Meta-WebIndexer, a crawler that navigates the open web to improve the quality of Meta AI’s search answers and, in Meta’s own words, to cite and link to a site’s content in those answers. For multi-location brands it means Meta AI, the assistant inside WhatsApp, Instagram, and Facebook, is becoming a discovery surface that reads their sites directly.

What happened

Meta documents a set of crawlers for webmasters, and the search-focused one is Meta-WebIndexer, which sends the user agent meta-webindexer/1.1. Meta’s guidance describes it as a crawler that navigates the web to improve Meta AI search result quality and says that allowing it helps Meta cite and link to your content in Meta AI’s responses. It sits alongside Meta-ExternalAgent, the broader crawler Meta has run since 2024 for training its models and indexing content, and it can be controlled through robots.txt in the same way.

The strategic backdrop has been reported for some time. The Information first reported in 2024 that Meta was building its own web index to reduce its reliance on Google and Bing for the live results that feed Meta AI. The documented Meta-WebIndexer crawler is the visible edge of that effort: a dedicated indexer whose stated job is to make Meta AI’s answers better and to attribute the sources it draws on.

Allowing Meta-WebIndexer helps Meta cite and link to your content in Meta AI’s responses.

Meta for Developers, web crawlers documentation

Why it matters

This is the same crawl-and-answer pattern that reshaped Google results, arriving on a set of apps where many brands still hold their largest audience. An AI assistant reads public web content, synthesises an answer, and may cite the sources it used. The brands more likely to be cited are the ones with accurate, crawlable, well-structured content. Block the crawler, or leave thin and inconsistent pages behind it, and a brand is far less likely to be surfaced or cited.

It also turns a technical setting into a strategic choice. A rule in a robots.txt file now decides whether Meta AI can represent a brand in its answers. Left to individual site owners, that decision gets made inconsistently across a large estate: some domains eligible for citation, others silently excluded, with no one owning the outcome.

What this means for multi-location brands

The first task is governance. Decide, centrally, whether Meta-WebIndexer is allowed across every property the brand controls, and apply that rule everywhere rather than discovering later that half the estate is unavailable to Meta-WebIndexer while the other half is not. A crawler policy is now part of brand governance, not a webmaster footnote.

The second task is the substance the crawler reads. Meta AI can only cite content that is accurate and structured enough to trust, so keep each location’s local business listing correct and consistent across markets, and manage presence for answer engines rather than for a single page view. Building the discipline to rank in AI search results and a structured presence layer such as Places AI helps a brand stay quotable across the assistants that now assemble answers, Meta AI included. The work that already applies to Google’s AI surfaces now has a Meta dimension, and the same estate-wide consistency is what carries a brand across both.

The bottom line

Meta-WebIndexer makes Meta AI one more answer engine reading the open web and deciding which brands to cite. The lever is unusually direct: a crawler rule and the quality of the content behind it. Enterprise teams that set one deliberate policy across the estate, and keep the underlying data accurate, will be the ones Meta AI has something to say about.