Key takeaway: Google says robots.txt governs compliant crawling, not later content use, so AI marketing teams need separate visibility and outcome checks.
Key Takeaway
If the practical question is whether adding a robots.txt rule lets a brand control whether it appears in AI search answers, the available evidence supports a limited answer: no. Google Search Central describes standard robots.txt directives as permissions for compliant crawlers, not as a complete rule for how content may be used after it has been crawled. Google-Extended can express preferences related to the use of Google-crawled content for future Gemini model training and for grounding in Gemini Apps and Vertex AI, but it is not a separate crawler and does not affect inclusion in Google Search. Separately, reporting on LLM visibility treats brand mentions, citations, impressions, share of voice, organic search performance, clicks, and sales as different measurements. For an AI marketing team, robots.txt is one part of access governance, not a guarantee of visibility, recommendation, or conversion.
What Google Robots.txt Controls for AI Search
Google’s Gary Illyes said that, under RFC 9309, standard robots.txt directives govern what compliant crawlers are allowed to crawl. A site can target a crawler through its user-agent token and use a Disallow rule for listed paths. Google’s crawler documentation identifies Google-Extended as the token for managing whether content crawled by Google may be used for training future Gemini models and for grounding in Gemini Apps and on Vertex AI. The same documentation says Google-Extended does not affect inclusion in Google Search.
That distinction matters for SEO and AI search operations. Blocking a crawl is not the same as withdrawing content that has already been obtained, and it is not a universal instruction for every later AI system. The source also notes that a proposed Content-Usage rule for expressing content-use preferences in robots.txt remains an Internet Engineering Task Force working-group draft and has not been approved. Teams should therefore avoid treating that proposal as a current standard or expanding one crawler token into a single control for every AI agent.
Google’s separate guidance on temporary PHP patch downtime illustrates another boundary. Google recommended returning a 503 status rather than a general 404 or 200 error page so already indexed URLs can be preserved while the site is offline. Its crawl-rate documentation advises limiting a 503 response to no longer than one to two days. That is a search availability and outage-handling recommendation, not evidence that a 503 or robots.txt rule improves a brand’s likelihood of being recommended in AI search. Crawl access, index preservation, and AI-answer visibility should be tracked as separate operational questions.
How AI Marketing Teams Should Measure Search Visibility
The second source describes LLM visibility as whether a brand appears in an AI-generated answer, which sources support that answer, and how often the brand is represented for relevant questions. The reported measurement framework includes AI mentions, citations, impressions, and share of voice, alongside Google organic visibility, content views, click-throughs, and sales tracked through affiliate links or UTM parameters. These metrics answer different questions. A rise in one does not establish a rise in all the others.
The reported agency results should also be treated as attributed claims from the agency’s media kit, not as Google’s guarantee about AI search rankings. The release says the agency combines Reddit marketing, high-authority editorial placements, and performance measurement. It reports more than 15,000 AI citations, more than 160 million views across ranked discussions, and a health and wellness campaign in which AI mentions increased by 200% over four months. The source also makes clear that platform-level figures do not mean any individual discussion will be cited by an AI system. For a reader evaluating an AI marketing program, the more defensible task is to establish a query-level baseline: whether the brand is mentioned, which sources are cited, how often competitors appear, and whether measurable website activity follows.
This is why SEO and AI marketing need parallel checks. Public discussions and editorial content may create different discovery paths through communities, Google Search, and AI-generated answers. Discovery does not equal citation, and citation does not equal a visit or a sale. Robots.txt can answer part of the question “which compliant crawler may access this path?” It cannot replace checks for relevance, source quality, query coverage, or commercial attribution.
Method and Limitations
A practical review can begin with the site’s robots.txt rules, checking the relevant Allow and Disallow directives and the crawler tokens they target. The team should then compare those rules with the current documentation for Google-Extended and the other platforms being evaluated, keeping the object and scope of each control explicit. Next, create a fixed test set of product, service, and comparison queries. Record whether the brand appears in Google Search, Gemini, or another AI search product, which sources are cited, and whether impressions, clicks, or UTM and affiliate activity can be observed. Reporting should keep crawling, indexing, mentions, citations, traffic, and sales in separate fields rather than turning correlation into causation.
The public material does not establish that any robots.txt configuration can guarantee or prevent a brand’s appearance in every AI search answer. It also does not establish that the agency’s reported method will transfer to every industry, query, or platform. Google notes that user-triggered fetchers occupy an area partly outside the robots.txt framework as Google defines it, so teams need to verify how a specific agent makes requests and which controls apply. Finally, the paid release’s agency-reported numbers should not be presented as an independent experiment, a Google ranking commitment, or a universal industry benchmark.