Can ChatGPT read and cite
a website? Three checks
Customers are finding businesses through AI answers. Nelson Lee’s guide examines three basics that help make website content accessible: text in the HTML, access for search crawlers and agent fetchers, and prices, locations and booking details in plain text. Individual checks and a combined audit prompt are included for Claude Code.
From AI answers to website visits
At a cooking class in Crete in summer 2026, independent writer Nelson Lee met a couple who had flown in from Martinique. They had found the class through ChatGPT. Everyone else named Google or a friend. Lee opened his September 11, 2026 guide with this encounter, arguing that more customers would find businesses this way.
ChatGPT can attach source links to its answers. These links are citations; clicking through to the website counts as an AI referral visit.
Data: traffic analytics company Similarweb, 2026 Generative AI Landscape report (official). The same report gives 22.6 percent for Travel on a US desktop basis; the source article rounds to “about 23 percent”.
The practical question is whether a website can be read by the assistant doing the citing. In Lee’s judgment, most small-business sites fail at least one of three checks: the text must be present in the HTML, the page’s source file; the AI crawlers, the programs AI companies send to read the web, must be allowed in; and the useful specifics must be written as plain text.
Each check comes with a prompt for Claude Code, Anthropic’s AI coding assistant, which runs in a computer terminal and can fetch pages, read files and run commands. A combined audit prompt brings the checks together at the end.
Most AI crawlers download the source and stop
A browser first downloads the page’s HTML, then runs its JavaScript. On many modern sites, that code fills the page with text and images. The browser renders the result as the finished page on screen.
Most AI crawlers download the HTML and stop, without running JavaScript.
A December 2024 analysis by website hosting company Vercel and search engine optimization (SEO) consultancy MERJ examined 569 million GPTBot fetches in a single month across Vercel’s network. Vercel’s “The rise of the AI crawler” reported that none of the major AI crawlers rendered JavaScript. Googlebot, Google Search’s crawler, does. GPTBot, ClaudeBot and PerplexityBot do not.
<!DOCTYPE html>
<html lang="en">
<head>
<title>Cooking Class</title>
<script src="/app.js"></script>
</head>
<body>
<div id="root"></div>
</body>
</html><!DOCTYPE html>
<html lang="en">
<head>
<title>Cooking Class in Chania</title>
</head>
<body>
<h1>Cooking Class in Chania, Crete</h1>
<p>A four-hour class in a village kitchen.
Cook five Cretan dishes, then eat them
with local wine.</p>
<ul>
<li>Price: €95 per person</li>
<li>Duration: 4 hours</li>
<li>Includes: lunch and wine</li>
<li>Book: hello@example.com</li>
</ul>
</body>
</html>Figure 2 · The same class page built two ways, as a crawler sees it through curl · adapted from the source article’s two illustrations (illustrative, not a real site)
The AI-generated summaries shown above Google’s conventional results, AI Overviews, sit on top of Google’s normal search index. Because Googlebot renders JavaScript, this first problem affects ChatGPT, Claude and Perplexity more directly. The next two fixes apply to all of them.
Get the content into the HTML
The first check asks whether the words on the page are also in its source. Lee offers a thirty-second test that needs no specialist tools, plus alternatives using JavaScript settings or the command line.
The simplest method is to open the homepage and choose View page source from the browser menu. In Chrome, use Cmd+Option+U on a Mac or Ctrl+U on Windows. Search for a sentence visible on the page. If it appears in the source, the page passes this test. If the source is a short file filled with script tags and the sentence is absent, it fails.
Another method is to temporarily switch off JavaScript. In Chrome, open the built-in developer panel, DevTools, press Cmd+Shift+P, type "Disable JavaScript", press Enter and reload. If the page becomes blank, that is what a crawler that does not run JavaScript sees.
A command-line alternative uses curl, which downloads a page without running its code:
curl -s https://yoursite.com | grep -c "a sentence from your page"A zero means the sentence is not in the HTML.
Sites built on the common platforms Shopify, Squarespace and WordPress generate their pages on the server before sending them to visitors, so they pass by default. Lee points to hand-built sites as the place where this goes wrong: a site built with the developer framework React, started with the Vite or Create React App tooling in its default form, ships a single empty container and builds the page entirely inside the browser.
Lee’s fix is server-side rendering or a static export. Either the server produces the content when the page is requested, or finished HTML files are created in advance. The frameworks Next.js, Astro and Nuxt do this by default. Another option is prerendering, which runs a JavaScript page beforehand and saves the completed result as HTML.
Compare what an AI crawler sees on [URL] with what a person sees.
Fetch the page with curl and no JavaScript, extract the visible text
and count the words. Then load the same page in a headless browser
with JavaScript on and do the same. Show me both counts and list any
content that only appears after JavaScript runs. If the raw HTML is
mostly an empty container and script tags, say so plainly.This prompt downloads the page with curl and no JavaScript, counts its words, then loads it with JavaScript enabled in a headless browser, one that runs in the background with no window. It compares the two word counts and lists anything that appears only after JavaScript runs.
Allow access for search and agent crawlers
Once the content is in the HTML, the next check is whether crawlers can reach it. A site can place a plain-text file at yoursite.com/robots.txt to specify what crawlers may read. Crawlers identify themselves using a User-agent name, and the file sets rules for each name.
Lee traces one problem to 2023 and 2024, when many sites blocked AI crawlers wholesale to keep content out of model training. Those rules can also catch the crawlers that find sources for AI answers.
Search crawlers
Index the site in advance so it can be cited later.
- OAI-SearchBotChatGPT
- Claude-SearchBotClaude
- PerplexityBotPerplexity
Agent fetchers
Fetch a relevant page in real time in response to a user’s request.
- ChatGPT-UserChatGPT
- Claude-UserClaude
- Perplexity-UserPerplexity
Training crawlers
Collect content to train models.
- GPTBotOpenAI
- ClaudeBotAnthropic
- CCBotCommon Crawl
- Google-ExtendedGoogle
Figure 3 · The source article’s three crawler groups and names. Lee recommends allowing search crawlers and agent fetchers, while leaving training access to the owner.
Lee’s conclusion is that search crawlers and agent fetchers must be allowed for a site to appear in answers. Whether to allow training crawlers remains the site owner’s choice. The following robots.txt contains the 6 allow-list entries for the first two groups:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /For sites on Cloudflare there is another gate in front of robots.txt. Many websites route their traffic through Cloudflare for speed and protection before it reaches the site itself. Its AI Crawl Control panel classifies AI traffic as Search, Agent or Training and allows each category to be permitted or blocked.
On July 1, 2026, Cloudflare announced defaults taking effect on September 15, 2026. For new domains, new sites added to existing accounts, and free-plan accounts that have not changed their settings, Training and Agent crawlers are blocked by default on pages that display ads. Search remains allowed.
Mixed-purpose crawlers require particular care. Googlebot, Applebot and Bingbot, the search crawlers operated by Google, Apple and Microsoft, can be used for both search indexing and training-related purposes. Cloudflare evaluates them under the most restrictive applicable rule. Blocking Training can therefore block Googlebot as well. Lee’s instruction is to check the setting under Security in the Cloudflare dashboard and keep Search and Agent allowed.
Cloudflare
Traffic passes through this acceleration and protection layer. AI Crawl Control sets access separately for Search, Agent and Training.
robots.txt
A text file at the site root specifies what each named crawler may read.
Your page
Once the request reaches the site, the crawler reads the HTML content.
Figure 4 · The access path for a site using Cloudflare. Its protection settings and robots.txt need to be checked separately.
One point not covered in the source article comes from OpenAI’s and Perplexity’s official documentation: ChatGPT-User may not be governed by robots.txt, and Perplexity-User generally does not read it. These user-triggered fetches differ from automated search indexing; robots.txt alone does not establish whether they can access a site.
Lee also points to settings on two website platforms. Squarespace has a toggle under Settings for blocking known AI crawlers; he recommends leaving it off. Shopify allows robots.txt changes through its robots.txt.liquid template, which may contain blocking rules added by a previous developer.
The prompt reads robots.txt and reports whether each of 9 crawler names is allowed, disallowed or not mentioned. It then requests the homepage as OAI-SearchBot and ClaudeBot and reports the HTTP status, the response code returned by the site. A 403 means the request was refused and something in front of the site is blocking bots. A 200 means the page was served normally, but it does not prove the crawler is allowed because some firewalls also verify the crawler’s IP address.
Fetch [URL]/robots.txt. For each of GPTBot, OAI-SearchBot, ChatGPT-User,
ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User
and Google-Extended, tell me whether it is allowed, disallowed, or not
mentioned. Then fetch the homepage twice more, once with the User-Agent
set to OAI-SearchBot and once as ClaudeBot, and report the HTTP status
each time. A 403 means something in front of the site is blocking bots.
A 200 does not prove the bot is allowed, because some firewalls verify
IP ranges, so note that.Write the specifics in plain text
An accessible page still needs to supply the facts an answer calls for. For a request such as "cooking class in Chania for two people", ChatGPT needs the price, duration, what is included, location and booking method.
authentic culinary experiences in the heart of Crete
The promotional line leaves out the price, duration, precise location and booking method.
Four hours. €95 per person. Five dishes, lunch and wine included. Village kitchen twenty minutes from Chania. Book by email or WhatsApp
Price, duration, inclusions, location and booking method are all stated in text.
Lee lists six details that most local businesses need to state:
- Price, in a currency, per person or per unit
- Duration or opening hours
- What is included and what is not
- Where it is, town name in text, not only on a map embed
- How to book, the actual email, phone or link
- A short FAQ, the questions people ask on the phone
Lee gives two rules for presenting these details: put them in text, preferably in complete sentences. Prices shown only in an image or PDF menu are invisible to these crawlers. "Price: €95" works, but "A four-hour class costs €95 per person, including lunch and wine" is easier to quote directly in an answer.
Lee also suggests an optional addition: structured data to make the details unambiguous. A small JSON-LD block in the page head, the setup section of the HTML, describes the business in a format machines can read. LocalBusiness, Product and Event are available types for a local business, product or event. This is not required for the preceding fixes to work.
This prompt takes the page text and a one-sentence description of the business, identifies every customer question whose answer is missing or vague, then rewrites the page in plain text. Each fact becomes a complete sentence an AI assistant could quote, while retaining the owner’s tone and adding no new claims.
Here is the text of my page: [paste]. The business is [one sentence].
List every fact a customer would ask about that is missing or vague:
price, duration or hours, what is included, location, how to book.
Then rewrite the page in plain text so each of those is a complete
sentence an AI assistant could quote directly. Keep my tone. Do not
add claims I did not make.The combined audit, with llms.txt as an option
In 2024 someone proposed a file called llms.txt that would describe a website to language models in markdown, a plain-text format with simple markers for headings and lists. Much current SEO advice presents it as mandatory. In Lee’s view, it is not.
As of summer 2026, no major AI company had confirmed that its search crawlers read the file. John Mueller, a public voice on Google’s Search team, has also said that server logs show AI systems are not requesting it. Lee’s advice is to do it last, and not to expect it to do the work of the three fixes.
Audit this website for AI search visibility: [URL]
Run these checks and show your work.
1. Raw HTML. Fetch the URL with curl and no JavaScript. Extract the
visible text. Report the word count, the title, the H1 and H2s, and
whether the body is mostly an empty container with script tags.
2. Rendered text. If a headless browser is available (playwright or
chromium), load the same URL with JavaScript on and extract the visible
text. Compare the two word counts and list any content that only appears
after JavaScript runs. If no browser is available, say so and rely on
check 1.
3. Bot access. Fetch /robots.txt. For each of GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot,
Perplexity-User and Google-Extended, report whether it is allowed,
disallowed, or not mentioned. Then fetch the homepage again with the
User-Agent set to OAI-SearchBot, and again as ClaudeBot, and report the
HTTP status each time. A 403 means the edge is blocking bots. A 200 does
not prove the bot is allowed, because some firewalls verify IP ranges,
so say that in the report.
4. Specifics. From the raw HTML text, report whether a price, an address
or town, hours or duration, a phone number or booking link, and a plain
one-sentence description of what the business does are present. Check
for JSON-LD structured data and name the type if found.
5. llms.txt. Check whether /llms.txt exists. Treat it as optional, since
no major AI company has confirmed its crawlers use it. Offer to draft
one from the site's main pages.
Finish with a fix list ranked by impact, one line each, most important
first. Do not fix anything yet.The combined prompt runs five checks: read the raw HTML; compare it with JavaScript-rendered text if a headless browser is available; check access rules for 9 crawlers and test two User-agent names; look for business details and JSON-LD; and check for the optional llms.txt file. It returns a fix list ranked by impact, one line per item, without changing the site. Lee suggests running it on one’s own site first, then on a competitor’s.
Lee closes with the same three checks: content in the HTML, access for search crawlers and agent fetchers, and specific facts on the page. In his view, a business whose site passes is ahead of most of the businesses ChatGPT is choosing between. Any remaining issues appear on the audit’s fix list.