What AI models actually read on your website in 2026

Last week we ran valiss.co through a tool that measures AI visibility. Our llms.txt scored 9 out of 10. The missing point was for length: the file was too long, because "AI summaries do not truncate it". And the tool recommended HowTo or Article schema on the pages where it made sense. Both points were fair, and both are fixed now. But the remark about length started something. What does an AI model actually read when it visits a website? Not what we hope it reads. What it actually fetches.

These are notes from finding out, on our own site and on the sites we measure for customers. Some of the answers surprised us.

Two kinds of visit, and only one is about you

There are two different ways an AI model comes by your website, and they get mixed up all the time.

The first visit is the training crawler. GPTBot from OpenAI, ClaudeBot from Anthropic, Google's crawler for Gemini training, CCBot from Common Crawl. They fetch pages at scale and use them to train the next model version. What they fetch today shows up in a model months from now at the earliest, and for a company with 40 employees in Herning the effect of that visit is close to zero. The model does not memorise your price list.

The second visit is the one that matters. When a user asks ChatGPT, Perplexity or Gemini for "an accountant for a trades business in Herning", the assistant queries a search index and typically fetches three to ten pages live, right there, while the user waits. That is OpenAI's search crawler and its user triggered fetcher, Anthropic's search crawler and its user agent, PerplexityBot and its user triggered counterpart. That visit decides whether you are named in the answer.

The distinction changes what you optimise. The training crawler does not care about your page. The live fetch reads it under time pressure, with a limited amount of text it is allowed to take along. That is the reader you are writing for.

AI crawlers do not run JavaScript

This is the single most important thing we have learned, and it explains most of the sites that score badly with us without anyone understanding why.

When Vercel analysed AI crawler traffic on its own platform at the end of 2024, the conclusion was simple: none of them executed JavaScript. GPTBot, ClaudeBot, PerplexityBot, all of them fetched the HTML file and stopped there. Googlebot renders the page like a browser and has done so for years. The AI crawlers do not, and that has not changed in any meaningful way since.

It means something concrete. If your website is built as an app, where the content is fetched and drawn by JavaScript in the browser, an AI crawler sees an empty page with a logo and a menu. Your prices do not exist. Your opening hours do not exist. Your page about who you are does not exist. You can have the best copy in your industry and be invisible.

The test takes five seconds: open your site, switch JavaScript off in the browser, and reload. What you see now is what the model sees. That is exactly why we build AI optimised websites as static HTML, where all the content is in the file before any script runs.

What llms.txt can do, and what it cannot

llms.txt is a text file at the root of your domain, written in markdown, that explains to an AI model who you are, what you do, and which pages matter. The proposal came from Jeremy Howard of Answer.AI in September 2024, and today Anthropic, Stripe and Cloudflare, among others, publish one on their documentation sites.

Then comes the uncomfortable part. Google's John Mueller has publicly compared the file to the old meta keywords field, and none of the major assistants have confirmed that they fetch it automatically when answering a question. If someone sells you llms.txt as the one thing that gets you into ChatGPT, they are selling you something that does not exist.

So why do we have one ourselves, and why did we spend an afternoon making it shorter? Three reasons. First, some tools and agents do fetch it, especially those built by developers who know the convention, and their number is growing. Second, it is a discipline: it forces you to write the one sentence you want a model to repeat about you, and that sentence usually ends up on the front page too. Third, it costs nothing.

Length is the point the tool got right. A model that fetches the file takes a limited slice along. If the most important thing sits on line 300, it does not make it. Our own file went from 17 kilobytes to just over 11, and the essentials now sit in the first 20 lines: who we are in two sentences, the four things we work on, and the page to send people to.

Schema, and why we added HowTo even though Google does not show it

Schema is structured data in JSON LD, a small block in the page code that tells a machine what the page is about without having to guess from the text. Organisation, address, product with price, questions and answers, an article with author and date, a procedure in steps.

Here is a nuance most people miss. Google removed HowTo results from search in September 2023, and FAQ results were restricted the same year to government and health sites. Measured by what can be seen in Google, HowTo schema is dead. So why does an AI visibility tool recommend it, and why did we follow the advice?

Because the two things are not the same. Google no longer shows the steps as a rich result, but the structure is still in the page, and a model that fetches the page gets the steps served with name, order, duration and price, without having to infer them from a long text. Our method page has four steps: Target, Plan, Pilot and Embedding. They now sit in schema with "30 minutes, free" on the first. When a model has to answer "how does an engagement with Valiss work", that block is the easiest thing to quote correctly.

The priority order is simple, and it applies to every industry:

• Organization on every page, with address, company registration number, phone and the topics you know about.
• Product or Service with price on the pages that sell something. The Rich Results Test must come up green.
• Article or BlogPosting on everything that has a date and an author.
• FAQPage where you actually answer questions, even though Google no longer shows it.
• HowTo wherever there is a procedure in steps.

Dates, and the test that decides everything

The models have become strikingly sensitive to freshness. Perplexity shows the date on its sources. ChatGPT prefers newer pages when two say the same thing. A page without a date is treated as old, and a page dated 2023 is treated as outdated, even if the content still holds. Set datePublished and dateModified in the schema, and show the date visibly on the page. And only update it when something has actually changed, because a date that moves every night while the content stays still is something the model quickly learns to ignore.

Then the test that ties it together. Ask ChatGPT, Perplexity and Gemini a question the way a customer would ask it, and see whether you are in the answer. We described the method in Ask AI like your customers do. If you are not in it, work the list from here: switch JavaScript off and see whether your site still exists. Check that price, address and the essentials about you are there as text and in schema. Add dates. Write a short llms.txt, and make it shorter again.

Those are precisely the things the Signal audit scores, page by page, together with what others write about you. But the first half you can do yourself this afternoon. The model reads less than you think. Make sure what it reads is the right thing.