Server-rendered pages and AI search: what crawlers see
If your content only appears after JavaScript runs, some crawlers never see it. How to check what a crawler receives, and the machine-readable files that help.
- Check what a crawler receives: view the page source, or fetch it with JavaScript off.
- Client-side rendering can hide content from crawlers that don't run scripts.
- A sitemap, llms.txt and structured data help machines find and read your pages.
- None of this guarantees ranking or citation. We haven't measured either on our own sites.
Quick answer
Search engines and AI crawlers read the HTML your server sends. If your text is in that HTML and visible without running a script, they can read it. If a script builds the page, or hides it until you scroll, some crawlers see less than you do. Check with three tests: view source, turn JavaScript off and fetch the page plainly.
The short answer
Your page has two versions: the one a person sees after the browser has run every script, and the one a crawler gets from the first response. If those differ, you have a visibility problem you can’t see from a normal browser.
Google runs JavaScript, with delays and limits. Other crawlers vary, and some fetch the HTML and read only that. Don’t build a business on a crawler running your script. Put the content in the HTML and treat scripts as an enhancement.
Why buyers ask this
You may have heard that AI tools now answer questions your buyers used to Google, and that being cited matters. You may have a site built as a single-page app, or a template with fancy animations, and wonder whether those tools can read it. The honest answer is that it depends on how the page is built, and you can find out in ten minutes.
Three things a crawler might see
| Case | What arrives in the first response | What a script-less crawler reads |
|---|---|---|
| Client-side single-page app | An empty shell and a bundle | Almost nothing |
| Server-rendered page | The full text, in the HTML | Everything |
| Server-rendered, but hidden by style until a script runs | The full text, with a style that hides it | The text is in the HTML, but a tool that renders the page may show it blank |
The third case surprises people. We found it on our own site.
What we found on ours
On CrawlTheory, our SEO publication, a root-cause analysis in June 2026 found that the public pages were server-rendered: the headings and articles were present in the HTML. But every content block carried a style that set it invisible, and a script revealed each block as it scrolled into view. A crawler that doesn’t run the script, or runs it but never scrolls, can end up with sections at zero opacity. Search tools that take a screenshot of the rendered page can see a blank.
The fix direction is straightforward: show content by default, and let the animation be an enhancement. As of 26 October 2026 the rule is still in our stylesheet, and we are saying so because it is exactly the sort of problem that hides in a good-looking site.
We have also diagnosed the more serious version, client-side rendering hiding content from AI crawlers, on a client’s blog and on our own.
The three checks
Do these on your own site, on a page that matters.
- View the page source. Not the inspector, which shows the page after scripts have run. In a browser, use “view source”. Search for a sentence from your page. If it’s there, the server sent it.
- Turn JavaScript off and reload. Does the page still say what you sell? Are your headings and first paragraphs visible, or does it look blank?
- Fetch the page plainly with a command-line request, then search the output for the same sentence. This is closest to what a basic crawler sees.
If any of them fails, treat it as a bug in the build, not a quirk.
What to put in the HTML
- The text, headings and first paragraphs, in the first response.
- Real links, with proper anchor tags a crawler can follow, not click handlers.
- Titles, descriptions and canonical tags set on the server for each page.
- Structured data (JSON-LD) that matches the page, generated from the same fields.
- A sitemap built from published content, so new pages are listed without anyone remembering.
- A short standalone answer near the top. Tools that quote pages favour text that makes sense out of context.
llms.txt: what it is and isn’t
An llms.txt file is a plain-text map of your site for AI tools: a title, a short summary, and a list of your best pages with one-line descriptions. We generate ours from the published content, so it can’t drift from the site, and the title and summary are editable in the admin.
Be careful with expectations. It is a convention, not a standard that every tool honours, and we have no measured evidence that it changes whether or how often we are cited. It’s cheap to generate and harmless to publish. It is not a substitute for readable pages.
Rendering choices, in plain words
| Option | In practice |
|---|---|
| Static generation | Pages are built ahead of time. Fast and crawler-friendly. Best for content that changes rarely |
| Server rendering | Pages are built when requested. Good for content that changes often |
| Incremental refresh | Static pages that rebuild on a schedule or when content changes. A good default for blogs |
| Client-side only | The browser builds the page. Fine behind a login, risky for anything meant to be found |
What to ask your developer or agency
- Where is each page rendered: on the server, ahead of time or in the browser?
- If I turn JavaScript off, what do I see?
- Does any content start hidden and get revealed by a script?
- How are the sitemap, titles, canonical tags and structured data produced, and what happens when someone adds a page?
- How do you know, and can I check it myself?
An answer of “it works in Google” isn’t an answer. Ask for the source and the plain fetch.
This explains how pages are read. It doesn’t show that fixing rendering raises rankings or AI citations, and we’ve recorded no such result for our own sites. Behind a login, none of it matters, because crawlers can’t see those pages anyway. A small, static, hand-written site is already in good shape.
Next: moving a React app to Next.js without losing search traffic and will the site get found: SEO and AI search.
The CrawlTheory findings come from our own root-cause analysis dated 17 June 2026 and a re-check of the stylesheet and sitemap route on 26 October 2026. The llms.txt description comes from our route code. The client-side-rendering diagnosis on a client’s blog and our own comes from our project notes. How other crawlers behave varies and changes, and we have not tested each one, so verify against current documentation for any crawler you care about. Proof grade 4: our own work.
More by GarunaCDX
Let's talk about your build.
Tell us what you are building. A founding engineer replies within one working day with how we would approach it.
