Do AI Crawlers Run JavaScript? How to Test Your Own Site

No company running a major AI crawler publicly commits to executing your JavaScript. GPTBot, ClaudeBot and PerplexityBot fetch your HTML. We have not found documentation that promises they render it the way Google's renderer does. So assume your raw HTML is what gets read — then verify it on your own site in about two minutes.

Two versions of your page exist: the file your server sends, and what a browser shows after it runs the scripts inside that file. On a lot of modern sites they are completely different. The server sends a shell of empty divs and script tags; the browser fills it with your headline, your service list, your FAQs.

A browser runs those scripts every time; a crawler only does it if somebody built a rendering pipeline. Google did — it has documented for years that it renders pages. We have not found an equivalent public commitment from the AI vendors. If you find one, it beats this post. That gap is the risk: you can pass in Google and still hand an AI assistant a page with nothing in it.

Some AI answers are grounded on a search index rather than a fresh crawl — Google's AI features run on Google's index, and that index is rendered — so a JavaScript-heavy site can still surface that way. The exposure is the direct fetchers: the crawler that pulls your page itself and reads whatever comes back. Google isn't the only judge anymore.

These are the user-agent tokens publicly documented by the companies that operate them. A documented token tells you what identifies itself at your server. It does not tell you what that fetcher does with the file.

• GPTBot, OAI-SearchBot and ChatGPT-User — OpenAI's documented tokens for training crawl, search indexing and user-triggered fetches. Three jobs, three tokens.
• ClaudeBot, Claude-User and Claude-SearchBot — Anthropic's documented tokens for crawl, user-triggered fetches and search.
• PerplexityBot — Perplexity's documented crawler token.
• Google-Extended and Applebot-Extended — not crawlers at all. They are robots.txt control tokens covering whether content Googlebot or Applebot already fetched may be used in generative products such as Gemini. You will not see them in your logs. Blocking them does not block Googlebot or Applebot, and does not remove you from Search.
• Meta-ExternalAgent, Amazonbot, CCBot — other documented tokens you will see in server logs.

What none of them publishes in plain language is: we execute your JavaScript and read the result. Some documentation is silent on it. Some describes what the crawler collects without saying how the page is processed. Independent researchers publish their own crawl tests, and those are worth reading — but each is a snapshot of somebody else's sample, on a date that has passed, that no vendor has committed to.

Steps 1 and 2 need no tool and no developer. Step 3 takes a terminal, or two minutes of your developer's time. Run it on a page that earns — your furnace repair page, your Barrie service-area page, the post that brings you calls. For a known-good result, view source on thecritterguy.ca first and search for a sentence you can see on the page.

Right-click the page and choose View Page Source, or press Ctrl+U. On a Mac that is Cmd+Option+U in Chrome or Firefox; in Safari, switch on the Develop menu in Settings first, or use the curl command below. What opens is the exact file your server sent — what a non-rendering fetcher receives.

That is not the Elements panel in DevTools. Elements shows the page after your scripts have run and rebuilt it, so it will almost always look complete. Judging crawlability from Elements is how this test produces a confident false pass.

Or from a terminal:

curl -sL -o raw.html https://yoursite.ca/your-page/

Pick a distinctive phrase of five to eight words from the middle of your body copy. Not the page title, not a nav link — those sit in the shell even when the real content does not. Search the raw source for it: Ctrl+F in the View Source tab, or from the terminal:

grep -c "your distinctive phrase here" raw.html

On Windows:

Select-String "your distinctive phrase here" raw.html

Any result above zero is a pass. Pick a phrase with no apostrophes, quotation marks or ampersands: the character in your source may not be the one you typed — a curly apostrophe where you typed a straight one, or an ampersand written as an entity — so your search can miss text that is genuinely there.

Found it? Your content is in the HTML, and a fetcher that does not run JavaScript can read it. Not found, and the source is mostly empty divs, a root div and script tags? Your content only exists after JavaScript runs.

This checks whether your server, your CDN (the caching layer in front of your site) or your bot protection treats crawlers differently from a browser.

curl -sL -A "GPTBot" -o bot.html https://yoursite.ca/your-page/
curl -sL -A "PASTE-YOUR-OWN-BROWSER-USER-AGENT" -o browser.html https://yoursite.ca/your-page/

Copy a full browser user-agent string from your own browser for the second command — a stub like Mozilla/5.0 can trip the same bot rules you are testing for. Compare size and word count. Then repeat with ClaudeBot and PerplexityBot.

Setting a user-agent string does not make you GPTBot. It only shows whether your stack varies its response based on that string. Worth knowing: a firewall rule serving a challenge page to non-browser agents never shows up in a browser, so nobody notices until somebody checks the raw fetch.

Open yoursite.ca/robots.txt and read it too. It will not change what the test returns — robots.txt is an instruction, not a server behaviour — but it is a second way to be invisible.

• You only tested the homepage. Homepages are often hand-built and static while blog and location templates are not. Test at least three page types.
• Your phrase is split by markup. A bolded word or a link inside your sentence breaks the exact string, and so does a line break in HTML that is not minified. If a five-word phrase misses, try three plain words from the same sentence before calling it a fail.
• You tested staging. Test the live URL a crawler would hit, with the live CDN in front of it.

First, page problem or template problem? Run Step 1 on two more pages from the same template. If all three fail, it is the template, and one fix covers every page. If only one fails, it is usually a single block loading its content late.

1. Server-render or statically generate the pages that make money. Service pages, location pages, the posts people find you through. Ask your developer for it in those words: return the page copy in the first HTML response, not after the scripts run.
2. Get critical copy into the first response. The review widget, the FAQ contents, the service description from your CMS — if it arrives in a second request after load, it may as well not exist to a fetcher that does not wait.
3. Stop hiding key content behind an interaction. Tabs and accordions are fine if the text is in the HTML and only the display toggles. "Click to load more" is not.
4. Then check robots.txt and your bot rules. No point rendering perfectly for a crawler you block at the door.

Every item is standard technical SEO and performance work. Server-rendered copy in the first response is what Google indexes soonest — rendering is a second, deferred pass — and it is what stops a visitor staring at a blank rectangle on a slow connection.

It is also how we build. The sites we build ship as static HTML, so the copy is in the markup before anything runs. View source on thecritterguy.ca or batsandwildlife.ca and search for a sentence you can see on the page. Perfect Lighthouse scores are the build standard, not a highlight. More on how we build sites.

Usually not. This is not an argument against JavaScript or frameworks. Server rendering is the default in most current frameworks — Next.js, Nuxt, SvelteKit and Astro all ship pages as HTML unless you opt out. A site can pass this test with no changes at all, which is exactly why you run it before you spend anything.

Passing does not mean you will get cited, either. Readable is the floor, not the finish line, and what happens next depends on whether the page is clear, structured and actually answers the question.

Do not trust a blog post from a year ago — including this one, written on 8 September 2026. Crawler behaviour changes without an announcement, so any claim about what a crawler executes has a shelf life. Trust your own most recent test.

• Re-run it quarterly, on the same three page types.
• Re-run it after a redesign, a framework upgrade, a CDN or firewall change, or a new third-party script.
• Keep a one-line log: date, URL, user agent used, pass or fail, one note.

There is an order of operations, and people routinely start at step four.

1. Can the page be fetched at all? Not blocked, not challenged, no error code.
2. Is your copy in the HTML that comes back? That is the test above.
3. Is it structured so a machine knows what it is describing? That is the schema a service business actually needs — worth nothing on a page that returns an empty shell.
4. Is your business described clearly and consistently enough to be summarized and recommended? That is entity work, plus the file covered in llms.txt for small business — and it decides why ChatGPT recommends your competitor instead of you.

Schema on an empty shell is a beautiful label on an empty box. Fix the box first. The whole sequence is on the AI SEO page.

You will know where you stand in under two minutes. If your sentence is not in the source and you want it fixed properly, use the form. One form, a real reply from the people who will do the work — not a sales rep. Everything is delivered in-house from Mississauga: same time zone, one thread from start to finish, real recourse. Send the URL and what your test returned, and we will tell you straight whether it is a small fix or a rebuild.