Want more qualified Shopify traffic from AI search?
Run a free Naridon scan to see which prompts, products, and AI engines can send more ready-to-buy visitors.
There is a single line in a file most merchants never open that can erase them from AI answers. It lives in robots.txt, and it tells crawlers what they may read. Get it wrong, block the bots that power ChatGPT, Perplexity, Gemini, or Copilot, and no amount of great product content will help, because the engines never see it. Get it right and you clear the precondition for every other GEO effort.
This guide names the bots that matter, explains the one real decision (answering versus training), and shows how to set robots.txt on Shopify without accidentally locking yourself out.
The crawlers that decide your AI visibility
Each major engine relies on specific user agents to fetch your pages. If you want to be cited, these need access.
| Crawler | Run by | Powers | Allow for visibility? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search and live answers | Yes |
| GPTBot | OpenAI | Data gathering, model improvement | Usually |
| PerplexityBot | Perplexity | Perplexity answers and citations | Yes |
| Googlebot | Search and AI Overviews eligibility | Yes | |
| Google-Extended | Gemini and Google AI features | Yes | |
| BingBot | Microsoft | Bing index and Copilot grounding | Yes |
| ClaudeBot | Anthropic | Claude retrieval | Yes |
The pattern is simple: for AI visibility, the default answer is allow. Blocking any of these removes you from the surface it feeds.
The one real decision: answering vs training
The nuance worth understanding is that some crawlers serve two different purposes, and you might feel differently about each. OpenAI splits this cleanly: OAI-SearchBot fetches pages to build the answers users see now, while GPTBot is associated with gathering data that can help train and improve models. Google mirrors it: Googlebot handles indexing, while Google-Extended governs use in Gemini and AI features.
If your only goal is visibility, allow the search and answer bots without hesitation, that is what puts you in the answer. The training bots are the judgment call: allow them if you are comfortable with your content contributing to model improvement, block them if you object. But be clear-eyed about the trade. Blocking the answer and search bots to avoid training is a mistake, because it costs you the visibility, not just the training. Separate the two decisions.
How to set robots.txt on Shopify
Shopify generates robots.txt for you, but it is customizable. You edit the robots.txt.liquid template in your theme to add or remove rules for specific user agents. After saving, verify the live file at yourstore.com/robots.txt to confirm your changes rendered as intended.
Two cautions. First, a broad or careless disallow can catch bots you meant to keep, so make each rule deliberate and re-read the rendered file. Second, robots.txt is not your only gate: a firewall, CDN rule, or bot-management layer can silently block a crawler that robots.txt allows, so if a bot never appears in your logs, check those layers too. If editing Liquid is not comfortable, use an app that manages crawler rules safely rather than risk locking yourself out.
Do the named bots actually obey it?
Yes, and that is the point. The major crawlers from OpenAI, Perplexity, Anthropic, Google, and Microsoft publish their user agents and are built to respect robots.txt. That is precisely why the file is a real control surface rather than a suggestion. Disreputable scrapers may ignore it, but the bots that feed the mainstream answers you want to win honor it. So configuring it correctly is meaningful work, not theater.
Confirm access, then earn the citation
Allowing the crawlers is necessary but not sufficient. Once they can read you, you still have to be the clearest, best-structured answer, which is where schema, factual copy, and llms.txt come in. Access gets you considered; quality gets you cited. The full picture is in the complete guide to GEO for Shopify.
Let Naridon handle the plumbing
Install Naridon free from the Shopify App Store, free forever at $0 with 150 credits a month. It checks that the AI crawlers can actually reach your pages, flags access problems, and then does the harder part, writing the schema, FAQ, and llms.txt that turn access into citations, verified before publish and applied to your catalog. Paid plans from $49/mo with a 7-day trial. See the pricing page.
The bots that power AI answers respect robots.txt, which makes that file one of the highest-leverage things you control. Allow the search and answer crawlers, decide the training bots on their own merits, set the rules deliberately on Shopify, and confirm access in your logs. Then earn the citation with the content behind the door.
Frequently asked
- Which AI crawlers should I allow on my Shopify store?
- If you want to appear in AI answers, allow the crawlers that power live retrieval and search: OAI-SearchBot and GPTBot for ChatGPT, PerplexityBot for Perplexity, Google-Extended and Googlebot for Gemini and AI Overviews, and BingBot for Copilot. These are the bots that let an engine read your pages and cite them. Blocking them removes you from the answers they generate. The nuance is that some of these bots serve both live answers and model training, and you may feel differently about those two uses, but for visibility the default is to allow them.
- Does blocking GPTBot hurt my Shopify store?
- It can. GPTBot is OpenAI's crawler, and blocking it reduces what ChatGPT can read from your site, which can keep your products out of ChatGPT's answers. OpenAI also runs OAI-SearchBot specifically for its search and live-answer features, so if your goal is visibility you generally want both allowed. The one reason to block GPTBot is if you object to your content being used for model training and are willing to trade some AI-answer visibility for that. For most merchants chasing sales, the trade is not worth it.
- What is the difference between GPTBot and OAI-SearchBot?
- They are both OpenAI crawlers with different jobs. GPTBot is associated with gathering data that can be used to train and improve models. OAI-SearchBot is associated with OpenAI's search and live-answer features, fetching pages to build answers users see now. If you care about showing up in ChatGPT's answers, OAI-SearchBot access is the one you especially want. Many merchants allow both; some allow the search bot and block the training bot. Decide based on how you feel about training versus answering, but do not block the search bot if you want visibility.
- How do I edit robots.txt on Shopify for AI crawlers?
- Shopify generates robots.txt automatically, but you can customize it by editing the robots.txt.liquid template in your theme, where you can add or remove rules for specific user agents. Add allow or disallow directives for the AI bots you care about, save, and verify at yourstore.com/robots.txt. Be careful: a broad disallow can accidentally block bots you wanted, so change rules deliberately and re-check the live file. If you are not comfortable editing Liquid, an app that manages this for you is the safer route.
- Should I use Google-Extended on Shopify?
- Google-Extended is a control that governs whether Google can use your content for Gemini and related AI features, separate from normal Googlebot indexing. If you want to appear in Gemini and Google's AI experiences, do not disallow Google-Extended. Disallowing it can keep you out of those AI surfaces while leaving classic search indexing intact. So the choice mirrors the others: allow it for AI visibility, disallow only if you have a deliberate reason to keep your content out of Google's AI features.
- Will allowing AI crawlers slow down my store?
- In almost all cases the crawl load is negligible for a normal Shopify catalog, and Shopify's infrastructure handles it. Well-behaved bots respect crawl-delay and do not hammer your store. If you ever see an aggressive or misbehaving crawler, you can rate-limit or block that specific agent, but that is an exception, not a reason to block AI bots broadly. Do not trade away AI visibility to prevent a performance problem you most likely do not have.
- How do I know if AI crawlers are visiting my store?
- Check your server or CDN logs for the AI user agents, GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, and BingBot, to confirm they are reaching your pages. On Shopify you may need your hosting or a log-analysis tool for this, since raw access logs are less exposed than on self-hosted sites. If you never see a given bot, check that robots.txt is not blocking it and that no firewall or bot-management layer is silently refusing it. Access is the precondition for citation.
- Do AI crawlers respect robots.txt?
- The major, named crawlers from OpenAI, Perplexity, Anthropic, Google, and Microsoft publish their user agents and are designed to respect robots.txt directives. That is exactly why robots.txt is your control surface: allowing or disallowing these agents actually changes what they fetch. Less reputable scrapers may ignore robots rules, but the bots that feed the mainstream AI answers you care about honor it, so setting it correctly is both meaningful and worth doing carefully.
Key concepts
Plain-language definitions of the terms in this guide.
Ready to rank for these conversations?
Join early adopters who are already capturing AI search traffic.