Editing the Shopify robots.txt: What You Can (and Can't) Change

Shopify generates a default robots.txt file for every store at yourdomain.com/robots.txt. Since the rollout of the robots.txt.liquid template, you can edit parts of it directly through your theme code — but not everything. Shopify locks the rules that protect cart, checkout, and account URLs from being edited or removed, while handing you full control over custom disallow rules for collections, filters, and duplicate-content paths. Get this wrong and you can deindex your entire product catalog. Get it right and you clean up the crawl budget waste that Shopify performance and SEO optimization work depends on.

This guide covers what's editable, what Shopify won't let you touch, the mistakes merchants make most often, and how to add safe custom rules without breaking indexation.

How robots.txt Works on Shopify

Every Shopify store ships with a default robots.txt that blocks crawlers from cart, checkout, account, and internal search result pages while allowing product, collection, and blog URLs. This default file lives at the theme level and generates automatically. You don't need to create it from scratch, and most stores run for years without anyone touching it.

Since the robots.txt.liquid editor became available on Online Store 2.0 themes, merchants can open Online Store > Themes > Edit code and find a file named robots.txt.liquid in the templates folder. If it isn't there, add it yourself. Shopify merges your custom rules with its own baseline output rather than replacing the whole file.

The file uses standard Liquid objects like {{ shop.permanent_domain }} and loops through Shopify's default group rules. The output isn't a plain static text file — it's rendered per request, on the fly. That matters because you can add conditional logic with Liquid, but you can also break the file with a single syntax error. That's the most common cause of a robots.txt outage we see on Shopify stores.

What You Can Edit vs. What Shopify Locks

Shopify splits robots.txt rules into two categories: default groups you can override or extend, and a small set of hardcoded disallow rules no theme edit can touch. Knowing the difference before you edit saves wasted effort and protects your crawl coverage.

Shopify's own documentation confirms the platform reserves the right to override certain default groups even after you've edited the file, specifically around checkout and customer account paths. That's a deliberate security and PCI-compliance decision, not a bug. Shopify would rather a merchant lose some SEO control than accidentally expose payment flows to crawler indexing.

The practical upshot for most Mumbai and Delhi-based DTC brands we work with: don't waste time trying to "clean up" the cart or checkout rules Shopify already manages. Spend that effort on what you actually control — collection filters, tag pages, and internal search results. That's where real crawl budget gets wasted.

Common SEO Mistakes Merchants Make

Most robots.txt damage on Shopify comes from copy-pasting rules written for a different platform, or over-blocking in a well-meaning attempt to clean up crawl budget. Three mistakes account for the majority of the indexation drops we see when auditing new client stores.

Blocking Entire Collections by Accident

A wildcard rule like Disallow: /collections/ looks like it targets filtered URLs, but it blocks every collection page on the store, including the main category pages that carry the bulk of organic traffic. We've seen stores lose 40-60% of organic sessions within three weeks of a broad collections block going live. Google simply stops crawling the pages that convert.

Disallowing /products/ Globally

Some merchants block /products/ thinking it reduces duplicate content from variant URLs, but this removes every single product page from Google's crawl queue. Product pages are the primary revenue-driving URLs on any ecommerce site. One misplaced line can take a store off Google for every transactional query it should be ranking for.

Blocking Cart or Checkout That's Already Blocked

Merchants migrating from WooCommerce or Magento often add explicit disallow rules for /cart and /checkout, not realizing Shopify already blocks these by default at the platform level. The redundant rule does no harm on its own, but it's usually a sign the merchant copied rules blind rather than auditing what Shopify actually outputs. That habit tends to cause damage elsewhere in the same file.

Forgetting the Liquid Syntax Is Live Code

Because robots.txt.liquid renders per request, a stray curly brace or an unclosed {% for %} loop doesn't throw a visible storefront error. It can silently corrupt the entire file output, and Google won't tell you — it just stops crawling whatever the broken output disallows. Always preview the rendered file after any change, not just the code editor view.

Step 1 — Audit the Current robots.txt Output

Visit yourdomain.com/robots.txt in a browser before making any changes and save a copy of the current output as plain text. This gives you a rollback reference, and it shows you exactly which default groups Shopify is already rendering, so you don't duplicate or contradict an existing rule.

Step 2 — Identify URLs That Actually Need Blocking

Pull a crawl report or check Google Search Console's Page Indexing report for URLs marked "Duplicate without user-selected canonical" — these are your real candidates for new disallow rules, not a guess. Typical culprits on Shopify stores are faceted filter combinations like /collections/all?color=red&size=m and internal search result pages at /search?q=. Both multiply endlessly without adding unique content.

Step 3 — Edit robots.txt.liquid Safely

Open Online Store > Themes > Edit code, locate or create robots.txt.liquid, and add your custom rule inside the existing {% for group in robots.default_groups %} loop structure rather than replacing it outright. Removing Shopify's default loop entirely strips out the sitemap reference and the built-in cart and checkout protections. Always extend the file — never overwrite it wholesale.

A safe pattern for blocking filtered URLs while preserving the base collection page looks like this:

Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*?*sort_by*

This targets parameter-stacked filter URLs specifically, leaving the clean /collections/category-name page fully crawlable and eligible to rank.

Step 4 — Test Before Publishing

Preview the theme change and load the staging robots.txt output, checking that your core collection and product URLs still return as allowed, not disallowed. A single misplaced asterisk in a Liquid loop can silently block far more than intended, so diff the before-and-after output line by line before you publish live.

How robots.txt Interacts With Canonical Tags and Sitemap.xml

Robots.txt controls crawling, canonical tags control indexing preference, and sitemap.xml tells Google what you actually want indexed. These are three separate signals, and confusing them is where most technical SEO damage on Shopify stores happens. Blocking a URL in robots.txt prevents Google from crawling it at all — Google can't even see the canonical tag on that page telling it to consolidate ranking signal elsewhere.

Shopify auto-generates canonical tags on filtered collection and variant URLs, pointing back to the clean parent URL. That's often enough to solve duplicate content without touching robots.txt at all. The two tools solve overlapping problems differently: canonical tags let Google crawl a page and then defer ranking signal to another URL, while robots.txt disallow rules stop the crawl before Google ever reads the page.

Because of this, Shopify's sitemap.xml only ever lists canonical, indexable URLs. It never includes cart, checkout, or account paths, and it automatically excludes pages marked noindex. If you've added a disallow rule for a URL pattern that's still appearing in your sitemap, Google receives conflicting signals: submitted for indexing, but blocked from crawling. That conflict usually resolves in Google's favor over time, but it wastes weeks of ambiguity your store doesn't need during a launch or migration window.

Robots.txt During a Platform Migration

Migrations are the single highest-risk moment for robots.txt errors, because merchants often carry over rules from their old platform without checking whether they still apply. A store moving off WooCommerce, Magento, or WordPress typically inherits a robots.txt built for a different URL structure. Pasting those old rules into robots.txt.liquid can block URLs that don't even exist on the new platform, while missing the ones that do.

The safer approach on any migration is to start from Shopify's default output, add only the custom rules your crawl audit justifies, and verify against your new sitemap before the old domain redirects go live. We build this check into every Shopify migration we run, precisely because a broken robots.txt during a domain cutover can cost weeks of recrawl time you don't get back.

When to Get Expert Help

A Liquid syntax error in robots.txt.liquid can silently deindex thousands of URLs without throwing any visible storefront error. That's why we treat this file as a change-controlled asset on every migration and SEO optimization engagement. If your store carries more than a few hundred SKUs or runs a faceted navigation system, the interaction between filters, canonicals, and crawl budget gets complex fast, and manual edits get risky without a testing process.

Before touching robots.txt on a live revenue-generating store, run a full technical audit against your current index coverage, crawl stats, and sitemap so you know your baseline first. We cover the exact checklist for this in our companion guide on running a technical SEO audit for Shopify stores, which pairs directly with the fixes here.

Frequently Asked Questions

Can I completely remove Shopify's default robots.txt rules?

No. Shopify always renders its default groups even if you edit robots.txt.liquid, and core protections like the cart and checkout disallow rules can't be removed through theme code. You can add to or reorganize the custom sections around them, but the platform-level defaults persist in the final output regardless of your edits.

Will editing robots.txt fix duplicate content from filtered collection URLs?

Robots.txt stops Google from crawling filtered URLs, which reduces crawl budget waste. Shopify's auto-generated canonical tags already handle most duplicate content signals without any robots.txt change. Combining both is fine, but robots.txt alone won't remove already-indexed duplicate URLs — that requires a canonical tag, a noindex tag, or a manual removal request in Search Console.

Does blocking a URL in robots.txt remove it from Google's index?

No, blocking crawling isn't the same as removing indexing. A URL that was indexed before you added a disallow rule can remain in Google's index indefinitely, since Google can no longer crawl it to see a noindex tag. To actually deindex a page, add a noindex meta tag first and let Google recrawl and drop it, then apply the robots.txt block afterward if you still want to stop future crawling.

Why does my Shopify robots.txt still show a Disallow for /cart even though I never added one?

Shopify hardcodes disallow rules for /cart, /checkout, and /account at the platform level as a baseline security and crawl-efficiency measure, independent of any theme customization. This rule appears on every Shopify store by default and can't be edited out through robots.txt.liquid.

Should I add a Crawl-delay directive to slow down bot traffic?

Googlebot ignores the Crawl-delay directive entirely, so adding it won't reduce Google's crawl rate on your store, though some other search engines and scrapers do respect it. If server load from crawling is a genuine problem, that's usually a sign of a deeper performance issue better solved through caching or app cleanup than through robots.txt rules.