Blog post — Merchant Audit

A well-configured robots.txt file is crucial for steering search engine spiders through your e-commerce store. Improper settings can inadvertently block critical parts of your site from being indexed, impacting visibility and ad performance. This guide covers best practices for managing robots.txt on Shopify, WooCommerce, and other platforms to ensure Googlebot, AdsBot-Google, and other important crawlers can access what they need.

What is robots.txt and Why Does It Matter?

The robots.txt file is a simple text file that lives at the root of your website (e.g., yourstore.com/robots.txt). It's a directive file that communicates with web crawlers, telling them which parts of your site they can or cannot access. Think of it as a bouncer at the door of your website, guiding bots to the right sections and keeping them out of others.

For e-commerce stores, robots.txt is vital for several reasons:

Controlling Indexing: Preventing irrelevant or duplicate content (like identical product variants, internal search results, or cart pages) from appearing in search results.
Managing Crawl Budget: Directing search engines to the most important pages, ensuring efficient use of your site's allocated crawl budget. This is particularly important for large stores with thousands of product pages.
Protecting Sensitive Areas: Shielding administrative URLs or staging environments from public indexing.

While robots.txt can tell crawlers not to crawl a page, it doesn't guarantee that the page won't be indexed if linked from elsewhere. For definitive blocking from search results, use the noindex meta tag. However, googlebot ecommerce activity often starts with robots.txt, making it a critical first step.

Key Directives: User-agent, Disallow, Allow, and Sitemap

Understanding these core directives is essential for any robots.txt configuration:

User-agent: Specifies which crawler the following rules apply to. Common user-agents include:
- User-agent: * (applies to all crawlers)
- User-agent: Googlebot (Google's main crawler)
- User-agent: AdsBot-Google (Google Ads crawler for landing page quality checks)
- User-agent: Googlebot-Image (Google Images crawler)
- User-agent: Googlebot-News
- User-agent: Googlebot-Video
- User-agent: Storebot-Google (for Google Shopping related activities)
Disallow: Informs the user-agent not to access specific URLs or directories.
- Disallow: /admin/ (blocks the entire /admin/ directory)
- Disallow: /search (blocks pages starting with /search)
Allow: Used in conjunction with Disallow to create exceptions, allowing crawling of a sub-directory or file within a disallowed directory.
- Disallow: /private/
- Allow: /private/public-doc.html
Sitemap: Points crawlers to your XML sitemap(s), helping them discover all the important pages on your site.
- Sitemap: https://www.yourstore.com/sitemap.xml

robots.txt for Shopify Stores

Shopify automatically generates a robots.txt file for all stores. You cannot directly edit this file via FTP or your dashboard. Shopify's default robots.txt is generally well-optimized for e-commerce, designed to disallow common problematic URLs while allowing critical product, collection, and static pages.

Here's an example of what a typical robots.txt shopify file looks like:

# We disallow crawling of these paths by default, as they are not useful to store owners.
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /collections/*filter*
Disallow: /collections/*page*
Disallow: /products/*?*
Disallow: /search
Disallow: /policies
Disallow: /*/policies
Disallow: /*/orders
Disallow: /*/checkouts
Disallow: /*/account
Disallow: /apps/store-reviews/
Disallow: /collections/*/
Disallow: /collections/*/*/*

# Allow AdsBot-Google to crawl all pages
User-agent: AdsBot-Google
Disallow:

Sitemap: https://yourstore.com/sitemap.xml
Sitemap: https://yourstore.com/sitemap_products_1.xml
Sitemap: https://yourstore.com/sitemap_collections_1.xml
Sitemap: https://yourstore.com/sitemap_pages_1.xml

Key observations for Shopify's robots.txt:

Disallow: /collections/*sort_by* etc.: Shopify wisely blocks facets, filters, and sorting parameters to prevent duplicate content issues.
User-agent: AdsBot-Google Disallow:: This crucial directive explicitely allows AdsBot-Google to crawl all pages. This is vital for your Google Shopping and Performance Max campaigns, as AdsBot-Google needs to access your product landing pages to verify their quality and relevance. Never block AdsBot-Google.
Sitemaps: Shopify automatically lists your main sitemap and partitioned sitemaps (products, collections, pages). These are dynamically updated.

How to modify (within limits):

While you can't directly edit robots.txt, you can influence indexing behavior using theme customizations and apps.

Meta noindex Tag: For specific pages you want to block from indexing but keep discoverable by users, add a noindex meta tag within the <head> section of your theme.liquid file or specific template files.
- Example: To noindex specific collection pages based on a tag, you might add:
```
{% if template contains 'collection' and collection.handle contains 'private' %}
<meta name="robots" content="noindex">
{% endif %}
```
X-Robots-Tag HTTP Header: Use apps or custom code to add X-Robots-Tag headers for certain content types, which can act similarly to noindex but is sent in the HTTP header instead of the HTML.

robots.txt for WooCommerce Stores

WooCommerce stores, being built on WordPress, offer more flexibility in managing robots.txt but also demand more careful configuration. WordPress automatically generates a basic robots.txt, but it's often insufficient for e-commerce.

Default WordPress robots.txt (example):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.yourstore.com/sitemap_index.xml

This default is very minimal. You'll need to expand on it.

Best practices for WooCommerce robots.txt:

Using SEO Plugins: Plugins like Yoast SEO or Rank Math allow you to edit your robots.txt directly from your WordPress dashboard. They also generate comprehensive sitemaps linked in robots.txt.

Disallowing Admin and Internal Paths:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /feed/
Disallow: /*?add-to-cart=* # Blocks add-to-cart URLs
Disallow: /*/feed/$
Disallow: /*/comment-page-*
Disallow: /*?c=
Disallow: /*?s= # Prevents internal search results from indexing
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /product-category/*?filter_*
Disallow: /product-tag/*?filter_*
Disallow: /*?orderby=*
Disallow: /*?filter_*

Explicitly Allowing AdsBot-Google: Just like Shopify, ensure AdsBot-Google isn't blocked. If you use Disallow: / for User-agent: *, you must then explicitly allow AdsBot-Google.
```
User-agent: AdsBot-Google
Allow: /
```
Include Sitemaps: Always include the URL to your main sitemap (e.g., Sitemap: https://www.yourstore.com/sitemap_index.xml if using Yoast SEO).

Table: Common WooCommerce Disallows

Path to Disallow	Purpose	Example Result
`/wp-admin/`	Prevent indexing of backend areas	`yourstore.com/wp-admin/`
`/wp-includes/`	Prevent indexing of core WordPress files	`yourstore.com/wp-includes/css/`
`/?add-to-cart=`	Block duplicate URLs from cart actions	`yourstore.com/product/xyz/?add-to-cart=123`
`/cart/`	Cart page	`yourstore.com/cart/`
`/checkout/`	Checkout pages	`yourstore.com/checkout/`
`/my-account/`	Customer account pages	`yourstore.com/my-account/`
`/?orderby=`	Sorting parameters for categories/products	`yourstore.com/shop/?orderby=price`
`/?filter_`	Product filters based on attributes/price etc.	`yourstore.com/shop/?filter_color=red`
`/*?s=`	Internal search results	`yourstore.com/?s=shoes`
`Disallow: /feed/`	RSS Feeds	`yourstore.com/feed/`

Ensuring Googlebot-Image Accessibility

Many e-commerce stores rely heavily on product images for traffic from Google Images. Ensure Googlebot-Image can access your product images. Typically, if Googlebot is allowed to crawl your product pages, Googlebot-Image will also be able to access the images linked from those pages.

The primary concern is accidentally disallowing image folders, e.g., Disallow: /wp-content/uploads/ (on WooCommerce) or asset domains. Shopify handles image hosting automatically, so this is less of a concern there. For custom setups, double-check that your image directories are not blocked by a broad Disallow: rule.

Testing Your robots.txt

After making any changes (especially on WooCommerce or custom platforms), it's crucial to test your robots.txt file.

Google Search Console robots.txt Tester: Navigate to Google Search Console > Crawl > robots.txt Tester. You can paste your robots.txt content here and test specific URLs for various user-agents (including Googlebot, AdsBot-Google, and Googlebot-Image).
Inspect Live URL: In Google Search Console, use the "URL Inspection" tool for a specific product page. Check the "Crawl" section to see if it reports "Crawl allowed by robots.txt."
Manual Check: Make sure yourstore.com/robots.txt is accessible and displays the correct content.

Remember that changes to robots.txt can take some time to be picked up by crawlers. However, for critical directives like those affecting AdsBot-Google or product feeds, Google typically processes these fairly quickly.

Potential Pitfalls and How Merchant Audit Helps

Misconfiguring your robots.txt can lead to serious e-commerce issues:

Products Not Showing in Google Shopping: If AdsBot-Google is blocked from accessing product pages, your Google Merchant Center feed will likely show disapprovals for "Invalid URL" or "Landing page not accessible." This directly impacts your ability to run Google Shopping Ads.
Important Pages Not Indexed: Accidentally blocking categories, product pages, or even your homepage can severely damage your organic search visibility.
Crawl Budget Waste: If you don't block irrelevant pages, crawlers might spend too much time on low-value content, potentially delaying the indexing of your new products or important updates.

Merchant Audit scans your Shopify or WooCommerce store continuously, identifying these and many other Google Merchant Center compliance issues. It can flag if AdsBot-Google is blocked by robots.txt, if your sitemap is missing, or if noindex tags are incorrectly applied. This proactive monitoring ensures your products remain discoverable and eligible for advertising.

FAQ

Q: Can I really edit robots.txt on Shopify? A: No, you cannot directly edit the robots.txt file on Shopify. Shopify generates and manages it automatically. You can, however, use liquid code in your theme files to add noindex meta tags to individual pages or templates if you need to prevent specific pages from being indexed.

Q: What is the most critical user-agent to ensure is NOT blocked for an e-commerce store? A: AdsBot-Google is arguably the most critical to allow access for e-commerce. Blocking it will prevent your product landing pages from being crawled for Google Shopping, leading to product disapprovals and issues with your Google Merchant Center feed.

Q: I blocked Googlebot by accident. How long does it take to fix? A: Once you correct your robots.txt file, Googlebot will eventually re-crawl and discover the changes. This can range from a few hours to a few days, depending on your site's crawl rate and how frequently Google crawls it. You can expedite the process by submitting your sitemap in Google Search Console after the fix.

Ensure your robots.txt file is a helpful guide for search engines, not a barrier to your online success.