Gatsby robots.txt — Correct Configuration

Gatsby generates robots.txt using the gatsby-plugin-robots-txt plugin or by placing a static file in the /static/ directory. The plugin supports environment-specific configuration — useful for staging vs production.

Method 1: gatsby-plugin-robots-txt (Recommended)

// gatsby-config.js
{
  resolve: 'gatsby-plugin-robots-txt',
  options: {
    host: 'https://yourdomain.com',
    sitemap: 'https://yourdomain.com/sitemap-index.xml',
    policy: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'GPTBot', allow: '/' },
      { userAgent: 'ClaudeBot', allow: '/' }
    ]
  }
}

Environment-Specific: Block Staging

// gatsby-config.js
const robotsPolicy = process.env.NODE_ENV === 'production'
  ? [{ userAgent: '*', allow: '/' }]
  : [{ userAgent: '*', disallow: ['/'] }]

{
  resolve: 'gatsby-plugin-robots-txt',
  options: { policy: robotsPolicy }
}

Validate your robots.txt live — fetch any URL and get AI bot coverage + URL tester.

Open robots.txt Validator →

Frequently Asked Questions

Where does robots.txt go in Gatsby without a plugin?

Place a static robots.txt file in the /static/ directory. Gatsby copies all files in /static/ to the root of the build output, making static/robots.txt accessible at /robots.txt.

How do I block Gatsby staging from being indexed?

Use environment-based configuration in gatsby-plugin-robots-txt with Disallow: / for non-production environments. Or add the X-Robots-Tag: noindex header in your staging server/CDN configuration — this is more reliable than robots.txt for preventing accidental indexing.

What is the correct sitemap URL for Gatsby?

With gatsby-plugin-sitemap, the default output is /sitemap-index.xml (Gatsby 4+) or /sitemap.xml (Gatsby 3 and earlier). Check your gatsby-config.js sitemap plugin output option to confirm.