ConfigClarity › Fix Guides › robots.txt › Fix: robots.txt Case-Sensitive Path Issues

Fix: robots.txt Case-Sensitive Path Issues

robots.txt paths are case-sensitive on Linux servers. Disallow: /Admin/ does not block /admin/. If your server serves URLs at both cases or if a URL in your robots.txt does not match the actual case of the path, the directive has no effect.

The Problem

CMS platforms like WordPress serve paths in lowercase by default. But some content management systems or frameworks generate URLs with mixed case (camelCase, PascalCase) or inconsistent capitalisation. If your robots.txt blocks /Products/ but the actual URL is /products/, the Disallow rule never fires.

The Fix

Match exact case of your live URLs
# Check your actual URL structure first:
# curl -sI https://yourdomain.com/Admin/ | grep -i location
# (If it redirects to /admin/, use /admin/ in robots.txt)

# WRONG — wrong case:
# Disallow: /Admin/
# Disallow: /WordPress/wp-admin/

# CORRECT — match actual URL case:
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /wp-login.php

Use ConfigClarity's robots.txt Validator URL Tester to test paths with different cases. Verify your actual URL paths by checking your server's access logs or by using curl -sI https://yourdomain.com/PATH/ to see if the server redirects to a differently-cased URL.

Validate your robots.txt live — fetch any URL and get a corrected file in one click.

Open robots.txt Validator →

Frequently Asked Questions

Are robots.txt directives case-sensitive?
The paths in Disallow and Allow directives are case-sensitive. Googlebot treats /Admin/ and /admin/ as different paths. The directives themselves (User-agent, Disallow, Allow) are case-insensitive.
Does capitalisation matter for User-agent in robots.txt?
No. User-agent, Disallow, Allow, Crawl-delay, and Sitemap directives are case-insensitive. User-agent: Googlebot, USER-AGENT: Googlebot, and user-agent: googlebot are all equivalent.
How do I check the actual case of my site's URLs?
Run curl -sI https://yourdomain.com/YOURPATH/ and check the Location header in the response. If the server redirects to a lowercase URL, use that lowercase path in robots.txt.

Related Guides