What is Custom Robots.txt

 What is Custom Robots.txt

A custom robots.txt file is a modified version of the standard robots.txt file used to control how web crawlers, or "robots," access and index your website. It allows you to specify which parts of your site should or should not be crawled by search engines and other automated bots.

User-agent: *

Sitemap: https://www.indiastark.com/sitemap.xml

Infrastructure

A robots.txt file contains one or more rules that specify which user agents (web crawlers) can or cannot access certain parts of your site. Here is a basic structure:

Plain text

Copy the code.

User-Agent: [Crawler Name]

Disallow: [URL path that should not be crawled]

Allow: [URL path that can be crawled]

Sitemap: [URL to your sitemap]

Examples

Block all crawlers from the entire site.

Plain text

Copy the code.

User Agent: *

Do not allow :/

Allow all crawlers to access everything.

Plain text

Copy the code.

User Agent: *

Do not allow:

Block a specific crawler.

Plain text

Copy the code.

User Agent: BadBot

Do not allow :/

Block access to a specific folder.

Plain text

Copy the code.

User Agent: *

Do not allow: /private-folder/

Allow access to specific folders while blocking others.

Plain text

Copy the code.

User Agent: *

Disallow :/

Allow: /public-folder/

Define a Sitemap.

Plain text

Copy the code.

Sitemap: http://www.example.com/sitemap.xml

Suggestions

Place the robots.txt file in the root directory: it should be located at the root of your domain (for example, http://www.example.com/robots.txt).

Be specific: Make sure your Disallow and Allow instructions are precise to avoid unintentional blocking or allowing.

Case sensitivity: URLs in the robots.txt file are case sensitive, so ensure consistency in URL paths.

Use wildcards carefully: Wildcards like * and $ can help define patterns, but use them correctly to avoid accidentally blocking content.

Testing

You can use different tools to test your robots.txt file:

Google Search Console: To test and validate how the Google bot reads your robots.txt.

Online Tools: Various online tools can help you correct the syntax and effectiveness of your robots.txt file.