Robots.txt is a text file that sits on your servers, which webmasters use to instruct web robots (typically search engine bots) how to crawl on the website. It is part of the web standards that regulate how robots crawl the web, access and index content

Why Carry Out This Service?

Search engine shave 2 main jobs:

1. to crawl the web and discover new content &
2. indexing that content so it can be served up to searchers who are looking for information.

Because the robots.txt contain information about how the search engine should crawl your website, the information found there will instruct further bots action on this the site. If you do not instruct the bots and crawlers correctly, your site will not be indexed correctly.

How Is It Done?

We’ll review your websites robots.txt file if it has one and optimise it correctly to which search engines, if not all you want it optimised towards. There are certain rules that should be implemented and therefore we’ll provide you the code for your web development team to implement

When Can It Take Place?

1 Week