Bots

Information for Webmasters

What is Linkdex?

Linkdex is an enterprise-class platform that combines team management software with search engine optimization tools. Sometimes we crawl websites to understand backlink profiles or gather information for our users so they can improve their websites and SEO strategies.

Why have you crawled my website?

Unlike some crawlers, our bots don’t randomly trawl the Internet for information. We only crawl pages which our users deem relevant.

If your site has been crawled, our bots will appear in your analytics/server logs with the following details:

  • linkdexbot v2.0 checks for links between Web pages. This helps us build a fresh, accurate picture of links between websites.
    Mozilla/5.0 (compatible; linkdexbot/2.0; +https://www.linkdex.com/en-us/about/bots/)
  • linkdexbot v2.1 is our more general site crawler that helps marketers detect technical issues with a website’s pages, and works in 2 versions: a standard Web Search and a Mobile Search configuration.
    Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/)
    Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +https://www.linkdex.com/en-us/about/bots/)
  • linkdexbot v2.2 checks the content of pages that are ranked by search engines. This helps us identify who are the influential websites, brands and authors writing about different topics.
    Mozilla/5.0 (compatible; linkdexbot/2.2; +https://www.linkdex.com/en-us/about/bots/)
  • Earlier versions of the bots had the name linkdex.com/v2.0 and linkdex.com/v2.1

Responsible Crawling

Linkdex will respect the crawl frequency limitation you have outlined in your robots.txt file. We also crawl relatively slowly, often waiting up to 5 seconds between pages. This ensures your website’s performance is not affected in any way.

Contact information

If you would like to discuss our site crawlers, please email us at [email protected] with Site Crawler in the subject line.