We blogged about our onsite crawler a while back. After the original announcement we had a group of long standing and trusted users test it fully and help us iron out the wrinkles. Well today our site crawler comes out of beta and is available as part of the live product for all paying users. We have even created a downloadable help guide to get you started.

The response from those trying out the crawler whilst in beta was amazing and for nearly everyone it quickly became the best onsite tool they use. In fact, the feature has also been shortlisted for a European Search Award in the category ‘Best SEO Software’. Because of the crawler’s versatility it is hard – even for us – to comprehend its many uses and applications, so we were delighted when they announced the nomination. Here’s some of the kind things that have been said:

“Having the ability to search through a website’s content by keyword theme using operators like intitle: and inheading: and pinpoint which content has rankings, natural search traffic and third-party links means that in seconds, not hours, we have the data needed to plan and execute the perfect content and link building strategy.” Lisa Myers, CEO, Verve Search

“Technical SEO is important to Razorfish and the new Linkdex crawler has had a huge impact already on the the strategies being shaped and decisions we’re making on how best to leverage our global clients’ web properties.”
David Harling, Head of SEO, Razorfish London

So What’s Special About The New Site Crawler?

I won’t go into the features you should expect an onsite crawler to do. I’ll leave you to read about them in our help guides. Instead I’ll focus on what I consider to be the innovation.

As well as discovering website issues, we wanted to make it easy for you to discover where your performing content and under-performing content is and make informed decisions on how to improve pages and site architecture, not just generally but by keyword themes!

A Site Crawler With A Search Engine Attached

You’re probably used to using search operators like intitle: so we’ve utilized the operators you know and added a few you’ve never been given to allow you to search through content on your website.

  • intitle:keyword
  • inurl:keyword
  • inheading:keyword
  • inintanchor:keyword
  • inextanchor:keyword
  • indesc:keyword
  • infolder:keyword
  • inalt:keyword
  • inpage:keyword

So if you had a financial services comparison website and wanted to find all of your site pages on a specific subject like “credit cards” you might use intitle:”credit card” inurl:”credit card”.

You could then narrow the list further by finding pages that have keywords in internal or external anchors.

To do this we have to hold a copy of your page’s content in our databases.

Relative Page Performance

Finding your content on specific keyword topics is a great start. Understanding which content is performing is the next step to optimizing your website.

To allow you to do this we provide page level information on:

  • Top 10 Ranks – Number of keywords being rank tracked in the top 10
  • Direct Natural Visits – Natural search visits entering at a specific page URL
  • Phrases Providing Visits – Number of unique phrases providing natural search visits to a specific page URL (a page’s keyword reach)
  • Internal Links – Number of internal links pointing at a specific page URL
  • External Links – Number of external links pointing at a specific page URL
  • Top Internal Link Anchors – The top internal link anchors and count pointing at a specific page URL
  • Top External Link Anchors – The top external link anchors and count pointing at a specific page URL

Making Informed Decisions

You can then see the pages that have rankings and / or traffic, and links and start to make decision on whether to:

  • Edit content
  • Build external links
  • Improve internal link volume / context
  • Redirect pages with 301’s
  • Use rel=canonicals

New Site Optimization Widgets

We’ve also built some new site optimization reporting widgets.

Site Summary

Get a summary of lots of key performance metrics in one widget.

Site Summary

Internal & External Link Distribution Chart

This is an interesting one. We’ve sorted the pages we’ve crawled by internal link count and charted the internal link counts on the left Y Axis. On the other we’ve created a scatter chart of the external links. The charts below shows 3 different sites, one significant brand, one huge vertical leader and one site that does pretty well but could do better.

Can you guess which one is which from just these charts? How will you compare?

Feedback Welcome

I hope you enjoy this feature and get to use its many features fully. If you have any suggestions for improvement I’d love to hear them.

There are certainly new features / improvements planned. We’re also adding more reporting widgets to our dashboards tab.