SEO Tools UK
Click infographic for full view. Copyright: Third Door Media. Written by Search Engine Land, Design by Column Five
Crawlers are largely a separate product category. There is some overlap with the self-service keyword tools (Ahrefs, for instance, does both), but crawling is another important piece of the puzzle. We tested several tools with these capabilities either as their express purpose or as features within a larger platform. Ahrefs, DeepCrawl, Majestic, and LinkResearchTools are all primarily focused on crawling and backlink tracking. Moz Pro, SpyFu, SEMrush, and AWR Cloud all include domain crawling and/or backlink tracking features as part of their SEO arsenals.
There are three types of crawling, all of which provide useful data. Internet-wide crawlers are for large-scale link indexing. It's a complicated and often expensive process but, as with social listening, the goal is for SEO experts, business analysts, and entrepreneurs to map how websites link to one another and extrapolate larger SEO trends and growth opportunities. Crawling tools generally do this with automated bots continuously scanning the web. As is the case with most of these SEO tools, many businesses use internal reporting features in tandem with integrated business intelligence (BI) tools to identify even deeper data insights. Ahrefs and Majestic are the two clear leaders in this type of crawling, and have invested more than a decade's worth of time and resources compling and indexing millions and billions, respectively, of crawled domains and pages.
Website-specific crawlers, or software that crawls one particular website at a time, are great for analyzing your own website's SEO strengths and weaknesses; they're arguably even more useful for scoping out the competition. Website crawlers analyze a website's URL, link structure, images, CSS scripting, associated applications, and third-party services in order to evaluate SEO. Not unlike how a website monitoring tool scans for a webpage's overall "health, " website crawlers can identify factors such as broken links and errors, website lag, and content or metadata with low keyword density and SEO value while mapping a website's architecture. Website crawlers can help your business improve website user experience (UX) while identifying key areas of improvement to help pages rank better. DeepCrawl is by far the most granular and detailed site crawler in this roundup, though Ahrefs and Majestic also provide comprehensive domain crawling and site optimization recommendations.
What are Backlinks?
The third type of crawling tool that emerged during testing is backlink tracking. Backlinks are one of the building blocks of good SEO. Analyzing the quality of your site's inbound backlinks and how they're feeding into your domain architecture can give your SEO team insight into everything from your site's strongest and weakest pages to search visibility on particular keywords against competing brands.
LinkResearchTools makes backlink tracking its core mission and provides a wide swath of backlink analysis tools. LinkResearchTools and Majestic provide the best backlink crawling of the bunch. Aside from these two backlink powerhouses, many of the other tools we tested, such as Moz Pro, SpyFu, SEMrush, Searchmetrics, and Ahrefs, also include solid backlink tracking capabilities.
The Enterprise Tier
The last piece of the complicated SEO tool ecosystem is the enterprise tier. This roundup is geared toward SEO at the small to midsize businesses (SMBs) level for which these platforms are likely priced out of reach. But the handful of enterprise SEO software providers out there essentially roll all of the self-service tools into one comprehensive platform that combines ongoing position monitoring, deep keyword research, and crawling with customizable reports and analytics.
The enterprise platforms can also tie in larger content marketing campaigns and ROI metrics for integrated digital marketing teams. Large enterprise organizations with flush SEO and digital marketing budgets may want to consider enterprise SEO platforms such as Brightedge, Conductor, and Linkdex. For SMBs, a smart combination of self-service tools is the much better bang for your SEO buck.
Running the Numbers
Evaluating which self-service SEO tools are best suited to your business incorporates a number of factors, features, and SEO metrics. Ultimately, though, when we talk about "optimizing, " it all comes down to how easy the tool makes it to get, understand, and take action on the SEO data you need. Particularly when it comes to ad hoc keyword investigation, it's about the ease with which you can zero in on the ground where you can make the most progress. In business terms, that means making sure you're targeting the most opportune and effective keywords available in your industry or space—the words for which your customers are searching.
The terms SEO experts often start with are page authority (PA) and domain authority (DA). DA, a concept in fact coined by Moz, is a 100-point scale that predicts how well a website will rank on search engines. PA is the modern umbrella term for what started as Google's original