Xenu Link Crawling Tool

SEO Crawler Tool

Tools / December 4, 2020

Site Auditor has been a major focus at Raven, and for good reason. Site Auditor has crawled and recrawled hundreds of millions of pages and has discovered billions of SEO issues. These kinds of numbers deserve attention.

So, we’re happy to announce that Raven has rebuilt Site Auditor’s crawlers from the ground up. It now has the ability to crawl up to 10, 000 pages per site. That’s 10 times more pages in a single crawl! So, bring out your big sites, and start going after those larger-site clients.

Wait, is This Even Fair?

We all know that Raven’s Site Auditor is powerful because of all of the information it gives you. And combined with Raven’s other SEO tools (Backlink Explorer, Keyword Rankings, Site Finder, and lest we forget: Research Central), you’re gonna be the SEO superhero you’ve always dreamed of being. Who cares about fair, just feed all this data goodness into your Reports and forget about your competition.

So, What’s Really Different About Site Auditor?

Over the past several months, we’ve been quietly pushing out some changes in Site Auditor to get ready for this big reveal. Here’s the rundown of all the changes:

1. Ability to Crawl 10, 000 Pages/Site: We used to cap site crawls at 1, 000 pages, but with our new crawling engine you can now crawl up to 10, 000 pages per site! Now you can crawl that big site with the same clarity and accuracy you’ve come to expect from the Raven Site Auditor.

2. No Daily Page Limit: You saw that right! You can crawl as many pages as you want in a day. We used to limit you to 10, 000 pages per day to save room for all the other folks, but we’re opening the gates, so bring all your sites to the party. Remember, we only crawl one site per account at a time, so have patience if you’ve got several whoppers. In place of a daily limit, you can crawl up to 300, 000 pages per billing cycle whenever you’d like.

3. Schedule Your Crawl Date: We’ve always let you choose Daily, Weekly or Monthly crawls, but now you can set your start date and the Auditor will schedule out from there, or hit the new “Run Crawl” button anytime. No more logging into Raven on Saturday morning to set up that scheduled crawl. Just visit Tool Options > Customize Settings for the date option.

4. Canonical Link Element Support: We added support for the rel=”canonical” tag, which enables SEOs and Webmasters to communicate to Google (and other search engines) which content you want treated as the original on your site. If Site Auditor finds duplicate content, but that content points to the same canonical tag, Site Auditor will no longer report it as duplicate content.

5. Custom Page Title Length Setting: We all have different standards for what an acceptable title, meta description or site content length is. However, until now, Raven’s dictated the character count for long and short Page Titles. Now you can set your own character count in Tool Options > Customize Settings to match your internal guides.

ICYMI (In Case You Missed It)

These are some other changes that have happened in Site Auditor over the past several months. Some are front-end facing, while some are behind the scenes, but we thought you’d like to know:

  • Increased crawl delay to 2 seconds for all crawls.
  • Replaced YSlow with Google PageSpeed reports for both mobile and desktop
  • Improvements for scalability—bring on all the sites, we can handle them.
  • And coming soon: AJAX website crawling

Great! Now What?

If you’re not yet a user, sign up for your free trial. If you’re already a Raven user, log in to your account and run a crawl in SEO > Site Auditor. Then, set up a new report in the Report Builder, and include your data. The Site Auditor reports make a powerful impression on both prospects and existing clients. They’re an easy way to show what you’ll do for a future client, and how you’ve been improving clients’ visibility and accessibility.

Source: raventools.com