Got 404 errors? Here’s the 4-1-1 to 409 those Web page issues

Two clients of Standing Dog Interactive have recently completed site migrations. Common with this process, several important pages fell off the wagon during the transition, causing 404 errors.

Despite using popular tools such as Google Webmaster Tools and Screaming Frog to find the majority of these 404 pages, we were still noticing pages with inbound links that these tools hadn’t identified.

Generally speaking, 404 pages are bad for search spiders to encounter. If Googlebot, for instance, encounters a large number of 404s on a site, it may stop indexing important pages, which could lead to a rankings drop for important, traffic-driving keywords.

Therefore, it’s imperative to find and fix as many important pages showing 404s as quickly as possible, to boot.

404 errors

404 error cartoon

There are some great tools out there to identify 404s, including Screaming Frog and the list that Google Webmaster Tools gives website owners. However, even Screaming Frog may not encounter and return all the important pages showing 404 errors.

Google Analytics to the rescue

While Google Analytics has many uses, its assistance in identifying 404s may not be readily apparent. Here’s how you can harness its 404-finding power.

First, navigate to the organic section of Google Analystics (Traffic sources>Search>Organic) and then change the primary dimension to “Landing Page,” like so:

Organic section of Google Analytics

Organic section of Google Analytics

I like this process because I know that these URLs were actually getting organic traffic prior to the migration, even if they didn’t have backlinks pointing to them. Are these pages important? If they drove visits to a site in organic search, you bet they are! I don’t want to lose a single potential visitor, so I want to ensure that all of these landing pages are properly redirected.

Next, set the dates in the comparison timeline to several weeks before and several weeks after the migration. Our migration happened on Aug. 5, 2013:

Comparison timeline for 404 removal

Google Analytics: Comparison timeline for 404 removal

By doing this, there won’t be a lot of clutter. This timeline showed 10,850 landing page URIs, definitely manageable since many of these may have been properly redirected already.

Once I set the timeline dates, I go to the bottom and see my total landing pages for that time period:

Google Analytics: Landing pages for 404 removal

Google Analytics: Landing pages for 404 removal

While you’re down there, set the total to 25 because you’ll need that to be able to export all of your landing page data.

Next, go to the address bar and look toward the end of the string for “D25” and change it to the total landing pages (mine was 10,850), like I did below:

Address bar

Address bar

After exporting the results and saving the URLs as a text file, I use Screaming Frog’s list mode to identify which of these pages are 404s and which have been redirected.

Screaming Frog

Screaming Frog

For really large sites or if you want to lengthen the timeline to include more landing pages, it’s helpful to download the results from Google Analytics, open them in Excel and then filter by the results to only show pages that have gotten little to no traffic after the migration.

Filtered results in Excel

Filtered results in Excel

While it’s possible that these pages simply didn’t appear in a natural search, there’s no reason to let that assumption cloud the reality that the page may be 404ing.

Happy 404 hunting!

About Shawn Cohen

Shawn Cohen is director of SEO at Standing Dog.
This entry was posted in SEO, web design, Web traffic and tagged , , , , , , , , . Bookmark the permalink.

One Response to Got 404 errors? Here’s the 4-1-1 to 409 those Web page issues

  1. Jeremy Estes says:

    Awesome post Shawn. In 28 days, I will be scouring analytics looking for these pages on a site we’re migrating.

    Fingers crossed.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>