Has this happened to you? You check the “Crawl Errors” report in Google Search Console (formerly known as Webmaster Tools) and you see so many crawl errors that you don’t know where to start. Loads of 404s, 500s, “Soft 404s”, 400s, and many more… Here’s how I deal with big amounts of crawl errors.
Note: This guide was first published on rebelytics.com in 2015 and has since then been updated several times and moved to this blog.
If you don’t find a solution to your problem in this article, feel free to leave me a comment at the bottom of this page. I normally reply within a couple of days.
Here’s an overview of what you will find in this article:
- Don’t panic!
- First, mark all crawl errors as fixed
- Check your crawl errors report once a week
- The classic 404 crawl error
- 404 errors caused by faulty links from other websites
- 404 errors caused by faulty internal links or sitemap entries
- Mystery 404 errors
- What are “Soft 404” errors?
- What to do with 500 server errors?
- Other crawl errors: 400, 503, etc.
- List of all crawl errors I have encountered in “real life”
- Crawl error peak after a website migration
So let’s get started. First of all:
Crawl errors are something you normally can’t avoid and they don’t necessarily have an immediate negative effect on your SEO performance. Nevertheless, they are a problem you should tackle. Having a low amount of crawl errors in Search Console is a positive signal for Google, as it reflects a good overall website health. Also, if the Google bot encounters less crawl errors on your page, users are less likely to see website and server errors.
First, mark all crawl errors as fixed
This may seem like a stupid piece of advice at first, but it will actually help you tackle your crawl errors in a more structured way. When you first look at your crawl errors report, you might see hundreds and thousands of crawl errors from way back when. It will be very hard for you to find your way through these long lists of errors.
My approach is to mark everything as fixed and then start from scrap: Irrelevant crawl errors will not show up again and the ones that really need fixing will soon be back in your report. So, after you have cleaned up your report, here is how to proceed:
Check your crawl errors report once a week
Pick a fixed day every week and go to your crawl errors report. Now you will find a manageable amount of crawl errors. As they weren’t there the week before, you will know that they have recently been encountered by the Google bot. Here’s how to deal with what you find in your crawl errors report once a week:
The classic 404 crawl error
This is probably the most common crawl error across websites and also the easiest to fix. For every 404 error the Google bot encounters, Google lets you know where it is linked from: Another website, another URL on your website, or your sitemaps. Just click on a crawl error in the report and a lightbox like this will open:
Please note that the info in the “Linked from” tab is not always up-to-date. It can contain URLs that don’t exist anymore or that don’t link to the error URL anymore. This is because in this tab, Google lets us know where it found the error URL, not where it is currently linked (as the name might suggest).
Did you know that you can download a report with all of your crawl errors and where they are linked from? That way you don’t have to check every single crawl error manually. Check out this link to the Google API explorer. Most of the fields are already prefilled, so all you have to do is add your website URL (the exact URL of the Search Console property you are dealing with) and hit “Authorize and execute”. Let me know if you have any questions about this!
Now let’s see what you can do about different types of 404 errors.
404 errors caused by faulty links from other websites
If the false URL is linked to from another website, you should simply implement a 301 redirect from the false URL to a correct target. You might be able to reach out to the webmaster of the linking page to ask for an adjustment, but in most cases it will not be worth the effort.
404 errors caused by faulty internal links or sitemap entries
If the false URL that caused the 404 error for the Google bot is linked from one of your own pages or from a sitemap, you should fix the link or the sitemap entry. In this case it is also a good idea to 301 redirect the 404 URL to the correct destination to make it disappear from the Google index and pass on the link power it might have.
Sometimes you will run into weird 404 errors that, according to Google Search Console, several or all of your pages link to. When you search for the links in the source code, you will find they are actually relative URLs that are included in scripts like this one (just a random example I’ve seen in one of my Google Search Console properties):
According to Google, this is not a problem at all and this type of 404 error can just be ignored. Read paragraph 3) of this post by Google’s John Mueller for more information (and also the rest of it, as it is very helpful):
Mystery 404 errors
In some cases, the source of the link remains a mystery. The data that Google provides in the crawl error reports is not always 100% reliable. For example, the information in the “Linked from” tab is not always up-to-date and can contain URLs that haven’t existed for many years or don’t link to the error URLs anymore. In such cases, you can still set up a 301 redirect for the false URL.
Remember to always mark all 404 crawl errors that you have taken care of as fixed in your crawl error report. If there are 404 crawl errors that you don’t know what to do about, you can still mark them as fixed and collect them in a “mystery list”. Should they keep showing up again, you know you will have to dig deeper into the problem. If they don’t show up again, all the better.
If you have a case of mystery 404 errors, feel free to leave me a comment at the end of this article. I’ll be happy to check out your problem.
Let’s have a look at the strange species of “Soft 404 errors” now.
What are “Soft 404” errors?
This is something Google invented, isn’t it? At least I’ve never heard of “Soft 404” errors anywhere else. A “Soft 404” error is an empty page that the Google bot encountered that gave back a 200 status code.
So it’s basically a page that Google THINKS should be a 404 page, but that isn’t. In 2014, webmasters started getting “Soft 404” errors for some of their actual content pages. This is Google’s way of letting us know that we have “thin content” on our pages.
Dealing with “Soft 404” errors is just as straightforward as dealing with normal 404 errors:
- If the URL of the “Soft 404” error is not supposed to exist, 301 redirect it to an existing page. Also make sure that you fix the problem of non-existent URLs not giving back a proper 404 error code.
- If the URL of the “Soft 404” page is one of your actual content pages, this means that Google sees it as “thin content”. In this case, make sure that you add valuable content to your website.
After working through your “Soft 404” errors, remember to mark them all as fixed. Next, let’s have a look at the fierce species of 500 server errors.
What to do with 500 server errors?
500 server errors are probably the only type of crawl errors you should be slightly worried about. If the Google bot encounters server errors on your page regularly, this is a very strong signal for Google that something is wrong with your page and it will eventually result in worse rankings.
This type of crawl error can show up for various reasons. Sometimes it might be a certain subdomain, directory or file extension that causes your server to give back a 500 status code instead of a page. Your website developer will be able to fix this if you send him or her a list of recent 500 server errors from Google’s Webmaster Tools.
Sometimes 500 server errors show up in Google’s Search Console due to a temporary problem. The server might have been down for a while due to maintenance, overload, or force majeure. This is normally something you will be able to find out by checking your log files and speaking to your developer and website host. In a case like this you should try to make sure that such a problem doesn’t occur again in future.
Pay attention to the server errors that show up in your Google Webmaster Tools and try to limit their occurrence as much as possible. The Google bot should always be able to access your pages without any technical barriers.
Let’s have a look at some other crawl errors you might stumble upon in your Google Webmaster Tools.
Other crawl errors: 400, 503, etc.
We have dealt with the most important and common crawl errors in this article: 404, “Soft 404” and 500. Once in a while, you might find other types of crawl errors, like 400, 503, “Access denied”, “Faulty redirects” (for smartphones), and so on.
In many cases, Google provides some explanations and ideas on how to deal with the different types of errors.
In general, it is a good idea to deal with every type of crawl error you find and try to avoid it showing up again in future. The less crawl errors the Google bot encounters, the more Google trusts your site health. Pages that constantly cause crawl errors will be thought to also provide a poor user experience and will be ranked lower than healthy websites.
You will find more information about different types of crawl errors in the next part of this article:
List of all crawl errors I have encountered in “real life”
I thought it might be interesting to include a list of all of the types of crawl errors I have actually seen in Google Search Console properties I have worked on. I don’t have much info on all of them (except for the ones discussed above), but here we go:
Server error (500)
In this report, Google lists URLs that returned a 500 error when the Google bot attempted to crawl the page. See above for more details.
These are URLs that returned a 200 status code, but should be returning a 400 error, according to Google. I suggested some solutions to this above.
Access denied (403)
Here, Google lists all URLs that returned a 403 error when the Google bot attempted to crawl them. Make sure you don’t link to URLs that require authentication. You can ignore “Access denied” errors for pages that you have included in your robots.txt file because you don’t want Google to access them. It might be a good idea though to use nofollow links when you link to these pages, so that Google doesn’t attempt to crawl them again and again.
Not followed (301)
The error “not followed” refers to URLs that redirect to another URL, but the redirect fails to work. Fix these redirects!
Other (400 / 405 / 406)
Here, Google groups everything it doesn’t have a name for: I have seen 400, 405 and 406 errors in this report and Google says it couldn’t crawl the URLs “due to an undetermined issue”. I suggest you treat these errors just like you would treat normal 404 errors.
Flash content (Smartphone)
This report simply lists pages with a lot of flash content that won’t work on most smartphones. Get rid of flash!
This error refers to pages that could be accessed by the Google bot, but were blocked for the mobile Google bot in your robots.txt file. Make sure you let all of Google’s bots access the content you want indexed!
Please let me know if you have any questions or additional information about the crawl errors listed above or other types of crawl errors.
Crawl error peak after a website migration
You can expect a peak in your crawl errors after a website migration. Even if you have done everything in your power to prepare your migration from an SEO perspective, it is very likely that the Google bot will encounter a big number of 404 errors after the relaunch.
If the number of crawl errors in your Google Webmaster Tools rises after a migration, there is no need to panic. Just follow the steps that have been explained above and try to fix as many crawl errors as possible in the weeks following the migration.
- Mark all crawl errors as fixed.
- Go back to your report once a week.
- Fix 404 errors by redirecting false URLs or changing your internal links and sitemap entries.
- Try to avoid server errors and ask your developer and server host for help.
- Deal with the other types of errors and use Google’s resources for help.
- Expect a peak in your crawl errors after a website migration.
If you have any additional ideas on how to deal with crawl errors in Google Webmaster Tools, I would be grateful for your comments.