Why are pages missing from Google's index, and what helps?

Crawling, rendering, indexing

Google describes search in three stages. During crawling, Googlebot downloads text, images and videos from pages it has found through links or sitemaps. Google renders the page and runs JavaScript in the process. During indexing, Google analyzes the content and stores it in the index. Only then can a page be served for a search query.

When indexing, Google groups similar pages and picks one of them as the canonical URL. As a rule, only that URL appears in the results. Google also states explicitly that not every processed page gets indexed. There is no guarantee even when everything is technically correct.

Reading the “Page indexing” report

You find the report in Search Console under “Indexing” and “Pages”. It splits known URLs into indexed and not indexed and gives a reason for each group. The example list for a reason shows 1,000 URLs at most. On large websites you therefore only see a sample there.

A reason is just a description at first. Whether you need to act depends on whether the URL belongs in the index.

There are also warnings for pages that are in the index. “Indexed, though blocked by robots.txt” means that Google knows the URL from links and has added it without its content. If it should go, allow the URL in robots.txt and set noindex.

How serious are 404 and server errors?

A 404 on one URL does not harm the other pages of your website. Google says so on the Search Central Blog. According to the current documentation, 4xx errors other than 429 don't slow down crawling either. Google simply fetches the affected URL less often over time. Lots of 404 errors in the report are therefore not yet a ranking problem. They matter when the affected pages have visitors or links.

• You link to the URL yourself or it is in the sitemap. Then correct the link or the sitemap entry. Search Console Help recommends dealing with these cases first.
• An external link points to a wrong URL and brings visitors, for example because of a typo. Then redirect the wrong URL to the correct one with a 301, as Google suggests for this case.
• The URL had clicks or backlinks and the content now lives elsewhere. A 301 redirect to the matching page helps here too.
• All other 404s can stay. That includes external links to URLs that never existed on your site and that nobody visits.

How to find 404 errors with GA4, Search Console and a crawl, and how to sort them by impact, is covered in the 404 errors guide.

Repeated server errors

Server errors weigh more. With 5xx responses, Google slows down crawling of the website, and the more URLs are affected, the more it slows down. URLs that are already indexed stay in the index at first. If they keep returning a server error, Google removes them. A short outage therefore doesn't drop pages from the index straight away. If 5xx errors keep showing up in the report, use the server logs to see which directories or file types are affected, and work out the cause with your developers or hosting provider.

Where to find server errors, robots.txt fetch problems and the other crawl errors in Search Console is shown in the guide Crawl errors in Search Console.

404 errors for URLs from scripts

Sometimes Search Console reports 404 errors for addresses that no page links to. In the source code they only appear inside a script, often as a relative path in a string. Googlebot also looks for links in JavaScript and fetches such strings. According to Google, you can ignore these 404 errors if the URLs aren't meant to exist on your website. This is quick to check. Search for the path in the source code of a page. If it only appears inside a script element, this is exactly that case.

Common causes

noindex in the wrong place

A noindex from the staging environment that goes live with a deployment is a classic. Google reads the directive in the meta tag or in the HTTP header. Both have the same effect.

The order of the blocks matters. If robots.txt blocks the URL, Google never sees the noindex. The page can then still end up in the index if other pages link to it.

noindex via JavaScript

If noindex is in the original HTML, Google may skip rendering. A script that removes the noindex later then never runs. This is what Google's JavaScript SEO basics say. For indexing, what counts is what the server delivers and what is in the HTML after rendering. You should check both.

Canonical to another URL

A canonical tells Google which URL you prefer. If it accidentally points to the homepage, to a redirect or to another language version, the actual page drops out of the index. More on this in the canonical guide.

Quality and duplicates

Many near-identical pages, such as filter combinations in a shop, often end up under “Crawled – currently not indexed” or as duplicates. A request in Search Console changes little here. Fewer and more distinct pages help.

Inspecting URLs

The URL Inspection tool in Search Console shows the last indexed version of a single URL. With “Test live URL” it fetches the page again. These fields matter for indexing.

A successful live test only means that Google can fetch the page and finds no block. It does not check quality, manual actions or the choice of canonical URL. For many URLs at once, the tool is not enough. A crawl of the whole website that reports status codes, noindex, canonicals and robots.txt for every URL together helps here.

Speeding up indexing

With “Request indexing” in the URL Inspection tool, you ask Google to recrawl a URL. Requests are limited per day, and submitting the same URL several times does not help. According to Google, a crawl can take a few days to a few weeks, and even then inclusion is not guaranteed. For many URLs, Google recommends a sitemap.

1. Use the URL Inspection tool to check whether crawling and indexing are allowed.
2. Link to the page internally from pages Google visits regularly.
3. Add the URL to the XML sitemap with its canonical address. Details in the XML sitemap guide.
4. Set a self-referencing canonical.
5. Request indexing once and wait.
6. If the page is still listed under “Crawled – currently not indexed” after weeks, work on the content.

Common mistakes

• robots.txt blocks pages that should leave the index via noindex. Google then cannot see the noindex.
• After a release, the noindex from the test environment is on the live site.
• Sitemap and canonical name different URLs for the same page.
• Every URL that is not indexed is treated as an error. Redirects, alternates with a canonical and intended noindex pages are normal.
• The same URL is resubmitted every day.
• After a migration, the errors in the report are left unattended. In the weeks that follow, the report belongs on the weekly list.

Frequently asked questions